Monthly Archives: February 2008

Compiling and installing by hand

If you’re not using a package manager, or if you are, but there is no package available for a piece of software you’d like to install, you’ll find yourself compiling the software by hand. Generally, you start by locating the official web page of the software, downloading an appropriate version of the source code, and extracting the tar file to a directory somewhere.

At this point in the process, you are not doing anything as the root user. You’ll become root much later in this process.

The next thing you’ll do is look in the top level of the extracted directory for promising looking files, like README, INSTALL, or Makefile. It is likely that you will see an executable script called “configure”. It’s always a good idea to start by looking at the README and INSTALL files, if present. They may be in the toplevel directory, or in a documentation directory, which will often have a name like “doc”, “docs”, or “documentation”, possibly with different capitalizations.

If The Package Came With A Makefile

If there’s a Makefile in the toplevel, that’s usually because the software package is fairly small. You will want to look over the Makefile to ensure that it is correct for your intended installation. The most important things to look for are the installation directory and any optional features that might have to be turned on by editing the Makefile. If you can’t find the installation directory, type the command:

make -n install

This will ask “make” to print out the sequence of commands that it will be using to install the package. Since you haven’t compiled anything yet, it will start with the sequence of commands required to compile your software, so look for the installation commands to occur near the end of the output generated by this command.

If your package came with a Makefile, you will now modify the Makefile if necessary, perhaps changing the installation directory of the product. You should do this before compiling it, because sometimes character strings holding the pathnames of configuration files are inserted into the compiled binary, so changing the installation target after compiling may result in an installation that doesn’t work correctly. Editing the Makefile will usually not force a recompilation of the objects under its control, that is the Makefile is not, by default, considered a dependency for the targets in the file.

After this, you will, still as your non-root user, compile the package. This is usually done by simply entering the command make. If errors are encountered during the compile, you’ll have to figure out what happened and how to fix it. The most common causes of errors are:

  • missing include files – you might have to add a “-I” to the CFLAGS, CXXFLAGS, or CPPFLAGS variables in your Makefile.
  • missing libraries – you might have to add a “-L” to the LDFLAGS variable in your Makefile.
  • bad version – the compilation may depend on a library you have on your machine, but the version you have may not be compatible with the software package. You might have to download a different version of that library and install it before you can continue with the software package.
  • apparent code errors – the compiler may generate errors related to missing variables, bad function declarations, or syntax errors. Resist the urge to correct these immediately, and try to understand why you are seeing these errors. Remember, this package probably compiled for somebody before they released it, why doesn’t it work for you? Is it that your compiler is a different version, and flags as errors things that used to be warnings? Is the Makefile configured for the wrong architecture or platform? Something else?

Once you get a clean compile, you’re almost ready for the install. I usually prefer to run the command

make -n install | less

once and read through the output, just to make sure that the install isn’t going to do something weird. Look for things like configuration files going into /usr/etc, which might not be what you expect, or binaries going into /bin (you should try to keep in that directory only those executables that are necessary to get the computer to boot through its startup scripts up to the point where the network starts up).

At this point, move down to the section of the text called “Installing The Software”.

You Have A “configure.am” Script, But No “configure” Script

If you have a “configure.am” script, but no “configure” script, you’ll have to generate the configure script. If there is an executable in this directory with a name like “autogen.sh”, run it. This should be sufficient to set up the configure script. If you don’t have an autogen script, you should run the commands automake then autoconf. This will often generate warnings, but unless the configure script you generate doesn’t run, you can ignore those. So, now you have a configure script, you continue to the next section.

You Have A “configure” Script

If you generated the configure script yourself, you know that it’s an autoconf configure script. Sometimes, though, software is produced that has a completely different script that happens to be called “configure”. This can be confusing if it doesn’t recognize the switch “–help”. Start by typing:

./configure --help | less

and look at the output. If it produces a list of options that are available to you, review them carefully and see if there are any optional behaviours that you would like to turn on, or unwanted options that you want to remove (possibly you don’t have library support for these, and don’t need them). If, instead, the configure script appears to run and do things, you don’t have an autoconf configure script, go back and look at the documentation again to see how to use their particular configuration script.

There are a few things to look at in the options you get from “configure”. One of them is the prefix location, and choosing that properly can require some care, which is discussed here. For now, let’s assume that you’ve chosen a set of options that look suitable. You re-run the configure script with those options, and without the “–help” option. It will do some things, it may take a considerable amount of time to run. Eventually, the script should exit, sometimes generating a list of all options and whether or not they are active. Examine this list if present, there might be an option that you want to enable that has been turned off because the configure script failed to find a particular library, in which case you’ll have figure out why that option was disabled and figure out how to get it working. When you’re satisfied with the compilation options, type “make”. If an error is encountered, see the possibilities mentioned in the earlier section referring to building from a Makefile. If you succeed in compiling the software package, go to next section, “Installing The Software”.

Installing The Software

Now, you can become the root user. Change directory to the location where you compiled the binary, and run

make install

If the thing you’re installing has any shared objects (libraries, usually with names that end in “.so”, possibly followed by more dots and numerals), you should type

ldconfig

to make sure that the dynamic linker knows where to find the libraries you’ve just installed.

Many packages these days produce a pkg-config file. This is usually a filename that ends in “.pc”, and is installed in a directory like “…/lib/pkgconfig/”. The pkg-config application often looks for these files when “configure” is being run, but it has a fairly definite idea of where to look. If your .pc file was installed into a directory where pkg-config doesn’t normally look, you’ll have to find some way to make this file visible to that program. There are three ways you can handle this:

  • Add the appropriate directory to the system-wide environment variable PKG_CONFIG_PATH. Usually this means editing /etc/profile. You likely want it set at least to “/usr/lib/pkgconfig:/usr/local/lib/pkgconfig:/usr/X11/lib/pkgconfig”, but you may want to add more search directories to it, if you expect many packages to be installed in the same prefix.
  • Copy the .pc file into a directory that pkg-config searches. This is unwise, you may install another version of the software some time later, and unless you remember this step your .pc file will still be the old one, causing much aggravation as “configure” insists you still have a version of the package that you know you just replaced.
  • Put a symbolic link to the file from a directory that is searched by pkg-config. Do this if you’ve got only one or two .pc files in this prefix, and don’t expect to put in more.

Test your newly-installed software. It’s best to find problems now, when you’ve just finished installing it and remember what you did, than two weeks from now and have to go through the whole thing again just to figure out how it’s set up.

Two more hints: “configure” writes its command line into a comment near the top of the file “config.log”. If you need to remember how you last ran “configure”, you will find the options you used there.

If you have a particularly detailed set of configure options, you might want to record them in a directory somewhere for future reference, both to see quickly what options you enabled when you compiled the software and to re-use the command the next time you recompile it after downloading a new version.

Keeping sensitive data on the crypto disks

Previously, I described how to create one or more crytpographic partitions. The data stored on those partitions is not retrievable without the 32-digit hexadecimal key that protects it, the key being constructed from a passphrase input by the user. It may seem that this is sufficient to protect sensitive data, making sure simply to create and edit your files only in that partition. However, there are some subtle details that have to be kept in mind.

Information stored on an unencrypted ext2 or ext3 partition has an unknown persistence. A file that was stored there, and later deleted, may be partially or fully recoverable at some time in the future. To be sure of the confidentiality of your data, you have to make sure that it has never been stored to an unencrypted partition.

If you start up your favourite text editor, telling it to create a new file in some place, let’s call it /crypto/sensitive.txt, and then start typing, you may expect that the data never lands on an unencrypted partition. However, there are at least four things to be careful of:

  1. The editor may store information in your home directory, which may not be on the encrypted partition. It might store some of the file contents there, or it might store file metadata. Your editor may keep a table of filenames recently visited in /home, with information about the line number last visited. Your editor might be configured to store crash-recovery autosave files in a directory under your /home directory.
  2. The editor may sometimes store the contents of a working buffer to a file in /tmp.
  3. The computer may come under memory pressure, resulting in some of your data being sent to the swap device.
  4. Your backups may not be as well protected as the files on the cryptographic disk.

The first two points are probably best addressed by ensuring that all of the directories writable by the unprivileged user are on cryptographic partitions. If you only have write permission to the crypto drives, you won’t store any files in plaintext. Note, however, that you typically need /tmp to exist and be writable during the bootup of your system, so that partition can’t be protected with a passphrase if you care about the system successfully performing an unattended reboot.

So, what do we do about /tmp? Well, one simple solution is an overmount. While you normally mount a partition onto an empty directory, it is legal to mount onto a directory that is not empty. The files that were present in that directory are mostly inaccessible after that (a process with access to file descriptors that it opened before the mount will still be able to operate on those files, but they will be invisible to new open operations by pathname).

We’re assuming you have at least one cryptographic partition. So, create a directory on that partition, let’s say /crypto/tmp. After you have formatted and mounted your cryptographic partition, run this command. You only have to do this once, the first time you set up cryptographic disks.

mkdir --mode=01777 /crypto/tmp

Now, you can add the following command to the end of the script in the previous post, the script that mounts your formatted disks:

mount --bind /crypto/tmp /tmp

After you’ve done this, the system will still boot up as usual, using its unencrypted /tmp partition. Then, the root user can run the script from the previous post, now modified to have this extra mount line on the end of it. After entering the passphrase the script will do its work and exit, at which time your /tmp partition will have been replaced with the one in /crypto. Note that if your system starts up in X, with a graphical login screen, you will have to restart it after you have overmounted /tmp, otherwise you will find that X programs fail to work at all. I usually restart X by issuing a simple “killall X” command, and letting the xdm or gdm program start it back up again. This is a lot of trouble, but all manner of things can be stored on your /tmp disk. Firefox will store downloaded files such as PDFs there when there is a helper application ready to use them.

That leaves us with swap. Encrypting the swap space is actually very easy:

# Encrypt the swap partition
hashed=`dd if=/dev/urandom bs=1 count=64 | md5sum | awk ' { print $1 } '`
dmsetup create SWP <<DONE
0 `blockdev --getsize /dev/hda6` crypt aes-plain $hashed 0 /dev/hda6 0
DONE
mkswap /dev/mapper/SWP
swapon /dev/mapper/SWP

This can run unattended during the bootup. It creates a random cryptographic key using /dev/urandom, a device especially designed to produce true random numbers even during a system bootup sequence. This random key is used to create an encrypted interface to /dev/hda6. It is formatted as a swap partition, and then enabled. A new key will be generated each time the system boots, so nothing in swap space will survive a reboot. Note that there do exist suspend-to-disk procedures for Linux that store a memory image on the swap partition. If you intend to use such a suspend system, you will have to ensure that it does not attempt to write to the cryptographic swap partition, or you’ll have to defer mounting the swap partition until the root user can enter a specific passphrase, thereby allowing you to preserve the contents across a reboot. If you’re supplying a passphrase to handle encryption on the swap space, you should not run mkswap, except the first time you set up the partition (think of mkswap as being a reformat).

The question of how to protect your backup copies of sensitive files is entirely dependent on what system you use for backups. You may be able to pipe your backups through the des binary, or you may be able to store the backups on encrypted filesystems, but there are too many variations for me to offer much advice here. The security of your backups is not something that can be ignored, as has been made all to obvious with the various data disclosure scares that occur with alarming regularity when shipments of tapes or CDs fail to arrive at their destinations.

UPDATE

See my followup article for a warning about a vulnerability in this technique.

Cryptographic mounts

Some of the data on my computers is stuff that I’d rather not let into the hands of a random stranger. Work-related files, proprietary data or source code, banking information, or other sensitive files. A laptop can go missing, an entire desktop computer can be carried away. It would be nice if the sensitive data were inaccessible in that event.

This leads us to cryptographic mounts. Partitions whose contents cannot be read without the knowledge of a secret that is not stored in the computer. I use a passphrase, but if you are the kind of person who memorizes 32 digit hexadecimal numbers, you can skip the passphrase. The appropriate features to enable in the kernel, either as modules or compiled directly in, are MD (the same subsystem that controls RAID) and two features in that subsystem, BLK_DEV_MD, and DM_CRYPT. You also need a cryptographic algorithm available. I use AES encryption on my partitions, but there are many others available. I have activated the CRYPTO_AES module, plus the appropriate architecture specific module, CRYPTO_AES_X86_64 for my desktop machine and CRYPTO_AES_586 for my laptop.

So, let’s say you have one or more blank partitions that you’d like to set up as a cryptographic partitions, all with the same passphrase. You start with this script:

#! /bin/sh
#

partition=/dev/sda6
partition2=/dev/sdc6
mtpt=/crypto
mapname1=Crypto1
mapname2=Crypto2

holdIFS="$IFS"
IFS=""

echo -n "Enter the passphrase: "
read -s oneline

IFS="$holdIFS"

{ hashed=`md5sum | awk ' { print $1 } '` ; }<<DONE
$oneline
DONE

dmsetup create $mapname1 <0 `blockdev --getsize $partition` crypt aes-plain $hashed 0 $partition 0
DONE
dmsetup create $mapname2 <0 `blockdev --getsize $partition2` crypt aes-plain $hashed 0 $partition2 0
DONE

What this script does is to prompt the user for a passphrase, without echoing it to the screen. Once the passphrase is entered, it is converted to a 32 character hexadecimal string with the MD5 program. I use a here document, marked with the << characters, because that way the hexadecimal string does not appear in the process status list. Simply using echo risks having the secret visible to any user who types ps at the correct moment. Then, the dmsetup program creates the cryptographic mapping, using the hex sequence as the cryptographic key.

You will have to change the values of the $partition and $partition2 variables to correspond to those on your system. Note that volume labels are unavailable, because the system can’t read the label off a cryptographic partition before the passphrase has been supplied.

Run this script, entering the passphrase. It’s important that you do this through the script, and not manually at the command line, because later you’ll modify the script to mount your cryptographic partitions, and you want to ensure that exactly the same code read your passphrase when you created the partitions as will read your passphrase when you try to mount the partitions after a reboot some time in the future.

When the script exits, you will have two new objects appearing in the /dev/mapper directory. In this case, they are /dev/mapper/Crypto1 and /dev/mapper/Crypto2. So, in this example, /dev/sda6 is the encrypted volume, and /dev/mapper/Crypto1 is the decrypted version of the same volume. You do all of your work on /dev/mapper/Crypto1. You format and mount that device, never /dev/sda6.

This command will create an ext3 filesystem with 0 bytes reserved for the superuser.

/sbin/mke2fs -j -m 0 /dev/mapper/Crypto1

Now, you can mount /dev/mapper/Crypto1 onto a mount point, and start copying files there as usual. Until you remove the cryptographic mapping, the data is available as a normal mounted partition. So, we now append some code to the script above to allow the partitions to be mounted by the root user after a reboot. Take the script above and add the following lines to the bottom:

/sbin/e2fsck -y /dev/mapper/$mapname1 || \
{ dmsetup remove $mapname1 ; echo "" ; echo "fsck failed"; exit 1; }

/sbin/e2fsck -y /dev/mapper/$mapname2 || \
{ dmsetup remove $mapname1; dmsetup remove $mapname2 ;\
echo "" ; echo "fsck failed"; exit 1; }

mount -onodiratime /dev/mapper/$mapname1 $mtpt1 || \
{ dmsetup remove $mapname1 ; dmsetup remove $mapname2 ; \
echo "" ; echo "Failed" ; exit 1 ; }

mount -onodiratime /dev/mapper/$mapname1 $mtpt2 || \
{ umount $mtpt ; \
dmsetup remove $mapname1 ; dmsetup remove $mapname2 ; \
echo "" ; echo "Failed" ; exit 1 ; }
echo ""

This runs fsck on the partitions, if necessary (remember, fstab can’t fsck these partitions because it doesn’t know the passphrase). Note that if you entered the wrong passphrase, you’ll find out at this point, when e2fsck fails to identify the partition as being an ext2 or ext3 partition.

It then manually mounts the cryptographic partitions onto the mountpoints in $mtpt1 and $mtpt2. In the event of a mount failure, it unmounts everything and removes the cryptographic mappings.

The next time the computer is rebooted, the root user will have to run this script and enter the correct passphrase before the data on those drives is readable. If somebody else obtains your laptop, any mounted cryptographic partitions will be unavailable if the computer is rebooted, or the drive removed from the laptop and inserted into another machine.

This is only half the story. In a later post I’ll describe the care you have to take to make sure your sensitive data does not wind up as readable plaintext somewhere on your filesystem.

Why do I have so many hard drives?

There are five hard drives in my main computer. There is no RAID setup. Why?

Hard drives fail. I’ve had the drive holding my root partition fail more than once. When that happens, I used to restore from backup. I would make a backup tape at least once a week, but a badly timed disk failure could still result in the loss of a lot of work.

My solution to this has been to buy my hard drives in matched pairs. I partition them equally, format them the same way, and install them both in the computer. One of them is the live disk, the other is the spare. The spare is kept unmounted and spun down. Every night around 3:00 AM, a cron job spins up the spares drives. Then, one partition at a time is fsck-ed, mounted, and copied to. The shell script uses rdist to synchronize the contents of the two partitions. Finally, I take special care to make the backup drive bootable. I use the LILO boot loader, so, when the root partition is mounted under /mnt/backup, the script executes the command:

/sbin/lilo -r /mnt/backup -b /dev/sdc

which, on my system, writes the LILO boot magic to the backup boot drive, which appears as /dev/sdc when it is the spare in my system. My lilo.conf file, on both the live system and the spare, refer to the boot drive as being /dev/sda, but this ‘-b’ switch overrides that, so that the information is written to the boot block of the current /dev/sdc, but is written so that is appropriate for booting the device at /dev/sda (which it will appear to be should my live boot drive fail and be removed from the system).

Next, I use volume labels to mount my partitions. You can’t have duplicate labels in the system, so my spare drive has labels with the suffix “_bak” applied. That means that the /etc/fstab file suitable for the live drive would not work if the spare were booted with that fstab. To solve this problem, the copying script runs this command after it finishes copying the files in /etc:

sed -e 's|LABEL=\([^ \t]*\)\([ \t]\)|LABEL=\1_bak\2|' /etc/fstab > /mnt/backup/etc/fstab

which has the effect of renaming the labels in the fstab to their versions with the _bak suffix, so they match the volume partitions on the spare hard drive.

OK, that sounds like a lot of work, why do I do it? What does it buy me?

First of all, it gives me automatic backups. Every night, every file is backed up. When I go to the computer at the beginning of the day, the spare drive holds a copy of the filesystem as it appeared when I went to sleep the night before. Now, if I do something really unwise, deleting a pile of important files, or similarly mess up the filesystem, I have a backup that I haven’t deleted. If I were to use RAID, deleting a file would delete it immediately from my backup, which isn’t what I want. As long as I realize there’s a problem before the end of the evening, I can always recover the machine to the way it looked before I started changing things in the morning. If I don’t have enough time to verify that the things I’ve done are OK, I turn off the backup for a night by editing the script.

Another important thing it allows me to do is to test really risky operations. For instance, replacing glibc on a live box can be tricky. In recent years, the process has been improved to the point that it’s not really scary to type “make install” on a live system, but ten years ago that would almost certainly have confused the dynamic linker enough that you would be forced to go to rescue floppies. Now, though, I can test it safely. I prepare for the risky operation, and then before doing it, I run the backup script. When that completes, I mount the complete spare filesystem under a mountpoint, /mnt/chroot. I chroot into that directory, and I am now running in the spare. I can try the unsafe operation, installing a new glibc, or a new bash, or something else critical to the operation of the Linux box. If things go badly wrong, I type “exit”, and I’m back in the boot drive, with a mounted image of the damage in /mnt/chroot. I can investigate that filesystem, figure out what went wrong and how to fix it, and avoid the problem when the time comes to do the operation “for real”. Then, I unmount the partitions under /mnt/chroot and re-run my backup script, and everything on the spare drive is restored. Think of it as a sort of semi-virtual machine for investigating dangerous filesystem operations.

The other thing this gives me is a live filesystem on a spare drive. When my hard drive fails (not “if”, “when”, your hard drive will fail one day), it’s a simple matter of removing the bad hardware from the box, re-jumpering the spare if necessary, and then rebooting the box. I have had my computer up and running again in less than ten minutes, having lost, at most, the things I did earlier in the same day. While you get this benefit with RAID, the other advantages listed above are not easily available with RAID.

Of course, this is fine, but it’s not enough for proper safety. The entire computer could catch fire, destroying all of my hard drives at once. I still make periodic backups to writable DVDs. I use afio for my backups, asking it to break the archive into chunks a bit larger than 4 GB, then burn them onto DVDs formatted with the ext2 filesystem (you don’t have to use a UDF filesystem on a DVD, ext2 works just fine, and it’s certain to be available when you’re using any rescue and recovery disk). Once I’ve written the DVDs, I put them in an envelope, mark it with the date, and give it to relatives to hang onto, as off-site backups.

So, one pair of drives is for my /home partition, one pair for the other partitions on my system. Why do I have 5 drives? Well, the fifth one isn’t backed up. It holds large data sets related to my work. These are files I can get back by carrying them home from the office on my laptop, so I don’t have a backup for this drive. Occasionally I put things on that drive that I don’t want to risk losing, and in that case I have a script that copies the appropriate directories to one of my backed-up partitions, but everything else on that drive is expendable.

There are two problems that can appear with large files.

  • rdist doesn’t handle files larger than 2 GB. I looked through the source code to see if I could fix that shortcoming, and got a bit worried about the code. So I’m working on writing my own replacement for rdist with the features I want. In the mean time, I rarely have files that large, and when I do, they don’t change often, so I’ve been copying the files to the backup manually.
  • Sometimes root’s shells, even those spawned by cron, have ulimit settings. If you’re not careful, you’ll find that cron jobs cannot create a file in excess of some maximum size, often 1 GB. This is an inconvenient restriction, and one that I have removed on my system.