When the company I work for, a civil engineering and surveying firm, decided to move all its AutoCad drawings onto a central fileserver, we were presented with a backup situation orders of magnitude larger than anything we had confronted before. We had at that time (now considerably larger) about 120,000 files, totaling 200GB, that were in active change and needed to be backed up at least daily.

My first thoughts were of some sort of tape backup system, but as I began to research them, I was shocked at the prices I encountered. A tape autoloader large enough to contain our filesystem ran about $12,000 and a 40Gig tape was $89. When I first convinced my boss to let me run Linux on our servers, cheap was a big selling point. So, what are the alternatives?

I had been using a removable 120GB IDE drive to transfer data back and forth from home; it had cost me $101 for the drive and $17 for the removable bay. I also had a retired 1GHz P4 box that had cost about $800--the math was starting to look interesting. If I could find a way to stick 120GB drives together, I would be home free. That's when I discovered Linux LVM (Logical Volume Manager).

With LVM you can, if you like, treat the removable drives as large, cheap tapes and use any of the traditional rotation schemes of incremental and full backups, such as the Tower of Hanoi. We took a somewhat different course for our needs. It has always seemed awkward to me that in order to recover a file from tape you had to search for it sequentially, restore the full backup, apply the incremental backups in order and, finally, arrive at the version you wanted. I wanted a backup system that would let me open and examine drawings to determine if they were the appropriate version before restoration, and with LVM that's what I got. It takes a few more drives to do it this way, but the 12,000 bucks I saved on the autoloader buys a whole lot of drives.

I probably should take this opportunity to warn everybody that I'm still new enough to Linux that I do things that doubtless make more experienced people shudder in horror. All I can say in my own defense is I'm still learning, and many of these things have worked so well in the real world I have found no need to change them.

We started by setting up the P4 box with one IDE drive for the operating system, a CD-ROM drive (to load the operating system) and two removable drive bays, each containing a 120GB IDE drive. It has since grown to four drives on a RAID controller as JBOD, but let's keep this example simple. We loaded Red Hat 8.0 on the backup server--the first Red Hat version to include LVM--and gave it an IP address on the network. Configure your box in a way appropriate to your own network.

Because we were going to be moving these drives in and out of the box and off site, we needed a way to keep track of which drive was which. I used stick-on letters and labelled the part of the first drive bay that remained in the case A and the part that came out A1. The part of the second drive bay that remained in the case was B and the part that came out was B1. These were our backups for one day. Subsequent days were called A2B2, A3B3 and so on.

Having done this, we were ready to configure our drives, in this case /dev/hdb and /dev/hdd, for LVM. Start by logging in as root and type fdisk /dev/hdb. fdisk returns a helpful warning that because we are using 120GB drives, the number of cylinders is set higher than a certain number, but we can probably safely ignore this. Trust me, we can.

Assuming we are starting with a new unformatted disk, we wish to create a new partition. Type n and then press Enter. fdisk now asks if we wish to create an extended or primary partition. We want a primary, so type p. fdisk now wants to know the number of the partition (1-4), so type 1.

Next, fdisk wants to know the beginning cylinder of our partition. The default is 1; accept it by pressing Enter. fdisk now wants to know the ending cylinder of our partition. The default is the last available cylinder on the drive; again, accept this by pressing Enter.

Let's take a look at what we've created so far. To see the partition table, type p and press Enter. We should see something like the following:

DEVICE   BOOT  START  END    BLOCKS    ID    SYSTEM

/dev/hdb1       1    15017   (large    83     Linux
                             number)

Up to this point we haven't actually modified the partition table. These changes are being held in memory by fdisk until we tell it to write them. At any time, we can exit fdisk without making permanent changes by typing q.

So far everything looks good except for the partition type, which seems to be 83 Linux. We need to change this to Linux LVM, which we do by typing t and then 8e. Now, when we read the partition table by typing p, we should see:

DEVICE   BOOT  START  END    BLOCKS    ID    SYSTEM

/dev/hdb1       1    15017   (large    8e    Linux LVM
                             number)

This is what we want, so we now officially change the partition table by typing w. At this point, fdisk becomes quite excited and tells us The partition table has been changed and returns some ongoing information about what it's doing before unceremoniously dumping us back at the command prompt.

We're done with the hairy part, except that we need to repeat the process for any other drives we are going to include in our logical volume set.

At this point we are ready to begin creating the logical volume. Much useful information can be found in the lvm man pages. Typing man lvm puts you at the parent page, which provides directions to the subheading man pages. Now that our disks are formatted as Linux LVM, we need to create them as physical volumes for inclusion in a volume group. We do this by typing pvcreate /dev/hdb1 /dev/hdd1.

We now have to join the physical volumes we created into a volume group from which one or more logical volumes can be created. A peculiarity of LVM is we must at this time set a physical extents size for our volume group. That is, we have to define a certain sized unit in terms of megabytes, called a physical extent. Any logical volume we create from this volume group can contain no more than 64k of these physical extents. If we suspect our logical volume eventually may contain a terabyte of data, we need to set our physical extents to 16MB. We do this while creating our volume group by typing:

vgcreate -s 16M a1b1 /dev/hdb1 /dev/hdd1     

We now have a volume group called a1b1 composed of the two physical volumes, /dev/hdb1 and /dev/hdd1. Let's create a logical volume that uses all the available room on this volume group by typing lvcreate -L 226g a1b1.

The -L option specifies the size (226GB). For some reason, I've never been able to get a drive advertised as 120GB to yield more than 113GB at this point. In the lvcreate line above we actually accepted the default logical volume name (lvol1), so our logical volume is now /dev/a1b1/lvol1. Notice that the last character is the numeral one and not a lower case l. Now we need to create a filesystem on our logical volume. I prefer to use the ext3 filesystem for our drawing files because of their large size. We can create this by typing mke2fs -j /dev/a1b1/lvol1.

It takes a little while for the filesystem to be created, but when it's done the only thing left to do is to create a mountpoint. We do this by typing mkdir /mnt/back. We mount our logical volume by typing:

mount /dev/a1b1/lvol1 /mnt/back  

We can test it by typing df -h /mnt/back; the operating system should report that we have about 226GB available on /dev/a1b1/lvol1.

If we need to add drives to this volume group in the future, we can do so on the fly with vgextend and e2fsadm. We now need to repeat the process for each volume group that we wish to create. I use one for each day of the week, and once a month I pull one out of rotation and save it for a year. This doesn't have to be done all at once; I did one a day until all were complete.

Our network is composed of mostly Windows workstations connecting to Linux servers through Samba. So, we set up a Samba configuration to make our backup server available in the same way. First, copy the default Samba configuration file to a safe place. On Red Hat systems this is /etc/samba/smb.conf. Now, we compose a new simpler smb.conf. Using your favorite editor, type:

encrypt passwords=yes

netbios name=yourbackupserver

workgroup=yourworkgroupname

[back]

read only=no

path=/mnt/back

Save this as /etc/samba/smb.conf. Now we'll restart the Samba server by typing /etc/rc.d/init.d/smb restart. On the first night, we copy the entire filesystem to a1b1 with cp. You may have to unalias cp on your system to make sure the -i flag is not set. This cp process may take 10-12 hours for 200GB. On the next night, we copy the entire filesystem to a2b2, and so on, until we have a complete copy for each day of the week. On subsequent nights, we use cp -au and only copy files that have changed since the previous backup. This only takes an hour or two and has the virtue that any files accidentally deleted from the main server are preserved on the backup.

When changing drives each day, it is somewhat tedious to activate the volume group and mount the logical volume, so we wrote a little Perl script to do it for us. Using our favorite editor, we put this together: like this:

#!/usr/bin/perl


$hold = `vgscan | grep a[0-9]b[0-9]`;

$hold2 = substr($hold, 39, 4);

`vgchange -a y $hold2`;

`mount /dev/$hold2/lvol1 /mnt/back`;

print "$hold2 mounted\n";

and saved it as /home/me/mountback.pl. We make this executable by typing chmod 700 /home/me/mountback.pl.

On the first line, our script creates a variable called $hold, runs the operating system command vgscan to search for volume groups, pipes the output of this command to grep, which searches for a line containing an a, followed by a digit from 0-9, followed by a b, followed by a digit from 0-9, and places it in $hold. On the next line we create a variable called $hold2, run the substring command on the line of text placed in $hold, grab the four characters after the first 39 and place them in $hold2.This should be the name of our volume group, a1b1.

After running vgscan manually a few times it becomes obvious where the name will appear in the line of text. On the third line we run vgchange with the -a option (activate) and give it the y option for yes and the name of the current volume group, which should be in $hold2. On the fourth line we mount the logical volume at our previously created mount point. On the last line we print to the screen the name of the volume we have mounted. If we have done something wrong, we get some strange messages here. I have always intended to go back and add some error checking to this script, but somehow or other I've never gotten around to it. It's been working fine day after day, year after year.

So, to sum up our procedure, to change out our backup drives we type shutdown -h now. When the machine is finished shutting down, we remove the drives labeled a1 and b1 and replace them with those labeled a2 and b2. Then, we start up the machine and log on as root. Type /home/me/mountback.pl, and we should see a2b2 mounted.

We're now ready for the current night's backup to happen. This is best run as a cron job from the main server, which presumably has more and faster processors, RAM and so on. I've been contentedly using a shell script based on cp -au for some time and have recently begun experimenting with rsync. I have every reason to believe tar would work fine as well. A simple script would be something like:

#!/bin/bash

echo $0 >> /var/spool/mail/me

echo "START:" >> /var/spool/mail/me

date >> /var/spool/mail/me

mount -t smbfs -o username=me,password=mepassword

//backupserver/back /mnt/mountpoint

(above 2 lines should be on 1 line in your script)

cp -au /files /mnt/mountpoint

umount /mnt/mountpoint

echo "FINISH:" >> /var/spool/mail/me

date >> /var/spool/mail/me

This mounts the logical volume from our backup server to a mountpoint on the main server, copies all files that have changed since the last backup and sends us an e-mail message giving the name of our backup cron job, the time the backup started and the time it finished. The script then unmounts the logical volume.

I hope this article is helpful to anyone who has found him or herself in a similar backup situation and is considering removable IDE drives as a possible solution. When I first started in this line of work, I would study articles that showed how to do things I desperately needed to learn, follow them carefully and get a good grasp on the process, only to reach the end and have the author say something like, "Now all you have to do is make a few changes to the usual configuration files, recompile the kernel and you're done", which would render the whole thing useless to me. I have tried not to let that happen here, but if anyone tries to implement this solution and runs into a snag, drop me an e-mail at mikef@farnerbarley.com, and I will try to help you over the hump.