I decided that it was about time that I started using a real backup system. The script that I put together a few years ago now has done me well but it just isn't good enough for the number of files that I now want to backup. About 9 months ago the backup set grew larger than could be stored on a single DVD and has now swollen to some 8GB. I might get a tape drive at some point but before I do that I need some way of getting all those files into one place. The option I have decided to go for is Bacula, this is probably the most popular enterprise open source backup solution and provides all the facilities I need in one convenient if complex package.

The manual for Bacula is generally very good and it would silly of me to duplicate it here but I will go through a few points that I became confused on and show a fairly standard small install set up. Bacula has been designed from the ground up to be enterprise class through and through. As such most of the power is only used when you are backing up a large number of machines. I haven't seen figures but looking at it I would imagine Bacula could handle hundreds of machines. My set up is, by contrast, tiny. I'm mainly backing up one server and perhaps a few files on the local machine. I may in the future backup some files on other client machines as well but for now it's pretty much just one machine (note that client means backup client in Bacula terms). 

Installing the Catalogue Database

This was really hard work because I answered no during the install of bacula-director-pgsql when it asked me if I wanted to create the database. The reason I answered no was because I wanted to do a database upgrade from Postgres 7.4 to 8.1. I thought, incorrectly it would seem, that I could just come back to this package and reconfigure it to make it create the datebase. If only it was that simple. Apparently because the package isn't being upgraded the reconfigure won't actually create the database. This left with in the sticky position of having no database set up so Bacula wouldn't run. The solution is rather manual but it seems to work (this assumes you have a fairly standard set up of postgres):

1. Login to a terminal as the user postgres.
2. Issue the command "createuser bacula".
3. Answer no to all the questions.
4. Issue the command "psql".
5. Issue the command "ALTER USER bacula WITH PASSWORD password;".
6. Issue the command "CREATE DATABASE bacula WITH OWNER bacula ENCODING 'UTF8';".
7. Leave psql with the command "\q".
8. Issue the command "psql -h 127.0.0.1 -U bacula -W -f /usr/share/dbconfig-common/data/bacula-director-pgsql/install/pgsql".

This should create all the required tables and leave you ready to go with the director.

Rotating File Storate Backup Routine

I have decided to implement a disk based backup system that is very similar to the example given in the Bacula manual. The biggest change I will make is to store only three months worth of data rather than the six given in the example. The reason for this is two fold. Firstly, the data is all private information and as such I don't need a long history of changes. Secondly, I am trying to use the resources I currently have at my disposal which is about 60GiB of free space on the backup server rather than install a new drive.

I have estimated my backup size as follows:

Full Set: 10GiB, Weekly Differential: 1.4GiB, Daily Incremental: 0.2GiB

This gives a steady state of 10 * 3 + 1.4 * 5 + 0.2 * 7 = 38.4GiB. In reality I think I have grossly over estimated the daily change. I suspect that my actual daily change is probably going to be less than 100MiB on average but I want to make sure that I am well within the capacity of the drive. Update: I've noticed that a lot of files get touched everytime I log in to my client machine and a daily differential of 0.2 GiB is about what I am seeing. I suspect that a lot to the changed files are related to software development.

As with the example given in the Bacula manual I have specified more than the bare minimum for the weekly differential and the daily incremental backups. As the manual points out this is a little wasteful and could be tightened up but the waste should be minimal compared to the size of the monthly full pool. I have reduced the daily incremental retention period to 14 days rather than the specified 20 and the maximum volumes is 4 rather than 5.

Backing up the Catalog 

I use Postgres to hold my catalog so the backup of the catalog is just a dump of the database. I use the following expect script called from a bash script that provides the required settings to automatically dump the database each night:

#!/usr/bin/expect -f
set file [lindex $argv 0]
set server [lindex $argv 1]
set database [lindex $argv 2]
set username [lindex $argv 3]
set password [lindex $argv 4]
send "Backing up database $database to file $file\n\r"
spawn /usr/bin/pg_dump -U $username -h $server -W -f $file $database
expect "Password: "
sleep 1
send "$password\r"
wait

The expect script is called like this database-backup.exp /backup/location/catalog.sql 127.0.0.1 database username password. The catalog file is then written into the nightly backup of any files on the backup machine.

Restoring Files  

In order to correctly restore files with Bacula it is necessary to understand the way backups work. Many backup routines include at least some incremental or differential backups to save disk space. The down side of using incremental or differential backups is that in order to perform a full restore of a file set two or more jobs generally need to be included in the restore process and the order those jobs are specified is important.

Before proceeding it is necessary to explain the various different backup strategies and the strengths and weaknesses of each. The simplest backup strategy is the full backup. This is the equivalent of just taking a copy of all of the files on the client machine and is useful in situations where the bulk of the files that need to be backed up change between backups. The full backup strategy also provides for the simplest restore since each job contains a complete copy of all the files on the client machine. When a restore is needed it's a simple as choosing the previously run job.

Incremental backups copy all the files that have changed since the last successful backup irrespective of type. This is the best method for saving space on the backup media as generally only a small subset of the files in a file set will change between backups. The downside of extensively using incremental backups is that full restores can become tedious, under some situations files can be missed and files that have been deleted on the client can come back from the dead. Each of these problems will be dealt with below.

Restores with incremental backups can become tedious because multiple jobs almost always need to be specified. In fact the only time you won't need to specify two or more jobs is when a full backup has just been carried out. After the first full backup has been carried out (there is always a full backup at the start of an incremental backup series, if one isn't found an incremental backup is promoted to a full backup) each subsequent backup job then only stores a small number of the files in the file set. Imagine the situation where a full backup is carried out: this is job 1 and includes the file test.txt that contains just one character "a". A file is then created on the client called example.txt; over night the incremental backup runs: this is job 2 and it includes only example.txt. The test.txt file is then change from containing "a" to "b" and the over night incremental backup runs again: this is job 3 and contains only test.txt.

The client now contains the files test.txt which contains the text "b" and example.txt. To do a full restore of the client from the backup we would have to restore jobs 1,2 and 3 in that order. If we only restored jobs 1 and 3 we would miss the creation of example.txt. If we restored the jobs in the order 3,2 and 1 we would have all the files but test.txt would contain "a" since job 1 was the last restored.

It may now be a little clearer why files come back from the dead when incremental backups are used. Without a great deal of computation (a rolling differential) it isn't possible to detect file deletions and each incremental job would have to maintain markers for files that were deleted between the current and previous job. In the example above imagine what would happen if example.txt was deleted after job 2 ran but before job 3 ran. Job 3 wouldn't include a hole where example.txt was. After job 3 the client would only contain the file test.txt with the content "b" but the restore process would have no way of knowing this. In our contrived example we could skip job 2 in the restore process because we know everything about the client and the backup jobs but, sadly, reality is much more complex than this situation and there are normally multiple changed files in each job some of which will still exist.

The final problem with incremental backups is missed files. This can happen because an incremental backup relies on the last modified time of a file changing when the file is changed. Generally this happens automatically but if a file is moved its last modified date is not normally updated. This means that a restore will result in the files or directories appearing in the wrong place.

Differential backups suffer the same problems as incremental backups but on a much smaller scale. A differential backup saves everything that has changed since the last full backup so is a compromise between media storage space and restore simplicity. If there are long periods of time between full backups and a moderate number of changes take place between each job a differential backup can become almost as large as a full backup.  The restore problems detailed for incremental backups are so much smaller for differential backups because the file hierarchy can only be at most one backup period (typically a day) out of date. With incremental backups changes that may have taken place many backups ago can reappear.

Database Upgrade

I recently upgraded Bacula from 1.38 to 2.0 which meant an update to the database schema. For some reason this wasn't automatic although I suspect that has something to do with the aborted original install. The updgrade was painless enough though as it simply involved running the upgrade script found in the dbconfig-common directory in much the same way as was done to install the database in the first place.