Encrypted Remote Backups with Sparse Bundles

From Bubba.org

Jump to: navigation, search

Introduction

OS X Leopard introduced a new type of encrypted disk image called Sparse Image Bundles. This type of disk image actually consists of a directory with many small 8MB chunks (called bands). All of the information is automatically sliced into 8MB chunks and put into these bands. OS X takes all the legwork out of breaking down the files and reassembling them. Such slicing is very conducive to many popular backup methods - one being Rsync.

The premise behind using encrypted sparse bundles for remote backup is simple:

  1. OS X Keychain handles local authentication so interaction with sparse bundle can occur without user intervention (ie. cronjob)
  2. Local files are Rsync'ed to the mounted sparse bundle and automatically sliced into 8MB chunks
  3. Bundle slices are then Rsync'ed to a remote server
  4. Once the encrypted bundle is removed from the authenticated machine it becomes useless without proper credentials

Rinse and repeat. Only the changed data on the bundle will be synced back to the remote site each time (give or take a few extra MB because not everything slices nicely and space for bundle overhead).

Instructions

For Offsite Backup using Sparse Bundle Image

  1. OS X 10.5 (Leopard) is required to use Sparse Bundles
  2. Create your Sparse Image Bundle:
    1. Open Disk Utility and create a Disk Image.
    2. Accept all defaults except:
      1. Volume Name (should match VOLUMENAME in script)
      2. Size (I choose custom and make it several gigs - doesn't actually take up all that space unless completely filled)
      3. Encryption - I choose AES128
      4. Image Format - sparse bundle disk image.
      5. For this to work (automounting/unmounting of bundle), you need to store your passphrase in your Keychain
  3. Edit the backup_local() function below to run your local (or remote) rsync commands. I actually have a script on another server that performs the mount over ssh, rsyncs, then unmounts the volume, but you can rsync everything from this script should you choose. See below for an example of the rsync excludes.
  4. Edit the variables in the script to match everything from #2 and to configure your remote location to rsync your sparsebundle. This all assumes you have ssh public key authentication working properly.


Alternative use: Backup to remotely accessible (afp/smb/nfs) Sparse Image

It may also be possible to utilize this script to mount a regular sparse image file on a remote share and perform your rsync -X (to save xattrs) directly on your Mac. This assumes that the sparse image file is on a local (LAN) network share because of the network overhead involved for this to happen. To accomplish this:

  1. OS X 10.4/10.5 (Leopard) is required to use sparse image files
  2. Create your Sparse Image:
    1. Open Disk Utility and create a Disk Image.
    2. Accept all defaults except:
      1. Volume Name (should match VOLUMENAME in script)
      2. Size (I choose custom and make it several gigs - doesn't actually take up all that space unless completely filled)
      3. Encryption - I choose AES128
      4. Image Format - sparse disk image.
      5. For this to work (automounting/unmounting of image), you need to store your passphrase in your Keychain
  3. Copy your sparse image file off to your remote share (hopefully smb or afp accessible location)
  4. Edit the backup_local() function below to run your local (or remote) rsync commands (you can use the rsync -X command here since the sparse image will allow you to preserve your resource forks/extended attributes. See below for an example of the rsync excludes.
  5. Edit the variables in the script to match everything from #2 and to configure your remote smb/afp location of your sparse image file. Note: The "REMOTE" variables are not required for this use.
  6. Uncomment the "return 1" in the backup_offsite() function. This will essentially bypass backing up the files off site. Since your sparse image is on a remote location and not on your machine, this eliminates the need to ship your data off site. Be warned.

Cavaets

  1. Things to keep in mind; there is some overhead to the sparse bundles and encryption; even an empty sparse bundle may be ~50MB big.
  2. Mac->Mac backups should use rsync's -E option to backup extended attributes; if you're going to another platform you can re-generate these extended attributes upon a restore (see Restore section below).
  3. Since all authentication relies on the Keychain being unlocked, you need to be logged in and your keychain cannot be locked for the job to run. This means that you need to not lock your Keychain when your screen is locked. Yes, probably not the most secure thing to do, but if your machine is rebooted your Keychain will remain locked until you login. One fix from Erwin Kooi was to replace:
                open -W ${IMAGEFILE} 2>&1 >> ${LOGFILE}

with

                echo $KEY | /usr/bin/hdiutil mount $CONTAINER -stdinpass

Where $KEY is the passphrase appended with a \c and $CONTAINER is the sparsebundle to mount. While this is more insecure, you don't necessarily have a copy of this script on your remove server, so the risks are lessened. It would be a good idea to use a passphrase that you do not use on anything else in case it does become compromised.

Script

#!/bin/sh
# 
# sparse_backup.sh
# 
# Encrypted remote backup using Mac OS X Sparsebundle Disk Images
# bubbaATbubba.org
# 
 
VOLUMENAME="backup_offsite"
IMAGEFILE="/Users/user/Documents/backup_offsite.sparsebundle";
LOCALHOST="macbook"
REMOTEHOST="backuphost"
REMOTEUSER="backupuser"
REMOTEDIR="/home/backupuser/backup/remote"
LOGFILE="/tmp/backup.log"
EXCLUDEFILE="/Users/user/rsync.excludes"
 
RET=0;
 
# local rsync commands to sync local mac files w/ mounted sparsebundle
# 
backup_local()
{
        DATE=`date`
        echo "Local rsync starting: ${DATE}" 2>&1 >> ${LOGFILE}
        #### CHANGE EVERYTHING BELOW HERE ####
        mkdir -p /Volumes/${VOLUMENAME}/${HOSTNAME} 2>&1 >> ${LOGFILE}
        # v to q to make quiet
        rsync -var --exclude-from=${EXCLUDEFILE} --delete /Users/user /Volumes/${VOLUMENAME}/${HOSTNAME} 2>&1 >> ${LOGFILE}
        #### STOP WITH YOUR CHANGES ####
        DATE=`date`
        echo "Local rsync ending: ${DATE}" 2>&1 >> ${LOGFILE}
}
 
# rsync the sparsebundle offsite
backup_offsite()
{
        # Uncomment this if you're using the alternative method mentioned in the documentation.  
        # return 1
        DATE=`date`
        echo "Offsite rsync starting: ${DATE}" 2>&1 >> ${LOGFILE}
        rsync -e ssh -var --delete ${IMAGEFILE} ${REMOTEUSER}@${REMOTEHOST}:"${REMOTEDIR}" 2>&1 >> ${LOGFILE}
        DATE=`date`
        echo "Offsite rsync ending: ${DATE}" 2>&1 >> ${LOGFILE}
}
 
# function to umount sparseimage
unmount_sparse() 
{
 
        if [ ! -d "/Volumes/${VOLUMENAME}" ]; then
                #echo "Volume /Volumes/${VOLUMENAME} not mounted"
                RET=1;
        else
                /usr/bin/hdiutil unmount "/Volumes/${VOLUMENAME}" 2>&1 >> ${LOGFILE}
                if [ -d "/Volumes/${VOLUMENAME}" ]; then
                         echo "Unable to unmount volume /Volumes/${VOLUMENAME}"
                         RET=1;
                fi
        fi
}
 
# function to mount sparseimage
mount_sparse()
{
        if [ ! -d "/Volumes/${VOLUMENAME}" ]; then
                open -W ${IMAGEFILE} 2>&1 >> ${LOGFILE}
                if [ ! -d "/Volumes/${VOLUMENAME}" ]; then
                        echo "Unable to attach ${IMAGEFILE} / unable to mount /Volumes/${VOLUMENAME}."
                        RET=1;
                fi
        fi
}
 
getpid()
{
        PID=`ps -axwww | grep rsync | grep ${IMAGEFILE} | grep -v grep | awk '{print $1}'`
        if [ ${PID}0 -ne 0 ]; then
                echo "$0 already running (PID ${PID}); Exiting."
                exit 1;
        fi
}
 
case "$1" in 
  mount) 
        getpid
        mount_sparse
        exit $RET;
        ;;
  unmount)
        unmount_sparse
        exit $RET;
        ;;
  backup)
        getpid
        mount_sparse
        if [ "$RET" = "0" ]; then
                # do backup stuff here
                backup_local
                unmount_sparse
                if [ "$RET" = "0" ]; then
                        backup_offsite
                else 
                        exit $RET;
                fi
        else
                exit $RET;
        fi
        ;;
  *)
        echo "Usage: $0 {mount|unmount|backup}"
        exit 1;
        ;;
esac

Usage

To run your backup:

% ./sparse_backup.sh backup

To mount your sparse bundle (handy if you're on a remote machine and want to mount the drive to perform an rsync):

% ./sparse_backup.sh mount

To unmount your sparse bundle (handy if you're on a remote machine and want to unmount the drive after an rsync):

% ./sparse_backup.sh unmount

If you choose to run this/schedule this from cron, you may need to run it as root (depending on what you backup). Since I only backup things in my /Users/user directory, I haven't had this requirement. Here is my cronjob that runs at 3am:

% crontab -l 
00      3       *       *       *       /Users/user/bin/sparse_backup.sh backup

Restore Process

If you're using Mac->Mac, then a simple rsync back to the local machine should suffice. If you're backing up to another flavor of UNIX, you'll probably have to re-generate the OS X extended attributes once the files have been restored.

To do this, you have two options:

  1. Make a new sparsebundle file (just like above; really a directory), then restoring files from the remote system to this new directory.
  2. Compile and run the code below on the sparse bundle after the data has been synced back to a Mac. This code will properly set the xattr extended attributes to the bundle so that it is clickable in the finder (you will need Dev Tools installed to compile).
/* 
 setxattrs - bubbaATbubba.org
 
 Properly sets sparse bundle extended attributes lost when rsyncing
 sparse bundle data to a platform that does not support extended 
 attributes.  This is only need when restoring/retrieving the
 bundle.
 
 To use, sync/copy all bundle files, then run this tool on the
 sparse bundle:
 % gcc -o setxattrs setxattrs.c
 % ./setxattrs mybackup.sparsebundle
 % xattr -l mybackup.sparsebundle
 
*/
 
#include <CoreFoundation/CoreFoundation.h>
#include <sys/xattr.h>
 
int main (int argc, const char * argv[]) {
 
   if (argc != 2) {
        printf("Usage: %s [sparse bundle directory]\n", argv[0]);
        printf("Sets extended attributes for sparse bundle disk image\n");
        return 0;
   }
   int sxr;
   int options = XATTR_NOFOLLOW;
   char theValue1[] = { 0x00, 0x00, 0x00, 0x00,
            0x00, 0x00, 0x00, 0x00, 0x20, 0x00, 0x00, 0x00,
            0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
            0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
            0x00, 0x00, 0x00, 0x00};
  char theValue2[] = { 0x59, 0x48, 0x60, 0x38,
            0x65, 0x7C, 0x09, 0x22, 0x33, 0xAD, 0xA5, 0x73,
            0x12, 0xAD, 0xF3, 0x7F, 0xEE, 0x90, 0x5B, 0x92};
 
   size_t   theSize1 = sizeof(theValue1);
   size_t   theSize2 = sizeof(theValue2);
 
   sxr = setxattr(argv[1], "com.apple.FinderInfo", (void *)theValue1, theSize1, 0, options);
   sxr = setxattr(argv[1], "com.apple.diskimages.fsck", (void *)theValue2, theSize2, 0, options);
 
   return 0;
}


rsync excludes

Make sure you at least exclude the sparse bundle itself. I also have all my VM's in a special directory so they don't get backed up. I'm also not backing up Music, Pictures or Movies as I back those up locally with backuppc and don't consider them required for disaster recovery.

Documents/Virtual\ Machines/*
Documents/backup_offsite.sparsebundle
Movies/**
.Trash/**
Cache**
Logs**
Desktop/**
Mail**
iChats**
Downloads/**
Music/**
Pictures/**
Application\ Support/**
Microsoft\ User\ Data/**

Linux Script

The point of this script is to allow a local Linux machine to connect to your Mac and rsync files to your sparse bundle. Of course for this to work, you need to setup ssh key authentication. You'll also need to make sure that /Volume/${VOLUMENAME}/${LOCALHOST} directory already exists in your image.

#!/bin/sh
 
LOGFILE="/var/log/backup_sparse.log"
MACHOST="macbook"
MACUSER="user"
VOLUMENAME="backup_offsite"
LOCALHOST="linux"
MACBACKUPCMD="/Users/user/bin/sparse_backup.sh"
EXCLUDEFILE="/etc/backup_sparse/excludes.dat"
 
# Run any local commands here like DB dumps, etc.  Just make sure the place
# that files are dumped is actually backed up below
# mysqldump -u user -ppass mydb> /etc/db_backups/mydb.db
 
ping $MACHOST -c 1 -W2 1>/dev/null 2>&1
RET=$?
 
if [ $RET -eq 0 ]; then
        ssh ${MACUSER}@${MACHOST} "${MACBACKUPCMD} mount"
        RET2=$?
        if [ $RET2 -eq 0 ]; then
                #### CHANGE RSYNC LINE TO BACKUP YOUR STUFF ####
                rsync -e ssh -azv --exclude-from=${EXCLUDEFILE} --delete /home /etc /root ${MACUSER}@${MACHOST}:"/Volumes/${VOLUMENAME}/${LOCALHOST}"  2>&1 > $LOGFILE
                #### STOP CHANGES ####
                ssh ${MACUSER}@${MACHOST} "${MACBACKUPCMD} unmount"
                RET3=$?
                if [ $RET3 -eq 0 ]; then
                        exit 0;
                fi
        fi
else 
     exit 1;
fi 
 
echo "Error running $0 (Ping: ${RET}, Mount: ${RET2}, Unmount: ${RET3})"