FreeBSD Full / Incremental Tape Backup Shell Script

by on June 11, 2009 · 13 comments

  1. #!/bin/sh
  2. # A FreeBSD shell script to dump Filesystem with full and incremental backups to tape device connected to server.
  3. # Tested on FreeBSD 6.x and 7.x - 32 bit and 64 bit systems.
  4. # May work on OpenBSD / NetBSD.
  5. # -------------------------------------------------------------------------
  6. # Copyright (c) 2007 nixCraft project <http://www.cyberciti.biz/fb/>
  7. # This script is licensed under GNU GPL version 2.0 or above
  8. # -------------------------------------------------------------------------
  9. # This script is part of nixCraft shell script collection (NSSC)
  10. # Visit http://bash.cyberciti.biz/ for more information.
  11. # ----------------------------------------------------------------------
  12. LOGGER=/usr/bin/logger
  13. DUMP=/sbin/dump
  14. # FSL="/dev/aacd0s1a /dev/aacd0s1g"
  15. FSL="/usr /var"
  16. NOW=$(date +"%a")
  17. LOGFILE="/var/log/dumps/$NOW.dump.log"
  18. TAPE="/dev/sa0"
  19.  
  20. mk_auto_dump(){
  21. local fs=$1
  22. local level=$2
  23. local tape="$TAPE"
  24. local opts=""
  25.  
  26. opts="-${level}uanL -f ${tape}"
  27. # run backup
  28. $DUMP ${opts} $fs
  29. if [ "$?" != "0" ];then
  30. $LOGGER "$DUMP $fs FAILED!"
  31. echo "*** DUMP COMMAND FAILED - $DUMP ${opts} $fs. ***"
  32. else
  33. $LOGGER "$DUMP $fs DONE!"
  34. fi
  35. }
  36.  
  37. dump_all_fs(){
  38. local level=$1
  39. for f in $FSL
  40. do
  41. mk_auto_dump $f $level
  42. done
  43. }
  44.  
  45. init_backup(){
  46. local d=$(dirname $LOGFILE)
  47. [ ! -d ${d} ] && mkdir -p ${d}
  48. }
  49.  
  50. init_backup
  51.  
  52. case $NOW in
  53. Mon) dump_all_fs 0;;
  54. Tue) dump_all_fs 1;;
  55. Wed) dump_all_fs 2;;
  56. Thu) dump_all_fs 3;;
  57. Fri) dump_all_fs 4;;
  58. Sat) dump_all_fs 5;;
  59. Sun) dump_all_fs 6;;
  60. *) ;;
  61. esac > $LOGFILE 2>&1

How do I run this script?

Download this script and unzip in /root. Open script and customize tape device ($TAPE variable) and file systems ($FSL). Operator can run this script from a shell prompt:
# /root/tapebackup.sh
Or via a cron job:
@midnight /root/tapebackup.sh



4000+ howtos and counting! If you enjoyed this article, join 45000+ others and get free email updates!

Click here to subscribe via email.

  • Hassan

    Nice script
    Can it be ported to solaris 10

  • Vivek Gite

    It should work under Solaris too with little changes.

  • Ville

    That’s very handy! Two questions: the `die()’ routine doesn’t seem to be used by anything? And `BAK’ seems to be undefined.

    I’ll use this as a base for a modified script (I save dumps onto a separate hard drive, and also want there to be separate dump files for each filesystem being dumped).

  • Vivek Gite

    Actually I’ve two scripts, BAK was used to dump to $BAK file system (e.g. BAK=/usr/backup) and another for tape. $BAK and die are from my dump to FS. Feel free to modify it as per your setup. I will update script to avoid confusion.

  • Ville

    Inspired by your script I wrote a new one. It can be found on my blog.

  • Aaron Hurt

    This gave me a the motivation to finally get around to writing a backup script for my server. It’s basic framework is a combination of this script and another one I found on this site. It’s optimized for a freebsd/zfs system with a local tapedrive…it’ll even automatically eject tapes and resume on a new tape load. Hope it helps someone.

    Main Part:

    #!/bin/bash
    ## /root/dobackup.sh
    ## last update 09162009 by ahurt
    #
    # datasets to backup - use zfs paths not mount points
    BACKUP_SETS="pool0/filebase pool0/vmware pool0/backuppc"
    # number of snapshots to keep of each dataset
    # snaps in excess of this number will be expired
    # oldest snaps deleted first...this must be non zero
    SNAP_KEEP="8"
    # where you want your log files
    LOGBASE=/root/logs
    # where your tape drive is located
    TAPE="/dev/nsa0"
    # the length of the tape (in KBs)
    TAPE_LENGTH="61440"
    # tape swap script
    TAPE_SWAP="/root/tapeswap.sh"
    # the tape blocksize in bytes
    BLOCKSIZE="32768"
    # the tar blocksize in 512byte sectors
    TAR_BLOCKS="64"
    # any additional tar arguments
    TAR_ARGS=""
    # path to binaries
    TAR=/usr/local/bin/gtar
    MT=/usr/bin/mt
    MKDIR=/bin/mkdir
    # get the current date info
    DOW=$(date +"%a")
    MOY=$(date "+%m")
    DOM=$(date "+%d")
    YR=$(date "+%Y")
    # backup log file
    LOGFILE="${LOGBASE}/${DOW}-backup.log"
    ############################################
    ##### warning gremlins live below here #####
    ############################################
    ## init our backup paths
    BACKUP_PATHS=""
    ## main backup function
    full_backup(){
            ## do our snapshots..we don't want to tar a live fs
            zfs_snap
            ## initiate our tape
            $MT -f $TAPE comp on
            $MT -f $TAPE blocksize $BLOCKSIZE
            ## build our tar command
            if [ "${TAR_ARGS}x" != 'x' ]; then
                    local tarcmd="$TAR $TAR_ARGS -F $TAPE_SWAP -H pax "
            else
                    local tarcmd="$TAR -F $TAPE_SWAP -H pax "
            fi
            ## build the rest and run it
            tarcmd+="-L $TAPE_LENGTH -b $TAR_BLOCKS --transform=${XFROM} "
            tarcmd+="--show-transformed-names -clpMSvf $TAPE $BACKUP_PATHS"
            echo "RUNNING: $tarcmd"
            $tarcmd
            ## check the tar return
            if [ $? -eq 0 ]; then
                    echo "FINISHED: Rewinding and ejecting tape"
                    $MT -f $TAPE rewind
                    $MT -f $TAPE offline
            else
                    echo "ERROR: $TAR exited abnormally please check logs"
                    exit 1
            fi
    }
    ## partial/incremental fucntion
    partial_backup(){
            ## we are doing an incremental...set our tar args
            ## only backup files newer than yesterday
            if [ "${TAR_ARGS}x" != 'x' ]; then
                    TAR_ARGS+=" -N yesterday"
            else
                    TAR_ARGS="-N yesterday"
            fi
            ## call our main backup function
            full_backup
    }
    ## create our zfs snapshots
    zfs_snap(){
            ## set our snap name
            local sname="${DOW}-${MOY}${DOM}${YR}"
            ## remove snapshot paths from filenames...this is passed to tar
            ## with --tramsform in the backup function above
            XFROM="s/\\/\\.zfs\\/snapshot\\/${sname}//g"
            ## generate snapshot list and cleanup old snapshots
            for dset in $BACKUP_SETS; do
                    ## get current existing snapshots that look like
                    ## they were made by this script
                    local temps=`zfs list|grep -e "${dset}\@[Mon|Tue|Wed|Thu]"|\
                    awk '{print $1}'`
                    ## just a counter var
                    local index=0
                    ## our snapshot array
                    declare -a snaps
                    ## to the loop...
                    for sn in $temps; do
                            ## while we are here...check for our current snap name
                            if [ $sn == $sname ]; then
                                    ## looks like it's here...we better kill it
                                    ## this shouldn't happen normally
                                    echo "Destroying OLD snapshot ${dset}@${sname}"
                                    zfs destroy ${dset}@${sname}
                            else
                                    ## append this snap to an array
                                    snaps[$index]=$sn
                                    ## increase our index counter
                                    let "index += 1"
                            fi
                    done
                    ## set our snap count and reset/reuse our index
                    local scount=${#snaps[@]}; index=0
                    ## how many snapshots did we end up with..
                    if [ $scount -ge $SNAP_KEEP ]; then
                            ## oops...too many snapshots laying around
                            ## we need to destroy some of these
                            while [ $scount -ge $SNAP_KEEP ]; do
                                    ## zfs list always shows newest last
                                    ## we can use that to our advantage
                                    echo "Destroying OLD snapshot ${snaps[$index]}"
                                    zfs destroy ${snaps[$scount]}
                                    ## decrease scount and increase index
                                    let "scount -= 1"; let "index += 1"
                            done
                    fi
                    ## come on already...make that snapshot
                    echo "Creating ZFS snapshot ${dset}@${sname}"
                    zfs snapshot ${dset}@${sname}
                    ## build our backup paths/snapshot paths
                    local mnt=`zfs get mountpoint ${dset}|tail -1|awk '{print $3}'`
                    ## add this to our global var
                    if [ "${BACKUP_PATHS}x" != 'x' ]; then
                            BACKUP_PATHS+=" ${mnt}/.zfs/snapshot/${sname}"
                    else
                            BACKUP_PATHS="${mnt}/.zfs/snapshot/${sname}"
                    fi
            done
    }
    ## make sure our log dir exits
    [ ! -d $LOGBASE ] && $MKDIR -p $LOGBASE
    ## this is where it all starts - do either a full
    ## or partial depening on day
    ## office is closed friday-sunday
    case $DOW in
            Mon)    full_backup;;
            Tue|Wed|Thu)    partial_backup;;
            *) ;;
    esac > $LOGFILE 2>&1
    [/code]
    Second Part: (this is the tar info-script)
    [code]
    #! /bin/sh
    ##
    ### tar info script to automate tape changes
    ### below should match settings in backup script
    ##
    # where your tape drive is located
    TAPE="/dev/nsa0"
    # path to binaries
    MT=/usr/bin/mt
    ## script loop below ##
    echo "Preparing volume $TAR_VOLUME of $TAR_ARCHIVE...."
    sleep 15s
    echo "Rewinding tape $TAPE ..."
    $MT -f $TAPE rewind
    sleep 5s
    echo "Ejecting tape $TAPE ..."
    $MT -f $TAPE offline
    sleep 5s
    echo "Waiting for next tape..."
    while true; do
            ## wait 15 seconds between checks
            sleep 15s
            ## issue a status command and squelch output
            ## this will return an error if there is no tape
            $MT -f $TAPE status >/dev/null 2>&1
            ## check if status returned error or good
            if [ $? -eq 0 ];
            then
                    ## if we are in here status worked
                    echo "New tape detected...preparing..."
                    ## make sure take is ready...and let's quick erase it
                    $MT -f $TAPE rewind
                    $MT -f $TAPE erase 0
                    ## that's it...
                    echo "Tape ready...resuming operation..."
                    ## exit clean and tar will continue
                    exit 0
            fi
    done
    

    Not sure if anyone else could use it...but I hope it helps someone get a start anyways.

  • Aaron Hurt

    Err…that’s not exactly what I wanted…guess [code] tags aren't supported here. It looses all the formatting/tabs like that...but it should still work just doesn't look very pretty.

  • Vivek Gite

    I’ve edited out your script and posted using <pre> … </pre> tags

    Thanks for sharing your code!

  • Aaron Hurt

    No problem, just a little update I basically rewrote it again and think I finally have what I really want. I use ZFS here on my freebsd system and I wanted to make sure 1) my backups were sane and stable ….and 2) I didn’t want to always have to goto tape to grab a file unless there was just a total catastrophic failure. Hence I integrated ZFS snapshot creation into the process. Then just to make the archives pretty, I told tar to transform the snapshot paths to the actual system paths before writing it to tape. This uses the same code for tapeswap…here it is…

    #!/bin/bash
    ## /root/dobackup.sh
    ## last update 09212009 by ahurt
    #
    # datasets to backup - use zfs paths not mount points
    BACKUP_SETS="pool0/filebase pool0/vmware pool0/backuppc"
    # number of snapshots to keep of each dataset
    # snaps in excess of this number will be expired
    # oldest snaps deleted first...this must be non zero
    SNAP_KEEP="8"
    # where you want your log files
    # and gnu tar incremental snaphots
    LOGBASE=/root/logs
    # where your tape drive is located
    TAPE="/dev/nsa0"
    # the length of the tape (in KBs)
    TAPE_LENGTH="62914560"
    # tape swap script
    TAPE_SWAP="/root/tapeswap.sh"
    # the tape blocksize in bytes
    BLOCKSIZE="32768"
    # the tar blocksize in 512byte sectors
    TAR_BLOCKS="64"
    # any additional tar arguments
    TAR_ARGS=""
    # path to binaries
    TAR=/usr/local/bin/gtar
    MT=/usr/bin/mt
    MKDIR=/bin/mkdir
    GZIP=/usr/bin/gzip
    # get the current date info
    DOW=$(date +"%a")
    MOY=$(date "+%m")
    DOM=$(date "+%d")
    YR=$(date "+%Y")
    # backup log file
    LOGFILE="${LOGBASE}/${DOW}-backup.log"
    # gnu tar incremental snapshot file
    SNAR="${LOGBASE}/gtar-backup.snar"
    ############################################
    ##### warning gremlins live below here #####
    ############################################
    ## init our backup paths
    BACKUP_PATHS=""
    ## main backup function
    do_backup(){
    	## do our snapshots..we don't want to tar a live fs
    	zfs_snap
    	## initiate our tape
    	$MT -f $TAPE comp on
    	$MT -f $TAPE blocksize $BLOCKSIZE
    	## if it's monday and we have a tar snapfile
    	## we need to delete this to get a full backup
    	if [ -f ${SNAR} ] && [ $DOW == 'Mon' ]; then
    		rm ${SNAR}
    	fi
    	## build our tar command
    	if [ "${TAR_ARGS}x" != 'x' ]; then
    		local tarcmd="$TAR $TAR_ARGS -F $TAPE_SWAP -H pax "
    	else
    		local tarcmd="$TAR -F $TAPE_SWAP -H pax "
    	fi
    	## build the rest and run it
    	tarcmd+="-L $TAPE_LENGTH -b $TAR_BLOCKS --transform=${XFROM} "
    	tarcmd+="--show-transformed-names --listed-incremental=${SNAR} "
    	tarcmd+="-clpMSvf $TAPE $BACKUP_PATHS"
    	echo "RUNNING: $tarcmd"
    	$tarcmd
    	## check the tar return
    	if [ $? -eq 0 ]; then
    		echo "FINISHED: Rewinding and ejecting tape"
    		$MT -f $TAPE rewind
    		$MT -f $TAPE offline
    	else
    		echo "ERROR: $TAR exited abnormally please check logs"
    	fi
    	## finish up...zip the log...they get huge
    	echo "Cleaning up...zipping log ${LOGFILE}.gz"
    	echo "This log can be viewed with zcat"
    	$GZIP ${LOGFILE}
    	## delete lockfile
    	if [ -f "${LOGBASE}/.backup.lock" ]; then
    		rm "${LOGBASE}/.backup.lock"
    	fi
    }
    ## create and manage our zfs snapshots
    zfs_snap(){
    	## set our snap name
    	local sname="${DOW}-${MOY}${DOM}${YR}"
    	## remove snapshot paths from filenames...this is passed to tar
    	## with --tramsform in the backup function above
    	XFROM="s/\\/\\.zfs\\/snapshot\\/${sname}//g"
    	## generate snapshot list and cleanup old snapshots
    	for dset in $BACKUP_SETS; do
                    ## get current existing snapshots that look like
    		## they were made by this script
    		local temps=`zfs list|grep -e "${dset}\@[Mon|Tue|Wed|Thu]"|\
    		awk '{print $1}'`
    		## just a counter var
    		local index=0
    		## our snapshot array
    		declare -a snaps
    		## to the loop...
    		for sn in $temps; do
    			## while we are here...check for our current snap name
                    	if [ $sn == $sname ]; then
                            	## looks like it's here...we better kill it
    							## this shouldn't happen normally
                            	echo "Destroying OLD snapshot ${dset}@${sname}"
                            	zfs destroy ${dset}@${sname}
                    	else
    				## append this snap to an array
    				snaps[$index]=$sn
    				## increase our index counter
    				let "index += 1"
    			fi
    		done
    		## set our snap count and reset/reuse our index
    		local scount=${#snaps[@]}; index=0
    		## how many snapshots did we end up with..
    		if [ $scount -ge $SNAP_KEEP ]; then
    			## oops...too many snapshots laying around
    			## we need to destroy some of these
    			while [ $scount -ge $SNAP_KEEP ]; do
    				## zfs list always shows newest last
    				## we can use that to our advantage
    				echo "Destroying OLD snapshot ${snaps[$index]}"
    				zfs destroy ${snaps[$scount]}
    				## decrease scount and increase index
    				let "scount -= 1"; let "index += 1"
    			done
    		fi
    		## come on already...make that snapshot
    		echo "Creating ZFS snapshot ${dset}@${sname}"
    		zfs snapshot ${dset}@${sname}
    		## build our backup paths/snapshot paths
    		local mnt=`zfs get mountpoint ${dset}|tail -1|awk '{print $3}'`
    		## add this to a global
    		if [ "${BACKUP_PATHS}x" != 'x' ]; then
    			BACKUP_PATHS+=" ${mnt}/.zfs/snapshot/${sname}"
    		else
    			BACKUP_PATHS="${mnt}/.zfs/snapshot/${sname}"
    		fi
    	done
    }
    ## check/create lock file and proceed
    init(){
    	## check our lockfile status
    	if [ -f "${LOGBASE}/.backup.lock" ]; then
    		## get lockfile contents
    		local lpid=`cat "${LOGBASE}/.backup.lock"`
    		## see if this pid is still running
    		local ps=`ps auxww|grep $lpid|grep -v grep`
    		if [ "${ps}x" != 'x' ]; then
    			## looks like it's still running
    			echo "ERROR: This script is already running as: $ps"
    		else
    			## well the lockfile is there...stale?
    			echo "ERROR: Lockfile exists '${LOGBASE}/.backup.lock'"
    			echo -n "However, the contents do not match any "
    			echo "currently running process...stale lockfile?"
    		fi
    		## tell em what to do...
    		echo -n "To force script run please delete "
    		echo "'${LOGBASE}/.backup.lock'"
    		## exit now...
    		exit 0
    	else
    		## well no lockfile..let's make a new one
    		echo $$ > "${LOGBASE}/.backup.lock"
    	fi
    	## start our backup function
    	do_backup
    }
    ## make sure our log dir exits
    [ ! -d $LOGBASE ] && $MKDIR -p $LOGBASE
    ## this is where it all starts
    ## office is closed friday-sunday
    case $DOW in
    	Mon|Tue|Wed|Thu) init;;
    	*) ;;
    esac > $LOGFILE 2>&1
  • Aaron Hurt

    It doesn’t like my tags … guess I don’t have html tag permissions ;p

  • Vivek Gite

    You need to put them between pre tags. BTW, I’ve edited out your post. Thanks for sharing your solution.

  • Aaron Hurt

    I’ve updated the above to include lockfile management, enabling setting snapshot only days, tape full days and, tape incremental days or just to skip certain days. You can see the full script here: http://woodstock.anbcs.com/shell/dobackup.sh

    Thanks again Vivek for giving me the motivation to get that setup…it’s been a real lifesaver for us here.

    – Aaron

  • William

    Hi Aaron
    I’m customizing your script to my site. Where can I find the tapeswap.sh script?
    thanks
    It’s still very useful

Previous Script:

Next Script: