Backup with Hanoi Strategy

Question

I want your suggestions about this shell script. I use it to backup some configurations on my home media server.

What improvements would you do? Keep in mind, I want to optimize space and speed and at the same time, backup the maximum configurations over time.

p.-s. For more informations about the hanoi scheme: https://en.wikipedia.org/wiki/Backup_rotation_scheme.

ideas :

• Only use the hanoi scheme when the limit space is reach by checking the space required by rsync.

#!/bin/bash

# ==================================================================================
# Description:  BackUp script following a Hanoi scheme using rsync.
# Author:       Hugo Lapointe Di Giacomo
# ==================================================================================

set -o errexit
set -o nounset
#set -o xtrace

# ----------------------------------------------------------------------------------
# Options
# ----------------------------------------------------------------------------------
setsMax=16 # ................................ Maximum number of sets bakcup.
cmdPrefix="" # .............................. "echo" if dry-run enable, "" otherwise.
srcDir="/appdata" # ......................... Source directory to backup.
dstDir="/mnt/backup/snapshots" # ............ Destination directory to backup in.
logDir="/mnt/backup/logs" # ................. Logs directory.


# ----------------------------------------------------------------------------------
# Constants
# ----------------------------------------------------------------------------------
SEC_PER_DAY=86400; # ........................ Number of seconds per day.
LATEST_SNAP="$dstDir/latest"; # ............. The "latest" snap.
LATEST_LOG="$logDir/latest"; # .............. The "latest" log.
DATE_FORMAT="%Y-%m-%d"; # ................... The format of a date.
HOUR_FORMAT="%H:%M"; # ...................... The format of a hour.


# ----------------------------------------------------------------------------------
# Show usage of this script.
# ----------------------------------------------------------------------------------
function showUsage() {
    echo "Usage: $0 [OPTIONS]..."
    echo ""
    echo "Options available:"
    echo " -h, --help   Show usage of the script."
    echo " -m, --max    Maximum sets of backup."
    echo " -s, --src    Source directory to backup."
    echo " -d. --dst    Destination directory to backup in."
    echo " -l, --log    Log directory."
    echo " -r, --dry    Enable dry-run."
}


# ----------------------------------------------------------------------------------
# Parse arguments.
# ----------------------------------------------------------------------------------
function parseArgs() {
    while [ $# -gt 0 ]; do
        case $1 in
            -h|--help)
                showUsage $@
                exit 0
            ;;

            -m|--max)
                setsMax=${2:-$setsMax}
                shift 2
            ;;

            -s|--src)
                if [ -d "$2" ]; then
                    srcDir="$2"
                fi
                shift 2
            ;;

            -d|--dst)
                if [ -d "$2" ]; then
                    dstDir="$2"
                fi
                shift 2
            ;;

            -l|--log)
                if [ -d "$2" ]; then
                    logDir="$2"
                fi
                shift 2
            ;;

            -r|--dry)
                cmdPrefix="echo"
                shift 1
            ;;

            *)
                echo "Unknown option or value: $1"
                showUsage $@
                exit 1
            ;;
        esac
    done
}


# ----------------------------------------------------------------------------------
# Create a backup with rsync.
# ----------------------------------------------------------------------------------
function createBackUp() {   
    snapDir="$dstDir/$1" # ................. Name of the snap dir.
    logFile="$logDir/$1.log" # ............. Name of the log file.

    RSYNC_CMD="rsync " # ................... Rsync Command.
    RSYNC_CMD+="--archive " # .............. Enable recursion and preserve infos.
    RSYNC_CMD+="--verbose " # .............. Increase amount of infos printed.
    RSYNC_CMD+="--human-readable " # ....... Output number in readable format.
    RSYNC_CMD+="--progress " # ............. Show progress during transfer.
    RSYNC_CMD+="--delete " # ............... Delete files from receiving side.
    RSYNC_CMD+="--link-dest=$LATEST_SNAP " # "latest" symbolic link to hardlink.
    RSYNC_CMD+="$srcDir " # ................ Source directory to backup.
    RSYNC_CMD+="$snapDir " # ............... Destination directory to backup in.

    # Create backup and save the output in a log file.
    $cmdPrefix $RSYNC_CMD 2>&1 | tee "$logFile"
}

# ----------------------------------------------------------------------------------
# Update the latest links.
# ----------------------------------------------------------------------------------
function updateLatestLinks() {
    $cmdPrefix rm -f "$LATEST_SNAP"
    $cmdPrefix ln -s "$snapDir" "$LATEST_SNAP"

    $cmdPrefix rm -f "$LATEST_LOG"
    $cmdPrefix ln -s "$logFile" "$LATEST_LOG"
}

# ----------------------------------------------------------------------------------
# Calculate the previous move in the cycle.
# https://en.wikipedia.org/wiki/Backup_rotation_scheme
# ----------------------------------------------------------------------------------
function calculateDaysFromBackupOfSameSet() {
    local todayInSec=$(date +%s)
    local todayInDay=$(($todayInSec / $SEC_PER_DAY))
    
    local daysToBackup=$((2 ** ($setsMax - 1))) 

    local daysElapsed=0
    local i=1

    while [ $i -lt $(($daysToBackup / 2)) ]; do
        local rotation=$(($todayInDay & $i))
        
        if [ $rotation -eq 0 ]; then
            daysElapsed=$(($i * 2))
            break
        fi

        daysElapsed=$(($daysToBackup / 2))
        i=$((2 * $i))
    done
    
    echo $daysElapsed
}


# ----------------------------------------------------------------------------------
# Main function.
# ----------------------------------------------------------------------------------
function main() {
    parseArgs $@
    
    local todayDatetime=$(date +$DATE_FORMAT.$HOUR_FORMAT)

    if createBackUp $todayDatetime; then
        updateLatestLinks
        
        local daysElapsed=$(calculateDaysFromBackupOfSameSet)
        local expiredDay=$(date -d "$daysElapsed days ago" +$DATE_FORMAT)
        $cmdPrefix rm -frv "$dstDir/$expiredDay."*
    fi
}

main $@

Answer 1

Do not silently ignore invalid arguments

Your script allows to specify the source directory (and other locations). Here is the relevant part in function parseArgs():

            -s|--src)
                if [ -d "$2" ]; then
                    srcDir="$2"
                fi
                shift 2
            ;;

If --src <sourceDir> is specified with an invalid source directory then this argument is (silently) ignored. The consequence is that even a simple typo

backup --src /my_importatn_data

causes the default directory to be backed up, and not /my_important_data.

Wrong arguments (here: an invalid or not existing directory) should print an error message (to the standard error) and terminate the shell script (with a non-zero exit code).

Answer 2

Double-quote variables used as command arguments

Although in many places in the posted code the variables used as command arguments are correctly double-quoted, there are more than a few exceptions, for example parseArgs $@, which should be parseArgs "$@" to preserve arguments with spaces.

It's a good habit to systematically double-quote variables used as command arguments.

Collect argument lists in arrays instead of strings

The createBackUp function collects arguments for the rsync command in a string and executes it. This will not work if some elements (here $srcDir, $snapDir, or $LATEST_SNAP) contain spaces.

You can make it safe by using an array instead:

createBackUp() {   
    snapDir="$dstDir/$1" # ................. Name of the snap dir.
    logFile="$logDir/$1.log" # ............. Name of the log file.

    RSYNC_CMD=("rsync") # ................... Rsync Command.
    RSYNC_CMD+=("--archive") # .............. Enable recursion and preserve infos.
    RSYNC_CMD+=("--verbose") # .............. Increase amount of infos printed.
    RSYNC_CMD+=("--human-readable") # ....... Output number in readable format.
    RSYNC_CMD+=("--progress") # ............. Show progress during transfer.
    RSYNC_CMD+=("--delete") # ............... Delete files from receiving side.
    RSYNC_CMD+=("--link-dest=$LATEST_SNAP") # "latest" symbolic link to hardlink.
    RSYNC_CMD+=("$srcDir") # ................ Source directory to backup.
    RSYNC_CMD+=("$snapDir") # ............... Destination directory to backup in.

    # Create backup and save the output in a log file.
    $cmdPrefix "${RSYNC_CMD[@]}" 2>&1 | tee "$logFile"
}

I also dropped the function keyword which is not recommended in Bash.

Simplify using arithmetic context

The calculateDaysFromBackupOfSameSet uses several arithmetic operations and conditions which could be simplified and made more readable using arithmetic context:

calculateDaysFromBackupOfSameSet() {
    local i todayInDay daysToBackup
    local todayInSec=$(date +%s)

    ((todayInDay = todayInSec / SEC_PER_DAY))
    ((daysToBackup = 2 ** (setsMax - 2)))
    local daysElapsed=daysToBackup

    for ((i = 1; i < daysToBackup; i *= 2)); do
        if ! ((todayInDay & i)); then
            ((daysElapsed = i * 2))
            break
        fi
    done

    echo "$daysElapsed"
}

I also eliminated some repeated daysToBackup / 2. I believe this is equivalent to the original, but I haven't tested it. Excuse me if I made an error, the point is more to demonstrate the power of the arithmetic context.