6
\$\begingroup\$

I want your suggestions about this shell script. I use it to backup some configurations on my home media server.

What improvements would you do? Keep in mind, I want to optimize space and speed and at the same time, backup the maximum configurations over time.

p.-s. For more informations about the hanoi scheme: https://en.wikipedia.org/wiki/Backup_rotation_scheme.

ideas :

  • Only use the hanoi scheme when the limit space is reach by checking the space required by rsync.
#!/bin/bash

# ==================================================================================
# Description:  BackUp script following a Hanoi scheme using rsync.
# Author:       Hugo Lapointe Di Giacomo
# ==================================================================================

set -o errexit
set -o nounset
#set -o xtrace

# ----------------------------------------------------------------------------------
# Options
# ----------------------------------------------------------------------------------
setsMax=16 # ................................ Maximum number of sets bakcup.
cmdPrefix="" # .............................. "echo" if dry-run enable, "" otherwise.
srcDir="/appdata" # ......................... Source directory to backup.
dstDir="/mnt/backup/snapshots" # ............ Destination directory to backup in.
logDir="/mnt/backup/logs" # ................. Logs directory.


# ----------------------------------------------------------------------------------
# Constants
# ----------------------------------------------------------------------------------
SEC_PER_DAY=86400; # ........................ Number of seconds per day.
LATEST_SNAP="$dstDir/latest"; # ............. The "latest" snap.
LATEST_LOG="$logDir/latest"; # .............. The "latest" log.
DATE_FORMAT="%Y-%m-%d"; # ................... The format of a date.
HOUR_FORMAT="%H:%M"; # ...................... The format of a hour.


# ----------------------------------------------------------------------------------
# Show usage of this script.
# ----------------------------------------------------------------------------------
function showUsage() {
    echo "Usage: $0 [OPTIONS]..."
    echo ""
    echo "Options available:"
    echo " -h, --help   Show usage of the script."
    echo " -m, --max    Maximum sets of backup."
    echo " -s, --src    Source directory to backup."
    echo " -d. --dst    Destination directory to backup in."
    echo " -l, --log    Log directory."
    echo " -r, --dry    Enable dry-run."
}


# ----------------------------------------------------------------------------------
# Parse arguments.
# ----------------------------------------------------------------------------------
function parseArgs() {
    while [ $# -gt 0 ]; do
        case $1 in
            -h|--help)
                showUsage $@
                exit 0
            ;;

            -m|--max)
                setsMax=${2:-$setsMax}
                shift 2
            ;;

            -s|--src)
                if [ -d "$2" ]; then
                    srcDir="$2"
                fi
                shift 2
            ;;

            -d|--dst)
                if [ -d "$2" ]; then
                    dstDir="$2"
                fi
                shift 2
            ;;

            -l|--log)
                if [ -d "$2" ]; then
                    logDir="$2"
                fi
                shift 2
            ;;

            -r|--dry)
                cmdPrefix="echo"
                shift 1
            ;;

            *)
                echo "Unknown option or value: $1"
                showUsage $@
                exit 1
            ;;
        esac
    done
}


# ----------------------------------------------------------------------------------
# Create a backup with rsync.
# ----------------------------------------------------------------------------------
function createBackUp() {   
    snapDir="$dstDir/$1" # ................. Name of the snap dir.
    logFile="$logDir/$1.log" # ............. Name of the log file.

    RSYNC_CMD="rsync " # ................... Rsync Command.
    RSYNC_CMD+="--archive " # .............. Enable recursion and preserve infos.
    RSYNC_CMD+="--verbose " # .............. Increase amount of infos printed.
    RSYNC_CMD+="--human-readable " # ....... Output number in readable format.
    RSYNC_CMD+="--progress " # ............. Show progress during transfer.
    RSYNC_CMD+="--delete " # ............... Delete files from receiving side.
    RSYNC_CMD+="--link-dest=$LATEST_SNAP " # "latest" symbolic link to hardlink.
    RSYNC_CMD+="$srcDir " # ................ Source directory to backup.
    RSYNC_CMD+="$snapDir " # ............... Destination directory to backup in.

    # Create backup and save the output in a log file.
    $cmdPrefix $RSYNC_CMD 2>&1 | tee "$logFile"
}

# ----------------------------------------------------------------------------------
# Update the latest links.
# ----------------------------------------------------------------------------------
function updateLatestLinks() {
    $cmdPrefix rm -f "$LATEST_SNAP"
    $cmdPrefix ln -s "$snapDir" "$LATEST_SNAP"

    $cmdPrefix rm -f "$LATEST_LOG"
    $cmdPrefix ln -s "$logFile" "$LATEST_LOG"
}

# ----------------------------------------------------------------------------------
# Calculate the previous move in the cycle.
# https://en.wikipedia.org/wiki/Backup_rotation_scheme
# ----------------------------------------------------------------------------------
function calculateDaysFromBackupOfSameSet() {
    local todayInSec=$(date +%s)
    local todayInDay=$(($todayInSec / $SEC_PER_DAY))

    local daysToBackup=$((2 ** ($setsMax - 1))) 

    local daysElapsed=0
    local i=1

    while [ $i -lt $(($daysToBackup / 2)) ]; do
        local rotation=$(($todayInDay & $i))

        if [ $rotation -eq 0 ]; then
            daysElapsed=$(($i * 2))
            break
        fi

        daysElapsed=$(($daysToBackup / 2))
        i=$((2 * $i))
    done

    echo $daysElapsed
}


# ----------------------------------------------------------------------------------
# Main function.
# ----------------------------------------------------------------------------------
function main() {
    parseArgs $@

    local todayDatetime=$(date +$DATE_FORMAT.$HOUR_FORMAT)

    if createBackUp $todayDatetime; then
        updateLatestLinks

        local daysElapsed=$(calculateDaysFromBackupOfSameSet)
        local expiredDay=$(date -d "$daysElapsed days ago" +$DATE_FORMAT)
        $cmdPrefix rm -frv "$dstDir/$expiredDay."*
    fi
}

main $@

EDIT 01

#!/bin/bash

# =============================================================================
# Description:  BackUp script following a Hanoi scheme using rsync.
# Author:       Hugo Lapointe Di Giacomo
# =============================================================================

#set -o errexit
#set -o nounset
#set -o xtrace

# -----------------------------------------------------------------------------
# Options with default values
# -----------------------------------------------------------------------------
setsMax=16 # .................... Maximum number of sets bakcup.
cmdPrefix="" # .................. "echo" if dry-run enable, "" otherwise.
srcDir="/appdata" # ............. Source directory to backup.
dstDir="/mnt/backup/snapshots" #  Destination directory to backup in.
logDir="/mnt/backup/logs" # ..... Logs directory.


# -----------------------------------------------------------------------------
# Constants
# -----------------------------------------------------------------------------
SEC_PER_DAY=86400 # ......................... Number of seconds per day.
LATEST_SNAP="$dstDir/latest" # .............. The "latest" snap.
LATEST_LOG="$logDir/latest" # ............... The "latest" log.
DATE_FORMAT="%Y-%m-%d" # .................... The format of a date.
HOUR_FORMAT="%H:%M" # ....................... The format of a hour.
DATETIME_FORMAT="$DATE_FORMAT.$HOUR_FORMAT" # The format of a datetime.
EMPTY_DIR="/dev/empty" # .................... Empty directory to work with.


# -----------------------------------------------------------------------------
# Show usage of this script.
# -----------------------------------------------------------------------------
showUsage() {
    local scriptName="$0"

    echo "Usage: $scriptName [OPTIONS] ..."
    echo ""
    echo "Options available:"
    echo " -h, --help   Show usage of the script."
    echo " -m, --max    Maximum sets of backup."
    echo " -s, --src    Source directory to backup."
    echo " -d. --dst    Destination directory to backup in."
    echo " -l, --log    Log directory."
    echo " -r, --dry    Enable dry-run."
}


# -----------------------------------------------------------------------------
# Parse arguments.
# -----------------------------------------------------------------------------
parseArgs() {
    while (($# > 0)); do
        case $1 in
            -h|--help)
                showUsage "$@"
                exit 0
            ;;

            -m|--max)
                setsMax=${2:-$setsMax}
                shift 2
            ;;

            -s|--src)
                if [ -d "$2" ]; then
                    srcDir="$2"
                else
                    echo "The source directory $2 must exist."
                    echo "The source directory by default will be use : $srcDir."
                fi
                shift 2
            ;;

            -d|--dst)
                if [ -d "$2" ]; then
                    dstDir="$2"
                else
                    echo "The destination directory $2 must exist."
                    echo "The source directory by default will be use : $dstDir."
                fi
                shift 2
            ;;

            -l|--log)
                if [ -d "$2" ]; then
                    logDir="$2"
                else
                    echo "The log directory $2 must exits."
                    echo "The log directory by default will be use : $logDir."
                fi
                shift 2
            ;;

            -r|--dry)
                cmdPrefix="echo"
                shift 1
            ;;

            *)
                echo "Unknown option or value: $1"
                showUsage "$@"
                exit 1
            ;;
        esac
    done
}


# -----------------------------------------------------------------------------
# Create a backup with rsync.
# -----------------------------------------------------------------------------
createBackUp() {    
    local snapDir="$1"
    local logFile="$2"

    rsyncCmd=("rsync") # ................... Rsync Command.
    rsyncCmd+=("--archive") # .............. Recursion and preserve infos.
    rsyncCmd+=("--verbose") # .............. Increase amount of infos printed.
    rsyncCmd+=("--human-readable") # ....... Output number in readable format.
    rsyncCmd+=("--progress") # ............. Show progress of each file.
    rsyncCmd+=("--delete") # ............... Delete files from receiving side.
    rsyncCmd+=("--link-dest=$LATEST_SNAP") # Symbolic link to hardlink.
    rsyncCmd+=("$srcDir/") # ............... Source directory to backup.
    rsyncCmd+=("$snapDir/") # .............. Directory to backup in.

    # Create backup and save the output in a log file.
    $cmdPrefix "${rsyncCmd[@]}" 2>&1 | tee "$logFile"
}


# -----------------------------------------------------------------------------
# Remove directory efficiently.
# -----------------------------------------------------------------------------
removeDirEfficiently() {
    local dirToRemove="$1"

    if ! [ -d "$EMPTY_DIR" ]; then
        mkdir "$EMPTY_DIR"
    fi

    rsyncCmd=("rsync") # ............ Rsync Command.
    rsyncCmd+=("--archive") # ....... Recursion and preserve infos.
    rsyncCmd+=("--verbose") # ....... Increase amount of infos printed.
    rsyncCmd+=("--human-readable") #  Output number in readable format.
    rsyncCmd+=("--progress") # ...... Show progress of each file.
    rsyncCmd+=("--delete") # ........ Delete files from receiving side.
    rsyncCmd+=("$EMTPY_DIR/") # ..... Empty directory.
    rsyncCmd+=("$dirToRemove/") # ... Directory to remove.

    # Efficiently empty the directory passed in params.
    $cmdPrefix "${rsyncCmd[@]}" 2>&1 | tee "$logFile"
}


# -----------------------------------------------------------------------------
# Remove expired BackUps.
# -----------------------------------------------------------------------------
removeExpiredBackUps() {
    local todayDateTime="$1"

    local daysElapsed=$(calculateDaysFromBackupOfSameSet $1)
    local expiredDay=$(date -d "$daysElapsed days ago" +$DATE_FORMAT)

    for dir in $(find "$dstDir/$expiredDay.*" -type d 2> /dev/null); do
        removeDirEfficiently "$dir"
    done
}


# -----------------------------------------------------------------------------
# Update the latest links.
# -----------------------------------------------------------------------------
updateLatestLinks() {
    local snapDir="$1"
    local logFile="$2"

    $cmdPrefix rm -f "$LATEST_SNAP"
    $cmdPrefix ln -s "$snapDir/" "$LATEST_SNAP"

    $cmdPrefix rm -f "$LATEST_LOG"
    $cmdPrefix ln -s "$logFile" "$LATEST_LOG"
}


# -----------------------------------------------------------------------------
# Calculate the previous move in the cycle.
# https://en.wikipedia.org/wiki/Backup_rotation_scheme
# -----------------------------------------------------------------------------
calculateDaysFromBackupOfSameSet() {    
    local todayDateTime="$1"
    local backUpDateInSec=$(date -d "$todayDateTime" +%s)

    local i todayInDay daysToBackup
    ((todayInDay = backUpDateInSec / SEC_PER_DAY))
    ((daysToBackup = 2 ** (setsMax - 2)))

    local daysElapsed=daysToBackup

    for ((i = 1; i < daysToBackup; i *= 2)); do
            if ! ((todayInDay & i)); then
                    ((daysElapsed = i * 2))
                    break
            fi
    done

    echo $daysElapsed
}


# -----------------------------------------------------------------------------
# Main function.
# -----------------------------------------------------------------------------
main() {
    parseArgs "$@"

    local todayDateTime="$(date)"
    local backUpName="$(date -d "$todayDateTime" +$DATETIME_FORMAT)"
    local snapDir="$dstDir/$backUpName"
    local logFile="$logDir/$backUpName.log"

    if ! createBackUp "$snapDir" "$logFile"; then
        removeDirEfficiently "$snapDir"

        echo "The backUp of $dstDir was unsucessful."
        echo "Check the log $logFile for more details."
    fi

    updateLatestLinks "$snapDir" "$logFile"
    removeExpiredBackUps "$todayDateTime"
}

main "$@"
\$\endgroup\$
  • \$\begingroup\$ I have rolled back the last edit. Please do not update the code in your question to incorporate feedback from answers, doing so goes against the Question + Answer style of Code Review. This is not a forum where you should keep the most updated version in your question. Please see what you may and may not do after receiving answers. \$\endgroup\$ – Zeta Mar 15 '19 at 19:19
  • \$\begingroup\$ @Zeta, understood. \$\endgroup\$ – hlapointe Mar 15 '19 at 19:25
10
\$\begingroup\$

Do not silently ignore invalid arguments

Your script allows to specify the source directory (and other locations). Here is the relevant part in function parseArgs():

            -s|--src)
                if [ -d "$2" ]; then
                    srcDir="$2"
                fi
                shift 2
            ;;

If --src <sourceDir> is specified with an invalid source directory then this argument is (silently) ignored. The consequence is that even a simple typo

backup --src /my_importatn_data

causes the default directory to be backed up, and not /my_important_data.

Wrong arguments (here: an invalid or not existing directory) should print an error message (to the standard error) and terminate the shell script (with a non-zero exit code).

| improve this answer | |
\$\endgroup\$
8
\$\begingroup\$

Double-quote variables used as command arguments

Although in many places in the posted code the variables used as command arguments are correctly double-quoted, there are more than a few exceptions, for example parseArgs $@, which should be parseArgs "$@" to preserve arguments with spaces.

It's a good habit to systematically double-quote variables used as command arguments.

Collect argument lists in arrays instead of strings

The createBackUp function collects arguments for the rsync command in a string and executes it. This will not work if some elements (here $srcDir, $snapDir, or $LATEST_SNAP) contain spaces.

You can make it safe by using an array instead:

createBackUp() {   
    snapDir="$dstDir/$1" # ................. Name of the snap dir.
    logFile="$logDir/$1.log" # ............. Name of the log file.

    RSYNC_CMD=("rsync") # ................... Rsync Command.
    RSYNC_CMD+=("--archive") # .............. Enable recursion and preserve infos.
    RSYNC_CMD+=("--verbose") # .............. Increase amount of infos printed.
    RSYNC_CMD+=("--human-readable") # ....... Output number in readable format.
    RSYNC_CMD+=("--progress") # ............. Show progress during transfer.
    RSYNC_CMD+=("--delete") # ............... Delete files from receiving side.
    RSYNC_CMD+=("--link-dest=$LATEST_SNAP") # "latest" symbolic link to hardlink.
    RSYNC_CMD+=("$srcDir") # ................ Source directory to backup.
    RSYNC_CMD+=("$snapDir") # ............... Destination directory to backup in.

    # Create backup and save the output in a log file.
    $cmdPrefix "${RSYNC_CMD[@]}" 2>&1 | tee "$logFile"
}

I also dropped the function keyword which is not recommended in Bash.

Simplify using arithmetic context

The calculateDaysFromBackupOfSameSet uses several arithmetic operations and conditions which could be simplified and made more readable using arithmetic context:

calculateDaysFromBackupOfSameSet() {
    local i todayInDay daysToBackup
    local todayInSec=$(date +%s)

    ((todayInDay = todayInSec / SEC_PER_DAY))
    ((daysToBackup = 2 ** (setsMax - 2)))
    local daysElapsed=daysToBackup

    for ((i = 1; i < daysToBackup; i *= 2)); do
        if ! ((todayInDay & i)); then
            ((daysElapsed = i * 2))
            break
        fi
    done

    echo "$daysElapsed"
}

I also eliminated some repeated daysToBackup / 2. I believe this is equivalent to the original, but I haven't tested it. Excuse me if I made an error, the point is more to demonstrate the power of the arithmetic context.

| improve this answer | |
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.