Script to record an inventory of a backup

Question

This is a script designed to make an inventory file which records the contents of a directory allowing verification that a backup has the same data. The same script can also

A full github repo is available at https://github.com/mikedlr/backup-tools including various test cases using python behave.

This is written as a shell script for reasonably rapid development and maxium ease of rapid portability. If it's useful for others I might rewrite in another language/other languages as well.

Comments appreciated especially for

• style
• safety / correctness
• portability

I have deleted a few irrelevant comments from the code to save your reading time. I'd happily make this portable to non bash shells but I think the use of arrays is pretty unavoidable.

#!/bin/bash

# inventory - record an inventory with checksums of the files in a directory
# Copyright (C) 2016 Michael De La Rue

# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as
# published by the Free Software Foundation, either version 3 of the
# License, or (at your option) any later version.

# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU Affero General Public License for more details.

# You should have received a copy of the GNU Affero General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

TEMP=$(getopt -o ho:i:x:cv --long help,output:,input:,check,verbose -n 'inventory.sh' -- "$@" )

if [ $? != 0 ] ; then echo "Argument parsing fail; terminating..." >&2 ; exit 1 ; fi

eval set -- "$TEMP"

usage() { 
  cat <<EOF 
inventory - create or verify an inventory of a directory, typically for backup verification
 -h --help  - output usage information
 -c --check - read inventory file and verify directory
 -v --verbose - verbose output
 -o --output <file> - output to <file>
 -x --exclude <file> - exclude filenames matching expressions in <file>
 -i --input <file> - use <file> for input (as an inventory file), together with -c
EOF
}

OFILE=""
IFILE=""
XFILE=""
CHECK="false"
VERBOSE="false"
while true ; do
    case "$1" in
        -h|--help) usage; exit 0;;
        -c|--check) CHECK="true" ; shift ;;
        -v|--verbose) VERBOSE="true" ; shift ;;
        -o|--output)
        if [ "" != "$OFILE" ]
        then
            echo "Only one output file allowed.  Terminating" >&2
            exit 1
        fi
        OFILE=$2; shift 2 ;;
        -i|--input)
        if [ "" != "$IFILE" ]
        then
            echo "Only one input file allowed.  Terminating" >&2
            exit 1
        fi
        IFILE=$2; shift 2 ;;
        -x|--exclude)
        if [ "" != "$XFILE" ]
        then
            echo "Only one exclude file currently allowed.  Terminating" >&2
            exit 1
        fi
        XFILE=$2; shift 2 ;;
        --) shift ; break ;;
        *) echo "Internal error!" ; exit 1 ;;
    esac
done

if [ "" = "$1" ]
then
echo "must have at least one directory argument to run inventory on" >&2
exit 1
fi

if [ "false" = "$CHECK" ]
then
if [ "" != "$IFILE" ]
then
    echo "cannot give input file when generating inventory" >&2
    exit 1
fi
fi


check_inventory ()
{
local check_dir="$1"
if [ "false" = "$VERBOSE" ]
then
    SHAOPTS="--check --quiet"
else
    SHAOPTS="--check"
fi

if [ "" != "$OFILE" ]
then
    echo "cannot give output file when checking inventory" >&2
    exit 1
fi
( if [ "" = "$IFILE" ]
then
    head -n-2
else
    head -n-2 "$IFILE"
fi )| tail -n+2 | (cd "$check_dir"; sha384sum $SHAOPTS - )
SHASUMRES=$?
exit $SHASUMRES
}

create_inventory ()
{
local record_dir="$1"
set -e
local TEMPFILE
TEMPFILE=$(mktemp)
( echo inventoryfile-0 at "$(date --rfc-3339=seconds --utc)" directory: "$(readlink -f "$1")" ) > "$TEMPFILE"
( cd "$record_dir"
    {   find . -type f "${FINDFILTER[@]}" -exec sha384sum {} + | sort -k 2
    echo -----------------------------------------------
    } >> "$TEMPFILE"
    #split to two lines to avoid reading and writing at the same time
    FOOT="inventory checksum $(sha384sum "$TEMPFILE" | sed 's/ .*//')"
    echo "$FOOT" >>  "$TEMPFILE"
)
if [ "" = "$OFILE" ]
then
   cat "$TEMPFILE"
else
   mv "$TEMPFILE" "$OFILE"
fi
}

if [  "true" = "$CHECK" ]
then
check_inventory "$1"
fi

FINDFILTER=()

if [ "" != "$XFILE" ]
then
if [ ! -r "$XFILE" ] 
then
    echo "$XFILE no such file or directory" >&2
    exit 5
fi
while read line           
do           
    #TODO: comments
    if [ "" != "$line" ]
    then
       FINDFILTER=("${FINDFILTER[@]}" -not -path "$line")
    fi
done <"$XFILE"
fi

create_inventory "$1"

Answer 1

Use a fatal function

This kind of code appears in many places:

echo "Only one output file allowed.  Terminating" >&2
exit 1

I suggest to introduce a fatal function:

fatal() {
    echo "$*  Terminating" >&2
    exit 1
}

So that you can replace the example above with a simpler single line:

fatal "Only one output file allowed."

Checking for empty value

Instead of this:

if [ "" != "$IFILE" ]

A shorter way to accomplish the same:

if [ ! "$IFILE" ]

Appending to arrays

Instead of this:

FINDFILTER=("${FINDFILTER[@]}" -not -path "$line")

The correct way to append to an array is using the += operator:

FINDFILTER+=(-not -path "$line")

Pointless local variable

Instead of this:

SHASUMRES=$?
exit $SHASUMRES

You could skip the local variable:

exit $?

Too compact writing style

Instead of this:

if [ $? != 0 ] ; then echo "Argument parsing fail; terminating..." >&2 ; exit 1 ; fi

It will be much more readable to split to multiple lines:

if [ $? != 0 ]; then
    echo "Argument parsing fail; terminating..." >&2
    exit 1
fi

Inconsistent indentation

I see multiple indentation styles for nested code blocks (for example if-statements): indent with 2 spaces, 4 spaces, or none. I suggest to indent consistently (I like 4 spaces).