Long-term archiving + encrypting shell script

Question

Yesterday, I very quickly, in an hour, wrote the very first version of my long-term archiving + encrypting shell script.

I am aware it may have too many flaws, but none of them really prevent it from normal operation, so before I take another step and restructure it with printf and better error reporting, structure, exit codes, etc. I would like you to have a look first.

What it does:

1. It archives a given directory into a tarball.
2. It compresses (with maximum effort) that tarball using xz.
3. It encrypts that xz archive using openssl and AES-256.
4. It creates and checks the created SHA-512 sum file.
5. It then decrypts the created file to a temporary file.
6. Finally, it compares the decrypted file with the original xz archive.

I am prepared to waste time and effort in order to be 100% sure before burning my precious data to Blu-ray M-disc.

I know I should not have to do steps 5 and 6 just because there is almost a 0% chance for the data to get corrupted in the process. But since I have a rather large NVMe SSD drive, and relatively fast CPU, it should not be so shocking that I wrote it with motto better safe than sorry.

#!/bin/sh

[ -z "$1" ] && echo "You need to give me one directory!" && exit 1

[ ! -d "$1" ] && echo "This is not a directory!" && exit 1

[ ! -w "$PWD" ] && echo "The current directory is not writable by you!" && exit 1

dir=$(basename "$1")
today=$(date +%Y-%m-%d)

backupdir="${dir}_${today}"
# backupdir="${dir}"

backupfile="${backupdir}.tar"

bold=$(tput bold)
red=$(tput setaf 1)
yellow=$(tput setaf 3)
nocolor=$(tput sgr0)
bold_red="$bold$red"
bold_yellow="$bold$yellow"

echo
echo "${bold_yellow}1/6: TAR is archiving your directory '${bold_red}$dir${bold_yellow}' into '${bold_red}$backupfile${bold_yellow}'${nocolor}"
echo

if ! tar -cf "$backupfile" "$1" --totals
then
    echo "TAR failed to archive your directory"
    exit 1
fi

echo
echo "${bold_yellow}2/6: XZ is compressing your '${bold_red}$backupfile${bold_yellow}' into '${bold_red}$backupfile.xz${bold_yellow}'${nocolor}"
echo

if ! xz --format=xz --check=sha256 -9 --threads=8 --keep --verbose --verbose --extreme "$backupfile"
then
    echo "XZ failed to compress your backup file"
    exit 1
fi

echo
echo "${bold_yellow}3/6: OpenSSL is encrypting your '${bold_red}$backupfile.xz${bold_yellow}' into '${bold_red}$backupfile.xz.enc${bold_yellow}'${nocolor}"
echo

if ! pv -W "$backupfile.xz" | openssl enc -aes-256-cbc -md sha256 -salt -out "$backupfile.xz.enc" -pass file:/home/vlastimil/.blablablabla 2> /dev/null
then
    echo "OpenSSL failed to encrypt your compressed file"
    exit 1
fi

echo
echo "${bold_yellow}4/6: sha512sum is creating hash file '${bold_red}$backupfile.xz.enc.SHA512SUM${bold_yellow}'${nocolor}"
echo

if ! sha512sum -b "$backupfile.xz.enc" > "$backupfile.xz.enc.SHA512SUM"
then
    echo "sha512sum failed to compute hashsum file"
    exit 1
fi

sha512sum -c "$backupfile.xz.enc.SHA512SUM"

echo
echo "${bold_yellow}5/6: Decrypting your '${bold_red}$backupfile.xz.enc${bold_yellow}' into a temporary file '${bold_red}$backupfile.xz.dec${bold_yellow}'${nocolor}"
echo

if ! pv -W "$backupfile.xz.enc" | openssl enc -aes-256-cbc -md sha256 -salt -out "$backupfile.xz.dec" -d -pass file:/home/vlastimil/.blablablabla 2> /dev/null
then
    echo "OpenSSL failed to decrypt your encrypted file"
    exit 1
fi

echo
echo "${bold_yellow}6/6: Comparing your un-encrypted, compressed, archived, directory file '${bold_red}$backupfile.xz${bold_yellow}' with a temporary file '${bold_red}$backupfile.xz.dec${bold_yellow}'${nocolor}"
echo

if ! cmp "$backupfile.xz" "$backupfile.xz.dec"
then
    echo "Failed to compare files"
    exit 1
else
    echo "Your directory '${bold_red}$dir${nocolor}' has been successfully stored as '${bold_red}$backupfile.xz.enc${nocolor}'"
    rm "$backupfile.xz.dec"
    rm "$backupfile.xz"
    rm "$backupfile"
fi

Answer 1

It's a fine script. Even shellcheck doesn't find anything in it. Well done!

I only have minor comments.

You can safely omit the sha512sum -c. It would only fail if the previous command failed. For the rest, I cannot say it's complete paranoia.

The password file used with openssl commands is buried in the script and repeated twice. It would be better to put that value in a variable and define at the top of the file.

A helper function would be more convenient to print the headers for the steps, instead of blocks of echos.

before I take another step and restructure it with printf and better error reporting, structure, exit codes, etc. I would like you to have a look first

I don't see what benefits printf will bring, and the error reporting, structure, exit codes look just fine.

I would rename bold_red and bold_yellow to be more about the purpose than presentation. Like <em> in HTML is for emphasis, and <strong> for strong emphasis, And the stylesheet (in this case the actual value of the variables) define the presentation, nicely segregated from the code.

Answer 2

Self-review for the follow-up question, which I will post in the following days.

Treat unset variables as an error when substituting

Add this to the beginning of the script:

set -o nounset

[ "$#" -eq 0 ] instead of [ -z "$1" ]

It seems testing number of arguments is a better approach than testing if the first argument is set. Also, I added:

[ "$#" -gt 1 ] && echo "Multiple arguments are not supported!" && exit 1

for an obvious reason.

As pointed out by Janos, put password file path into variable

And put it in the beginning of the script (maybe with a comment).

# In order to compress many directories with the same password,
# I decided to create a password file to avoid too many annoying prompts
passwd_file="/home/vlastimil/.blablablabla"

Test for the existence of the password file

It may happen that you migrate to a different platform and / or accidentally delete the file.

[ ! -f "$passwd_file" ] && echo "The password file does not exist!" && exit 1

You should also test for existence of the processed files

Like this, the comments are optional:

# TAR is archiving the directory into the following temporary file,
# it gets deleted on successful processing
[ -f "${backup_file}" ] && echo "File ${backup_file} exists, exiting." && exit 1

# XZ is compressing the archive into the following temporary file,
# it gets deleted on successful processing
[ -f "${backup_file}.xz" ] && echo "File ${backup_file}.xz exists, exiting." && exit 1

# OpenSSL is decrypting the file (for verification) into the following temporary file,
# it gets deleted on successful processing
[ -f "${backup_file}.xz.dec" ] && echo "File ${backup_file}.xz.dec exists, exiting." && exit 1

# OpenSSL is encrypting the compressed file into the following permanent file,
# this is the result of successful processing
[ -f "${backup_file}.xz.enc" ] && echo "File ${backup_file}.xz.enc exists, exiting." && exit 1

# SHA512SUM is computing a hash sum of the encrypted file into the following permanent file,
# this is the result of successful processing
[ -f "${backup_file}.xz.enc.SHA512SUM" ] && echo "File ${backup_file}.xz.enc.SHA512SUM exists, exiting." && exit 1

Define all common colors (optional)

It happens one might like a change and / or later wants to re-define the styling. So this may be good idea, just comment out the colors that you don't use.

red=$(tput setaf 1)
blue=$(tput setaf 4)
cyan=$(tput setaf 6)
green=$(tput setaf 2)
white=$(tput setaf 7)
yellow=$(tput setaf 3)
magenta=$(tput setaf 5)

As pointed out by Janos, color definitions should be renamed

I would rename bold_red and bold_yellow to be more about the purpose than presentation.

And I agree:

action_number_color="${bold}${red}"
action_color="${bold}${yellow}"
file_color="${bold}${white}"
success_color="${bold}${green}"

Again, as pointed out by Janos, there should be a helper printing function

A helper function would be more convenient to print the headers for the steps, instead of blocks of echos.

Like this, I added some leading and trailing coloring:

print_action()
{
    echo
    echo "${action_number_color}$1${no_color}"
    echo
}

There could be a helper error function too

For instance:

print_error()
{
    echo
    echo "$1"
    exit 1
}

There are too long lines of strings

Both in the code, and in the output. It could be simplified for example:

print_action "1/7:${action_color}
    TAR is archiving the directory
    ${file_color}${dir}${action_color}
    into
    ${file_color}${backup_file}"

tar --verify for enhanced integrity

The man page says:

-W, --verify: attempt to verify the archive after writing it

Just to note: I take it, this option is only usable when creating the archive, and not if later checking the integrity.

ifs can be completely pulled out of the whole script

More importantly, it makes the script more readable this way:

tar --create --file="${backup_file}" "$1" --totals --verify ||

print_error "TAR failed to archive the directory!"

Too long lines of code could be split

There were also places where the code itself could be formatted in two lines simply with the already present | (pipe), which acts as delimiter too.

pv --progress --timer --eta --rate --average-rate --bytes "${backup_file}.xz" |
openssl enc -aes-256-cbc -md sha256 -salt -out "${backup_file}.xz.enc" -pass file:"$passwd_file" 2> /dev/null ||

print_error "OpenSSL failed to encrypt the compressed file!"

Average rate could further be added to the pv statement

As shown above, this could be somewhat useful to me at least if the file is very large.

Remove -W parameter from pv

The --wait parameter is useless in case like this, where we are not waiting for user input. What's more, it caused pv not to output anything on small files.

As pointed out by Janos, remove the sha512sum --check

Not only it is useless in this case, but it takes a few seconds. Better yet, you could replace it with showing the SHA-512 hash sum like this:

awk '{ print $1 }' "${backup_file}.xz.enc.SHA512SUM"

Use cmp with --silent parameter

As you are not interested in actual differences in those files, and only wish to check the return value of cmp, this is the right way:

cmp --silent "${backup_file}.xz" "${backup_file}.xz.dec" &&

echo "Files are identical." ||

print_error "Failed to compare files!"

Re-write awkward texts

I have re-written most of them, but one example:

Comparing your un-encrypted, compressed, archived, directory file

could sound better as

CMP is comparing the un-encrypted compressed file

Current code looks like this

#!/bin/sh

# Treat unset variables as an error when substituting
set -o nounset

[ "$#" -eq 0 ] && echo "You need to give me one directory!" && exit 1
[ "$#" -gt 1 ] && echo "Multiple arguments are not supported!" && exit 1
[ ! -d "$1" ] && echo "This is not a directory!" && exit 1
[ ! -w "$PWD" ] && echo "The current directory is not writable by you!" && exit 1

# In order to compress many directories with the same password,
# I decided to create a password file to avoid too many annoying prompts
passwd_file="/home/vlastimil/.blablablabla"

[ ! -f "$passwd_file" ] && echo "The password file does not exist!" && exit 1

dir=$(basename "$1")
today=$(date +%Y-%m-%d)

backup_dir="${dir}_${today}"
# backup_dir="${dir}"

backup_file="${backup_dir}.tar"

# TAR is archiving the directory into the following temporary file,
# it gets deleted on successful processing
[ -f "${backup_file}" ] && echo "File ${backup_file} exists, exiting." && exit 1

# XZ is compressing the archive into the following temporary file,
# it gets deleted on successful processing
[ -f "${backup_file}.xz" ] && echo "File ${backup_file}.xz exists, exiting." && exit 1

# OpenSSL is decrypting the file (for verification) into the following temporary file,
# it gets deleted on successful processing
[ -f "${backup_file}.xz.dec" ] && echo "File ${backup_file}.xz.dec exists, exiting." && exit 1

# OpenSSL is encrypting the compressed file into the following permanent file,
# this is the result of successful processing
[ -f "${backup_file}.xz.enc" ] && echo "File ${backup_file}.xz.enc exists, exiting." && exit 1

# SHA512SUM is computing a hash sum of the encrypted file into the following permanent file,
# this is the result of successful processing
[ -f "${backup_file}.xz.enc.SHA512SUM" ] && echo "File ${backup_file}.xz.enc.SHA512SUM exists, exiting." && exit 1

bold=$(tput bold)
no_color=$(tput sgr0)

red=$(tput setaf 1)
#blue=$(tput setaf 4)
#cyan=$(tput setaf 6)
green=$(tput setaf 2)
white=$(tput setaf 7)
yellow=$(tput setaf 3)
#magenta=$(tput setaf 5)

action_number_color="${bold}${red}"
action_color="${bold}${yellow}"
file_color="${bold}${white}"
success_color="${bold}${green}"

print_action()
{
    echo
    echo "${action_number_color}$1${no_color}"
    echo
}

print_error()
{
    echo
    echo "$1"
    exit 1
}

#------------------------------------------------------------------------------------------------------------------

print_action "1/7:${action_color}
    TAR is archiving the directory
    ${file_color}${dir}${action_color}
    into
    ${file_color}${backup_file}"

tar --create --file="${backup_file}" "$1" --totals --verify ||

print_error "TAR failed to archive the directory!"

#------------------------------------------------------------------------------------------------------------------

print_action "2/7:${action_color}
    XZ is compressing the archive
    ${file_color}${backup_file}${action_color}
    into
    ${file_color}${backup_file}.xz"

xz --format=xz --check=sha256 -9 --threads=8 --keep --verbose --extreme "${backup_file}" ||

print_error "XZ failed to compress the backup file!"

#------------------------------------------------------------------------------------------------------------------

print_action "3/7:${action_color}
    OpenSSL is encrypting the compressed file
    ${file_color}${backup_file}.xz${action_color}
    into
    ${file_color}${backup_file}.xz.enc"

pv --progress --timer --eta --rate --average-rate --bytes "${backup_file}.xz" |
openssl enc -aes-256-cbc -md sha256 -salt -out "${backup_file}.xz.enc" -pass file:"$passwd_file" 2> /dev/null ||

print_error "OpenSSL failed to encrypt the compressed file!"

#------------------------------------------------------------------------------------------------------------------

print_action "4/7:${action_color}
    SHA512SUM is computing a hash sum of the encrypted file
    ${file_color}${backup_file}.xz.enc${action_color}
    into
    ${file_color}${backup_file}.xz.enc.SHA512SUM"

sha512sum --binary "${backup_file}.xz.enc" > "${backup_file}.xz.enc.SHA512SUM" ||

print_error "SHA512SUM failed to compute the hash sum!"

#------------------------------------------------------------------------------------------------------------------

awk '{ print $1 }' "${backup_file}.xz.enc.SHA512SUM"

#------------------------------------------------------------------------------------------------------------------

print_action "5/7:${action_color}
    OpenSSL is decrypting the file (for verification)
    ${file_color}${backup_file}.xz.enc${action_color}
    into
    ${file_color}${backup_file}.xz.dec"

pv --progress --timer --eta --rate --average-rate --bytes "${backup_file}.xz.enc" |
openssl enc -aes-256-cbc -md sha256 -salt -out "${backup_file}.xz.dec" -d -pass file:"$passwd_file" 2> /dev/null ||

print_error "OpenSSL failed to decrypt the encrypted file!"

#------------------------------------------------------------------------------------------------------------------

print_action "6/7:${action_color}
    CMP is comparing the un-encrypted compressed file
    ${file_color}${backup_file}.xz${action_color}
    with
    ${file_color}${backup_file}.xz.dec"

cmp --silent "${backup_file}.xz" "${backup_file}.xz.dec" &&

echo "Files are identical." ||

print_error "Failed to compare files!" &&

{

print_action "7/7:${action_color}
    Cleaning up"

rm "${backup_file}" &&
echo "${backup_file} removed"

rm "${backup_file}.xz" &&
echo "${backup_file}.xz removed"

rm "${backup_file}.xz.dec" &&
echo "${backup_file}.xz.dec removed"

echo
echo "${success_color}Your directory ${file_color}${dir}${success_color} has been successfully stored as ${file_color}${backup_file}.xz.enc${no_color}"

exit 0

}

And how it appears in Cygwin