7
\$\begingroup\$

I just wrote a little bash script for rsync push backups of my laptop to my Synology-Diskstation at home.

Since I am a bash beginner and don't have any experience with ssh, rsync and backups I am asking for suggestions how to improve the performance, reliablility, usability and security of my script and for suggestions to improve the coding style. I have listed some problems and notes below.

The hardrive of my laptop is encrypted and the backup should be encrypted as well.

Here is the script:

#!/bin/bash

bold=`tput bold`
normal=`tput sgr0`

dirlist=/home/myuser/bin/dirlist
pass=mypassword

date=`date "+%Y-%m-%dT%H_%M_%S"`

echo "${bold}Mounting encrypted backup volume${normal}"
ssh -n -i /home/myuser/.ssh/id_rsa  root@192.168.0.70 "/usr/syno/sbin/synoshare --enc_mount MyuserBackup $pass" 

sleep 5

echo "${bold}Checking and eventually creating dirs on backup server${normal}"
## Creating directories
while read dir; do   
    echo "Create incomplete$dir if not existent"
    ## http://stackoverflow.com/questions/9393038/ssh-breaks-out-of-while-loop-in-bash
    ssh -n -i /home/myuser/.ssh/id_rsa root@192.168.0.70 "mkdir -p /volume1/MyuserBackup/rsync_Backup/incomplete$dir"
done < $dirlist

while read dir; do
echo "${bold}Backing up $dir${normal}"
time sudo rsync -avX -u --compress-level=0 --stats -h  --exclude-from=/home/myuser/bin/excludelist  --numeric-ids "$@" --link-dest=/volume1/MyuserBackup/rsync_Backup/current$dir --rsh="ssh -i /home/myuser/.ssh/id_rsa" $dir root@192.168.0.70:/volume1/MyuserBackup/rsync_Backup/incomplete$dir
done < $dirlist


echo $date

echo "${bold}Updating Date-Structure${normal}"

echo "mv /volume1/MyuserBackup/rsync_Backup/incomplete /volume1/MyuserBackup/rsync_Backup/back-$date"

ssh -t  -i /home/myuser/.ssh/id_rsa root@192.168.0.70 "mv /volume1/MyuserBackup/rsync_Backup/incomplete /volume1/MyuserBackup/rsync_Backup/back-$date"


echo "rm -f /volume1/MyuserBackup/rsync_Backup/current"
ssh -t -i /home/myuser/.ssh/id_rsa root@192.168.0.70 "rm -f /volume1/MyuserBackup/rsync_Backup/current"

echo "ln -s /volume1/MyuserBackup/rsync_Backup/back-$date /volume1/MyuserBackup/rsync_Backup/current"

ssh -t -i  /home/myuser/.ssh/id_rsa root@192.168.0.70 "ln -s /volume1/MyuserBackup/rsync_Backup/back-$date /volume1/MyuserBackup/rsync_Backup/current"


echo "Showing the current dir structure:"

ssh -t -i /home/myuser/.ssh/id_rsa root@192.168.0.70 "ls -lhrt /volume1/MyuserBackup/rsync_Backup/"

echo "${bold}Unmounting encrypted volume${normal}"
ssh -t -i /home/myuser/.ssh/id_rsa root@192.168.0.70 "/usr/syno/sbin/synoshare --enc_unmount MyuserBackup"

cat dirlist

/home/
/etc/
/var/

cat excludelist

## Universal excludes

lost+found
ld.so.cache

# backup text files (e.g. from Emacs)
- *~
- \#*\#

# Commonly distributed Windows cache
- Thumbs.db

# Common package managers (apt, yum)
- *cache/apt/


# sshfs
- box1
- box2
- box3
- box4

# Cache
- .cache/
- **cache

# Temporary files / cache
- .local/share/Trash
- .cache
- .Trash
- **.Trash
- **.macromedia
- **.thumbnails
- .local/share/gvfs-metadata
- .aptitude
- .pulse
- .pulse-cookie
- .xournal
- .sage/temp
- tmp

- .mozilla/firefox/*/Cache
- .mozilla/firefox/*/minidumps    # in case Fx crashes dumps will be stored in this
- .mozilla/firefox/*/.parentlock       # session-specific           
- .mozilla/firefox/*/urlclassifier3.sqlite  # phishing database, recreated
- .mozilla/firefox/*/blocklist.xml   # blacklisted extensions
- .mozilla/firefox/*/extensions.sqlite  # extension database, recreated on startup
- .mozilla/firefox/*/extensions.sqlite-journal
- .mozilla/firefox/*/extensions.rdf
- .mozilla/firefox/*/extensions.ini
- .mozilla/firefox/*/extensions.cache
- .mozilla/firefox/*/XUL.mfasl     # cached UI data, recreated
- .mozilla/firefox/*/XPC.mfasl
- .mozilla/firefox/*/xpti.dat
- .mozilla/firefox/*/compreg.dat

- .compiz
- .dbus

# Private directory from ecryptfs
- .Private

# X Windows System
- .xsession-errors*

# Gnome temp stuff
- .compiz*/session
- .gksu.lock
- **.gvfs


# Common Applications

# Adobe Reader
- */.adobe/**/AssetCache/
- */.adobe/**/Cache/
- */.adobe/**/Temp/
- */.adobe/**/UserCache.bin

# Gimp
- /.gimp-*/tmp
- /.gimp-*/swap

# Mozilla Firefox
- .mozilla/firefox/*/Cache/
- .mozilla/firefox/*/lock
- .mozilla/firefox/*/.parentlock

# Mozilla Thunderbird
- .mozilla-thunderbird/*/lock
- .mozilla-thunderbird/*/.parentlock

- *recently-used*

Notes and Problems

  • Usually I execute the script with sudo, but sometimes it asks for the sudo password again when it comes to the next item in dirlist. I guess this is due to the fact that the sudo password is only saved for some time, 5 minutes or so. However this is bad since I don't want to control what the script is doing all the time.
  • Sometimes it gives something like

    rsync: send_files failed to open "/home/myuser/.thunderbird/myprofile.default/session-6.json": Input/output error (5)
    

    This happens only with some "exotic" files like the file above.

  • Sometimes it prints errors like

    rsync: mkstemp "/volume1/MyuserBackup/rsync_Backup/incomplete/home/" failed: No such file or directory (2)
    

    However I didn't find any files which were not transferred.

  • Is there some better way to handle the password instead of writing it in plaintext into the script?

  • The script seems to be really slow even if nothing changed on the disk it takes over 30 min

  • Currently I thin out the snapshots manually but I am thinking of writing a program which do this automatically. Something like this.

Versions

Here are the versions of the used software:

  • bash: 4.2.8(1)
  • rsync: 3.0.7
  • synology DSM: 4.2
  • OpenSSH 5.8p1 Debian-1ubuntu3

Hardware

\$\endgroup\$
7
\$\begingroup\$

DRY - Don't Repeat Yourself.

You repeat certain things a lot. I recommend extracting a function for your ssh commands:

function remotessh {
    echo "Running ssh : " + "$@"
    ssh -i /home/myuser/.ssh/id_rsa root@192.168.0.70 "$@"
}

This function should have some appropriate error handling too.

Then, your code can become simplified like:

echo "${bold}Mounting encrypted volume${normal}"
remotessh -n "/usr/syno/sbin/synoshare --enc_mount MyuserBackup $pass" 

.....

remotessh -t "mv /volume1/MyuserBackup/rsync_Backup/incomplete /volume1/MyuserBackup/rsync_Backup/back-$date"

remotessh -t "rm -f /volume1/MyuserBackup/rsync_Backup/current"

remotessh -t "ln -s /volume1/MyuserBackup/rsync_Backup/back-$date /volume1/MyuserBackup/rsync_Backup/current"

echo "Showing the current dir structure:"
remptessh -t "ls -lhrt /volume1/MyuserBackup/rsync_Backup/"

echo "${bold}Unmounting encrypted volume${normal}"
remotessh -t "/usr/syno/sbin/synoshare --enc_unmount MyuserBackup"

Error Handling

You should include error handling always in bash scripts.

RSync issues, etc.

rsync builds up a list of files to transfer, calculates the smallets dataset that needs to be migrated to accomplish the transfer, then it does the copy. If files change between when the calculation is done, and the transfer is done, it gives different errors/warnings.

This is what is happening with you, I believe.

Performance

Depending on the rsync options, it can take a while to both build the copy-list, and do the copy. Even though it may be inconvenient, you probably cannot beat rsync for performance.

Check each of your options to ensure they are the best. Kill those options that you don't need.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.