Can I speed up this simple versioning/backup script?

Question

I'm writing a simple automatic backup/versioning bash script.

It should basically mirror a directory structure somewhere, and then copy files over there when they've been altered. The ghost files should then be named filename.YYYYMMDD-HHMMSS.ext (the included date/time being the time of last modification).

This is what I came up with - it seems to work already, sort of. but since I've never written shell scripts before, I suspect I might be doing some things fundamentally wrong and inefficiënt.

How can I make this faster? The iterating over files seems to be really slow. Is this robust? It seems to "lock up" without error messages sometimes, and I can't explain or understand why. What am I doing wrong?

I intend to run this as a cron job, every 30 minutes or so.

#!/bin/sh

# $1 : backup "root"
# $2 : from: directory to be backed up
# $3 : to: destination

rpath=$1

for f in $(find $2); do
  r=$(./rel.sh $rpath $f)
  if [ -f $f ]
  then
    basename=$(basename $f)
    dir=$(dirname $r)
    name=$(echo $basename | cut -d'.' -f1)
    ext=$(echo $basename | cut -d'.' -f2)
    mod=$(stat --format=%y $f | awk -F'.' '{printf $1}' | sed 's/[-:]//g' | sed 's/ /-/g')

    if [ ! -f "$3/$dir/$name.$mod.$ext" ]
    then
      echo "+f   $mod $r"
      # cp $f "$3/$dir/$name.$mod.$ext"
    fi
  elif [ -d $f ]
  then
    if [ ! -d "$3/$r" ]
    then
      echo "+dir $r"
      mkdir "$3/$r"
    fi
  fi
done
exit

Answer 1

The previous answers addressed some alternative ways of accomplishing the backup and versioning goals; in this answer I'll comment on three or four possible improvements to your script.

• For clarity, I prefer at the start of a script like this to copy all of the $1, $2, $3 parameters to named variables, as you did for the first of them.
• The find $2 lists all the files and directories in $2 and below, but presumably the bulk of those files will have been treated already in previous runs. For example, to avoid processing files older than second-previous run, in each run write a time-marker file and use -newer:

mv Mark1 Mark0; mv Mark2 Mark1; touch Mark2
find -newer Mark0 ...  

• Your script runs four new processes while setting basename, dir, name, and ext. To avoid all those separate processes, use various shell parameter expansions as below. (Eg, when f=/home/tx.7/xyz.axi.pan, these produce xyz.axi.pan, /home/tx.7, xyz.axi, and pan, respectively. Note, these expansions should work for all the files in your directories as listed by find, but if used in other scripts will stumble when given names like . or / or some other edge cases.)

 basename=${f##*/}
 dir=${f%/*}
 name=${basename%.*}
 ext=${basename##*.}

• The for f in $(find $2) structure is likely to produce a large list of file names and then process it. That list need not be stored if you instead use (eg)

find $2 -exec filescript '{}' $3 \;

where filescript represents a separate script that does the stuff found inside your current for loop. Of course you can also add -newer Mark2 to the find command:

find $2-newer Mark2 -exec filescript '{}' $3 \;

I would expect using -exec to be faster than using the shell for loop, but it might be worthwhile to run a timing test.

Answer 2

Implementing solutions to problems like this is good for exercise purposes. It is good for learning.

But it is almost never good to re-invent the wheel in production systems. Please have a look at well-established data synchronization software such as rsync and perform some research around it and other data synchronization/backup/snapshot techniques.

So, your question is good until "I intend to run this as a cron job, every 30 minutes or so." :-) I am not that bash expert myself, to others might point out the potential weaknesses of your approach.

Answer 3

why not just tar up the entire folder:

tar -cvzf backup-`date  +%Y-%m-%d`.tar.gz /path/to/backup

or use find to do it all

# in script go to path
cd /path/to/folder;
# find all folders and make them in /tmp/backup-todays-date
find . -type d -exec mkdir /tmp/backup`date  +%Y-%m-%d`/{} \; 
# cp all files from current path to /tmp/backup-todays-date
find . -type f -exec cp {} /tmp/backup`date  +%Y-%m-%d`/{} \;

ls -l /tmp/backup2012-09-23/

using top method produced 1 file for all content and then you could use logrotate to rotate tar files after 10 backups or something. so you don't have backup's filling up disk

The 2nd method will continue on creating /tmp/backup-date folders no easy way of managing this unless you wrote another script to monitor them that did something like

find /tmp/backup* -mtime +10 -exec rm -rf {} \;

the initial rsync suggestion is great for server to server copy and could be used for local copying too, the thing that stood out - is your script adding dates to files which means you wish to be able to look back on same file X amount days ago, doing rsync version control visit http://www.howtoforge.com/backing-up-with-rsync-and-managing-previous-versions-history

The alternative if files are being backed up for version control purposes would be to use something like svn(subversion) or git to check in/out files this way there is an external process managing changes.

and finally using tar to do same thing

mkdir /tmp/backup`date  +%Y-%m-%d`; (cd /path/to/backup;  tar -cvzf - .) | (cd /tmp/backup`date  +%Y-%m-%d`;  tar -xvzf -)

all the best