8
\$\begingroup\$

First I create a .tar of the root directory with all the file permissions saved and excluding all irrelevant directories (are these the right folders to be excluding?).

sudo tar -cvpzf /HDD/systemBackup/backup.tar.gz --one-file-system --exclude=/HDD --exclude=/NAS --exclude=/proc exclude=/tmp --exclude=/mnt --exclude=/dev --exclude=/sys /

Then the .tar is split into smaller files of 100mb for easier upload efficiency.

split -b 100MB /HDD/systemBackup/backup.tar.gz "/HDD/systemBackup/$date-"

Then I upload the parts to the Internet.


I am looking to optimise ease of restoration and speed efficiency (ignoring (to some extent) the speed of the upload to the Internet).

Will this method also work on both a Raspberry Pi and an Ubuntu system?

Is this a good method of backing up my Linux system to the web?

\$\endgroup\$
  • \$\begingroup\$ Do you care about the ability to selectively restore only some files? \$\endgroup\$ – 200_success Jul 27 '15 at 14:50
  • \$\begingroup\$ That would be a bonus! But would that not impede massively on the restoration process?! \$\endgroup\$ – maxisme Jul 27 '15 at 23:23
  • \$\begingroup\$ As I am uploading the system to "the cloud" I can always just download it and decompress it and then get the specific files. What I am really interested in, is whether I will be able to restore the files (incase of a system failure) to a previous state, simply and efficiently. \$\endgroup\$ – maxisme Jul 28 '15 at 7:48
  • \$\begingroup\$ Not an answer to your question, but you might enjoy duplicity. \$\endgroup\$ – tarleb Jul 30 '15 at 13:10
  • \$\begingroup\$ Thank you! I have already looked into duplicity but not really what I am looking for! \$\endgroup\$ – maxisme Jul 30 '15 at 13:13
3
+50
\$\begingroup\$

When it comes to system backups, I think it's more important to know what you include than to know what you exclude. So instead of specifying what you exclude, I think it's better to think about what you need to include. It might be a shorter list, and less noisy without all the --exclude flags.

Your current method is wasteful: the backup takes twice the size it needs, once for the .tar.gz, and once for the split version. You could pipe the output of the tar command to split to save some space, for example:

sudo tar -cvpz --one-file-system /usr /etc /home ... | split -b 100MB - "/HDD/systemBackup/$date-"

Most importantly, as @tarleb also said, make sure to test restoring from backup. An all too common mistake is to not test restoring, and one day when you really need it, your backups turn out to be useless.

Will this method also work on both a Raspberry Pi and an Ubuntu system?

The list of directories you want to backup might be a little bit different. If you have to inspect both systems and see. Other than that, the same technique should work, yes.

Is this a good method of backing up my Linux system to the web?

This is one way to do it. It's a full backup. Its advantage is that it's simple. Its drawback is that it wastes a lot of space, because much of the data in each backup for different $date values will be duplicated. And transferring over the network will be relatively slow.

An alternative would be using incremental backup, which would save disk space and network bandwidth, at the expense of considerably more complex backup and restore procedure.

One more way would be to rsync the interesting directories, and take create a timestamped archive at the other side. That would save disk space locally, save network bandwidth, and still be relatively simple.

\$\endgroup\$
  • \$\begingroup\$ Thank you for your answer! Very rookie question but how do I go about testing a backup without removing everything in the first place? Also I delete the files after the backup so it is not a big issue, are you saying that if I pipe the output it will run faster as well? \$\endgroup\$ – maxisme Aug 3 '15 at 3:26
  • 1
    \$\begingroup\$ @Maximilian Testing backups: You'll need some kind of redundant system for that. If you are on a raspberry-pi, buying a second SD card would be a cheap option. Faster by piping: Yes, it's faster. The tar-generated data doesn't have to be written to disk twice (very slow operation) but can be passed via RAM till split writes it to disk. \$\endgroup\$ – tarleb Aug 5 '15 at 21:13
  • \$\begingroup\$ Sorry to ask another question and I can write a new SE question if you want but: What do folders should I exclude if I am wanting to replicate a system but still keep the system unique. I tried performing a restore (finally) and when running any commands on the new system I just get Exec format error which I think means it has a different ARM? I would like to be able to take into account that they could be different? \$\endgroup\$ – maxisme Sep 3 '15 at 9:41
  • \$\begingroup\$ I think all I really need is the software running (eg NGINX) and all the settings, and the home and root folders. But I do not want it to have the same system /etc/hostname and things like that? Sorry if that doesn't really make sense I do not really understand linux systems.... Haha. The reason I am being so unspecific is because I believe this is quite a common requirement of systems (please tell me if it is not and I will expand) \$\endgroup\$ – maxisme Sep 3 '15 at 9:44
  • \$\begingroup\$ I am wanting this so that I can setup cluster servers simply btw!! \$\endgroup\$ – maxisme Sep 3 '15 at 9:57
2
\$\begingroup\$

It's a solid way to do backups. I'm assuming you already know about rdiff-backup and wrappers like backupninja, so you've likely already decided that those don't fit your use-case.

Since your code is likely going to be wrapped in a shell script, the --exclude parameters could be created in a loop. This would prevent errors such as the missing -- prefix in the example you gave. Don't exclude / or you won't get any backups.

EXCLUDE_PATHS="/HDD /NAS /proc /tmp /mnt /dev /sys /var/lock /var/tmp"

EXCLUDE_PARAMS=""
for p in ${EXCLUDE_PATHS}
do
    EXCLUDE_PARAMS="${EXCLUDE_PARAMS} --exclude ${p}"
done

sudo tar -cvpzf /HDD/systemBackup/backup.tar.gz --one-file-system $EXCLUDE_PARAMS

Consider adding a longer suffix to the split fragments by adding -a 3 to the split command. Otherwise we might get unexpected results (e.g. irregular suffix lengths). However, given that you are on a (probably small) raspberry pi, and that the default of a 2-char suffix will work fine for up to 576 files, this can likely be skipped here.

Restoring the full tree will be possible with

 find /path/to/backups -name "${date}-*" | sort | xargs cat | tar zvxf -

Please test your backup/restoration process at least once to make sure everything works as intended. Don't wait till disaster strikes to possibly realize that important parts of the system were accidentally left out.

Since you plan to do a backup to the cloud, protecting your data from nosy people might be a concern. Integrating gpg into the pipeline should be fairly straightforward.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.