# a wrapper for bbcp

I am writing a small bash wrapper for bbcp. This is a fast network file transfer utility. bbcp requires the destination folder to be the last argument.

bbcp user@remote:/a/file1 user@remote:/b/file2 destination_directory


I would like to download all the files and then mv them into place in one go, preventing problems with half downloaded files, or half a set of downloaded files. I am trying to correctly deal with whitespace.

USER, REMOTE, and DESTDIR are parameters provided as environment variables.

I am surprised how hard I found it to manipulate arguments in bash, maybe there are some built in arg manipulation functions I am not aware of?

#!/bin/bash
printf "%s\0" "$@" \ | awk -v USER="$USER" -v REMOTE="$REMOTE" \ 'BEGIN{ORS="\0"; RS="\0"} {print USER"@"REMOTE":"$0}' \
| xargs -0 -r bash -sc 'bbcp "$@" '\'"$DESTDIR/bbcp_tmp"\' bash

basename -az "$@" \ | awk -v DEST="$DESTDIR" \
'BEGIN{ORS="\0"; RS="\0"}
{print DEST"/bbcp_tmp/"$0}' \ | xargs -0 -r mv -t "$DESTDIR"


This code is hard to read. Let's see if I got this right:

1. printf "%s\0" "$@" NUL-terminates the arguments. 2. The awk script produces $USER@$REMOTE:$N for each $N in $@

Those two can be combined much easier using arrays. Ditto for basename | awk.

1. You then run bash -sc 'bbcp "$@" '\'"$DESTDIR/bbcp_tmp"\' bash with that list of arguments. Why -s? Why the final bash argument? If that's the name of the target directory it's really confusing. You might want to use mktemp --directory (or -d) instead.
2. mv isn't atomic, so even if the downloads all succeed you're not guaranteed that after the mv command the files will all be there. If you want an atomic remote copy you could instead create a directory locally, copy all the remote files into that, and then move only the directory to where you want it. As long as the source and target directory are on the same partition that should be atomic.
3. This script could benefit from set -o errexit -o nounset -o pipefail.

• Brilliant thanks, could not find that array manipulation info in the manuals. – Evan Benn Aug 14 at 8:42
• I used bash -sc in order to place the fixed destdir argument as the last arg. (mv -t makes this easier). the extra bash is required, that bash becomes $0. – Evan Benn Aug 14 at 8:44 • I have replaced the first step with pref=( "${@/#/$USER@$REMOTE:}" ) bbcp "${pref[@]}" "$DESTDIR/bbcp_tmp" But having trouble with the second. I think I still need the basename, and then I can't use the bash variable manipulator. Looks like I might need a read loop, but would prefer to stick with the awk in that case. Any ideaS? – Evan Benn Aug 15 at 0:16
• You might want to post that separately on Stack Overflow. – l0b0 Aug 15 at 1:53

Sounds like you need to read about rsync
rsync -a user@remote:/a/file1 user@remote:/b/file2 destination_directory # two remote files