I am writing a small bash wrapper for bbcp. This is a fast network file transfer utility. bbcp requires the destination folder to be the last argument.

bbcp user@remote:/a/file1 user@remote:/b/file2 destination_directory

I would like to download all the files and then mv them into place in one go, preventing problems with half downloaded files, or half a set of downloaded files. I am trying to correctly deal with whitespace.

USER, REMOTE, and DESTDIR are parameters provided as environment variables.

I am surprised how hard I found it to manipulate arguments in bash, maybe there are some built in arg manipulation functions I am not aware of?

printf "%s\0" "$@" \
| awk -v USER="$USER" -v REMOTE="$REMOTE" \
    'BEGIN{ORS="\0"; RS="\0"} {print USER"@"REMOTE":"$0}' \
| xargs -0 -r bash -sc 'bbcp "$@" '\'"$DESTDIR/bbcp_tmp"\' bash

basename -az "$@" \
| awk -v DEST="$DESTDIR" \
    'BEGIN{ORS="\0"; RS="\0"}
    {print DEST"/bbcp_tmp/"$0}' \
| xargs -0 -r mv -t "$DESTDIR"

2 Answers 2


This code is hard to read. Let's see if I got this right:

  1. printf "%s\0" "$@" NUL-terminates the arguments.
  2. The awk script produces $USER@$REMOTE:$N for each $N in $@

Those two can be combined much easier using arrays. Ditto for basename | awk.

  1. You then run bash -sc 'bbcp "$@" '\'"$DESTDIR/bbcp_tmp"\' bash with that list of arguments. Why -s? Why the final bash argument? If that's the name of the target directory it's really confusing. You might want to use mktemp --directory (or -d) instead.
  2. mv isn't atomic, so even if the downloads all succeed you're not guaranteed that after the mv command the files will all be there. If you want an atomic remote copy you could instead create a directory locally, copy all the remote files into that, and then move only the directory to where you want it. As long as the source and target directory are on the same partition that should be atomic.
  3. This script could benefit from set -o errexit -o nounset -o pipefail.

More about arrays

  • \$\begingroup\$ Brilliant thanks, could not find that array manipulation info in the manuals. \$\endgroup\$
    – Evan Benn
    Commented Aug 14, 2019 at 8:42
  • \$\begingroup\$ I used bash -sc in order to place the fixed destdir argument as the last arg. (mv -t makes this easier). the extra bash is required, that bash becomes $0. \$\endgroup\$
    – Evan Benn
    Commented Aug 14, 2019 at 8:44
  • \$\begingroup\$ I have replaced the first step with pref=( "${@/#/$USER@$REMOTE:}" ) bbcp "${pref[@]}" "$DESTDIR/bbcp_tmp" But having trouble with the second. I think I still need the basename, and then I can't use the bash variable manipulator. Looks like I might need a read loop, but would prefer to stick with the awk in that case. Any ideaS? \$\endgroup\$
    – Evan Benn
    Commented Aug 15, 2019 at 0:16
  • \$\begingroup\$ You might want to post that separately on Stack Overflow. \$\endgroup\$
    – l0b0
    Commented Aug 15, 2019 at 1:53

would like to download all the files and then mv them into place in one go, preventing problems with half downloaded files, or half a set of downloaded files.

Sounds like you need to read about rsync

rsync -a user@remote:/a/file1 user@remote:/b/file2 destination_directory # two remote files
rsync -a user@remote:/a/ destination_directory # all files in dir ‘a’

Other arguments can limit bandwidth, or use compression to speed up transfer.

  • \$\begingroup\$ Hello, yes I know rsync. We use bbcp because we use > 10Gbit links and rsync is not built for this. (AFAIK!) \$\endgroup\$
    – Evan Benn
    Commented Aug 16, 2019 at 0:14
  • \$\begingroup\$ Although it surprised me to measure this, rsync can even speed up local transfers (all files on local disks of the same CPU). But the main thing is that rsync is intended for "preventing problems with half downloaded files, or half a set of downloaded files." Of course, I realize that by using multiple connections, bbcp probably exceeds rsync's speed when NOT trying to resume an interrupted operation. \$\endgroup\$
    – WGroleau
    Commented Aug 16, 2019 at 1:31
  • \$\begingroup\$ Yes rsync is a great tool and is often the right choice. bbcp works here because we need to use UDP to saturate our wide, reasonable short, connection transferring a single large file at a time. Maybe rsync could be good at that, in my measurements bbcp is >8x faster, as well as less load on cpu. \$\endgroup\$
    – Evan Benn
    Commented Aug 16, 2019 at 1:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.