3
\$\begingroup\$

I am writing a small bash wrapper for bbcp. This is a fast network file transfer utility. bbcp requires the destination folder to be the last argument.

bbcp user@remote:/a/file1 user@remote:/b/file2 destination_directory

I would like to download all the files and then mv them into place in one go, preventing problems with half downloaded files, or half a set of downloaded files. I am trying to correctly deal with whitespace.

USER, REMOTE, and DESTDIR are parameters provided as environment variables.

I am surprised how hard I found it to manipulate arguments in bash, maybe there are some built in arg manipulation functions I am not aware of?

#!/bin/bash
printf "%s\0" "$@" \
| awk -v USER="$USER" -v REMOTE="$REMOTE" \
    'BEGIN{ORS="\0"; RS="\0"} {print USER"@"REMOTE":"$0}' \
| xargs -0 -r bash -sc 'bbcp "$@" '\'"$DESTDIR/bbcp_tmp"\' bash

basename -az "$@" \
| awk -v DEST="$DESTDIR" \
    'BEGIN{ORS="\0"; RS="\0"}
    {print DEST"/bbcp_tmp/"$0}' \
| xargs -0 -r mv -t "$DESTDIR"
New contributor
Evan Benn is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
\$\endgroup\$
3
\$\begingroup\$

This code is hard to read. Let's see if I got this right:

  1. printf "%s\0" "$@" NUL-terminates the arguments.
  2. The awk script produces $USER@$REMOTE:$N for each $N in $@

Those two can be combined much easier using arrays. Ditto for basename | awk.

  1. You then run bash -sc 'bbcp "$@" '\'"$DESTDIR/bbcp_tmp"\' bash with that list of arguments. Why -s? Why the final bash argument? If that's the name of the target directory it's really confusing. You might want to use mktemp --directory (or -d) instead.
  2. mv isn't atomic, so even if the downloads all succeed you're not guaranteed that after the mv command the files will all be there. If you want an atomic remote copy you could instead create a directory locally, copy all the remote files into that, and then move only the directory to where you want it. As long as the source and target directory are on the same partition that should be atomic.
  3. This script could benefit from set -o errexit -o nounset -o pipefail.

More about arrays

\$\endgroup\$
  • \$\begingroup\$ Brilliant thanks, could not find that array manipulation info in the manuals. \$\endgroup\$ – Evan Benn Aug 14 at 8:42
  • \$\begingroup\$ I used bash -sc in order to place the fixed destdir argument as the last arg. (mv -t makes this easier). the extra bash is required, that bash becomes $0. \$\endgroup\$ – Evan Benn Aug 14 at 8:44
  • \$\begingroup\$ I have replaced the first step with pref=( "${@/#/$USER@$REMOTE:}" ) bbcp "${pref[@]}" "$DESTDIR/bbcp_tmp" But having trouble with the second. I think I still need the basename, and then I can't use the bash variable manipulator. Looks like I might need a read loop, but would prefer to stick with the awk in that case. Any ideaS? \$\endgroup\$ – Evan Benn Aug 15 at 0:16
  • \$\begingroup\$ You might want to post that separately on Stack Overflow. \$\endgroup\$ – l0b0 Aug 15 at 1:53
1
\$\begingroup\$

would like to download all the files and then mv them into place in one go, preventing problems with half downloaded files, or half a set of downloaded files.

Sounds like you need to read about rsync

rsync -a user@remote:/a/file1 user@remote:/b/file2 destination_directory # two remote files
rsync -a user@remote:/a/ destination_directory # all files in dir ‘a’

Other arguments can limit bandwidth, or use compression to speed up transfer.

New contributor
WGroleau is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
\$\endgroup\$
  • \$\begingroup\$ Hello, yes I know rsync. We use bbcp because we use > 10Gbit links and rsync is not built for this. (AFAIK!) \$\endgroup\$ – Evan Benn 2 days ago
  • \$\begingroup\$ Although it surprised me to measure this, rsync can even speed up local transfers (all files on local disks of the same CPU). But the main thing is that rsync is intended for "preventing problems with half downloaded files, or half a set of downloaded files." Of course, I realize that by using multiple connections, bbcp probably exceeds rsync's speed when NOT trying to resume an interrupted operation. \$\endgroup\$ – WGroleau 2 days ago
  • \$\begingroup\$ Yes rsync is a great tool and is often the right choice. bbcp works here because we need to use UDP to saturate our wide, reasonable short, connection transferring a single large file at a time. Maybe rsync could be good at that, in my measurements bbcp is >8x faster, as well as less load on cpu. \$\endgroup\$ – Evan Benn 2 days ago

Your Answer

Evan Benn is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.