4
\$\begingroup\$

I needed smth that could send (scp/rsync) many files in parallel but that would not overload machine, firewalls on the way by starting e.g. 600 simultaneous uploads, so upload should be done in batches.

Most of the utilities like aria2c are download managers and I needed something to send many files in parallel from a machine behind NAT to a machine on the internet (so no dl from internet possible).

Pls keep in mind that it's the draft that I quote below, I do not need details on general good practices such as setting paths and counters using variables and not literal values.

Any better approach? Problems? I've noticed that I have to sleep 1 between starting scp cmds in background or else target server refuses some connections.

#!/bin/bash


FLIST=file_paths_list.txt

COUNTER=0
PIDS=()

cat ${FLIST} | while read F; do
                COUNTER=$((COUNTER + 1))
                echo COUNTER $COUNTER
                sleep 1
                scp ${F} root@host:/data/tmp &
                PID=$!
                echo Adding PID $PID
                PIDS+=($PID)
                if [ $COUNTER -lt 20 ]; then
                        continue
                fi
                # wait for uploads batch to complete
                for PID in ${PIDS[@]}; do
                        echo waiting for PID $PID
                        wait $PID
                done
                # reset PIDS array
                PIDS=()
                COUNTER=0
done
\$\endgroup\$
4
\$\begingroup\$

It is very, very unlikely that running multiple files in parallel will go any faster than running them all sequentially (your bottleneck will be 'internet' bandwidth regardless).

Is there something wrong with doing something like:

#!/bin/bash

SCPCMD="scp \"\$@\" root@host:/data/tmp"
# echo The scp command is: $SCPCMD

cat ${FLIST} | tr '\n' '\000' | xargs --null bash -c "${SCPCMD}"

The above script will convert newlines to null characters in the input list, then copy the files to the server.

You can add the -n 10 argument to xargs to do 10 files are a time, or whatever works for you. I like the -t argument as well which echos the xarg command to stderr before it runs it.

The above will copy only one stream at a time, but, it will do the files in bulk operations, and you will likely be limited by your network bandwidth, not the parallelism of your copies.

EDIT:

If you want to run the scp's in parallel, you can add the -P xx (--max-procs)argument to xargs, which will run as many as xx scp's in parallel for you. So, for example, if you have hundreds of files, you can scp them in 5 parallel streams, 5 files at a time, with:

cat ${FLIST} | tr '\n' '\000' | xargs --null -P 5 -n 5 bash -c "${SCPCMD}"
\$\endgroup\$
  • \$\begingroup\$ It is very much likely that they will be faster in parallel bc I have demonstrable evidence that they do: it seems that many companies, ISPs etc implement some sort of load balancing anyway, and even inside the companies: at mine the network admins do some magic that cause my scp of a single file max out at 10-20 MB/s which makes the same set of files use up 5 hrs to complete sending while the script I posted sends it in less than 30 mins. In principle all that you have to do is tracking tcp/ip connections which stateful firewalls do anyway. \$\endgroup\$ – LetMeSOThat4U Jan 16 '14 at 14:49
  • \$\begingroup\$ Of course if you have low bw ADSL uplink you can saturate it with sequential upload as well. But once your data leaves "last mile" the ISPs do stuff to your connections. A friend of mine is sending 400GB of physics data today to US from Europe and sees rsync speed vary from 10 KB/s to 20 MB/s from moment to moment. \$\endgroup\$ – LetMeSOThat4U Jan 16 '14 at 14:53
  • \$\begingroup\$ I can see where multi-link protocols over the net could help with bandwidth... 200's answer is probably closer to the mark - using the right tools for the job. \$\endgroup\$ – rolfl Jan 16 '14 at 14:59
  • \$\begingroup\$ nevertheless the trick with -n argument to xargs is a very nice one which is why I upvoted your answer, thanks! \$\endgroup\$ – LetMeSOThat4U Jan 16 '14 at 15:04
  • \$\begingroup\$ @JohnDoe - xargs can run things in parallel too ;-) \$\endgroup\$ – rolfl Jan 16 '14 at 17:08
3
\$\begingroup\$

The best code would be no code at all.

Using sftp(1) with the -R num_requests option (and/or -l bandwidth_limit), you can do a put of multiple files simultaneously.

Alternatively, use a client such as lftp(1) that specializes in batch operations, and has robustness-enhancing features like automatic retry. To do parallel uploads, use lftp -e 'mirror --reverse --parallel'.

\$\endgroup\$
  • \$\begingroup\$ Sadly, I have seen only modest speedup (30%-50%) with using -R option for sftp (which they even say in manpage IIRC), because those are outstanding requests that are parallelized, not separate connections it seems. Have to try lftp though. \$\endgroup\$ – LetMeSOThat4U Jan 17 '14 at 10:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.