I needed smth that could send (scp/rsync) many files in parallel but that would not overload machine, firewalls on the way by starting e.g. 600 simultaneous uploads, so upload should be done in batches.
Most of the utilities like aria2c
are download managers and I needed something to send many files in parallel from a machine behind NAT to a machine on the internet (so no dl from internet possible).
Pls keep in mind that it's the draft that I quote below, I do not need details on general good practices such as setting paths and counters using variables and not literal values.
Any better approach? Problems? I've noticed that I have to sleep 1
between starting scp cmds in background or else target server refuses some connections.
#!/bin/bash
FLIST=file_paths_list.txt
COUNTER=0
PIDS=()
cat ${FLIST} | while read F; do
COUNTER=$((COUNTER + 1))
echo COUNTER $COUNTER
sleep 1
scp ${F} root@host:/data/tmp &
PID=$!
echo Adding PID $PID
PIDS+=($PID)
if [ $COUNTER -lt 20 ]; then
continue
fi
# wait for uploads batch to complete
for PID in ${PIDS[@]}; do
echo waiting for PID $PID
wait $PID
done
# reset PIDS array
PIDS=()
COUNTER=0
done