4
\$\begingroup\$

This is a simple tool to parallelize execution of multiple tasks in Bash, since env_parallel can't handle the environment size (or something of the sort - dies for mysterious reasons.) I'd love to hear your critique.

#
# Simple parallel executor. Takes a list of commands and executes each in its own child process. Assumes commands are distinctly named.
# Returns with 0 if all are successful. Otherwise, prints out their stdout/stderr outputs and returns with 1.

function simple_par () {

    local TMP_FILE="${WRITABLE_DIR}/simple_par_$(uuidgen)"
    touch $TMP_FILE

    for prog in "$@"
    do
        local PROG_TMP="${TMP_FILE}_${prog}_$(uuidgen)"
        (
            ( $prog 1>$PROG_TMP 2>&1 )
            echo "$?" >> $TMP_FILE
        ) &
    done

    while true
    do
        RESULT_COUNT=$(wc -l ${TMP_FILE} | awk '{ print $1 }')

        if [[ $RESULT_COUNT -eq $# ]]; then
            if [[ $(cat $TMP_FILE) == *"1"* ]]; then
                echo "Failed at least one command"
                for prog in "$@"
                do
                    echo "OUTPUT FOR: $prog"
                    cat ${TMP_FILE}_${prog}_*
                    rm ${TMP_FILE}_${prog}_*
                done

                rm $TMP_FILE
                return 1
            else
                return 0
            fi
        fi

        sleep 5
    done
}
\$\endgroup\$
1
\$\begingroup\$

A couple of things could be written better, let's go from top to bottom.


It's recommended to double-quote variables when used as paths, to avoid globbing and word-splitting. Even if you're certain that $TMP_FILE won't have any troublesome characters, it's a good practice to double-quote anyway:

touch "$TMP_FILE"

Similarly, you should double-quote $prog in $prog 1>$PROG_TMP 2>&1.


You can simplify for arg in "$@"; do loop as for arg; do.


In Bash, instead of prog 1> somewhere 2>&1 you can write prog &> somewhere.


Instead of this:

RESULT_COUNT=$(wc -l ${TMP_FILE} | awk '{ print $1 }')

You could write simpler without awk like this:

RESULT_COUNT=$(wc -l < "$TMP_FILE")

The trick is that when wc is used with stdin, it won't print a file name, only the number. Although RESULT_COUNT will have some whitespace padding, if you use it in [ ... ] without double-quoting, it will behave as expected.

Notice again that I double-quoted $TMP_FILE.


Don't use -eq in [[ ... ]], use == or = instead.


This is pretty ugly:

if [[ $(cat $TMP_FILE) == *"1"* ]]; then

A better way to achieve the same thing is using grep:

if grep -q 1 "$TMP_FILE"; then
\$\endgroup\$
  • \$\begingroup\$ Thanks a lot, @janos! I'm a bit of a bash dilettante, it's good to have good advice. Since then, I've rewritten the program to store the child processes' pids into an array/string, and wait on each of them. It seems to be a better use of the tools at hand. \$\endgroup\$ – vivri Dec 23 '16 at 3:58
  • \$\begingroup\$ Cool, wanna post that up for review? ;-) \$\endgroup\$ – Stop ongoing harm to Monica Dec 23 '16 at 4:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.