4
\$\begingroup\$

This code uploads an entire directory and allows for its reparenting on a remote server (an artifact repository).

Three concerns:

  1. Copying the files locally takes a lot of space. Ideally, I could use inotify and as soon as my wget -m had a file I could begin immediately uploading and then deleting. These repositories can be hundreds of gigs
  2. For performance reasons I fork/join everything. This definitely improves overall job performance but I get an error upload.sh: fork: retry: Resource temporarily unavailable which is the OS telling me I can't have more sub processes. My concern is that some of these processes will terminate without doing their job because of this.
  3. Lastly, and I can't be sure of this, but several of the larger files that upload seem to never complete when I use a fork. But when I do a 1-off they finish in an appropriate time

First I run:

wget --user cbongiorno --password abc123 --mirror --no-parent https://artifactory.stback.com/artifactory/BIF-Releases &> my.logs

Then I do the upload. It would be nice to upload and delete files as they come in

#!/usr/bin/env bash

[[ -z "$1" ]] && echo "no resource directory supplied" && exit -1
[[ -z "$2" ]] && echo "no destination repository supplied" && exit -1


upload() {
    local local_repo="$1"
    local remote_repo="$2"
    local ct
    local user=${3:-${USER:-$(read -p "Username: "  user && [ -n "${user}" ] && echo ${user})}}
    local pwd=$(read -s -p "Password: "  pwd && [ -n "${pwd}" ] && echo ${pwd})
    echo ""
    declare -a pids
    for art in $(find "${local_repo}" -not -path '*/\.*' -type f -not -name 'index.html' | sed -n 's;./\(.*\);\1;p'); do


#        file extension
        case "$(echo ${art} | rev | cut -d'.' -f1 | rev)" in
            jar)
                ct="application/java-archive"
            ;;
            md5)
                ct="application/x-checksum"
            ;;
            pom)
                ct="application/x-maven-pom+xml"
            ;;
            sha1)
                ct="application/x-checksum"
            ;;
            txt)
                ct="text/plain"
            ;;
            xml)
                ct="application/xml"
            ;;
        esac
        (
            echo "${art} -> ${remote_repo}/${art}"
            curl -v -u "${user}:${pwd}" -X PUT -d "@$art" -H "Content-Type: $ct" "$remote_repo/$art" 2&>1 &&
            curl -I -u "${user}:${pwd}" "$remote_repo/$art" 2&>1
        )  &

        pids[${i}]=$!

    done
    for pid in ${pids[*]}; do
        wait ${pid}
    done
}
upload "$@"
\$\endgroup\$
  • \$\begingroup\$ I have a totally updated script. Should I place it here or close it or add my own answer? \$\endgroup\$ – Christian Bongiorno Feb 9 at 0:30
  • \$\begingroup\$ you aren't supposed to edit your question with updated code: the existing answers will not make sense to future readers. Adding your own answer is better. You can even mark your own answer as accepted. \$\endgroup\$ – Oh My Goodness Feb 11 at 9:50
5
\$\begingroup\$

I think you'll find that inotify has problems of its own and that you're already on the right path.

The approach I'd use: get this working the way you want, then run it in a loop while your wget runs, removing successful files as it goes.

If you're concerned about polling overhead (I wouldn't be: the size of the directory should remain manageable so long as you're uploading/deleting along the way), have the wget process emit filenames into a log file or a named pipe, while the curl process consumes them.

You can attack the resource problem in a couple of ways. One, use exec to launch curl so that there isn't an intermediate bash process sitting around waiting for curl to exit. Two, keep count of the number of children, and reap them when the number gets too high. Three, check ulimit -u and consider increasing that number.

In the code below I've made a few lesser edits:

  • content-types in an associative array with sensible default
  • use [[ ]] instead of [ ] (the former is a bash builtin)
  • clean up the find command line a little. I've guessed that the sed invocation is meant to remove $local_repo?
  • omit forked echo and curl-HEAD; rely on exit status instead
  • use curl's -w to get a useful line of output on success
  • store pids in an associative array with filenames as values, to allow meaningful response to outcomes
  • "Content-type" has a lowercase t
  • run upload in infinite loop, keep count of files found, exit when no more files are found

There may be some bugs, since I don't have your environment to test. Proceed accordingly.

#!/usr/bin/env bash
reap() {
    for pid in ${!pid2fn[@]}
    do
        local fn=${pid2fn[$pid]} 
        if wait ${pid} 
        then 
            echo successful upload of $fn
            # rm $fn
        else
            echo $fn upload failed with status $?         
        fi
        unset pid2fn[$pid]
    done
}
upload() {
    local local_repo="${1:?no resource directory supplied}"
    local remote_repo="${2:?no destination repository supplied}"
    local default_ct=application/octet-stream
    local user=${3:-${USER:-$(read -p "Username: "  user && [[ -n $user ]] && echo $user) }}
    local pwd=$(read -s -p "Password: "  pwd && [[ -n $pwd ]] && echo $pwd )
    local max_children=100
    local found=0
    declare -A pid2fn
    declare -A types=(
        [jar]=application/java-archive
        [md5]=application/x-checksum
        [pom]=application/x-maven-pom+xml
        [sha1]=application/x-checksum
        [txt]=text/plain
        [xml]=application/xml
    )
    for art in $( find "$local_repo" -type f -name "[^.]*" -not -name index.html | sed s,^$local_repo/*,, )
    do
        let found=found+1
        local ct=${types[${art##*.}]:-$default_ct}
        local dest="$remote_repo/$art"
        local status="$art -> $dest | response %{http_code} sent %{size_upload} bytes\n"
        exec curl -u "$user:$pwd" -X PUT -d "@$art" -H "Content-type: $ct" -w "$status" "$dest" 2>&1 &
        pid2fn[$!]=$art
        [[ ${#pid2fn[@]} -ge $max_children ]] && reap
    done
    reap
    [[ $found -gt 0 ]]
}
while upload "$@"; do :; done
\$\endgroup\$
  • \$\begingroup\$ Thanks for the snippet - there are definitely some good ideas here. The sed in the original for loop was meant to remove ./foo and make it just foo if you gave the local repo as .. I also need to exclude any dot files and directories. I was going to go with an associative array but found the syntax cumbersome - I like your approach. Let me chew on it some more. I am guessing thr ${1:?...} syntax exits if empty? \$\endgroup\$ – Christian Bongiorno Feb 5 at 16:34
  • \$\begingroup\$ Oh yeah, I had no idea you could do that with curl \$\endgroup\$ – Christian Bongiorno Feb 5 at 16:35
  • \$\begingroup\$ to remove ./ you have to escape the dot in sed, like \./, because plain dot matches any character. Should anchor to the beginning of the string too, so that dot at end of a directory name doesn't match: try sed 's,^\./,,'. \$\endgroup\$ – Oh My Goodness Feb 5 at 17:56
  • \$\begingroup\$ re: comment about HEAD - the head command is to verify the artifact exists. You're going on the assumption that if the server responded with 201 that the resource is there. With this server, it's always a question. Maybe it's overkill, but with so many artifacts to verify I can't do it manually. \$\endgroup\$ – Christian Bongiorno Feb 5 at 20:13
  • \$\begingroup\$ this line doesn't produce the file extension: ct=${types[${art#*.}]:-${default_ct}} it always resolves to the default. \$\endgroup\$ – Christian Bongiorno Feb 5 at 21:57
0
\$\begingroup\$

Here is the final result. It doesn't handle failure/retry as I continued to have issues with using a FIFO, but it does do a pretty good job. I ended up using xargs -P to get the parallelism since it behaves as a thread pool which is really what I needed.

#!/usr/bin/env bash

declare -A types=(
    [jar]=application/java-archive
    [md5]=application/x-checksum
    [pom]=application/x-maven-pom+xml
    [sha1]=application/x-checksum
    [txt]=text/plain
    [xml]=application/xml
    [nupkg]=application/x-nupkg
)

errMsg(){
  printf '[1;31m' >&2
  echo "$@" >&2
  printf '[0m' >&2
}
export -f errMsg

okMsg(){
  printf '[1;32m'
  echo "$@"
  printf '[0m'
}
export -f okMsg

warnMsg(){
  printf '[1;33m' >&2
  echo "$@" >&2
  printf '[0m' >&2
}
export -f warnMsg

normalizeUrl() {
    echo ${1}| sed 's;\(.*\);\1/;g' | sed 's;\(.*\)//;\1/;g'
}
upone() {
    local user="${1:?User not supplied}"
    local pwd="${2:?Password not supplied}"
    local remote="${3:?No remote repo}"
    local art="${4:?Artifact not supplied}"

    local ct=${types[${art##*.}]:-application/octet-stream}
    local dest="$remote$art"
    local status_code="500" # fake an error
    local returnCode=0

    if ! exists ${user} ${pwd} ${dest}; then

      for attempt in 1 2 3; do
            status_code=$(
                curl -u "${user}:${pwd}" -X PUT -d "@${art}" \
                -H "Content-type: $ct" -w "%{http_code}" "$dest" 2> /dev/null
            )
            case "${status_code}" in
                5*)
                    errMsg  "${status_code} $art -> $dest"
                    returnCode=1
                    # exponential back-off just in case we are overloading the server
                    sleep $(( attempt * 2 ))
                ;;
                4*)
                    warnMsg "${status_code} $art -> $dest"
                    break
                ;;
                2*)
                    okMsg "${status_code} $art -> $dest"
                    break
                ;;
            esac
        done
    fi
    return ${returnCode}
}
export -f upone

exists(){
    local user="${1:?User not supplied}"
    local pwd="${2:?Password not supplied}"
    local resource="${3:?Destination not supplied}"
    local status_code
    status_code=$(curl -I -u "${user}:${pwd}" "${resource}" 2> /dev/null |  head -1 | awk '{print $2}')
    [[ $? == 0 ]] && grep -qE "2[0-9]{2}" <<< ${status_code}
}
export -f exists

upload() {
    local user="${1:?User not supplied}"
    local pwd="${2:?Password not supplied}"
    local remote="${3:?No remote repo}"
    local local="${4:?No local repo supplied}"

    cat <<< $(localFiles ${local}) | xargs -n 1 -P 50 -I {} bash -c 'upone "$@"' _ ${user} ${pwd} ${remote} {}

}
verify() {
    local user="${1:?User not supplied}"
    local pwd="${2:?Password not supplied}"
    local remote="${3:?No remote repo}"
    local local="${4:?No local repo supplied}"
    cat <<< $(localFiles ${local}) | xargs -n 1 -P 50 -I {} bash -c 'verifyone "$@"' _ ${user} ${pwd} ${remote}{}

}


localFiles(){
    find "${1}" -not -path '*/\.*' -type f -not -name 'index.html' | sed -n 's;./\(.*\);\1;p'
}

validateCreds(){
    local user="${1:?User not supplied}"
    local pwd="${2:?Password not supplied}"
    local remote="${3:?No remote repo}"

    local status_code
    status_code=$(curl -I -u "${user}:${pwd}"  -w "%{http_code}"  "${remote}" |  head -1 | awk '{print $2}')
    if [[ $? != 0 || ${status_code} -eq 401 || ${status_code} -eq 403 ]]; then
        errMsg "${status_code} - curl test fail"  && return -1
    fi

}

remote="${1:?$(errMsg No destination repository supplied)}"
grep -qE "http[s]?://.*" <<< ${remote} || (errMsg "Not a valid URL '${remote}'" && exit -1)


remote=$(normalizeUrl ${remote})

user=${2:-${USER:-$(read -p "Username: "  user && [ -n "${user}" ] && echo ${user})}}
local="${3:-.}"
pwd=$(read -s -p "Password: "  pwd && [ -n "${pwd}" ] && echo ${pwd})
echo ""

validateCreds ${user} ${pwd} ${remote}

upload ${user} ${pwd} ${remote} ${local}

verify ${user} ${pwd} ${remote} ${local}
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.