4
\$\begingroup\$

This actually works nicely (we have > 2k repos) but I would really like it to start producing results right away as soon as it has any. Mostly so it seems faster than it is (by being more responsive).

   bbrepos(){
        #clone[1] is ssh using a filter made the escaping really ugly. 
        #If it becomes necessary we can use a jq arg
        # get max number of repos, ceil(repos / 50), then create a page 
        #sequence and curl them all
        local url=https://api.bitbucket.org/2.0/repositories/twengg
        local project=${1:+"q=project.key=\"$1\""}
        curl -snL  "$url?pagelen=0&page=1" | jq '"$(((\(.size) / 50) + 1 ))"' \
            | xargs bash -c 'eval "echo $1"' _ | xargs seq 1 | xargs -L 1 printf "?pagelen=50&page=%s&${1:-$project}\n" \
            | xargs -I {} -L 1 -P 20 -I {} bash -c 'curl -snL  "$1/$2" | jq -er .values[].links.clone[1].href' _ $url {} \ #clone[1] is ssh
            | sort -u
    }

Sample workflow:

bbsearch something | xargs -P 20 git clone

Alternatively, it might also be handy to have these other workflows:

bbsearch something other thing | xargs -P 20 git clone

cat things-to-find.txt | bbsearch | xargs -P 20 git clone

Sample payload (it's normally huge, so I stripped it down to what is needed):

{
  "pagelen": 1,
  "size": 3054,
  "values": [
    {
      "links": {
        "clone": [
          {
            "href": "https://chb0bitbucket@bitbucket.org/twengg/development-process.git",
            "name": "https"
          },
          {
            "href": "git@bitbucket.org:twengg/development-process.git",
            "name": "ssh"
          }
        ],
        "self": {
          "href": "https://api.bitbucket.org/2.0/repositories/twengg/development-process"
        }
      }
    }
  ],
  "page": 1,
  "next": "https://api.bitbucket.org/2.0/repositories/twengg?page=2"
}
\$\endgroup\$
  • 1
    \$\begingroup\$ Please describe the flow appropriately and add a sample fragment of .values[].links structure \$\endgroup\$ – RomanPerekhrest Nov 19 '19 at 12:25
  • 1
    \$\begingroup\$ This could be a great question if it described the work flow. Who doesn't want to speed up their workflow. A real example might also be nice. \$\endgroup\$ – pacmaninbw Nov 19 '19 at 15:28
2
\$\begingroup\$

eval is evil

Always look for ways to avoid eval.

Relevant snippet from the posted code:

curl -snL  "$url?pagelen=0&page=1" \
    | jq '"$(((\(.size) / 50) + 1 ))"' \
    | xargs bash -c 'eval "echo $1"' _

The code uses jq to create a formatted expression to evaluate by Bash, with .size injected in the middle of the expression.

A better way to achieve the same thing would be to make jq simply output the variable itself, and then write the expression in Bash:

curl -snL  "$url?pagelen=0&page=1" \
    | jq -r .size \
    | xargs bash -c 'echo $((($1 / 50) + 1))' _

Unnecessary pipeline

It's strange to use xargs to process one line of input, and it's strange to spawn another Bash just to evaluate an expression. There's really no need to force a chain of operations in a single pipeline.

I'd rewrite the above snippet like this:

repos_count=$(curl -snL "$url?pagelen=0&page=1" | jq -r .size)
((pages_count = (repos_count / 50) + 1))

Store positional arguments in variables with descriptive names

It's a mental burden to have to remember what $1 and $2 mean. I'd store them in local variables with descriptive names at the beginning of the function.

Avoid seq

Is it obsolete, and not available in all systems. Use a Bash counting loop instead:

for ((page = 1; page <= pages_count; page++)); do
    ...
done \
| xargs -L 1 ...

Break lines in long pipelines

It's easier to read code when there's one statement per line. This is especially true when a line is so long you have to scroll to the right to see it. So instead of this:

| xargs bash -c 'eval "echo $1"' _ | xargs seq 1 | xargs -L 1 printf "?pagelen=50&page=%s&${1:-$project}\n" \
| xargs -I {} -L 1 -P 20 -I {} bash -c 'curl -snL  "$1/$2" | jq -er .values[].links.clone[1].href' _ $url {} \ #clone[1] is ssh
| sort -u

I recommend to write like this:

| xargs bash -c 'eval "echo $1"' _ \
| xargs seq 1 \
| xargs -L 1 printf "?pagelen=50&page=%s&${1:-$project}\n" \
| xargs -I {} -L 1 -P 20 -I {} bash -c 'curl -snL  "$1/$2" \
| jq -er .values[].links.clone[1].href' _ $url {} \
| sort -u

Always double-quote variables used as command line arguments

In | jq -er .values[].links.clone[1].href' _ $url {} you forgot to double-quote $url.

\$\endgroup\$
  • \$\begingroup\$ This is good feedback and I will incorporate it - however, it doesn't address how I get this to output immediately more input comes \$\endgroup\$ – Christian Bongiorno Dec 7 '19 at 20:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.