Follow-up from this question using @Toby Speight's answer:

The primary concern is jq improvement/optimization, but please detail any others.


set -eu

trap 'rm "$sources"' EXIT

curl -s -o "$sources" https://raw.githubusercontent.com/T145/packages/master/net/adblock/files/adblock.sources

for key in $(jq -r 'keys[]' "$sources")
    case $key in
        gaming | oisd_basic )
        # Ignore these lists
        * )
            url=$(jq -r ".$key.url" "$sources")
            rule=$(jq -r ".$key.rule" "$sources")

            curl -s "$url" |
                case $url in
                    *.tar.gz) tar -xOzf - ;;
                    *) cat ;;
                esac |
                gawk --sandbox -- "$rule"
done |
    sed -e 's/\r//g' -e 's/^/ /' | sort -u > the_blacklist.txt
# use sort over gawk to merge sort multiple temp files instead of using up limited memory
  • \$\begingroup\$ Please refrain from modifying the code. After getting an answer you are not allowed to change your code anymore. This is to ensure that answers do not get invalidated and have to hit a moving target. If you have changed your code you can either post it as an answer (if it would constitute a code review) or ask a new question with your changed code (linking back to this one as reference). See the section What should I not do? on What should I do when someone answers my question? for more information \$\endgroup\$ May 26, 2021 at 15:44
  • \$\begingroup\$ @SᴀᴍOnᴇᴌᴀ I'm aware; the only change was a minor typo. Nothing about how the overall program works has been altered, and the change doesn't affect the chosen answer at all. \$\endgroup\$
    – T145
    May 26, 2021 at 16:27

1 Answer 1


Don't use a for loop to iterate over program output. See http://mywiki.wooledge.org/BashFAQ/001

Try this -- extract the keys, urls, rules all at once:

jq -r 'keys[] as $k | [$k, .[$k].url, .[$k].rule] | @tsv' "$sources" |
while IFS=$'\t' read key url rule; do 
    case $key in 

If, for some reason, your shell does not understand $'\t', use

while IFS="$(printf '\t')" read ...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.