The Blacklist: Blocking Malicious domains using Bash

Question

I've made this script to automate blocking some deviant hosts on my router, and was curious if there's anything much else that can be done to make it quicker and more efficient.

Presently I'm restricted to using packages available on the Entware repository and the latest Busybox/ash shell environment. So, for example, I can't use commands like sort -S 25% --parallel=4 -u adblock.sources (though awk/gawk is faster and more memory efficient anyway).

update_the_blacklist.sh

curl -GOs https://raw.githubusercontent.com/T145/packages/master/net/adblock/files/adblock.sources

for key in $(jq -r 'keys[]' adblock.sources); do
        case $key in
        gaming | osid_basic )
                # Ignore these lists
                ;;
        * )
                url=$(jq -r ".$key.url" adblock.sources)
                rule=$(jq -r ".$key.rule" adblock.sources)

                case $url in
                *.tar.gz )
                        curl -s $url | \
                        tar -xOzf - | \
                        gawk "$rule" | \
                        sed "s/\r//g" | \
                        sed 's/^/0.0.0.0 /' >> the_blacklist.temp.txt
                        ;;
                * )
                        curl -s $url | \
                        gawk "$rule" | \
                        sed "s/\r//g" | \
                        sed 's/^/0.0.0.0 /' >> the_blacklist.temp.txt
                esac

                unset url
                unset rule
        esac
done

# Filter duplicate hosts
gawk '!a[$0]++' the_blacklist.temp.txt > the_blacklist.txt

rm the_blacklist.temp.txt
rm adblock.sources

Answer 1

We're missing a shebang line. The question is tagged bash, but that seems overkill, since there's nothing here that's not standard POSIX shell.

#!/bin/sh

We append to the_blacklist.temp.txt, but don't ensure that it starts off empty, or that it's even writeable. It's best to create temporary files using mktemp:

tmpfile=mktemp || exit $?
trap "rm $tmpfile" EXIT

Also, instead of directing final output to the_blacklist.txt, it would be more flexible to write to standard output, and let the user choose where to send it (perhaps pipe into another filter).

But there seems to be no reason to use a file here - just pipe the for loop directly into awk (or into sort -u - that may be a better choice if you're memory-constrained, as it will perform an external merge-sort if it needs to).

The duplicated sed pipe could be replaced with a single sed program of two commands:

for
    …
done |
    sed -e 's/\r//g' -e 's/^/0.0.0.0 /' | gawk '!a[$0]++'

There's a security issue here:

                    gawk "$rule" | \

We are executing downloaded content in awk. Because we can't be sure that the adblock.sources URL cannot be replaced with a more harmful version, we should run awk in its constrained "sandbox" mode, and use the -- separator so we don't interpret the script as arguments.

                    gawk --sandbox -- "$rule" | \

---

These changes give me

#!/bin/sh

set -eu

sources=$(mktemp)
trap 'rm "$sources"' EXIT

curl -s -o "$sources" https://raw.githubusercontent.com/T145/packages/master/net/adblock/files/adblock.sources

for key in $(jq -r 'keys[]' "$sources")
do
    case $key in
        gaming | osid_basic )
        # Ignore these lists
        ;;
        * )
            url=$(jq -r ".$key.url" "$sources")
            rule=$(jq -r ".$key.rule" "$sources")

            curl -s "$url" |
                case $url in
                    *.tar.gz) tar -xOzf - ;;
                    *) cat ;;
                esac |
                gawk --sandbox -- "$rule"
    esac
done |
    sed -e 's/\r//g' -e 's/^/0.0.0.0 /' | sort -u

Somebody more expert with jq could probably avoid the remaining temporary file by making a single pass over the retrieved data.