Skip to main content
Use `sort -u` to uniquify the result
Source Link
Toby Speight
  • 81.7k
  • 14
  • 101
  • 308

But there seems to be no reason to use a file here - just pipe the for loop directly into awk (or into sort -u - that may be a better choice if you're memory-constrained, as it will perform an external merge-sort if it needs to).

The duplicated sed pipe could be replaced with a single sed program of two commands:

#!/bin/sh

set -eu

sources=$(mktemp)
trap 'rm "$sources"' EXIT

curl -s -o "$sources" https://raw.githubusercontent.com/T145/packages/master/net/adblock/files/adblock.sources

for key in $(jq -r 'keys[]' "$sources")
do
    case $key in
        gaming | osid_basic )
        # Ignore these lists
        ;;
        * )
            url=$(jq -r ".$key.url" "$sources")
            rule=$(jq -r ".$key.rule" "$sources")

            curl -s "$url" |
                case $url in
                    *.tar.gz) tar -xOzf - ;;
                    *) cat ;;
                esac |
                gawk --sandbox -- "$rule"
    esac
done |
    sed -e 's/\r//g' -e 's/^/0.0.0.0 /' | gawksort '!a[$0]++'-u

But there seems to be no reason to use a file here - just pipe the for loop directly into awk (or into sort -u).

The duplicated sed pipe could be replaced with a single sed program:

#!/bin/sh

set -eu

sources=$(mktemp)
trap 'rm "$sources"' EXIT

curl -s -o "$sources" https://raw.githubusercontent.com/T145/packages/master/net/adblock/files/adblock.sources

for key in $(jq -r 'keys[]' "$sources")
do
    case $key in
        gaming | osid_basic )
        # Ignore these lists
        ;;
        * )
            url=$(jq -r ".$key.url" "$sources")
            rule=$(jq -r ".$key.rule" "$sources")

            curl -s "$url" |
                case $url in
                    *.tar.gz) tar -xOzf - ;;
                    *) cat ;;
                esac |
                gawk --sandbox -- "$rule"
    esac
done |
    sed -e 's/\r//g' -e 's/^/0.0.0.0 /' | gawk '!a[$0]++'

But there seems to be no reason to use a file here - just pipe the for loop directly into awk (or into sort -u - that may be a better choice if you're memory-constrained, as it will perform an external merge-sort if it needs to).

The duplicated sed pipe could be replaced with a single sed program of two commands:

#!/bin/sh

set -eu

sources=$(mktemp)
trap 'rm "$sources"' EXIT

curl -s -o "$sources" https://raw.githubusercontent.com/T145/packages/master/net/adblock/files/adblock.sources

for key in $(jq -r 'keys[]' "$sources")
do
    case $key in
        gaming | osid_basic )
        # Ignore these lists
        ;;
        * )
            url=$(jq -r ".$key.url" "$sources")
            rule=$(jq -r ".$key.rule" "$sources")

            curl -s "$url" |
                case $url in
                    *.tar.gz) tar -xOzf - ;;
                    *) cat ;;
                esac |
                gawk --sandbox -- "$rule"
    esac
done |
    sed -e 's/\r//g' -e 's/^/0.0.0.0 /' | sort -u
Source Link
Toby Speight
  • 81.7k
  • 14
  • 101
  • 308

We're missing a shebang line. The question is tagged , but that seems overkill, since there's nothing here that's not standard POSIX shell.

#!/bin/sh

We append to the_blacklist.temp.txt, but don't ensure that it starts off empty, or that it's even writeable. It's best to create temporary files using mktemp:

tmpfile=mktemp || exit $?
trap "rm $tmpfile" EXIT

Also, instead of directing final output to the_blacklist.txt, it would be more flexible to write to standard output, and let the user choose where to send it (perhaps pipe into another filter).

But there seems to be no reason to use a file here - just pipe the for loop directly into awk (or into sort -u).

The duplicated sed pipe could be replaced with a single sed program:

for
    …
done |
    sed -e 's/\r//g' -e 's/^/0.0.0.0 /' | gawk '!a[$0]++'

There's a security issue here:

                    gawk "$rule" | \

We are executing downloaded content in awk. Because we can't be sure that the adblock.sources URL cannot be replaced with a more harmful version, we should run awk in its constrained "sandbox" mode, and use the -- separator so we don't interpret the script as arguments.

                    gawk --sandbox -- "$rule" | \

These changes give me

#!/bin/sh

set -eu

sources=$(mktemp)
trap 'rm "$sources"' EXIT

curl -s -o "$sources" https://raw.githubusercontent.com/T145/packages/master/net/adblock/files/adblock.sources

for key in $(jq -r 'keys[]' "$sources")
do
    case $key in
        gaming | osid_basic )
        # Ignore these lists
        ;;
        * )
            url=$(jq -r ".$key.url" "$sources")
            rule=$(jq -r ".$key.rule" "$sources")

            curl -s "$url" |
                case $url in
                    *.tar.gz) tar -xOzf - ;;
                    *) cat ;;
                esac |
                gawk --sandbox -- "$rule"
    esac
done |
    sed -e 's/\r//g' -e 's/^/0.0.0.0 /' | gawk '!a[$0]++'

Somebody more expert with jq could probably avoid the remaining temporary file by making a single pass over the retrieved data.