4
\$\begingroup\$

One of my main goals regarding this script has been to figure out a way to use the "adblock.sources" file available on its main repository and not my fork. That version has some syntax issues with its JSON, preventing it from being directly piped into jq.

After some research, I came across the jq -n -f {file} command that successfully read in the ill-formatted JSON! I have a couple questions on this feature and one on the sed command.

  1. Is jq -n -f {file} being used properly?
  2. Can the two jq commands be blended into one command?
  3. Can the two sed parameters be blended into one parameter?
#!/bin/sh

set -eu

sources=$(mktemp)
trap 'rm "$sources"' EXIT

curl -s -o "$sources" https://raw.githubusercontent.com/openwrt/packages/master/net/adblock/files/adblock.sources

jq -n -f "$sources" | jq -r 'keys[] as $k | [$k, .[$k].url, .[$k].rule] | @tsv' |
while IFS=$'\t' read key url rule; do
    case $key in
        gaming | oisd_basic | yoyo )
        # Ignore these sources:
        # "gaming" blocks virtually all gaming servers
        # "oisd_basic" is included in "oisd_full"
        # "yoyo" is included in "stevenblack"
        ;;
        * )
            curl -s "$url" |
                case $url in
                    *.tar.gz) tar -xOzf - ;;
                    *) cat ;;
                esac | gawk --sandbox -- "$rule"
        ;;
    esac
done | sed -e 's/\r//g' -e 's/^/0.0.0.0 /' | sort -u >| the_blacklist.txt
# use sort over gawk to merge sort multiple temp files instead of using up limited memory
\$\endgroup\$

1 Answer 1

6
\$\begingroup\$

You might want to check again the link you've provided: on 2023-10-12, it is strictly valid JSON and as such you can remove the jq -n -f part completely.

  1. That being said, yes, if it worked for you, you were probably using jq -n -f correctly. The idea behind this workaround is the following: you are telling jq to ignore its input (-n), thereby not using its parser for data, and to read the filter to apply from your file (-f). Since jq's parser for filters is much more permissive than the one for data (e.g. since the filter language accepts #-style comments, you could use this to parse json data with #-style comments, or with unquoted identifiers -- see here), you can use this to obtain correct input from nearly-correct input. Note that this will not always work though, as "nearly correct" is defined as what the filter parser accepts.

  2. Yes, with some modification of the input file of the first command. Because in the jq -n -f command the filter is read from a file, and you use the filter to define the data, you can simply append the actual filter you're using at the end of the json data and use that for a single pass, with jq -n -r -f:

    In your "$sources" file:

    { nearly-json data ...} | keys[] as $k | [$k, .[$k].url, .[$k].rule] | @tsv
    

    You can either modify the "$sources" file after curl-ing, or use the fact that jq -f allows - as a filename to point to standard input, and construct the desired string beforehand in a pipe.

  3. Yes as well. Sed allows usage of ; to separate commands in scripts, so your above command can be written sed -e 's/\r//g; s/^/0.0.0.0 /'.

One final comment: when writing shell scripts, especially involved ones, it is useful to have shellcheck running in your editor. In your case, it gives the following two (useful) warnings:

shellcheck blacklist.sh 

In foo.sh line 11:
while IFS=$'\t' read key url rule; do
          ^---^ SC3003 (warning): In POSIX sh, $'..' is undefined.
                ^--^ SC2162 (info): read without -r will mangle backslashes.

For more information:
  https://www.shellcheck.net/wiki/SC3003 -- In POSIX sh, $'..' is undefined.
  https://www.shellcheck.net/wiki/SC2162 -- read without -r will mangle backslashes.
\$\endgroup\$
1
  • 2
    \$\begingroup\$ Pedantic note - GNU sed allows separating commands using ;. For standard (POSIX) sed only newline is permitted as separator, so pass the commands separately, e.g. sed -e 's/\r//g' -e s/^/0.0.0.0 /'` \$\endgroup\$ Oct 13, 2023 at 6:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.