Find download URL for a package, from almost-JSON with substitutions

Question

I have an almost-JSON document in a file named versions.json. It looks like this:

{
  "package_a": {
    "sha_darwin": "6d94f041d06670977b2e5604b827f7e8bd46a1d200d14545ba75c17314ec12ad",
    "sha_linux": "6daa0fae252dc32e0af6ad89119555b277456349c55dfc9f276ee83718c6a9e2",
    "url": "https://github.com/someone/package_a/releases/download/v\(.version)/package_a_\(.version)_\(.platform)_x86_64.tar.gz",
    "version": "1.2.3"
  }, "package_b": {
    "sha_darwin": "6d94f041d06670977b2e5604b827f7e8bd46a1d200d14545ba75c17314ec12ad",
    "sha_linux": "6daa0fae252dc32e0af6ad89119555b277456349c55dfc9f276ee83718c6a9e2",
    "url": "https://storage.googleapis.com/package_b/release/v\(.version)/package_b_\(.version)_\(.platform)/amd64/package_b",
    "version": "6.3.0"
  }
}

It's input from a third party and I can't change it. What I need to do is interpolate the platform and version into the URL. The problem is that that version value comes from the version in the file itself. I can't quite get it in the raw file because it's not actually valid JSON.

jq -r . versions.json
parse error: Invalid escape at line 5, column 129

I can get this in two passes:

echo '{"platform":"darwin","version":""}' |
  jq -f versions.json |
  jq -r '{version:.package_a.version,platform:"darwin"}' |
  jq -f versions.json |
  jq -r '"curl -sSL \(.package_a.url | @sh)"' |
  sh

But I'm not happy with the number of pipes and invocations of jq here, and looking to improve this all around. I would prefer a solution that involves a single invocation of jq, if that can be done.

Answer 1

Late answer, but here goes anyway

First off, I'm not a huge fan of piping the resulting commands to sh

As far as I can tell, we're just trying to generate a single curl command with a single parameter. A more natural way to do that would, in my opinion, likely be to just run a single curl command, using command substitution to generate the URL - kind of like curl -sSL "$(jq '#filters etc')"

Though perhaps this is intended to be able to be extended to handling multiple packages at once. Though in that case, I still find piping the entire curl commands to sh less natural than pipping the URLs themselves to xargs curl, like jq '#filters etc' | xargs curl -sSL

Alternatively, curl can take a config file on stdin, so something like jq '"generate url" | "--url=\(@sh)"' | curl --config - may also be an option

Beyond that, I find the use of versions.js as filter rather than input a bit awkward. True, seeing as it's not quite JSON, it's not acceptable input from the start, but the errors seem consistent and predictable - we may be able to replace the invalid escapes with placeholder text using a tool like sed, before processing the now-valid JSON in a single filter - which might perhaps look like so:

curl -sSL "$(sed 's/\\(/\\\\(/g' versions.json \
| jq -r '.package_a
    | .version as $version
    | .url
    | gsub("\\\\\\(\\.platform\\)"; "darwin")
    | gsub("\\\\\\(\\.version\\)"; $version)'
)"