I have an almost-JSON document in a file named versions.json. It looks like this:

  "package_a": {
    "sha_darwin": "6d94f041d06670977b2e5604b827f7e8bd46a1d200d14545ba75c17314ec12ad",
    "sha_linux": "6daa0fae252dc32e0af6ad89119555b277456349c55dfc9f276ee83718c6a9e2",
    "url": "https://github.com/someone/package_a/releases/download/v\(.version)/package_a_\(.version)_\(.platform)_x86_64.tar.gz",
    "version": "1.2.3"
  }, "package_b": {
    "sha_darwin": "6d94f041d06670977b2e5604b827f7e8bd46a1d200d14545ba75c17314ec12ad",
    "sha_linux": "6daa0fae252dc32e0af6ad89119555b277456349c55dfc9f276ee83718c6a9e2",
    "url": "https://storage.googleapis.com/package_b/release/v\(.version)/package_b_\(.version)_\(.platform)/amd64/package_b",
    "version": "6.3.0"

It's input from a third party and I can't change it. What I need to do is interpolate the platform and version into the URL. The problem is that that version value comes from the version in the file itself. I can't quite get it in the raw file because it's not actually valid JSON.

jq -r . versions.json
parse error: Invalid escape at line 5, column 129

I can get this in two passes:

echo '{"platform":"darwin","version":""}' |
  jq -f versions.json |
  jq -r '{version:.package_a.version,platform:"darwin"}' |
  jq -f versions.json |
  jq -r '"curl -sSL \(.package_a.url | @sh)"' |

But I'm not happy with the number of pipes and invocations of jq here, and looking to improve this all around. I would prefer a solution that involves a single invocation of jq, if that can be done.


1 Answer 1


Late answer, but here goes anyway

First off, I'm not a huge fan of piping the resulting commands to sh

As far as I can tell, we're just trying to generate a single curl command with a single parameter. A more natural way to do that would, in my opinion, likely be to just run a single curl command, using command substitution to generate the URL - kind of like curl -sSL "$(jq '#filters etc')"

Though perhaps this is intended to be able to be extended to handling multiple packages at once. Though in that case, I still find piping the entire curl commands to sh less natural than pipping the URLs themselves to xargs curl, like jq '#filters etc' | xargs curl -sSL

Alternatively, curl can take a config file on stdin, so something like jq '"generate url" | "--url=\(@sh)"' | curl --config - may also be an option

Beyond that, I find the use of versions.js as filter rather than input a bit awkward. True, seeing as it's not quite JSON, it's not acceptable input from the start, but the errors seem consistent and predictable - we may be able to replace the invalid escapes with placeholder text using a tool like sed, before processing the now-valid JSON in a single filter - which might perhaps look like so:

curl -sSL "$(sed 's/\\(/\\\\(/g' versions.json \
| jq -r '.package_a
    | .version as $version
    | .url
    | gsub("\\\\\\(\\.platform\\)"; "darwin")
    | gsub("\\\\\\(\\.version\\)"; $version)'
  • \$\begingroup\$ Absolutely right. Piping into a shell is always tricky, as it's so hard to get the quoting correct. Creating arguments/input for a command is much safer. \$\endgroup\$ Sep 2, 2022 at 5:58
  • \$\begingroup\$ Well, @sh exists for the purpose of getting quoting right. I'm not sure I agree that piping to sh is more cumbersome than xargs or a command substitution. But the idea of using curl's config file is a good one. I will have to think about this answer some more. \$\endgroup\$
    – kojiro
    Sep 2, 2022 at 11:08
  • \$\begingroup\$ I still think Sara's approach of "convert to valid JSON, then process as normal" is easier to follow. \$\endgroup\$ Sep 2, 2022 at 19:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.