Replacing values in json file with jq

Question

I have a JSON file like

{
    "Changes": [{
        "Action": "UPSERT",
        "ResourceRecordSet": {
            "Name": "RESOURCE_NAME",
            "Type": "A",
            "TTL": 60,
            "ResourceRecords": [{
                "Value": "RESOURCE_HOSTS"
            }]
        }
    }]
}

where I need to replace .Changes[].ResourceRecordSet.Name and .Changes[].ResourceRecordSet.ResourceRecords[] dynamically.

So, I wrote a script to replace required values with jq.

#!/bin/bash -xe

RESOURCE_NAME="example.com"
RESOURCE_HOSTS="1.1.1.1,2.2.2.2,"

cat resource-changes.json | \
jq '.Changes[].ResourceRecordSet.Name=env.RESOURCE_NAME' | \
jq 'del(.Changes[].ResourceRecordSet.ResourceRecords[])' \
> resource-changes.$$.json && \
mv resource-changes.$$.json resource-changes.json

for host in $(echo "$RESOURCE_HOSTS" | tr "," "\n"); do
  cat resource-changes.json | \
  jq --arg theHost $host '.Changes[].ResourceRecordSet.ResourceRecords |= (. + [{"Value":$theHost}])' \
  > resource-changes.$$.json
  mv resource-changes.$$.json resource-changes.json
done

jq '.' resource-changes.json

Which works fine and I can get required replacements. Though the script looks quite long and complicated IMO. Also, I do not really like joggling with temporary files. I'm looking for a way to simplify the script. Appreciate any comments!

Answer 1

Accessing environment variables in jq

This doesn't really work as it is...:

RESOURCE_NAME="example.com"
# ...

cat resource-changes.json | \
jq '.Changes[].ResourceRecordSet.Name=env.RESOURCE_NAME' | \
# ...

jq cannot access RESOURCE_NAME if it's not exported. The fix is simple:

export RESOURCE_NAME="example.com"

Iterating over a list of values

This is a very hacky way to iterate over a list of hosts:

RESOURCE_HOSTS="1.1.1.1,2.2.2.2,"

# ...

for host in $(echo "$RESOURCE_HOSTS" | tr "," "\n"); do

If the host names will never contain space, you could use a simple space-separated string:

RESOURCE_HOSTS="1.1.1.1 2.2.2.2"

# ...

for host in $RESOURCE_HOSTS; do

Or for maximum flexibility you could use a proper Bash array:

RESOURCE_HOSTS=(1.1.1.1 2.2.2.2)

# ...

for host in "${RESOURCE_HOSTS[@]}"; do

Streamlining by eliminating the loop

The current script creates many temporary files in the process, which is very ugly. The biggest obstacle to streamlining this into a single nice pipeline is the looping logic. You could prepare the host values into a JSON list and let jq take care of the rest.

Here's one way to create a JSON list from an array of hosts:

hosts=(1.1.1.1 2.2.2.2)

values="\"${hosts[0]}\""
for host in "${hosts[@]:1}"; do
    values="$values,\"$host\""
done

This assumes that there is at least one host. It initializes values with the first one, and then appends the others with a comma, adding double-quotes to make it valid JSON.

With this new variable, you can chain the jq calls like this:

cat resource-changes.json | \
jq '.Changes[].ResourceRecordSet.Name=env.RESOURCE_NAME' | \
jq 'del(.Changes[].ResourceRecordSet.ResourceRecords[])' | \
jq ".Changes[].ResourceRecordSet.ResourceRecords = ([$values] | map({\"Value\":.}))"

The cat command at the front is a bit ugly, because the filename could be a parameter of the first jq call, or replaced with an input redirection. But that would break the current nice aesthetic flow of the jq calls. This minor issue could be fixed by grouping and input redirection:

{
jq '.Changes[].ResourceRecordSet.Name=env.RESOURCE_NAME' | \
jq 'del(.Changes[].ResourceRecordSet.ResourceRecords[])' | \
jq ".Changes[].ResourceRecordSet.ResourceRecords = ([$values] | map({\"Value\":.}))"
} < resource-changes.json