bash script - sed - template file processing

Question

Using a bash script to process a file and pass it to AWS CLI. The script runs inside a docker container.

I'm using sed to do template string replace. When MY_VALUE contains special chars. (eg: MY_VALUE="http://aaa.com") the sed command fails.

This makes me question if sed is the right tool for this job, don't have a lot of control on the values that come in.

Because the template is a JSON file, I considered trying JQ or Python (something like this).

I haven't used bash or Unix tools a lot so I would like to hear your thoughts on:

1. Is sed the right tool for this type of job?
2. What else would you use?
3. What do you think about the script?

Simplified template (JSON)

{
    "networkMode": "bridge",
    "name": "[Placeholder.A]"     
}

Script

#!/bin/bash

MY_VALUE=${myvalue}
rm -f processed.json

cat /app/template.json | \
sed s/\\[Placeholder.A]/${MY_VALUE}/ > /app/processed.json

aws cli command -file processed.json

Answer 1

You are parsing a JSON file using some UNIX tools. This is risky, since those can have so many edge cases that you will end up refactoring over and over.

For example, you mention that sed s/\\[Placeholder.A]/${MY_VALUE}/ is giving errors when $MY_VALUE has slashes. This is normal, since this is how it gets expanded:

sed s/\[Placeholder.A]/${MY_VALUE}/
                           ↓    
sed s/\[Placeholder.A]/http://aaa.com/
#    ^                ^     ^^       ^

See? Too many slashes for a simple sed 's/find/replace/, thus it fails. You can solve this by using another separator:

$ sed 's/\[Placeholder.A\]/a/' file
{
    "networkMode": "bridge",
    "name": "a"     
}
$ MY_VALUE="http://aaa.com"
$ sed "s/\[Placeholder.A\]/$MY_VALUE/" file
sed: -e expression #1, char 27: unknown option to `s'

# Let's use ? as a separator, for example
$ sed "s?\[Placeholder.A\]?$MY_VALUE?" file
{
    "networkMode": "bridge",
    "name": "http://aaa.com"     
}

---

Instead, use jq, a command-line JSON processor. Given your file:

$ cat file
{
    "networkMode": "bridge",
    "name": "[Placeholder.A]"     
}

See how easy is to replace the value in the "name" key:

$ jq '.name="a"' file
{
  "networkMode": "bridge",
  "name": "a"
}

If you want to use a variable then you will need to do some quotes-opening-closing:

$ MY_VALUE="http://aaa.com"
$ jq '.name="'$MY_VALUE'"' file
{
  "networkMode": "bridge",
  "name": "http://aaa.com"
}