5
\$\begingroup\$

This is my rewrite of the program posted here, based on janos's comments. I also added the ability to read from stdin.

I've also included a test script. I wanted something lightweight I could easily re-run, but didn't want a dependency or full test framework like bats. Comments on the test script are also welcome.

New Program

#!/bin/bash

set -euo pipefail

die() {
  printf '%s\n' "$1" >&2
  exit 1
}

show_help() {
>&2 echo "Usage: 

  $ $0 <filename>

The program will read from stdin if <filename> is not given.

Description:

  Fills in template files with ENV variables.  <filename> is assumed to be a
  template file whose variables are enclosed in double braces like:

  Some content with {{ MY_VAR1 }}
  or {{ MY_VAR1 }} and {{ MY_VAR2 }}

  where MY_VAR1 and MY_VAR2 and ENV variables.  Assuming that the ENV variables
  are set:

  $ export MY_VAR1=value_1
  $ export MY_VAR2=value_2

  then executing this script on a file with the above content will output:

  Some content with value_1
  or value_1 and value_2
"
}

if [[ $# -gt 0 && ( $1 = '-h' || $1 = '--help' ) ]]; then
  show_help
  exit 0
fi

# If given, ensure arg is a file:

if [[ $# -gt 0 && ! -f $1 ]]; then
  die "'$1' is not a file." 
fi

# If we're reading from stdin, save its contents to a temp file
# This is because we need to read it twice: first to extract 
# the required vars, then again to replace them.
if [[ $# -eq 0 ]]; then
  # Read stdin into a temp file
  tmpfile=$(mktemp)
  cat /dev/stdin > "$tmpfile"

  # Clean it up, saving to FDs to read from
  exec 3< "$tmpfile"
  exec 4< "$tmpfile"
  rm "$tmpfile"
else
  exec 3< "$1"
  exec 4< "$1"
fi

# Gather all the required template variables

vars=()
while IFS= read -r line; do
  vars+=( "$line" )
done < <( grep -Eo '\{\{ ([-_[:alnum:]]*) }}' <&3 | \
          grep -Eo '([-_[:alnum:]]*)' | \
          sort -u )

# Verify that all template variables exist

missing=()
for var in "${vars[@]}"; do
  if [[ -z ${!var+x} ]]; then
    missing+=( "$var" )
  fi
done

if [[ ${#missing[@]} -gt 0 ]]; then
  >&2 echo "The following required variables have not been exported:"
  for var in "${missing[@]}"; do
    >&2 echo "${var}"
  done
  exit 1
fi

# Dynamically construct the sed cmd to do the replacement

sed_cmd=
for var in "${vars[@]}"; do
  # sanitize the user's input (ie, the var's value) by prepending a backslash
  # to our sed delimiter (#) as well as to backslashes themselves, to prevent
  # the input from being interpreted by sed as a special character like a
  # backreference (\1) or a tab (\t), etc
  escaped_val=$(printf "%s" "${!var}" | sed -E 's|([#\])|\\\1|g')
  sed_cmd+="s#\\{\\{ ${var} }}#${escaped_val}#g;"
done

sed -E "${sed_cmd}" <&4

Test Script

#!/bin/bash
# shellcheck disable=SC2030,SC2031
# disabling because we're modifying in a subshell by design

set -euo pipefail

# Do everything in a sub-process to keep parent process clean
(

new_test() {
  echo "$1"
  unset MY_VAR1
  unset MY_VAR2
}

  basic_tmpl='
hello there {{ MY_VAR1 }}
some other stuff

foo: {{ MY_VAR2 }}
line with both: {{ MY_VAR1 }} and {{ MY_VAR2 }}'

# Each test goes in a subshell too
(
  new_test "Should error when tmpl variables are missing"

  printf "%s" "$basic_tmpl" | ./fill_template 2>/dev/null \
    && echo 'FAIL' || echo 'PASS'
)

(
  new_test "Should succeed when tmpl variables are set"

  export MY_VAR1=val1
  export MY_VAR2=val2

  printf "%s" "$basic_tmpl" | ./fill_template >/dev/null 2>&1 \
    && echo 'PASS' || echo 'FAIL'
)

(
  new_test "Basic template should produce expected output"

  export MY_VAR1=val1
  export MY_VAR2=val2

  result=$(printf '%s' "$basic_tmpl" | ./fill_template 2> /dev/null)
  expected='
hello there val1
some other stuff

foo: val2
line with both: val1 and val2'

  [[ "$result" = "$expected" ]] && echo 'PASS' || echo 'FAIL'
)

(
  new_test "Values with spaces/slashes/metachars still work"

  export MY_VAR1='/some/path/and_\1_\t+\_'
  export MY_VAR2='blah _\\_ baz'

  result=$(printf '%s\n\n' "$basic_tmpl" | ./fill_template 2> /dev/null)
  expected='
hello there /some/path/and_\1_\t+\_
some other stuff

foo: blah _\\_ baz
line with both: /some/path/and_\1_\t+\_ and blah _\\_ baz'

  [[ "$result" = "$expected" ]] && echo 'PASS' || echo 'FAIL'
)

)
\$\endgroup\$
  • \$\begingroup\$ Can you clarify what you mean by "pure" Bash? I assumed it meant using only builtins, but I see grep, sed, mktemp, sort and more. sed in particular, being a program interpreter, seems far from any definition of "pure Bash" than I can conceive. \$\endgroup\$ – Toby Speight Apr 5 at 8:56
  • \$\begingroup\$ sed, awk, etc are fine. \$\endgroup\$ – Jonah Apr 5 at 13:45
3
\$\begingroup\$

The grep matches a lot of strings that can't be valid bash identifiers ("alphanumeric characters and underscores, beginning with an alphabetic character or an underscore"). There are some magical exceptions like $* and $@ but I'm assuming you aren't trying to support those in your templates.

Reading inputs twice is not necessary; a single-pass approach would remove a lot of complexity from your program, and make it possible to process infinitely large inputs on STDIN. The only real drawback is that it will exit on the first undefined value, instead of producing a list.

And finally, everyone has their own definitions, but—to me—"pure bash" means "without the use of external programs." Making your program meet this bar not only makes it faster, it removes even more complexity, because you don't need to filter values through an external program that might misunderstand them. It also means your template values need not be exported. They only need to be visible to the current shell.

This version, implemented as a function, uses only bash builtins and passes your tests once ./fill_template is replaced by fill_template:

fill_template() {
  (( ${#@} )) || set -- /dev/stdin
  local file line eof original name value
  for file; do
    while true; do
      read -r line
      eof=$?
      while [[ $line =~ \{\{" "*([a-zA-Z_][_a-zA-Z0-9]*)" "*\}\} ]]; do
        original=$BASH_REMATCH
        name=${BASH_REMATCH[1]}
        value=${!name?"unset template variable: $name"}
        line=${line//$original/$value}
      done
      printf -- %s "$line"
      (( eof )) && break || printf "\n"
    done <$file
  done
}
\$\endgroup\$
  • 1
    \$\begingroup\$ Very nice. I do still like the idea of printing out all missing ENV vars (I think it will be an easy thing to forget when using the script) but I much prefer the simplicity of the single pass. I'm thinking a better design would be a single pass like your suggestion as the default (with die on first error), and then a separate option to list all missing vars. \$\endgroup\$ – Jonah Apr 2 at 1:06
  • \$\begingroup\$ You could switch modes on error -- proceed normally until the first failure; on failure set a flag that suppresses normal template output and only prints the names of unset variables. \$\endgroup\$ – Oh My Goodness Apr 3 at 22:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.