Shell Template Files

We get lots of questions about how to use a template file, meaning a file with placeholders in it, which we want to replace with values at run time. There are a few different forms of this question, and as you would expect, a few different answers depending on the exact requirements.

The most basic requirement is what GNU autoconf uses: a pre-made template file with %PLACEHOLDER% things in it, and values in specific shell variables that we want to use. Sometimes, sed is OK for this particular task:

sed -e "s/%PLACEHOLDER%/$variable/g" \
    -e "s/%FIRSTNAME%/$firstname/g" \
    ... "$templatefile" > "$outputfile"

The notation %FOO% is what autoconf actually uses for its placeholders; hence its choice here.

If the variables can contain slashes or newlines in their contents, then appropriate steps must be taken. A different delimiter than slash can be used for the sed s/// command, but for newlines, you're advised to consult an advanced sed FAQ. Fortunately, the case where people want to include newlines in their substitutions, in this form of the question, is somewhat uncommon.

Generally speaking, this usage introduces a code injection vulnerability, in the sense that the contents of your variables are being parsed by sed and interpreted as sed commands. The slash and newline special cases are just one place where the use of sed backfires.

If you don't want to work around sed's limitations, or if you're rightfully unwilling to introduce potential code injection vulnerabilities, consider using perl instead:

PLACEHOLDER=$variable FIRSTNAME=$firstname \
perl -p -e 's/%PLACEHOLDER%/$ENV{"PLACEHOLDER"}/g;
    s/%FIRSTNAME%/$ENV{"FIRSTNAME"}/g;' \
    "$templatefile" > "$outputfile"

This is completely superior to the sed solution in every way except that sed is a standard POSIX command, guaranteed to be available, while perl is not. The perl solution doesn't break with newlines or slashes in the variable contents, and there is no possible code injection.

An alternative form of this question is, "I have a file with shell parameter expansions in it, like $foo. I don't know in advance what variables will be used, so I can't just make a list of all of them in a giant sed command." (Newlines in the variables are much more common here, but fortunately they do not present a problem with the solutions we're about to give.)

If you have GNU gettext installed on your system, you can use a program called envsubst to perform the template file substitutions:

export VAR1 VAR2 ...
envsubst < "$templatefile" > "$outputfile"

If you don't have envsubst but you have Bash, you can use an approach like this:

   1 # Bash 3.2 or higher
   2 LC_COLLATE=C
   3 while read -r; do
   4   while [[ $REPLY =~ \$(([a-zA-Z_][a-zA-Z_0-9]*)|\{([a-zA-Z_][a-zA-Z_0-9]*)\})(.*) ]]; do
   5     if [[ -z ${BASH_REMATCH[3]} ]]; then   # found $var
   6       printf %s "${REPLY%"$BASH_REMATCH"}${!BASH_REMATCH[2]}"
   7     else # found ${var}
   8       printf %s "${REPLY%"$BASH_REMATCH"}${!BASH_REMATCH[3]}"
   9     fi
  10     REPLY=${BASH_REMATCH[4]}
  11   done
  12   printf "%s\n" "$REPLY"
  13 done < "$templatefile" > "$outputfile"

The good:

• It works without exporting your variables.

• It doesn't suffer code injection exploits with $( ) or ` `, and EOF in the template won't break it.

The bad:

• There are some name collision issues: trying to replace $REPLY or $BASH_REMATCH will use their current value in the middle of the loop (but it won't loop forever on malicious input). The name collision can be worked around with a minor modification requiring a prefix for each variable:

     1 while read -r; do
     2   while [[ $REPLY =~ \$(([a-zA-Z_][a-zA-Z_0-9]*)|\{([a-zA-Z_][a-zA-Z_0-9]*)\})(.*) ]]; do
     3     var=X_${BASH_REMATCH[2]:-${BASH_REMATCH[3]}}
     4     printf %s "${REPLY%"${BASH_REMATCH[0]}"}${!var}"
     5     REPLY=${BASH_REMATCH[4]}
     6   done
     7   printf "%s\n" "$REPLY"
     8 done < "$templatefile" > "$outputfile"
  

  In this case, a ${REPLY} in the template file will be replaced by the content of the variable X_REPLY instead of REPLY.

If you need to do it in sh, a portable but potentially dangerous alternative involves constructing a HereDocument and feeding the result to a shell for expansion:

{ echo "cat <<EOF"
  cat "$templatefile"
  echo "EOF"
} | sh > "$outputfile"

NOTICE: If you get an unexpected result that includes EOF at the end of the output, your template file probably lacks a newline char at the end.

Recall that substitutions occur in the body of a here document as long as the sentinel (EOF in our case) is not quoted on the first line. We take advantage of that here.

Also, note that the substitutions are performed in a new sh shell (or bash or ksh, whatever you require), not in the current shell. Therefore, any variables you want to use must be exported.

Since any command substitutions in the template file will be executed as shell code, it is important that any template file used in this way be under your control, and not generated by a user. You've been warned.

CategoryShell