Diff for "BashFAQ/110"

How can i perform a substitution with arbitrary values ("s/$foo/$bar/") safely, without treating either value as a regular expression or worrying about other special characters?

Sed is not the right tool for this. Nor is ed. At best, attempting either will result in an escaping nightmare, and will be extremely prone to bugs.

First, what are we performing the substitution on? If it's a string, it can be done very simply with a parameter expansion.

var='some string'
search=some
rep=another
printf '%s\n' "${var//"$search"/$rep}"

This is discussed in more detail in Faq #100.

If it's a file or stream, things get a bit trickier. One way to accomplish this would be to combine the previous method with Faq #1.

search=foo
rep=bar

# file
while IFS= read -r line; do
  printf '%s\n' "${line//"$search"/$rep}"
done < "$file"

# command output
while IFS= read -r line; do
  printf '%s\n' "${line//"$search"/$rep}"
done < <(my_command)

my_command | while IFS= read -r line; do
  printf '%s\n' "${line//"$search"/$rep}"
done

Note that the last example creates a subshell. See Faq #24 for more information on that.

Both of the above examples print to stdout; neither actually edits the file in place. Of course this could be resolved with something like:

# create a temp file, die on failure
tmp=$(mktemp) || exit

while IFS= read -r line; do
  printf '%s\n' "${line//"$search"/$rep}"
done < "$file" > "$tmp" && mv "$tmp" "$file"

On large data sets, you'll notice that this is quite slow. The following functions use awk, and are quite a bit faster:

The first two here are similar to sed 's/STR/REP/', they only replaces the first instance on each line. The first function operates on stdin and writes to stdout, the second overwrites FILE.

# usage: sub_literal STR REP
sub_literal() {
  # string manip needed to escape '\'s, so awk doesn't expand '\n' and such
  awk -v str="${1//\\/\\\\}" -v rep="${2//\\/\\\\}" '
    # get the length of the search string
    BEGIN {
      len = length(str);
    }

    # if the search string is in the line
    (i = index($0, str)) {
      # replace the first occurance with rep
      $0 = substr($0, 1, i-1) rep substr($0, i + len);
    }

    # print each line
    1
  '
}

# usage: sub_literal_f STR REP FILE
sub_literal_f() {
  local tmp
  if ! [[ -f $3 && -r $3 && -w $3 ]]; then
    printf '%s does not exist or is not readable or writable\n' "$3" >&2
    return 1
  fi

  trap 'rm -rf "$tmp"' RETURN
  tmp=$(mktemp) && cp "$3" "$tmp" || return

  # string manip needed to escape '\'s, so awk doesn't expand '\n' and such
  awk -v str="${1//\\/\\\\}" -v rep="${2//\\/\\\\}" '
    # get the length of the search string
    BEGIN {
      len = length(str);
    }

    # if the search string is in the line
    (i = index($0, str)) {
      # replace the first occurance with rep
      $0 = substr($0, 1, i-1) rep substr($0, i + len);
    }

    # print each line
    1
  ' "$tmp" > "$3"
}

The next two functions are similar to 's/STR/REP/g', replacing every instance. Just like above, the first reads stdin and writes to stdout, the second actually edits FILE.

# usage: gsub_literal STR REP
gsub_literal() {
  # string manip needed to escape '\'s, so awk doesn't expand '\n' and such
  awk -v str="${1//\\/\\\\}" -v rep="${2//\\/\\\\}" '
    # get the length of the search string
    BEGIN {
      len = length(str);
    }

    {
      # empty the output string
      out = "";

      # continue looping while the search string is in the line
      while (i = index($0, str)) {
        # append everything up to the search string, and the replacement string
        out = out substr($0, 1, i-1) rep;

        # remove everything up to and including the first instance of the
        # search string from the line
        $0 = substr($0, i + len);
      }

      # append whatever is left
      out = out $0;

      print out;
    }
  '
}

# usage: gsub_literal_f STR REP FILE
gsub_literal_f() {
  local tmp
  if ! [[ -f $3 && -r $3 && -w $3 ]]; then
    printf '%s does not exist or is not readable or writable\n' "$3" >&2
    return 1
  fi

  trap 'rm -rf "$tmp"' RETURN
  tmp=$(mktemp) && cp "$3" "$tmp" || return

  # string manip needed to escape '\'s, so awk doesn't expand '\n' and such
  awk -v str="${1//\\/\\\\}" -v rep="${2//\\/\\\\}" '
    # get the length of the search string
    BEGIN {
      len = length(str);
    }

    {
      # empty the output string
      out = "";

      # continue looping while the search string is in the line
      while (i = index($0, str)) {
        # append everything up to the search string, and the replacement string
        out = out substr($0, 1, i-1) rep;

        # remove everything up to and including the first instance of the
        # search string from the line
        $0 = substr($0, i + len);
      }

      # append whatever is left
      out = out $0;

      print out;
    }
  ' "$tmp" > "$3"
}

For more information on how these work, and awk in general, visit the #awk channel on freenode.

The mktemp(1) command used in some of the examples above is not completely portable. While it will work on most systems, more information on safely creating temp files can be found in Faq #62.