Diff for "BashFAQ/110"

How can i perform a substitution with arbitrary values ("s/$foo/$bar/") safely, without treating either value as a regular expression or worrying about other special characters?

Sed is not the right tool for this. Nor is ed. At best, attempting either will result in an escaping nightmare, and will be extremely prone to bugs. (The exception to this rule is when you have sanitized input, but then you probably wouldn't be needing this faq, would you?)

First, what are we performing the substitution on? If it's a string, it can be done very simply with a parameter expansion.

var='some string'
search=some
replace=another
printf '%s\n' "${var//"$search"/$replace}"

This is discussed in more detail in Faq #100.

If it's a file or stream, things get a bit trickier. One way to accomplish this would be to combine the previous method with Faq #1.

search=foo
rep=bar

# file
while IFS= read -r line; do
  printf '%s\n' "${line//"$search"/$rep}"
done < "$file"

# command output
while IFS= read -r line; do
  printf '%s\n' "${line//"$search"/$rep}"
done < <(my_command)

my_command | while IFS= read -r line; do
  printf '%s\n' "${line//"$search"/$rep}"
done

The second of the two command examples there creates a subshell. See Faq #24 for more information on that.

Both of the above examples print to stdout; neither actually edits the file in place. Of course this could be resolved with something like:

# create a temp file, die on failure
tmp=$(mktemp) || exit

while IFS= read -r line; do
  printf '%s\n' "${line//"$search"/$rep}"
done < "$file" > "$tmp" && mv "$tmp" "$file"

On large data sets, you'll notice that this is quite slow. The following functions use awk, and are quite a bit faster:

# usage: sub_literal STR REP
# Replaces the first instance (on each line) of STR with REP, treating them as
# literal strings and not regexes. Reads stdin and writes to stdout.
# Similar to sed 's/STR/REP/'

sub_literal() {
  # string manip needed to escape '\'s, so awk doesn't expand '\n' and such
  awk -v str="${1//\\/\\\\}" -v rep="${2//\\/\\\\}" '
    # get the length of the search string
    BEGIN {
      len = length(str);
    }

    # if the search string is in the line
    (i = index($0, str)) {
      # replace the first occurance with rep
      $0 = substr($0, 1, i-1) rep substr($0, i + len);
    }

    # print each line
    1
  '
}

# usage: sub_literal_f STR REP FILE
# Replaces the first instance (on each line) of STR with REP in FILE, treating 
# them as literal strings and not regexes.
# Similar to sed -i 's/STR/REP/' FILE

sub_literal_f() {
  local tmp
  if ! [[ -f $3 && -r $3 && -w $3 ]]; then
    printf '%s does not exist or is not readable or writable\n' "$3" >&2
    return 1
  fi

  trap 'rm -rf "$tmp"' RETURN
  tmp=$(mktemp) && cp "$3" "$tmp" || return

  # string manip needed to escape '\'s, so awk doesn't expand '\n' and such
  awk -v str="${1//\\/\\\\}" -v rep="${2//\\/\\\\}" '
    # get the length of the search string
    BEGIN {
      len = length(str);
    }

    # if the search string is in the line
    (i = index($0, str)) {
      # replace the first occurance with rep
      $0 = substr($0, 1, i-1) rep substr($0, i + len);
    }

    # print each line
    1
  ' "$tmp" > "$3"
}


# usage: gsub_literal STR REP
# Replaces all instances of STR with REP, treating them as literal strings 
# and not regexes. Reads stdin and writes to stdout
# Similar to sed 's/STR/REP/g'

gsub_literal() {
  # string manip needed to escape '\'s, so awk doesn't expand '\n' and such
  awk -v str="${1//\\/\\\\}" -v rep="${2//\\/\\\\}" '
    # get the length of the search string
    BEGIN {
      len = length(str);
    }

    {
      # empty the output string
      out = "";

      # continue looping while the search string is in the line
      while (i = index($0, str)) {
        # append everything up to the search string, and the replacement string
        out = out substr($0, 1, i-1) rep;

        # remove everything up to and including the first instance of the
        # search string from the line
        $0 = substr($0, i + len);
      }

      # append whatever is left
      out = out $0;

      print out;
    }
  '
}

# usage: gsub_literal_f STR REP FILE
# Replaces all instances of STR with REP in FILE, treating them as literal 
# strings and not regexes.
# Similar to sed -i 's/STR/REP/g' FILE

gsub_literal_f() {
  local tmp
  if ! [[ -f $3 && -r $3 && -w $3 ]]; then
    printf '%s does not exist or is not readable or writable\n' "$3" >&2
    return 1
  fi

  trap 'rm -rf "$tmp"' RETURN
  tmp=$(mktemp) && cp "$3" "$tmp" || return

  # string manip needed to escape '\'s, so awk doesn't expand '\n' and such
  awk -v str="${1//\\/\\\\}" -v rep="${2//\\/\\\\}" '
    # get the length of the search string
    BEGIN {
      len = length(str);
    }

    {
      # empty the output string
      out = "";

      # continue looping while the search string is in the line
      while (i = index($0, str)) {
        # append everything up to the search string, and the replacement string
        out = out substr($0, 1, i-1) rep;

        # remove everything up to and including the first instance of the
        # search string from the line
        $0 = substr($0, i + len);
      }

      # append whatever is left
      out = out $0;

      print out;
    }
  ' "$tmp" > "$3"
}

The mktemp(1) command used in some of the examples above is not completely portable. While it will work on most systems, more information on safely creating temp files can be found in Faq #62.