Differences between revisions 51 and 52
Revision 51 as of 2012-07-18 12:29:30
Size: 11531
Editor: GreyCat
Comment: It could use some trimming, but it's not all noise.
Revision 52 as of 2012-07-18 12:36:38
Size: 1878
Editor: Lhunath
Comment: How about a link to ReplacingStrings?
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
== How can I replace a string with another string in a variable, a stream, a file, or in all the files in a directory? ==
There are a number of tools available for this. Which one to use depends on a lot of factors, the biggest of which is of course ''what we're editing''.
== How can I replace a string with another string in a file? ==
Line 6: Line 5:
<<TableOfContents>> We're trying to edit files, which means we need a file editor. Most systems come with a range of editors, but POSIX defines only two that we can use: `ed` and `ex`.
Line 8: Line 7:
=== Files === '''Warning''': A lot of bad advice on the Internet will recommend `sed -i` or `perl -i` for this. While on the surface it may look like these tools modify your files, what they ''really'' do is recreate your file and delete the original. This has many downsides and is often very risky. Open handles are destroyed, certain metadata will get lost, a lot of excess disk I/O is involved, and depending on the implementation, interrupted operations may cause your file to get lost. Just use a file editor.
Line 10: Line 9:
Actually editing files gets even trickier. The only tool listed that actually edits a file is `ed`. The other methods could be used, but to do so involves a temp file and `mv` (or POSIX extensions). For pure-bash or POSIX-sh solutions or to replace strings in variables or streams, see ReplacingStrings.
Line 12: Line 11:
`ed` is the standard UNIX command-based editor. Here are some commonly-used syntaxes for replacing the string `olddomain.com` by the string `newdomain.com` in a file named `file`. All four commands do the same thing, with varying degrees of portability and efficiency: `ed` is the standard UNIX command-based editor. `ex` is another POSIX command-based editor with syntax slightly similar to that of `sed`. Here are some commonly-used syntaxes for replacing the string `old` by the string `new` in a file named `file`.
Line 15: Line 14:
# Bash
ed -s file <<< $'g/olddomain\\.com/s//newdomain.com/g\nw\nq'

# Bourne (with printf)
printf '%s\n' 'g/olddomain\.com/s//newdomain.com/g' w q | ed -s file

printf 'g/olddomain\\.com/s//newdomain.com/g\nw\nq' | ed -s file

# Bourne (without printf)
ed -s file <<!
g/olddomain\\.com/s//newdomain.com/g
w
q
!
ex -sc '%s/old/new/ge|x' file
ed -s file <<< $'g/old/s//new/g\nw\nq'
Line 31: Line 18:
To replace a string in all files of the current directory: If you need to edit multiple files, `ex` can easily be combined with `find`. `ed` is hard to call from `find` since it takes commands as input.
Line 34: Line 21:
for file in ./*; do
    [[ -f $file ]] && ed -s "$file" <<< $'g/old/s//new/g\nw\nq'
done
}}}

To do this recursively, the easy way would be to enable globstar in bash 4 (`shopt -s globstar`, a good idea to put this in your `~/.bashrc`) and use:

{{{
for file in ./**/*; do
    [[ -f $file ]] && ed -s "$file" <<< $'g/old/s//new/g\nw\nq'
done
}}}

If you don't have bash 4, you can use [[UsingFind|find]]. Unfortunately, it's a bit tedious to feed `ed` stdin for each file hit:

{{{
find . -type f -exec ex -sc '%s/old/new/ge|x' {} \;
Line 53: Line 25:
`sed` is a '''Stream EDitor''', not a '''file''' editor. Nevertheless, people everywhere tend to abuse it for trying to edit files. It doesn't edit files. GNU `sed` (and some BSD `sed`s) have a `-i` option that makes a copy and replaces the original file with the copy. An expensive operation, but if you enjoy unportable code, I/O overhead and bad side effects (such as destroying symlinks), this would be an option: You may notice that this causes an `ex`/`ed` process per file. If your `ex` is provided by `vim` (NOT POSIX), you can optimize that into:
Line 56: Line 28:
sed -i 's/old/new/g' ./* # GNU
sed -i '' 's/old/new/g' ./* # BSD

# POSIX sed, uses a temp file and mv:

# remove all temp files on exit, in case sed fails and they weren't moved
trap 'rm -f "${temps[@]}"' EXIT

temps=()
for file in ./*; do
  if [[ -f $file ]]; then
    tmp=$(mktemp) || exit
    temps+=("$tmp")

    sed 's/old/new/g' "$file" > "$tmp" &&
    mv "$tmp" "$file"
  fi
done
find . -type f -exec ex -sc 'argdo %s/old/new/ge|x' {} +
Line 76: Line 31:
Those of you who have perl 5 can accomplish the same thing using this code:

{{{
perl -pi -e 's/old/new/g' ./*
}}}

Moreover, perl can be used to pass variables into both search and replace strings with no unquoting or potential for conflict with sigil characters:

{{{
in="input (/string" out="output string" perl -pi -e $'$quoted_in=quotemeta($ENV{\'in\'}); s/$quoted_in/$ENV{\'out\'}/g' ./*
}}}

Recursively using `find`:

{{{
find . -type f -exec perl -pi -e 's/old/new/g' {} \; # if your find doesn't have + yet
find . -type f -exec perl -pi -e 's/old/new/g' {} + # if it does
}}}

If you want to delete lines instead of making substitutions:

{{{
# Deletes any line containing the perl regex foo
perl -ni -e 'print unless /foo/' ./*
}}}

To replace for example all "unsigned" with "unsigned long", if it is not "unsigned int" or "unsigned long" ...:

{{{
find . -type f -exec perl -i.bak -pne \
    's/\bunsigned\b(?!\s+(int|short|long|char))/unsigned long/g' {} \;
}}}

----

All of the tools listed above use regular expressions, which means they have the same issue as the sed code earlier; trying to embed shell variables in them is a terrible idea, and treating an arbitrary value as a literal string is painful at best. This brings us back to our while read loop, or the awk function above.

The while read loop:

{{{
# overwrite a single file
tmp=$(mktemp) || exit
trap 'rm -f "$tmp"' EXIT

while IFS= read -r line; do
  printf '%s\n' "${line//"$search"/$rep}"
done < "$file" > "$tmp" && mv "$tmp" "$file"
}}}

Replaces all files in a directory:

{{{
trap 'rm -f "${temps[@]}"' EXIT

temps=()
for f in ./*; do
  if [[ -f $f ]]; then
    tmp=$(mktemp) || exit
    temps+=("$tmp")

    while IFS= read -r line; do
      printf '%s\n' "${line//"$search"/$rep}"
    done < "$f" > "$tmp" &&
    mv "$tmp" "$f"
  fi
done
}}}

The above glob could be changed to './**/*' in order to use globstar (mentioned above) to be recursive, or of course we could use `find`:

{{{
# this example uses GNU find's -print0. Using POSIX find -exec is left as an exercise to the reader
trap 'rm -f "${temps[@]}"' EXIT

temps=()
while IFS= read -rd '' f <&3; do
  tmp=$(mktemp) || exit
  temps+=("$tmp")

  while IFS= read -r line; do
    printf '%s\n' "${line//"$search"/$rep}"
  done < "$f" > "$tmp" &&
  mv "$tmp" "$f"
done 3< <(find . -type f -print0)
}}}

And of course, we can adapt the `AWK` function above. The following function replaces all instances of STR with REP in FILE, actually overwriting FILE:

{{{
# usage: gsub_literal_f STR REP FILE
# replaces all instances of STR with REP in FILE
gsub_literal_f() {
  local tmp
  # make sure FILE exists, is a regular file, and is readable and writable
  if ! [[ -f $3 && -r $3 && -w $3 ]]; then
    printf '%s does not exist or is not readable or writable\n' "$3" >&2
    return 1
  fi

  # STR cannot be empty
  [[ $1 ]] || return

  tmp=$(mktemp) || return
  trap 'rm -f "$tmp"' RETURN

  # string manip needed to escape '\'s, so awk doesn't expand '\n' and such
  awk -v str="${1//\\/\\\\}" -v rep="${2//\\/\\\\}" '
    # get the length of the search string
    BEGIN {
      len = length(str);
    }

    {
      # empty the output string
      out = "";

      # continue looping while the search string is in the line
      while (i = index($0, str)) {
        # append everything up to the search string, and the replacement string
        out = out substr($0, 1, i-1) rep;

        # remove everything up to and including the first instance of the
        # search string from the line
        $0 = substr($0, i + len);
      }

      # append whatever is left
      out = out $0;

      print out;
    }
  ' "$3" > "$tmp" && mv "$tmp" "$3"
}
}}}

This function, of course, could be called on all of the files in a dir, or recursively.

----

'''Notes:'''

For more information on `sed` or `awk`, you can visit the '''##sed''' and '''#awk''' channels on freenode, respectively.

''mktemp(1)'', used in many of the examples above, is not completely portable. While it will work on most systems, more information on safely creating temp files can be found in [[BashFAQ|Faq #62]].



=== Variables ===

If it's a variable, this can (and should) be done very simply with parameter expansion. Forking an external tool for string manipulation is extremely slow and unnecessary.

{{{
var='some string'; search=some; rep=another

# Bash
var=${var//"$search"/$rep}


# POSIX function

# usage: string_rep SEARCH REPL STRING
# replaces all instances of SEARCH with REPL in STRING
string_rep() {
  # initialize vars
  in=$3
  unset out

  # SEARCH must not be empty
  [[ $1 ]] || return

  while true; do
    # break loop if SEARCH is no longer in "$in"
    case "$in" in
      *"$1"*) : ;;
      *) break;;
    esac

    # append everything in "$in", up to the first instance of SEARCH, and REP, to "$out"
    out=$out${in%%"$1"*}$2
    # remove everything up to and including the first instance of SEARCH from "$in"
    in=${in#*"$1"}
  done

  # append whatever is left in "$in" after the last instance of SEARCH to out, and print
  printf '%s%s\n' "$out" "$in"
}

var=$(string_rep "$search" "$rep" "$var")

# Note: POSIX does not have a way to localize variables. Most shells (even dash and
# busybox), however, do. Feel free to localize the variables if your shell supports
# it. Even if it does not, if you call the function with var=$(string_rep ...), the
# function will be run in a subshell and any assignments it makes will not persist.
}}}

In the bash example, the quotes around "$search" prevent the contents of the variable to be treated as a shell pattern (also called a "glob"). Of course, if pattern matching is intended, do not include the quotes. If "$rep" were quoted, however, the quotes would be treated as literal.

Parameter expansions like this are discussed in more detail in [[BashFAQ/100|Faq #100]].

=== Streams ===

If it's a file or a stream, things get a little bit trickier. The standard tools available for this are `sed` or `AWK` (for streams), and `ed` (for files).

Of course, you could do it in bash itself, by combining the previous method with [[BashFAQ/001|Faq #1]]:
{{{
search=foo; rep=bar

while IFS= read -r line; do
  printf '%s\n' "${line//"$search"/$rep}"
done < <(some_command)

some_command | while IFS= read -r line; do
  printf '%s\n' "${line//"$search"/$rep}"
done
}}}

If you want to do more processing than just a simple search/replace, this may be the best option. Note that the last example runs the loop in a subshell. See [[BashFAQ/024|Faq #24]] for more information on that.

Another option would, of course, be `sed`:

{{{
# replaces all instances of "search" with "replace" in the output of "some_command"
some_command | sed 's/search/replace/g'
}}}

`sed` uses [[RegularExpression|regular expressions]]. Unlike the bash, "search" and "replace" would have to be rigorously escaped in order to treat the values as literal strings. This is very impractical, and attempting to do so will make your code extremely prone to bugs. Embedding shell variables in sed is '''never''' a good idea.

You may notice, however, that the bash loop above is very slow for large data sets. So how do we find something faster, that can replace literal strings? Well, you could use `AWK`. The following function replaces all instances of STR with REP, reading from stdin and writing to stdout.

{{{
# usage: gsub_literal STR REP
# replaces all instances of STR with REP. reads from stdin and writes to stdout.
gsub_literal() {
  # STR cannot be empty
  [[ $1 ]] || return

  # string manip needed to escape '\'s, so awk doesn't expand '\n' and such
  awk -v str="${1//\\/\\\\}" -v rep="${2//\\/\\\\}" '
    # get the length of the search string
    BEGIN {
      len = length(str);
    }

    {
      # empty the output string
      out = "";

      # continue looping while the search string is in the line
      while (i = index($0, str)) {
        # append everything up to the search string, and the replacement string
        out = out substr($0, 1, i-1) rep;

        # remove everything up to and including the first instance of the
        # search string from the line
        $0 = substr($0, i + len);
      }

      # append whatever is left
      out = out $0;

      print out;
    }
  '
}

some_command | gsub_literal "$search" "$rep"


# condensed as a one-liner:
some_command | awk -v s="${search//\\/\\\\}" -v r="${rep//\\/\\\\}" 'BEGIN {l=length(s)} {o="";while (i=index($0, s)) {o=o substr($0,1,i-1) r; $0=substr($0,i+l)} print o $0}'
}}}
This will run a single `ex` for as many files as fit on the command line. You can't do this with `ed`.

How can I replace a string with another string in a file?

We're trying to edit files, which means we need a file editor. Most systems come with a range of editors, but POSIX defines only two that we can use: ed and ex.

Warning: A lot of bad advice on the Internet will recommend sed -i or perl -i for this. While on the surface it may look like these tools modify your files, what they really do is recreate your file and delete the original. This has many downsides and is often very risky. Open handles are destroyed, certain metadata will get lost, a lot of excess disk I/O is involved, and depending on the implementation, interrupted operations may cause your file to get lost. Just use a file editor.

For pure-bash or POSIX-sh solutions or to replace strings in variables or streams, see ReplacingStrings.

ed is the standard UNIX command-based editor. ex is another POSIX command-based editor with syntax slightly similar to that of sed. Here are some commonly-used syntaxes for replacing the string old by the string new in a file named file.

ex -sc '%s/old/new/ge|x' file
ed -s file <<< $'g/old/s//new/g\nw\nq'

If you need to edit multiple files, ex can easily be combined with find. ed is hard to call from find since it takes commands as input.

find . -type f -exec ex -sc '%s/old/new/ge|x' {} \;
find . -type f -exec bash -c 'printf "%s\n" "g/old/s//new/g" w q | ed -s "$1"' _ {} \;

You may notice that this causes an ex/ed process per file. If your ex is provided by vim (NOT POSIX), you can optimize that into:

find . -type f -exec ex -sc 'argdo %s/old/new/ge|x' {} +

This will run a single ex for as many files as fit on the command line. You can't do this with ed.


CategoryShell

BashFAQ/021 (last edited 2022-11-03 23:42:27 by GreyCat)