<> == How can I replace a string with another string in a variable, a stream, a file, or in all the files in a directory? == There are a number of techniques for this. Which one to use depends on many factors, the biggest of which is ''what we're editing''. This page also contains contradictory advice from multiple authors. This is a deeply ''ugly'' topic, and there are no universally right answers (but plenty of universally ''wrong'' ones). <> === Files === Before you start, be warned that [[http://backreference.org/2011/01/29/in-place-editing-of-files/|editing files is a really bad idea]]. The preferred way to modify a file is to create a new file within the same file system, write the modified content into it, and then `mv` it to the original name. This is the '''only''' way to prevent data loss in the event of a crash while writing. However, using a temp file and `mv` means that you break hardlinks to the file (unavoidably), that you would convert a symlink to hard file, and that you may need to take extra steps to transfer the ownership and permissions (and possibly other metadata) of the original file to the new file. Some people prefer to roll the dice and accept the tiny possibility of data loss versus the greater possibility of hardlink loss and the inconvenience of `chown`/`chmod` (and potentially `setfattr`, `setfacl`, `chattr`...). The other major problem you're going to face is that all of the standard Unix tools for editing files expect some kind of regular expression as the search pattern. If you're passing input ''you did not create'' as the search pattern, it may contain syntax that breaks the program's parser, which can lead to failures, or CodeInjection exploits. ==== Just Tell Me What To Do ==== If your search string or your replacement string comes from an external source (environment variable, argument, file, user input) and is therefore not under your control, then this is your best choice: {{{ in="$search" out="$replace" perl -pi -e 's/\Q$ENV{"in"}/$ENV{"out"}/g' ./* }}} That will operate on all of the files in the current directory. If you want to operate on a full hierarchy (recursively), then: {{{ in="$search" out="$replace" find . -type f -exec \ perl -pi -e 's/\Q$ENV{"in"}/$ENV{"out"}/g' -- {} + }}} You may of course supply additional options to `find` to restrict which files are replaced; see UsingFind for more information. The critical reader may note that these commands use `perl` which is not a standard tool. That's because none of the standard tools can do this task safely. If you're stuck using standard tools due to a restricted execution environment, then you'll have to weigh the options below and choose the one that will do the least amount of damage to your files. ==== Using a file editor ==== The only standard tools that actually edit a file are `ed` and `ex` (`vi` is the visual mode for `ex`). `ed` is the standard UNIX command-based editor. `ex` is another standard command-line editor. Here are some commonly-used syntaxes for replacing the string `olddomain.com` by the string `newdomain.com` in a file named `file`. All four commands do the same thing, with varying degrees of portability and efficiency: {{{ ## Ex ex -sc '%s/olddomain\.com/newdomain.com/g|x' file ## Ed # Bash ed -s file <<< $'g/olddomain\\.com/s//newdomain.com/g\nw\nq' # Bourne (with printf) printf '%s\n' 'g/olddomain\.com/s//newdomain.com/g' w q | ed -s file printf 'g/olddomain\\.com/s//newdomain.com/g\nw\nq' | ed -s file # Bourne (without printf) ed -s file < tmp && mv -- tmp "$file" }}} {{{ # Using GNU tools to preseve ownership/group/permissions gsub_literal "$search" "$rep" < "$file" > tmp && chown --reference="$file" tmp && chmod --reference="$file" tmp && mv -- tmp "$file" }}} ==== Using nonstandard tools ==== `sed` is a '''Stream EDitor''', not a '''file''' editor. Nevertheless, people everywhere tend to abuse it for trying to edit files. It doesn't edit files. GNU `sed` (and some BSD `sed`s) have a `-i` option that makes a copy and replaces the original file with the copy. An expensive operation, but if you enjoy unportable code, I/O overhead and bad side effects (such as destroying symlinks), and CodeInjection exploits, this would be an option: {{{ sed -i 's/old/new/g' ./* # GNU, OpenBSD sed -i '' 's/old/new/g' ./* # FreeBSD }}} Those of you who have perl 5 can accomplish the same thing using this code: {{{ perl -pi -e 's/old/new/g' ./* }}} Recursively using `find`: {{{ find . -type f -exec perl -pi -e 's/old/new/g' -- {} \; # if your find doesn't have + yet find . -type f -exec perl -pi -e 's/old/new/g' -- {} + # if it does }}} If you want to delete lines instead of making substitutions: {{{ # Deletes any line containing the perl regex foo perl -ni -e 'print unless /foo/' ./* }}} To replace for example all "unsigned" with "unsigned long", if it is not "unsigned int" or "unsigned long" ...: {{{ find . -type f -exec perl -i.bak -pne \ 's/\bunsigned\b(?!\s+(int|short|long|char))/unsigned long/g' -- {} \; }}} All of the examples above use regular expressions, which means they have the same issue as the sed code earlier; trying to embed shell variables in them is a terrible idea, and treating an arbitrary value as a literal string is painful at best. If the inputs are not under your direct control, you can pass them as variables into both search and replace strings with no unquoting or potential for conflict with sigil characters: {{{ in="$search" out="$replace" perl -pi -e 's/\Q$ENV{"in"}/$ENV{"out"}/g' ./* }}} Or, wrapped in a useful shell function: {{{ # Bash # usage: replace FROM TO [file ...] replace() { in=$1 out=$2 perl -p ${3+'-i'} -e 's/\Q$ENV{"in"}/$ENV{"out"}/g' -- "${@:3}" } }}} This wrapper passes perl's `-i` option if there are any filenames, so that they are "edited in-place" (or at least as far as perl does such a thing -- see the perl documentation for details). === Variables === If you want to replace content within a variable, this can (and should) be done very simply with Bash's [[BashFAQ/100|parameter expansion]]: {{{ # Bash var='some string' var=${var//some/another} }}} However, if the replacement string is in a variable, one must be cautious. There are inconsistent behaviors across different versions of bash. {{{ # Bash var='some string' search=some; rep=another # Assignments work consistently. Note the quotes. var=${var//"$search"/"$rep"} # Expansions outside of assigments are not consistent. echo "${var//"$search"/"$rep"}" # Works in bash 4.3 and later. echo "${var//"$search"/$rep}" # Works in bash 5.1 and earlier. }}} The quotes around `"$search"` prevent the contents of the variable from being treated as a shell pattern (also called a [[glob]]). Of course, if pattern matching is intended, do not include the quotes. In Bash 4.2 and earlier, if you quote `$rep` in `${var//"$search"/"$rep"}` the quotes will be inserted literally. In Bash 5.2, you ''must'' either quote `$rep` in `${var//"$search"/"$rep"}` or disable ''patsub_replacement'' (`shopt -u patsub_replacement`) because otherwise `&` characters in `$rep` will be substituted by `$search`. The only way to get a consistent and correct result across all versions of bash is to use a temporary variable: {{{ # Bash tmp=${var//"$search"/"$rep"} echo "$tmp" }}} For compatibility with Bash 4.2 and earlier, make sure you ''do not'' put quotes around the assignment's right hand side. {{{ # In bash 4.2, this fails. You get literal quotes in the result. tmp="${var//"$search"/"$rep"}" }}} Replacements within a variable are even harder in POSIX sh: {{{ # POSIX function # usage: string_rep SEARCH REPL STRING # replaces all instances of SEARCH with REPL in STRING string_rep() { # initialize vars in=$3 unset -v out # SEARCH must not be empty case $1 in '') return; esac while # break loop if SEARCH is no longer in "$in" case "$in" in *"$1"*) ;; *) break;; esac do # append everything in "$in", up to the first instance of SEARCH, and REP, to "$out" out=$out${in%%"$1"*}$2 # remove everything up to and including the first instance of SEARCH from "$in" in=${in#*"$1"} done # append whatever is left in "$in" after the last instance of SEARCH to out, and print printf '%s%s\n' "$out" "$in" } var=$(string_rep "$search" "$rep" "$var") # Note: POSIX does not have a way to localize variables. Most shells (even dash and # busybox), however, do. Feel free to localize the variables if your shell supports # it. Even if it does not, if you call the function with var=$(string_rep ...), the # function will be run in a subshell and any assignments it makes will not persist. }}} === Streams === If you wish to modify a stream, and if your search and replace strings are known in advance, then use the '''s'''tream '''ed'''itor: {{{ some_command | sed 's/foo/bar/g' }}} `sed` uses [[RegularExpression|regular expressions]]. In our example, `foo` and `bar` are literal strings. If they were variables (e.g. user input), they would have to be rigorously escaped in order to prevent errors. This is very impractical, and attempting to do so will make your code extremely prone to bugs. Embedding shell variables in sed commands is '''never''' a good idea -- it is a prime source of CodeInjection bugs. You could also do it in Bash itself, by combining a parameter expansion with [[BashFAQ/001|Faq #1]]: {{{ search=foo rep=bar while IFS= read -r line; do printf '%s\n' "${line//"$search"/"$rep"}" done < <(some_command) # or some_command | while IFS= read -r line; do printf '%s\n' "${line//"$search"/"$rep"}" done }}} If you want to do more processing than just a simple search/replace, this may be the best option. Note that the last example runs the loop in a SubShell. See [[BashFAQ/024|Faq #24]] for more information on that. You may notice, however, that the bash loop above is very slow for large data sets. So how do we find something faster, that can replace literal strings? Well, you could use `awk`. The following function replaces all instances of STR with REP, reading from stdin and writing to stdout. {{{ # usage: gsub_literal STR REP # replaces all instances of STR with REP. reads from stdin and writes to stdout. gsub_literal() { # STR cannot be empty [[ $1 ]] || return str=$1 rep=$2 awk ' # get the length of the search string BEGIN { str = ENVIRON["str"] rep = ENVIRON["rep"] len = length(str); } { # empty the output string out = ""; # continue looping while the search string is in the line while (i = index($0, str)) { # append everything up to the search string, and the replacement string out = out substr($0, 1, i-1) rep; # remove everything up to and including the first instance of the # search string from the line $0 = substr($0, i + len); } # append whatever is left out = out $0; print out; } ' } some_command | gsub_literal "$search" "$rep" # condensed as a one-liner: some_command | s=$search r=$rep awk 'BEGIN {s=ENVIRON["s"]; r=ENVIRON["r"]; l=length(s)} {o=""; while (i=index($0, s)) {o=o substr($0,1,i-1) r; $0=substr($0,i+l)} print o $0}' }}} ---- CategoryShell