Diff for "BashFAQ/021"

Differences between revisions 21 and 36 (spanning 15 versions)

How can I replace a string with another string in a file, or in all the files in a directory?

ed is the standard UNIX command-based editor. Here are some commonly-used syntaxes for replacing the string olddomain.com by the string newdomain.com in a file named file. All four commands do the same thing, with varying degrees of portability and efficiency:

# Bash
ed -s file <<< $'g/olddomain\\.com/s//newdomain.com/g\nw\nq'

# Bourne (with printf)
printf '%s\n' 'g/olddomain\.com/s//newdomain.com/g' w q | ed -s file

printf 'g/olddomain\\.com/s//newdomain.com/g\nw\nq' | ed -s file

# Bourne (without printf)
ed -s file <<!
g/olddomain\\.com/s//newdomain.com/g
w
q
!

To replace a string in all files of the current directory:

for file in ./*; do
    [[ -f $file ]] && ed -s "$file" <<< $'g/old/s//new/g\nw\nq'
done

To do this recursively, the easy way would be to enable globstar in bash 4 (shopt -s globstar, a good idea to put this in your ~/.bashrc) and use:

for file in ./**/*; do
    [[ -f $file ]] && ed -s "$file" <<< $'g/old/s//new/g\nw\nq'
done

If you don't have bash 4, you can use find. Unfortunately, it's a bit tedious to feed ed stdin for each file hit:

find . -type f -exec bash -c 'printf "%s\n" "g/old/s//new/g" w q | ed -s "$1"' _ {} \;

sed is a Stream EDitor, not a file editor. Nevertheless, people everywhere tend to abuse it for trying to edit files. It doesn't edit files. GNU sed (and some BSD seds) have a -i option that makes a copy and replaces the original file with the copy. An expensive operation, but if you enjoy unportable code, I/O overhead and bad side effects (such as destroying symlinks), this would be an option:

sed -i    's/old/new/g' ./*  # GNU
sed -i '' 's/old/new/g' ./*  # BSD
for file in ./*              # Other
do
    [ -f "$file" ] &&
        sed 's/old/new/g' "$file" > "$file~" &&
        mv "$file~" "$file"
done

Those of you who have perl 5 can accomplish the same thing using this code:

perl -pi -e 's/old/new/g' ./*

Recursively using find:

find . -type f -exec perl -pi -e 's/old/new/g' {} \;   # if your find doesn't have + yet
find . -type f -exec perl -pi -e 's/old/new/g' {} +    # if it does

If you want to delete lines instead of making substitutions:

# Deletes any line containing the perl regex foo
perl -ni -e 'print unless /foo/' ./*

To replace for example all "unsigned" with "unsigned long", if it is not "unsigned int" or "unsigned long" ...:

find . -type f -exec perl -i.bak -pne \
    's/\bunsigned\b(?!\s+(int|short|long|char))/unsigned long/g' {} \;

Finally, for those of you who shun ed and have none of the useful things above, here's a script that may be useful:

# chtext - change text in several files

# neither string may contain '|' unquoted
old='olddomain\.com'
new='newdomain\.com'

# if no files were specified on the command line, use all files:
[ $# -lt 1 ] && set -- ./*

for file
do
    [ -f "$file" ] || continue # do not process e.g. directories
    [ -r "$file" ] || continue # cannot read file - ignore it
    # Replace string, write output to temporary file. Terminate script in case of errors
    sed "s|$old|$new|g" -- "$file" > "$file"-new || exit
    # If the file has changed, overwrite original file. Otherwise remove copy
    if cmp -- "$file" "$file"-new >/dev/null 2>&1
    then rm -- "$file"-new              # file has not changed
    else mv -- "$file"-new "$file"      # file has changed: overwrite original file
    fi
done

If the code above is put into a script file (e.g. chtext), the resulting script can be used to change a text e.g. in all HTML files of the current and all subdirectories:

find . -type f -name '*.html' -exec chtext {} \;

Many optimizations are possible:

use another sed separator character than '|', e.g. ^A (ASCII 0x01)
the find command above could use + instead of \; if available
the chtext script could be replaced with an ed command

Note: set -- ./* in the code above is safe with respect to files whose names contain spaces. The expansion of ./* by set is the same as the expansion done by for, and filenames will be preserved properly as individual parameters, and not broken into words on whitespace.

A more sophisticated example of chtext is here: http://www.shelldorado.com/scripts/cmds/chtext

CategoryShell

-  ⇤ ← Revision 21 as of 2009-09-15 17:56:53 → 
  Size: 3429
  Editor: GreyCat
  Comment: ./* and -- fixes
+   ← Revision 36 as of 2011-11-04 19:17:14 → ⇥
  Size: 4635
  Editor: GreyCat
  Comment: clarify question, un-indent examples, add find -exec +
-Deletions are marked like this.
+Additions are marked like this.
 Line 2:
-== How can I replace a string with another string in all files? ==
{{{sed}}} is a good command to replace strings, e.g.
+== How can I replace a string with another string in a file, or in all the files in a directory? ==
`ed` is the standard UNIX command-based editor.  Here are some commonly-used syntaxes for replacing the string `olddomain.com` by the string `newdomain.com` in a file named `file`.  All four commands do the same thing, with varying degrees of portability and efficiency:
 Line 6:
-    sed 's/olddomain\.com/newdomain.com/g' input > output
+# Bash
ed -s file <<< $'g/olddomain\\.com/s//newdomain.com/g\nw\nq'

# Bourne (with printf)
printf '%s\n' 'g/olddomain\.com/s//newdomain.com/g' w q | ed -s file

printf 'g/olddomain\\.com/s//newdomain.com/g\nw\nq' | ed -s file

# Bourne (without printf)
ed -s file <<!
g/olddomain\\.com/s//newdomain.com/g
w
q
!
-Line 8:
+Line 21:
-'''Note that input and output must differ.'''
-Line 13:
+Line 25:
-    for i in ./*; do
        sed 's/old/new/g' "$i" > atempfile && mv atempfile "$i"
    done
+for file in ./*; do
    [[ -f $file ]] && ed -s "$file" <<< $'g/old/s//new/g\nw\nq'
done
-Line 18:
+Line 30:
-GNU sed 4.x has a special {{{-i}}} flag which makes the loop and temp file unnecessary:
+To do this recursively, the easy way would be to enable globstar in bash 4 (`shopt -s globstar`, a good idea to put this in your `~/.bashrc`) and use:
-Line 21:
+Line 33:
-      sed -i 's/old/new/g' ./*
+for file in ./**/*; do
    [[ -f $file ]] && ed -s "$file" <<< $'g/old/s//new/g\nw\nq'
done
-Line 24:
+Line 38:
-On some (but not all) BSD systems, sed has a {{{-i}}} flag as well, but it takes a mandatory argument. The above example then becomes
+If you don't have bash 4, you can use [[UsingFind|find]].  Unfortunately, it's a bit tedious to feed `ed` stdin for each file hit:
-Line 27:
+Line 41:
-      sed -i '' 's/old/new/g' ./*
+find . -type f -exec bash -c 'printf "%s\n" "g/old/s//new/g" w q | ed -s "$1"' _ {} \;
-Line 30:
+Line 44:
-which in turn does not work with GNU sed. Effectively, whenever portability matters, {{{sed -i}}} should be avoided.
+`sed` is a '''Stream EDitor''', not a '''file''' editor.  Nevertheless, people everywhere tend to abuse it for trying to edit files.  It doesn't edit files.  GNU `sed` (and some BSD `sed`s) have a `-i` option that makes a copy and replaces the original file with the copy.  An expensive operation, but if you enjoy unportable code, I/O overhead and bad side effects (such as destroying symlinks), this would be an option:

{{{
sed -i    's/old/new/g' ./*  # GNU
sed -i '' 's/old/new/g' ./*  # BSD
for file in ./*              # Other
do
    [ -f "$file" ] &&
        sed 's/old/new/g' "$file" > "$file~" &&
        mv "$file~" "$file"
done
}}}
-Line 35:
+Line 60:
-    perl -pi -e 's/old/new/g' ./*
+perl -pi -e 's/old/new/g' ./*
-Line 38:
+Line 63:
-Recursively (requires GNU or BSD {{{find}}}):
+Recursively using `find`:
-Line 41:
+Line 66:
-    find . -type f -print0 | xargs -0 perl -pi -e 's/old/new/g'
+find . -type f -exec perl -pi -e 's/old/new/g' {} \;   # if your find doesn't have + yet
find . -type f -exec perl -pi -e 's/old/new/g' {} +    # if it does
-Line 47:
+Line 73:
-    perl -ni -e 'print unless /foo/' ./*
    # Deletes any line containing the perl regex foo
+# Deletes any line containing the perl regex foo
perl -ni -e 'print unless /foo/' ./*
-Line 54:
+Line 80:
-    find . -type f -print0 | xargs -0 perl -i.bak -pne \
 's/\bunsigned\b(?!\s+(int|short|long|char))/unsigned long/g'
+find . -type f -exec perl -i.bak -pne \
    's/\bunsigned\b(?!\s+(int|short|long|char))/unsigned long/g' {} \;
-Line 58:
+Line 84:
-Finally, for those of you with ''none'' of the useful things above, here's a script that may be useful:
+Finally, for those of you who shun `ed` and have ''none'' of the useful things above, here's a script that may be useful:
-Line 61:
+Line 87:
-    #!/bin/sh
    # chtext - change text in several files
+#!/bin/sh
# chtext - change text in several files
-Line 64:
+Line 90:
-    # neither string may contain '|' unquoted
    old='olddomain\.com'
    new='newdomain\.com'
+# neither string may contain '|' unquoted
old='olddomain\.com'
new='newdomain\.com'
-Line 68:
+Line 94:
-    # if no files were specified on the command line, use all files:
    [ $# -lt 1 ] && set -- ./*
+# if no files were specified on the command line, use all files:
[ $# -lt 1 ] && set -- ./*
-Line 71:
+Line 97:
-    for file
    do
        [ -f "$file" ] || continue # do not process e.g. directories
        [ -r "$file" ] || continue # cannot read file - ignore it
        # Replace string, write output to temporary file. Terminate script in case of errors
        sed "s|$old|$new|g" -- "$file" > "$file"-new || exit
        # If the file has changed, overwrite original file. Otherwise remove copy
        if cmp -- "$file" "$file"-new >/dev/null 2>&1
        then rm -- "$file"-new              # file has not changed
        else mv -- "$file"-new "$file"      # file has changed: overwrite original file
        fi
    done
+for file
do
    [ -f "$file" ] || continue # do not process e.g. directories
    [ -r "$file" ] || continue # cannot read file - ignore it
    # Replace string, write output to temporary file. Terminate script in case of errors
    sed "s|$old|$new|g" -- "$file" > "$file"-new || exit
    # If the file has changed, overwrite original file. Otherwise remove copy
    if cmp -- "$file" "$file"-new >/dev/null 2>&1
    then rm -- "$file"-new              # file has not changed
    else mv -- "$file"-new "$file"      # file has changed: overwrite original file
    fi
done
-Line 85:
+Line 111:
-If the code above is put into a script file (e.g. {{{chtext}}}), the resulting script can be used to change a text e.g. in all HTML files of the current and all subdirectories:
+If the code above is put into a script file (e.g. `chtext`), the resulting script can be used to change a text e.g. in all HTML files of the current and all subdirectories:
-Line 88:
+Line 114:
-    find . -type f -name '*.html' -exec chtext {} \;
+find . -type f -name '*.html' -exec chtext {} \;
-Line 92:
+Line 118:
- * use another {{{sed}}} separator character than '|', e.g. ^A (ASCII 0x01)
 * the [[UsingFind|find]] command above could use either {{{xargs}}} or the built-in {{{xargs}}} of POSIX find
+ * use another `sed` separator character than '|', e.g. ^A (ASCII 0x01)
 * the [[UsingFind|find]] command above could use `+` instead of `\;` if available
 * the `chtext` script could be replaced with an `ed` command
-Line 95:
+Line 122:
-Note: {{{set -- ./*}}} in the code above is safe with respect to files whose names contain spaces.  The expansion of `./*` by {{{set}}} is the same as the expansion done by {{{for}}}, and filenames will be preserved properly as individual parameters, and not broken into words on whitespace.
+Note: `set -- ./*` in the code above is safe with respect to files whose names contain spaces.  The expansion of `./*` by `set` is the same as the expansion done by `for`, and filenames will be preserved properly as individual parameters, and not broken into words on whitespace.
-Line 97:
+Line 124:
-A more sophisticated example of {{{chtext}}} is here: http://www.shelldorado.com/scripts/cmds/chtext
+A more sophisticated example of `chtext` is here: http://www.shelldorado.com/scripts/cmds/chtext

----
CategoryShell