|
Size: 101455
Comment: add newbie example to #55
|
Size: 2367
Comment: -> BashProgramming
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| #acl GreyCat:read,write,revert,admin,delete pgas:read,write,revert All:read | |
| Line 5: | Line 6: |
| These are answers to frequently asked questions on channel #bash on the [http://www.freenode.net/ freenode] IRC network. These answers are contributed by the regular members of the channel (originally heiner, and then others including greycat and r00t), and by users like you. If you find something inaccurate or simply misspelled, please feel free to correct it! | || '''Note''': The FAQ was split into individual pages for easier editing. Also, for faster loading of this page, the answers are no longer presented here in their entirety.<<BR>><<BR>>Readers, click the `[BashFAQ/nnn]` link at the bottom of each answer to read the rest of the answer.<<BR>><<BR>>Editors, click the '[edit]' link at the bottom of each entry. Don't add new ones to this page; create a new subpage with the next available question number instead.<<BR>><<BR>>Thank you.|| These are answers to frequently asked questions on channel #bash on the [[http://www.freenode.net/|freenode]] IRC network. These answers are contributed by the regular members of the channel (originally heiner, and then others including greycat and r00t), and by users like you. If you find something inaccurate or simply misspelled, please feel free to correct it! |
| Line 9: | Line 12: |
| ["BASH"] is a BourneShell compatible shell, which adds many new features to its ancestor. Most of them are available in the KornShell, too. If a question is not strictly shell specific, but rather related to Unix, it may be in the UnixFaq. | [[BASH]] is a BourneShell compatible shell, which adds many new features to its ancestor. Most of them are available in the KornShell, too. The answers given in this FAQ may be slanted toward Bash, or they may be slanted toward the lowest common denominator Bourne shell, depending on who wrote the answer. In most cases, an effort is made to provide both a portable (Bourne) and an efficient (Bash, where appropriate) answer. If a question is not strictly shell specific, but rather related to Unix, it may be in the UnixFaq. This FAQ assumes a certain level of familiarity with basic shell script syntax. If you're completely new to Bash or to the Bourne family of shells, you may wish to start with the BashGuide. More advanced users may wish to read BashPitfalls and BashProgramming. |
| Line 13: | Line 20: |
| [[TableOfContents]] | Chet Ramey's official [[http://tiswww.case.edu/php/chet/bash/FAQ|Bash FAQ]] contains many technical questions not covered here. |
| Line 15: | Line 22: |
| [[Anchor(faq1)]] == How can I read a file line-by-line? == {{{ while read line do echo "$line" done < "$file" }}} |
<<TableOfContents>> |
| Line 24: | Line 24: |
| If you want to operate on individual fields within each line, you may supply additional variables to {{{read}}}: | <<Include(^BashFAQ/[0-9]*$, , to="^$", editlink)>> |
| Line 26: | Line 26: |
| {{{ # Input file has 3 columns separated by white space. while read first_name last_name phone; do ... done < "$file" }}} If the field delimiters are not whitespace, you can set {{{IFS}}} (input field separator): {{{ while IFS=: read user pass uid gid gecos home shell; do ... done < /etc/passwd }}} Also, please note that you do ''not'' necessarily need to know how many fields each line of input contains. If you supply more variables than there are fields, the extra variables will be empty. If you supply fewer, the last variable gets "all the rest" of the fields after the preceding ones are satisfied. For example, {{{ while read first_name last_name junk; do ... done <<< 'Bob Smith 123 Main Street Elk Grove Iowa 123-555-6789' # Inside the loop, first_name will contain "Bob", and # last_name will contain "Smith". The variable "junk" holds # everything else. }}} The {{{read}}} command modifies each line read, e.g. it removes all leading whitespace characters (blanks, tab characters). If that is not desired, the {{{IFS}}} variable has to be cleared: {{{ OIFS=$IFS; IFS= while read line do echo "$line" done < "$file" IFS=$OIFS }}} As a feature, the {{{read}}} command concatenates lines that end with a backslash '\' character to one single line. To disable this feature, KornShell and ["BASH"] have {{{read -r}}}: {{{ OIFS=$IFS; IFS= while read -r line do echo "$line" done < "$file" IFS=$OIFS }}} Note that reading a file line by line this way is ''very slow'' for large files. Consider using e.g. ["AWK"] instead if you get performance problems. One may also read from a command instead of a regular file: {{{ some command | while read line; do other commands done }}} That may cause problems later on if the commands inside the body of the loop attempt to set variables which need to be used outside the loop; in that case, see [#faq24 FAQ 24], or use process substitution like: {{{ while read line; do other commands done < <(some command) }}} Sometimes it's useful to read a file into an array, one array element per line. You can do that with the following example: {{{ O=$IFS IFS=$'\n' arr=($(< myfile)) IFS=$O }}} This temporarily changes the Input Field Separator to a newline, so that each line will be considered one field by read. Then it populates the array {{{arr}}} with the fields. Then it sets the {{{IFS}}} back to what it was before. [[Anchor(faq2)]] == How can I store the return value of a command in a variable? == Well, that depends on exactly what you mean by that question. Some people want to store the command's ''output'' (either stdout, or stdout + stderr); and others want to store the command's ''exit status'' (0 to 255, with 0 typically meaning "success"). If you want to capture the output: {{{ var=$(command) # stdout only; stderr remains uncaptured var=$(command 2>&1) # both stdout and stderr will be captured }}} If you want the exit status: {{{ command var=$? }}} If you want both: {{{ var1=$(command) var2=$? # the assignment to var1 has no effect on command's exit status, which is still in $? }}} If you don't ''actually'' want the exit status, but simply want to take an action upon success or failure: {{{ if command then echo "it succeeded" else echo "it failed" fi }}} [[Anchor(faq3)]] == How can I insert a blank character after each character? == {{{ sed 's/./& /g' }}} Example: {{{ $ echo "testing" | sed 's/./& /g' t e s t i n g }}} [[Anchor(faq4)]] == How can I check whether a directory is empty or not? == We can test for the exit status of ls: {{{ if ls "$directory"/file.txt; then echo "file.txt found!" else echo "file.txt not found." fi }}} The following idea counts the number of entries in the specified directory (omitting ".." and "."): {{{ find "$dir" -maxdepth 0 -links 2 \ -exec echo "empty directory: {}" \; }}} Conversely, to find a non-empty directory: {{{ find "$dir" -maxdepth 0 -links +2 \ -exec echo "directory is non-empty" \; }}} Most modern systems have an "ls -A" which explicitly omits "." and ".." from the directory listing: {{{ if [ -n "$(ls -A somedir)" ] then echo directory is non-empty fi }}} This can be shortened to: {{{ if [ "$(ls -A somedir)" ] then echo directory is non-empty fi }}} Another way, using Bash features, involves setting the special shell option which changes the behavior of globbing. Some people prefer to avoid this approach, because it's so drastically different and could severely alter the behavior of scripts. Nevertheless, if you're willing to use this approach, it does greatly simplify this particular task: {{{ shopt -s nullglob if [[ -z $(echo *) ]]; then echo directory is empty fi }}} It also simplifies various other operations: {{{ shopt -s nullglob for i in *.zip; do blah blah "$i" # No need to check $i is a file. done }}} Without the {{{shopt}}}, that would have to be: {{{ for i in *.zip; do [[ -f $i ]] || continue # If no .zip files, i becomes *.zip blah blah "$i" done }}} (You may want to use the latter anyway, if there's a possibility that the glob may match directories in addition to files.) [[Anchor(faq5)]] == How can I convert all upper-case file names to lower case? == {{{ # tolower - convert file names to lower case for file in * do [ -f "$file" ] || continue # ignore non-existing names newname=$(echo "$file" | tr '[A-Z]' '[a-z]') # lower-case version of file name [ "$file" = "$newname" ] && continue # nothing to do [ -f "$newname" ] && continue # do not overwrite existing files mv "$file" "$newname" done }}} Purists will insist on using {{{ tr '[[:upper:]]' '[[:lower:]]' }}} in the above code, in case of non-ASCII (e.g. accented) letters in locales which have them. This technique can also be used to replace all unwanted characters in a file name e.g. with '_' (underscore). The script is the same as above, only the "newname=..." line has changed. {{{ # renamefiles - rename files whose name contain unusual characters for file in * do [ -f "$file" ] || continue # ignore non-existing names newname=$(echo "$file" | sed 's/[^a-zA-Z0-9_.]/_/g') [ "$file" = "$newname" ] && continue # nothing to do [ -f "$newname" ] && continue # do not overwrite existing files mv "$file" "$newname" done }}} The character class in {{{[]}}} contains all allowed characters; modify it as needed. If you have the utility "mmv" on your machine, you could simply do {{{ mmv "*" "#l1" }}} [[Anchor(faq6)]] == How can I use a logical AND in a shell pattern (glob)? == That can be achieved through the !() extglob operator. You'll need {{{extglob}}} set. It can be checked with: {{{ $ shopt extglob }}} and set with: {{{ $ shopt -s extglob }}} To warm up, we'll move all files starting with foo AND not ending with .d to directory foo_thursday.d: {{{ $ mv foo!(*.d) foo_thursday.d }}} For the general case: Delete all files containing Pink_Floyd AND not containing The_Final_Cut: {{{ $ rm !(!(*Pink_Floyd*)|*The_Final_Cut*) }}} By the way: these kind of patterns can be used with KornShell and KornShell93, too. They don't have to be enabled there, but are the default patterns. [[Anchor(faq7)]] == Is there a function to return the length of a string? == The fastest way, not requiring external programs (but usable only with ["BASH"] and KornShell): {{{ ${#varname} }}} or {{{ expr "$varname" : '.*' }}} ({{{expr}}} prints the number of characters matching the pattern {{{.*}}}, which is the length of the string) or {{{ expr length "$varname" }}} (for a BSD/GNU version of {{{expr}}}. Do not use this, because it is not ["POSIX"]). [[Anchor(faq8)]] == How can I recursively search all files for a string? == On most recent systems (GNU/Linux/BSD), you would use {{{grep -r pattern .}}} to search all files from the current directory (.) downward. You can use {{{find}}} if your {{{grep}}} lacks -r: {{{ find . -type f -exec grep -l "$search" '{}' \; }}} The {} characters will be replaced with the current file name. This command is slower than it needs to be, because {{{find}}} will call {{{grep}}} with only one file name, resulting in many {{{grep}}} invocations (one per file). Since {{{grep}}} accepts multiple file names on the command line, {{{find}}} can be instrumented to call it with several file names at once: {{{ find . -type f -exec grep -l "$search" '{}' \+ }}} The trailing '+' character instructs {{{find}}} to call {{{grep}}} with as many file names as possible, saving processes and resulting in faster execution. This example works for POSIX {{{find}}}, e.g. with Solaris. GNU find uses a helper program called {{{xargs}}} for the same purpose: {{{ find . -type f -print0 | xargs -0 grep -l "$search" }}} The {{{-print0}}} / {{{-0}}} options ensure that any file name can be processed, even ones containing blanks, TAB characters, or new-lines. 90% of the time, all you need is: Have grep recurse and print the lines (GNU grep): {{{ grep -r "$search" . }}} Have grep recurse and print only the names (GNU grep): {{{ grep -r -l "$search" . }}} The {{{find}}} command can be used to run arbitrary commands on every file in a directory (including sub-directories). Replace {{{grep}}} with the command of your choice. The curly braces {} will be replaced with the current file name in the case above. (Note that they must be escaped in some shells, but not in ["BASH"].) [[Anchor(faq9)]] == My command line produces no output: tail -f logfile | grep 'ssh' == Most standard Unix commands buffer their output if used non-interactively. This means, that they don't write each character (or even each line) as they are ready, but collect a larger number (e.g. 4 kilobytes) before printing it. In the case above, the {{{tail}}} command buffers its output, and therefore {{{grep}}} only gets its input in e.g. 4K blocks. Unfortunately there's no easy solution to this, because the behaviour of the standard programs would need to be changed. *See bottom of section before taking 'no easy solution' to heart* Some programs provide special command line options for this purpose, e.g. ||grep (e.g. GNU version 2.5.1)||{{{--line-buffered}}}|| ||sed (e.g. GNU version 4.0.6)||{{{-u,--unbuffered}}}|| ||awk (some GNU versions)||{{{-W interactive, or use the fflush() function}}}|| ||tcpdump, tethereal||{{{-l}}}|| The {{{expect}}} package (http://expect.nist.gov/) has an {{{unbuffer}}} example program, which can help here. It disables buffering for the output of a program. Example usage: {{{ unbuffer tail -f logfile | grep 'ssh' }}} There is another option when you have more control over the creation of the log file. If you would like to {{{grep}}} the real-time log of a text interface program which does buffered session logging by default (or you were using {{{script}}} to make a session log), then try this instead: {{{ $ program | tee -a program.log In another window: $ tail -f program.log | grep whatever }}} Apparently this works because {{{tee}}} produces unbuffered output. This has only been tested on GNU {{{tee}}}, YMMV. A solution to this is to use the 'less' command in follow mode. This is simple to do! {{{ $ less program.log }}} Then enter your search pattern (/ is search in less, like vi) /ssh Next, put less into follow mode by issuing shift+f Thats all there is to it! [[Anchor(faq10)]] == How can I recreate a directory structure, without the files? == With the {{{cpio}}} program: {{{ cd "$srcdir" find . -type d -print | cpio -pdumv "$dstdir" }}} or with GNU-{{{tar}}}, and less obscure syntax: {{{ cd "$srcdir" find . -type d -print | tar c --files-from - --no-recursion | tar x --directory "$dstdir" }}} This creates a list of directory names with find, non-recursively adds just the directories to an archive, and pipes it to a second tar instance to extract it at the target location. [[Anchor(faq11)]] == How can I print the n'th line of a file? == The dirty (but not quick) way would be {{{sed -n ${n}p "$file"}}} but this reads the whole input file, even if you only wanted the third line. The following {{{sed}}} command line reads a file printing nothing (-n). At line $n the command "p" is run, printing it, with a "q" afterwards: quit the program. {{{ sed -n "$n{p;q;}" "$file" }}} [[Anchor(faq12)]] == A program (e.g. a file manager) lets me define an external command that an argument will be appended to - but i need that argument somewhere in the middle... == {{{ sh -c 'echo "$1"' -- hello }}} [[Anchor(faq13)]] == How can I concatenate two variables? == There is no concatenation operator for strings (either literal or variable dereferences) in the shell. The strings are just written one after the other: {{{ var=$var1$var2 }}} If the right-hand side contains whitespace characters, it needs to be quoted: {{{ var="$var1 - $var2" }}} Braces can be used to disambiguate the right-hand side: {{{ var=${var1}xyzzy # without braces, var1xyzzy would be interpreted as a variable name # Another equivalent way would be: var="$var1"xyzzy }}} CommandSubstitution can be used as well. The following line creates a log file name {{{logname}}} containing the current date, resulting in names like e.g. {{{log.2004-07-26}}}: {{{ logname="log.$(date +%Y-%m-%d)" }}} Appending data to the end of a string doesn't require any black magic, either. {{{ string="$string more data here" }}} Bash 3.1 has a new += operator that you may see from time to time: {{{ string+=" more data here" # EXTREMELY non-portable! }}} It's generally best to use the portable syntax. [[Anchor(faq14)]] == How can I redirect the output of multiple commands at once? == Redirecting the standard output of a single command is as easy as {{{ date > file }}} To redirect standard error: {{{ date 2> file }}} To redirect both: {{{ date > file 2>&1 }}} In a loop or other larger code structure: {{{ for i in $list; do echo "Now processing $i" # more stuff here... done > file 2>&1 }}} However, this can become tedious if the output of many programs should be redirected. If all output of a script should go into a file (e.g. a log file), the {{{exec}}} command can be used: {{{ # redirect both standard output and standard error to "log.txt" exec > log.txt 2>&1 # all output including stderr now goes into "log.txt" }}} Otherwise command grouping helps: {{{ { date # some other command echo done } > messages.log 2>&1 }}} In this example, the output of all commands within the curly braces is redirected to the file {{{messages.log}}}. [[Anchor(faq15)]] == How can I run a command on all files with the extention .gz? == Often a command already accepts several files as arguments, e.g. {{{ zcat *.gz }}} (One some systems, you would use {{{gzcat}}} instead of {{{zcat}}}. If neither is available, or if you don't care to play guessing games, just use {{{gzip -dc}}} instead.) If an explicit loop is desired, or if your command does not accept multiple filename arguments in one invocation, the {{{for}}} loop can be used: {{{ for file in *.gz do echo "$file" # do something with "$file" done }}} To do it recursively, you should use a loop, plus the find command: {{{ while read file; do echo "$file" # do something with "$file" done < <(find . -name '*.gz' -print) }}} For more hints in this direction, see [#faq20 FAQ #20], below. To see why the find command comes after the loop instead of before it, see [#faq24 FAQ #24]. [[Anchor(faq16)]] == How can I remove a file name extension from a string, e.g. file.tar to file? == The easiest (and fastest) way is to use the following: {{{ $ name="file.tar" $ echo "${name%.tar}" file }}} The {{{${var%pattern}}}} syntax removes the pattern from the end of the variable. {{{${var#pattern}}}} would remove pattern from the start of the string. This could be used to rename all files from "*.doc" to "*.txt": {{{ for file in *.doc do mv "$file" "${file%.doc}".txt done }}} There's more to ParameterSubstitution, e.g. {{{${var%%pattern}, ${var##pattern}, ${var//old/new}}}}. Note that this extended form of ParameterSubstitution works with ["BASH"], KornShell, KornShell93, but not with the older BourneShell. If the code needs to be portable to that shell as well, {{{sed}}} could be used to remove the filename extension part: {{{ for file in *.doc do base=`echo "$file" | sed 's/\.[^.]*$//'` # remove everything starting with last '.' mv "$file" "$base".txt done }}} Finally, some GNU/Linux/BSD systems offer a {{{rename}}} command. There are multiple different {{{rename}}} commands out there with contradictory syntaxes. Consult your man pages to see which one you have (if any). [[Anchor(faq17)]] == How can I group expressions, e.g. (A AND B) OR C? == The TestCommand {{{[}}} uses parentheses () for expression grouping. Given that "AND" is "-a", and "OR" is "-o", the following expression {{{ (0<n AND n<=10) OR n=-1 }}} can be written as follows: {{{ if [ \( $n -gt 0 -a $n -le 10 \) -o $n -eq -1 ] then echo "0 < $n <= 10, or $n=-1" else echo "invalid number: $n" fi }}} Note that the parentheses have to be quoted: \(, '(' or "(". ["BASH"] and KornShell have different, more powerful comparison commands with slightly different (easier) quoting: * ArithmeticExpression for arithmetic expressions, and * NewTestCommand for string (and file) expressions. Examples: {{{ if (( (n>0 && n<10) || n == -1 )) then echo "0 < $n < 10, or n==-1" fi }}} or {{{ if [[ ( -f $localconfig && -f $globalconfig ) || -n $noconfig ]] then echo "configuration ok (or not used)" fi }}} Note that the distinction between numeric and string comparisons is strict. Consider the following example: {{{ n=3 if [[ n>0 && n<10 ]] then echo "$n is between 0 and 10" else echo "ERROR: invalid number: $n" fi }}} The output will be "ERROR: ....", because in a ''string comparision'' "3" is bigger than "10", because "3" already comes after "1", and the next character "0" is not considered. Changing the square brackets to double parentheses {{{((}}} makes the example work as expected. [[Anchor(faq18)]] == How can I use numbers with leading zeros in a loop, e.g. 01, 02? == As always, there are different ways to solve the problem, each with its own advantages and disadvantages. If there are not many numbers, BraceExpansion can be used: {{{ for i in 0{1,2,3,4,5,6,7,8,9} 10 do echo $i done }}} Output: {{{ 00 01 02 03 [...] }}} This gets tedious for large sequences, but there are other ways, too. If the command {{{seq}}} is available, you can use it as follows: {{{ seq -w 1 10 }}} or, for arbitrary numbers of leading zeros (here: 3): {{{ seq -f "%03g" 1 10 }}} If you have the {{{printf}}} command (which is a Bash builtin, and is also POSIX standard), it can be used to format a number, too: {{{ for ((i=1; i<=10; i++)) do printf "%02d " "$i" done }}} The KornShell and KornShell93 have the {{{typeset}}} command to specify the number of leading zeros: {{{ $ typeset -Z3 i=4 $ echo $i 004 }}} Finally, the following example works with any BourneShell derived shell to zero-pad each line to three bytes: {{{ i=0 while test $i -le 10 do echo "00$i" i=`expr $i + 1` done | sed 's/.*\(...\)$/\1/g' }}} In this example, the number of '.' inside the parentheses in the {{{sed}}} statement determins how many total bytes from the {{{echo}}} command (at the end of each line) will be kept and printed. One more addendum: in Bash 3, you can use: {{{ printf "%03d \n" {1..300} }}} Which is slightly easier in some cases. Also you can use the {{{printf}}} command with xargs and wget to fetch files: {{{ printf "%03d \n" {$START..$END} | xargs -i% wget $LOCATION/% }}} Sometimes a good solution. [[Anchor(faq19)]] == How can I split a file into line ranges, e.g. lines 1-10, 11-20, 21-30? == Some Unix systems provide the {{{split}}} utility for this purpose: {{{ split --lines 10 --numeric-suffixes input.txt output- }}} For more flexibility you can use {{{sed}}}. The {{{sed}}} command can print e.g. the line number range 1-10: {{{ sed -n '1,10p' }}} This stops {{{sed}}} from printing each line ({{{-n}}}). Instead it only processes the lines in the range 1-10 ("1,10"), and prints them ("p"). {{{sed}}} still reads the input until the end, although we are only interested in lines 1 though 10. We can speed this up by making {{{sed}}} terminate immediately after printing line 10: {{{ sed -n -e '1,10p' -e '10q' }}} Now the command will quit after reading line 10 ("10q"). The {{{-e}}} arguments indicate a script (instead of a file name). The same can be written a little shorter: {{{ sed -n '1,10p;10q' }}} We can now use this to print an arbitrary range of a file (specified by line number): {{{ file=/etc/passwd range=10 firstline=1 maxlines=$(wc -l < "$file") # count number of lines while (($firstline < $maxlines)) do ((lastline=$firstline+$range+1)) sed -n -e "$firstline,${lastline}p" -e "${lastline}q" "$file" ((firstline=$firstline+$range+1)) done }}} This example uses ["BASH"] and KornShell ArithmeticExpressions, which older [wiki:Self:BourneShell Bourne shells] do not have. In that case the following example should be used instead: {{{ file=/etc/passwd range=10 firstline=1 maxlines=`wc -l < "$file"` # count line numbers while [ $firstline -le $maxlines ] do lastline=`expr $firstline + $range + 1` sed -n -e "$firstline,${lastline}p" -e "${lastline}q" "$file" firstline=`expr $lastline + 1` done }}} [[Anchor(faq20)]] == How can I find and deal with file names containing newlines, spaces or both? == The preferred method is still to use {{{ find ... -exec command {} \; }}} or, if you need to handle filenames ''en masse'': {{{ find ... -print0 | xargs -0 command }}} for GNU {{{find}}}/{{{xargs}}}, or (POSIX {{{find}}}): {{{ find ... -exec command {} + }}} Use that unless you really can't. Another way to deal with files with spaces in their names is to use the shell's filename expansion (["globbing"]). This has the disadvantage of not working recursively (except with zsh's extensions), but if you just need to process all the files in a single directory, it works fantastically well. This example changes all the *.mp3 files in the current directory to use underscores in their names instead of spaces. (But it will not work in the original BourneShell.) {{{ for file in *.mp3; do mv "$file" "${file// /_}" done }}} You could do the same thing for all files (regardless of extension) by using {{{ for file in *\ *; do }}} instead of *.mp3. Another way to handle filenames recursively involes using the {{{-print0}}} option of {{{find}}} (a GNU/BSD extension), together with bash's {{{-d}}} option for read: {{{ unset a i while read -d $'\0' file; do a[i++]="$file" # or however you want to process each file done < <(find /tmp -type f -print0) }}} The preceding example reads all the files under /tmp (recursively) into an array, even if they have newlines or other whitespace in their names, by forcing {{{read}}} to use the NUL byte (\0) as its word delimiter. Since NUL is not a valid byte in Unix filenames, this is the safest approach besides using {{{find -exec}}}. [[Anchor(faq21)]] == How can I replace a string with another string in all files? == {{{sed}}} is a good command to replace strings, e.g. {{{ sed 's/olddomain\.com/newdomain\.com/g' input > output }}} To replace a string in all files of the current directory: {{{ for i in *; do sed 's/old/new/g' "$i" > atempfile && mv atempfile "$i" done }}} GNU sed 4.x (but no other version of sed) has a special {{{-i}}} flag which makes the temp file unnecessary: {{{ for i in *; do sed -i 's/old/new/g' "$i" done }}} Those of you who have perl 5 can accomplish the same thing using this code: {{{ perl -pi -e 's/old/new/g' * }}} Recursively: {{{ find . -type f -print0 | xargs -0 perl -pi -e 's/old/new/g' }}} To replace for example all "unsigned" with "unsigned long", if it is not "unsigned int" or "unsigned long" ...: {{{ perl -i.bak -pne 's/\bunsigned\b(?!\s+(int|short|long|char))/unsigned long/g' $(find . -type f) }}} Finally, here's a script that some people may find useful: {{{ : # chtext - change text in several files # neither string may contain '|' unquoted old='olddomain\.com' new='newdomain\.com' # if no files were specified on the command line, use all files: [ $# -lt 1 ] && set -- * for file do [ -f "$file" ] || continue # do not process e.g. directories [ -r "$file" ] || continue # cannot read file - ignore it # Replace string, write output to temporary file. Terminate script in case of errors sed "s|$old|$new|g" "$file" > "$file"-new || exit # If the file has changed, overwrite original file. Otherwise remove copy if cmp "$file" "$file"-new >/dev/null 2>&1 then rm "$file"-new # file nas not changed else mv "$file"-new "$file" # file has changed: overwrite original file fi done }}} If the code above is put into a script file (e.g. {{{chtext}}}), the resulting script can be used to change a text e.g. in all HTML files of the current and all subdirectories: {{{ find . -type f -name '*.html' -exec chtext {} \; }}} Many optimizations are possible: * use another {{{sed}}} separator character than '|', e.g. ^A (ASCII 1) * some implementations of {{{sed}}} (e.g. GNU sed) have an "-i" option that can change a file in-place; no temporary file is necessary in that case * the {{{find}}} command above could use either {{{xargs}}} or the built-in {{{xargs}}} of POSIX find Note: {{{set -- *}}} in the code above is safe with respect to files whose names contain spaces. The expansion of * by {{{set}}} is the same as the expansion done by {{{for}}}, and filenames will be preserved properly as individual parameters, and not broken into words on whitespace. A more sophisticated example of {{{chtext}}} is here: http://www.shelldorado.com/scripts/cmds/chtext [[Anchor(faq22)]] == How can I calculate with floating point numbers instead of just integers? == ["BASH"] does not have built-in floating point arithmetic: {{{ $ echo $((10/3)) 3 }}} For better precision, an external program must be used, e.g. {{{bc}}}, {{{awk}}} or {{{dc}}}: {{{ $ echo "scale=3; 10/3" | bc 3.333 }}} The "scale=3" command notifies {{{bc}}} that three digits of precision after the decimal point are required. {{{awk}}} can be used for calculations, too: {{{ $ awk 'BEGIN {printf "%.3f\n", 10 / 3}' /dev/null 3.333 }}} There is a subtle but important difference between the {{{bc}}} and the {{{awk}}} solution here: {{{bc}}} reads commands and expressions ''from standard input''. {{{awk}}} on the other hand evaluates the expression as ''part of the program''. Expressions on standard input are ''not'' evaluated, i.e. {{{echo 10/3 | awk '{print $0}'}}} will print {{{10/3}}} instead of the evaluated result of the expression. This explains why the example uses {{{/dev/null}}} as an input file for {{{awk}}}: the program evaluates the {{{BEGIN}}} action, evaluating the expression and printing the result. Afterwards the work is already done: it reads its standard input, gets an end-of-file indication, and terminates. If no file had been specified, {{{awk}}} would wait for data on standard input. Newer versions of KornShell93 have built-in floating point arithmetic, together with mathematical functions like {{{sin()}}} or {{{cos()}}} . [[Anchor(faq23)]] == How do I append a string to the contents of a variable? == The shell doesn't have a string concatenation operator like Java ("+") or Perl ("."). The following example shows how to append the string ".2004-08-15" to the contents of the shell variable {{{filename}}}: {{{ filename="$filename.2004-08-15" }}} If the variable name and the string to append could be confused, the variable name can be enclosed in braces, e.g. {{{ filename="${filename}old" }}} instead of {{{filename=$filenameold}}} [[Anchor(faq24)]] == I set variables in a loop. Why do they suddenly disappear after the loop terminates? == The following command always prints "total number of lines: 0", although the variable {{{linecnt}}} has a larger value in the {{{while}}} loop: {{{ linecnt=0 cat /etc/passwd | while read line do linecnt=`expr $linecnt + 1` done echo "total number of lines: $linecnt" }}} The reason for this surprising behaviour is that a {{{while/for/until}}} loop runs in a subshell when its input or output is redirected from a pipeline. For the {{{while}}} loop above, a new subshell with its own copy of the variable {{{linecnt}}} is created (initial value, taken from the parent shell: "0"). This copy then is used for counting. When the {{{while}}} loop is finished, the subshell copy is discarded, and the original variable {{{linecnt}}} of the parent (whose value has not changed) is used in the {{{echo}}} command. It's hard to tell when shell would create a new process for a loop: * BourneShell creates it when the input or output is redirected, either by using a pipeline or by a redirection operator ('<', '>'). * ["BASH"] creates a new process only if the loop is part of a pipeline * KornShell creates it only if the loop is part of a pipeline, but ''not'' if the loop is the last part of it. To solve this, either use a method that works without a subshell (shown below), or make sure you do all processing inside that subshell (a bit of a kludge, but easier to work with): {{{ linecnt=0 cat /etc/passwd | ( while read line ; do linecnt="$((linecnt+1))" done echo "total number of lines: $linecnt" ) }}} To avoid the subshell completely (not easily possible if the other part of the pipe is a command!), use redirection, which does not have this problem at least for ["BASH"] and KornShell (but still for BourneShell): {{{ linecnt=0 while read line ; do linecnt="$((linecnt+1))" done < /etc/passwd echo "total number of lines: $linecnt" }}} For ["BASH"], when the first part of the pipe is a command, you can use "process substitution". The command used here is a simple "echo -e $'a\nb\nc'" as a substitute for a command with a multiline output: {{{ while read LINE; do echo "-> $LINE" done < <(echo -e $'a\nb\nc') }}} A portable and common work-around is to redirect the input of the {{{read}}} command using {{{exec}}}: {{{ linecnt=0 exec < /etc/passwd # redirect standard input from the file /etc/passwd while read line # "read" gets its input from the file /etc/passwd do linecnt=`expr $linecnt + 1` done echo "total number of lines: $linecnt" }}} This works as expected, and prints a line count for the file /etc/passwd. But the input is redirected from that file permanently. What if we need to read the original standard input sometime later again? In that case we have to save a copy of the original standard input file descriptor, which we later can restore: {{{ exec 3<&0 # save original standard input file descriptor "0" as FD "3" exec 0</etc/passwd # redirect standard input from the file /etc/passwd linecnt=0 while read line # "read" gets its input from the file /etc/passwd do linecnt=`expr $linecnt + 1` done exec 0<&3 # restore saved standard input (fd 0) from file descriptor "3" exec 3<&- # close the no longer needed file descriptor "3" echo "total number of lines: $linecnt" }}} Subsequent {{{exec}}} commands can be combined into one line, which is interpreted left-to-right: {{{ exec 3<&0 exec 0</etc/passwd _...read redirected standard input..._ exec 0<&3 exec 3<&- }}} is equivalent to {{{ exec 3<&0 0</etc/passwd _...read redirected standard input..._ exec 0<&3 3<&- }}} [[Anchor(faq25)]] == How can I access positional parameters after $9? == Use {{{${10}}}} instead of {{{$10}}}. This works for ["BASH"] and KornShell, but not for older BourneShell implementations. Another way to access arbitrary positional parameters after $9 is to use {{{for}}}, e.g. to get the last parameter: {{{ for last do : # nothing done echo "last argument is: $last" }}} To get an argument by number, we can use a counter: {{{ n=12 # This is the number of the argument we are interested in i=1 for arg do if [ $i -eq $n ] then argn=arg break fi i=`expr $i + 1` done echo "argument number $n is: $argn" }}} This has the advantage of not "consuming" the arguments. If this is no problem, the {{{shift}}} command discards the first positional arguments: {{{ shift 11 echo "the 12th argument is: $1" }}} Although direct access to any positional argument is possible this way, it's hardly needed. The common way is to use {{{getopts(3)}}} to process command line options (e.g. "-l", or "-o filename"), and then use either {{{for}}} or {{{while}}} to process all arguments in turn. An explanation of how to process command line arguments is available here: http://www.shelldorado.com/goodcoding/cmdargs.html [[Anchor(faq26)]] == How can I randomize (shuffle) the order of lines in a file? == {{{ randomize(){ while read l ; do echo "0$RANDOM $l" ; done | sort -n | cut -d" " -f2- } }}} Note: the leading 0 is to make sure it doesnt break if the shell doesnt support $RANDOM, which is supported by ["BASH"], KornShell, KornShell93 and ["POSIX"] shell, but not BourneShell. The same idea (printing random numbers in front of a line, and sorting the lines on that column) using other programs: {{{ awk ' BEGIN { srand() } { print rand() "\t" $0 } ' | sort -n | # Sort numerically on first (random number) column cut -f2- # Remove sorting column }}} This is faster thAn the previous solution, but will not work for very old AWK implementations (try "nawk", or "gawk", if available). A related question we frequently see is, "How can I print a random line from a file?" The problem here is that you need to know in advance how many lines the file contains. Lacking that knowledge, you have to read the entire file through once just to count them -- or, you have to suck the entire file into memory. Let's explore both of these approaches. {{{ n=$(wc -l < "$file") # Count number of lines. r=$((RANDOM % n + 1)) # Random number from 1..n. sed -n "$r{p;q;}" "$file" # Print the r'th line. }}} (These examples use the answer from [#faq11 FAQ 11] to print the n'th line.) The first one's pretty straightforward -- we use {{{wc}}} to count the lines, choose a random number, and then use {{{sed}}} to print the line. If we already happened to know how many lines were in the file, we could skip the {{{wc}}} command, and this would be a very efficient approach. The next example sucks the entire file into memory. This approach saves time reopening the file, but obviously uses more memory. {{{ oIFS=$IFS IFS=$'\n' lines=($(<"$file")) IFS=$oIFS n=${#lines[@]} r=$((RANDOM % n)) echo "${lines[r]}" }}} Note that we don't add 1 to the random number in this example, because the array of lines is indexed counting from 0. Also, some people want to choose a random file from a directory (for a signature on an e-mail, or to chose a random song to play, or a random image to display, etc.). A similar technique can be used: {{{ files=(*.ogg) # Or *.gif, or * n=${#files[@]} # For aesthetics xmms "${files[RANDOM % n]}" # Choose a random element }}} [[Anchor(faq27)]] == How can two processes communicate using named pipes (fifos)? == NamedPipes, also known as FIFOs ("First In First Out") are well suited for inter-process communication. The advantage over using files as a means of communication is, that processes are synchronized by pipes: a process writing to a pipe blocks if there is no reader, and a process reading from a pipe blocks if there is no writer. Here is a small example of a server process communicating with a client process. The server sends commands to the client, and the client acknowledges each command: '''Server''' {{{ #! /bin/sh # server - communication example # Create a FIFO. Some systems don't have a "mkfifo" command, but use # "mknod pipe p" instead mkfifo pipe while sleep 1 do echo "server: sending GO to client" # The following command will cause this process to block (wait) # until another process reads from the pipe echo GO > pipe # A client read the string! Now wait for its answer. The "read" # command again will block until the client wrote something read answer < pipe # The client answered! echo "server: got answer: $answer" done }}} '''Client''' {{{ #! /bin/sh # client # We cannot start working until the server has created the pipe... until [ -p pipe ] do sleep 1; # wait for server to create pipe done # Now communicate... while sleep 1 do echo "client: waiting for data" # Wait until the server sends us one line of data: read data < pipe # Received one line! echo "client: read <$data>, answering" # Now acknowledge that we got the data. This command # again will block until the server read it. echo ACK > pipe done }}} Write both examples to files {{{server}}} and {{{client}}} respectively, and start them concurrently to see it working: {{{ $ chmod +x server client $ server & client & server: sending GO to client client: waiting for data client: read <GO>, answering server: got answer: ACK server: sending GO to client client: waiting for data client: read <GO>, answering server: got answer: ACK server: sending GO to client client: waiting for data [...] }}} [[Anchor(faq28)]] == How do I determine the location of my script? I want to read some config files from the same place. == This is a complex question because there's no single right answer to it. Even worse: it's not possible to find the location reliably in 100% of all cases. All ways of finding a script's location depend on the name of the script, as seen in the predefined variable {{{$0}}}. But providing the script name in {{{$0}}} is only a (very common) convention, not a requirement. The suspect answer is "in some shells, $0 is always an absolute path, even if you invoke the script using a relative path, or no path at all". That's not the case in ["BASH"]. But this isn't reliable across shells; some of them return the actual command typed in by the user instead of the fully qualified path. In those cases, if all you want is the fully qualified version of $0, you can use something like this (["POSIX"], non-Bourne): {{{ [[ $0 = /* ]] && echo $0 || echo $PWD/$0 }}} Or the BourneShell version: {{{ case $0 in /*) echo $0;; *) echo `pwd`/$0;; esac }}} However, this approach has some major drawbacks. The most important is, that the script name (as seen in {{{$0}}}) may not be relative to the current working directory, but relative to a directory from the program search path {{{$PATH}}} (this is often seen with KornShell). Another drawback is that there is really no guarantee that your script is still in the same place it was when it first started executing. Suppose your script is loaded from a temporary file which is then unlinked immediately... your script might not even exist on disk any more! The script could also have been moved to a different location while it was executing. Or (and this is most likely by far...) there might be multiple links to the script from multiple locations, one of them being a simple symlink from a common {{{PATH}}} directory like {{{/usr/local/bin}}}, which is how it's being invoked. Your script might be in {{{/opt/foobar/bin/script}}} but the naive approach of reading {{{$0}}} won't tell you that. (For a more general discussion of the Unix file system and how symbolic links affect your ability to know where you are at any given moment, see [http://www.cs.bell-labs.com/sys/doc/lexnames.html this Plan 9 paper].) So if the name in {{{$0}}} is a relative one, i.e. does not start with '/', we can still try to search the script like the shell would have done: in all directories from {{{$PATH}}}. The following script shows how this could be done: {{{ myname=$0 if [ -s "$myname" ] && [ -x "$myname" ] then # $myname is already a valid file name mypath=$myname else case "$myname" in /*) exit 1;; # absolute path - do not search PATH *) # Search all directories from the PATH variable. Take # care to interpret leading and trailing ":" as meaning # the current directory; the same is true for "::" within # the PATH. for dir in `echo "$PATH" | sed 's/^:/.:/g;s/::/:.:/g;s/:$/:./;s/:/ /g'` do [ -f "$dir/$myname" ] || continue # no file [ -x "$dir/$myname" ] || continue # not executable mypath=$dir/$myname break # only return first matching file done ;; esac fi if [ -f "$mypath" ] then : # echo >&2 "DEBUG: mypath=<$mypath>" else echo >&2 "cannot find full path name: $myname" exit 1 fi echo >&2 "path of this script: $mypath" }}} Note that {{{$mypath}}} is not necessarily an absolute path name. It still can contain relative parts like {{{../bin/myscript}}}. Generally storing data files in the same directory as their scripts is a bad practice. The Unix file system layout assumes that files in one place (e.g. /bin) are executable programs, while files in another place (e.g. /etc) are data files. (Let's ignore legacy Unix systems with programs in /etc for the moment, shall we....) It really makes the most sense to keep your script's configuration in a single, static location such as {{{$SCRIPTROOT/etc/foobar.conf}}}. If you need to define multiple configuration files, then you can have a directory (say, {{{/var/lib/foobar}}} or {{{/usr/local/lib/foobar}}}), and read that directory's location from a variable in {{{/etc/foobar.conf}}}. If you don't even want that much to be hard-coded, you could pass the location of {{{foobar.conf}}} as a parameter to the script. If you need the script to assume certain default in the absence of {{{/etc/foobar.conf}}}, you can put defaults in the script itself, and/or fall back to something like {{{$HOME/.foobar.conf}}} if {{{/etc/foobar.conf}}} is missing. (This depends on what your script does. In some cases, it may make more sense to abort gracefully.) [[Anchor(faq29)]] == How can I display value of a symbolic link on standard output? == The external command {{{readlink}}} can be used to display the value of a symbolic link. {{{ $ readlink /bin/sh bash }}} you can also use GNU find's %l directive, which is especially useful if you need to resolve links in batches: {{{ $ find /bin/ -type l -printf '%p points to %l\n' /bin/sh points to bash /bin/bunzip2 points to bzip2 ... }}} If your system lacks {{{readlink}}}, you can use a function like this one: {{{ readlink() { local path=$1 ll if [ -L "$path" ]; then ll="$(LC_ALL=C ls -l "$path" 2> /dev/null)" && echo "${ll/* -> }" else return 1 fi } }}} [[Anchor(faq30)]] == How can I rename all my *.foo files to *.bar? == Some GNU/Linux distributions have a rename command, which you can use for this purpose; however, the syntax differs from one distribution to the next, so it's not a portable answer. You can do it in POSIX shells like this: {{{ for f in *.foo; do mv "$f" "${f%.foo}.bar"; done }}} This invokes the external command {{{mv}}} once for each file, so it may not be as efficient as some of the {{{rename}}} implementations. If you want to do it recursively, then it becomes much more challenging. This example works (in ["BASH"]) as long as no files have newlines in their names: {{{ find . -name '*.foo' -print | while IFS=$'\n' read -r f; do mv "$f" "${f%.foo}.bar" done }}} Another common form of this question is "How do I rename all my MP3 files so that they have underscores instead of spaces?" You can use this: {{{ for f in *\ *.mp3; do mv "$f" "${f// /_}"; done }}} [[Anchor(faq31)]] == What is the difference between the old and new test commands ([ and [[)? == {{{[}}} ("test" command) and {{{[[}}} ("new test" command) are both used to evaluate expressions. Some examples: {{{ if [ -z "$variable" ] then echo "variable is empty!" fi if [ -f "$filename" ] then echo "not a valid, existing file name: $filename" fi }}} and {{{ if [[ -e $file ]] then echo "directory entry does not exist: $file" fi if [[ $file0 -nt $file1 ]] then echo "file $file0 is newer than $file1" fi }}} To cut a long story short: {{{[}}} implements the old, portable syntax of the command. Although all modern shells have built-in implementations, there usually still is an external executable of that name, e.g. {{{/bin/[}}}. {{{[[}}} is a new improved version of it, which is a keyword, not a program. This has benefical effects on the ease of use, see below. {{{[[}}} is understood by KornShell, ["BASH"] (e.g. 2.03), KornShell93, ["POSIX"] shell, but not by the older BourneShell. Although {{{[}}} and {{{[[}}} have much in common, and share many expression operators like "-f", "-s", "-n", "-z", there are some notable differences. Here is a comparison list: ||'''Feature'''||'''new test''' {{{[[}}}||'''old test''' {{{[}}}||'''Example'''|| ||<rowspan="4">string comparison||>||(not available)||-|| ||<||(not available)||-|| ||== (or =)||=||-|| ||!=||!=||-|| ||<rowspan="2">expression grouping||&&||-a||{{{[[ -n $var && -f $var ]] && echo "$var is a file"}}}|| ||{{{||}}}||-o||-|| ||Pattern matching||=||(not available)||{{{[[ $name = a* ]] || echo "name does not start with an 'a': $name"}}}|| ||In-process regular expression matching||=~||(not available)||{{{[[ $(date) =~ '^Fri ... 13 ' ]] && echo "It's Friday the 13th!"}}}|| Special primitives that {{{[[}}} is defined to have, but {{{[}}} may be lacking (depending on the implementation): ||'''Description'''||'''Primitive'''||'''Example'''|| ||entry (file or directory) exists||-e||{{{[[ -e $config ]] && echo "config file exists: $config"}}}|| ||file is newer/older than other file||-nt / -ot||{{{[[ $file0 -nt $file1 ]] && echo "$file0 is newer than $file1"}}}|| ||two files are the same||-ef||{{{[[ $input -ef $output ]] && { echo "will not overwrite input file: $input"; exit 1; } }}}|| ||negation||!||-|| But there are more subtle differences. * No field splitting will be done for {{{[[}}} (and therefore many arguments need not to be quoted) {{{ file="file name" [[ -f $file ]] && echo "$file is a file"}}} will work even though $file is not quoted and contains whitespace. With {{{[}}} the variable needs to be quoted: {{{ file="file name" [ -f "$file" ] && echo "$file is a file"}}} This makes {{{[[}}} easier to use and less error prone. * No file name generation will be done for {{{[[}}}. Therefore the following line tries to match the contents of the variable $path with the pattern {{{/*}}} {{{ [[ $path = /* ]] && echo "\$path starts with a forward slash /: $path"}}} The next command most likely will result in an error, because {{{/*}}} is subject to file name generation: {{{ [ $path = /* ] && echo "this does not work"}}} {{{[[}}} is strictly used for strings and files. If you want to compare numbers, use ArithmethicExpression ((''expression'')), e.g. {{{ i=0 while ((i<10)) do echo $i ((i=$i+1)) done}}} When should the new test command {{{[[}}} be used, and when the old one {{{[}}}? If portability to the BourneShell is a concern, the old syntax should be used. If on the other hand the script requires ["BASH"] or KornShell, the new syntax could be preferable. [[Anchor(faq32)]] == How can I redirect the output of 'time' to a variable or file? == The reason that 'time' needs special care for redirecting its output is one of those mysteries of the universe. The answer will probably be solved around the same time we find dark matter. * File Redirection {{{ bash -c "time ls" > /path/to/foo 2>&1 ( time ls ) > /path/to/foo 2>&1 { time ls; } > /path/to/foo 2>&1 }}} * Variable Redirection {{{ foo=$( bash -c "time ls" 2>&1 ) foo=$( ( time ls ) 2>&1 ) foo=$( { time ls; } 2>&1 ) }}} Note: Using 'bash -c' and ( ) creates a subshell, using { } does not. Do with that as you wish. [[Anchor(faq33)]] == How can I find a process ID for a process given its name? == Usually a process is referred to using its process ID (PID), and the {{{ps}}} command can display the information for any process given its process ID, e.g. {{{ $ echo $$ # my process id 21796 $ ps -p 21796 PID TTY TIME CMD 21796 pts/5 00:00:00 ksh }}} But frequently the process ID for a process is not known, but only its name. Some operating systems, e.g. Solaris, BSD, and some versions of Linux have a dedicated command to search a process given its name, called {{{pgrep}}}: {{{ $ pgrep init 1 }}} Often there is an even more specialized program available to not just find the process ID of a process given its name, but also to send a signal to it: {{{ $ pkill myprocess }}} Some systems also provide {{{pidof}}}. It differs from {{{pgrep}}} in that multiple output process IDs are only space separated, not newline separated. {{{ $ pidof cron 5392 }}} If these programs are not available, a user can search the output of the ps(1) command using {{{grep}}}. The major problem when grepping the ps output is that grep ''may'' match its own ps entry (try: ps aux | grep init). To make matters worse, this does not happen every time; the techicnal name for this is a "race condition". To avoid this, there are several ways: * Using grep -v at the end {{{ ps aux | grep name | grep -v grep }}} will throw away all lines containing "grep" from the output. Disadvantage: You always have the exit state of the grep -v, so you can't e.g. check if a specific process exists. * Using grep -v in the middle {{{ ps aux | grep -v grep | grep name }}} This does exactly the same, beside that the exit state of "grep name" is acessible and a representation for "name is a process in ps" or "name is not a process in ps". It still has the disadvantage to start a new process (grep -v). * Using [] in grep {{{ ps aux | grep [n]ame }}} This spawns only the needed grep-process. The trick is to use the {{{[]}}}-character class (regular expressions). To put only one character in a character group normally makes no sense at all, because a {{{[c]}}} will always be a "c". In this case, it's the same. {{{grep [n]ame}}} searches for "name". But as grep's own process list entry is what you executed ("grep [n]ame") and not "grep name", it will not match itself. ===BEGIN greycat rant=== Most of the time when someone asks a question like this, it's because they want to manage a long-running daemon using primitive shell scripting techniques. Common variants are "How can I get the PID of my foobard process.... so I can start one if it's not already running" or "How can I get the PID of my foobard process... because I want to prevent the foobard script from running if foobard is already active." Both of these questions will lead to seriously flawed production systems. If what you really want is to restart your daemon whenever it dies, just do this: {{{ #!/bin/sh while true; do mydaemon --in-the-foreground done }}} where --in-the-foreground is whatever switch, if any, you must give to the daemon to PREVENT IT from automatically backgrounding itself. (Often, -d does this and has the additional benefit of running the daemon with increased verbosity.) Self-daemonizing programs may or may not be the target of a future greycat rant.... If that's too simplistic, look into [http://cr.yp.to/daemontools.html daemontools] or [http://smarden.org/runit/ runit], which are programs for managing services. If what you really want is to prevent multiple instances of your program from running, then the only sure way to do that is by using a lock. For details on doing this, see ProcessManagement or [#faq45 FAQ 45]. ===END greycat rant=== [[Anchor(faq34)]] == Can I do a spinner in Bash? == Sure. {{{ i=1 sp="/-\|" echo -n ' ' while true do echo -en "\b${sp:i++%${#sp}:1}" done }}} You can also use \r instead of \b. You can use pretty much any character sequence you want as well. If you want it to slow down, put a {{{sleep}}} command inside the loop. A similar technique can be used to build progress bars. [[Anchor(faq35)]] == How can I handle command-line arguments to my script easily? == Well, that depends a great deal on what you want to do with them. Here's a general template that might help for the simple cases: {{{ while [[ $1 == -* ]]; do case "$1" in -h|--help) show_help; exit 0;; -v) verbose=1; shift;; -f) output_file=$2; shift 2;; esac done # Now all of the remaining arguments are the filenames which followed # the optional switches. You can process those with "for i" or "$@". }}} For more complex/generalized cases, or if you want things like "-xvf" to be handled as three separate flags, you can use getopts or getopt. (Heiner, that's your cue....) [[Anchor(faq36)]] == How can I get all lines that are: in both of two files (set intersection) or in only one of two files (set subtraction). == Use the comm(1) command. {{{ # intersection of file1 and file2 comm -12 <(sort file1) <(sort file2) # subtraction of file1 from file2 comm -13 <(sort file1) <(sort file2) }}} Read the comm(1) manpage for details. If for some reason you lack the core comm(1) program, you can use these other methods: an amazingly simple and fast implementation, that took just 20 seconds to match a 30k line file against a 400k line file for me. note that it probably only works with GNU grep, and that the file specified with -f is will be loaded into ram, so it doesn't scale for very large files. it has grep read one of the sets as a pattern list from a file (-f), and interpret the patterns as plain strings not regexps (-F), matching only whole lines (-x). {{{ # intersection of file1 and file2 grep -xF -f file1 file2 # substraction of file1 from file2 grep -vxF -f file1 file2 }}} an implementation using sort and uniq {{{ # intersection of file1 and file2 sort file1 file2 | uniq -d (Assuming each of file1 or file2 does not have repeated content) # file1-file2 (Subtraction) sort file1 file2 file2 | uniq -u # same way for file2 - file1, change last file2 to file1 sort file1 file2 file1 | uniq -u }}} another implementation of substraction: {{{ cat file1 file1 file2 | sort | uniq -c | awk '{ if ($1 == 2) { $1 = ""; print; } }' }}} This may introduce an extra space at the start of the line; if that's a problem, just strip it away. Also, this approach assumes that neither file1 nor file2 has any duplicates in it. Finally, it sorts the output for you. If that's a problem, then you'll have to abandon this approach altogether. Perhaps you could use awk's associative arrays (or perl's hashes or tcl's arrays) instead. [[Anchor(faq37)]] == How can I print text in various colors? == ''Do not'' hard-code ANSI color escape sequences in your program! The {{{tput}}} command lets you interact with the terminal database in a sane way. {{{ tput setaf 1; echo this is red tput setaf 2; echo this is green tput setaf 0; echo now we are back in black }}} {{{tput}}} reads the terminfo database which contains all the escape codes necessary for interacting with your terminal, as defined by the {{{$TERM}}} variable. For more details, see the {{{terminfo(5)}}} man page. If you don't know in advance what your user's terminal's default text color is, you can use {{{tput sgr0}}} to reset the colors to their default settings. This also removes boldface ({{{tput bold}}}), etc. [[Anchor(faq38)]] == How do Unix file permissions work? == See ["Permissions"]. [[Anchor(faq39)]] == What are all the dot-files that bash reads? == See DotFiles. [[Anchor(faq40)]] == How do I use dialog to get input from the user? == {{{ foo=$(dialog --inputbox "text goes here" 8 40 2>&1 >/dev/tty) echo "The user typed '$foo'" }}} The redirection here is a bit tricky. 1. The {{{foo=$(command)}}} is set up first, so the standard output of the command is being captured by bash. 1. Inside the command, the {{{2>&1}}} causes standard error to be sent to where standard out is going -- in other words, stderr will now be captured. 1. {{{>/dev/tty}}} sends standard output to the terminal, so the dialog box will be seen by the user. Standard error will still be captured, however. Another common {{{dialog(1)}}}-related question is how to dynamically generate a dialog command that has items which must be quoted (either because they're empty strings, or because they contain internal white space). One ''can'' use {{{eval}}} for that purpose, but the cleanest way to achieve this goal is to use an array. {{{ unset m; i=0 words=(apple banana cherry "dog droppings") for w in "${words[@]}"; do m[i++]=$w; m[i++]="" done dialog --menu "Which one?" 12 70 9 "${m[@]}" }}} In the previous example, the while loop that populates the '''m''' array could have been reading from a pipeline, a file, etc. Recall that the construction {{{"${m[@]}"}}} expands to the entire contents of an array, but with each element implicitly quoted. It's analogous to the {{{"$@"}}} construct for handling positional parameters. For more details, see [#faq50 FAQ50] below. Here's another example, using filenames: {{{ files=(*.mp3) # These may contain spaces, apostrophes, etc. cmd=(dialog --menu "Select one:" 22 76 16); n=6 i=0 for f in "${files[@]}"; do cmd[n++]=$((i++)); cmd[n++]="$f" done choice=$("${cmd[@]}" 2>&1 >/dev/tty) }}} The user's choice will be stored in the {{{choice}}} variable, as an integer, which can in turn be used as an index into the {{{files}}} array. A seperate but useful function of dialog is to track progress of a process that produces output. Below is an example that uses dialog to track processes writing to a log file. In the dialog window, there is a tailbox where output is stored, and a msgbox with a clickable Quit. Clicking quit will cause trap to execute, removing the tempfile, and destroying the tail process. {{{ #you can not tail a nonexistant file, so always ensure it pre-exists! rm -f dialog-tail.log; echo Initialize log >> dialog-tail.log date >> dialog-tail.log tempfile=`tempfile 2>/dev/null` || tempfile=/tmp/test$$ trap "rm -f $tempfile" 0 1 2 5 15 dialog --title "TAIL BOXES" \ --begin 10 10 --tailboxbg dialog-tail.log 8 58 \ --and-widget \ --begin 3 10 --msgbox "Press OK " 5 30 \ 2>$tempfile & mypid=$!; for i in 1 2 3; do echo $i >> dialog-tail.log; sleep 1; done echo Done. >> dialog-tail.log wait $mypid; }}} [[Anchor(faq41)]] == How do I determine whether a variable contains a substring? == {{{ if [[ $foo = *bar* ]] }}} The above works in virtually all versions of Bash. Bash version 3 also allows regular expressions: {{{ if [[ $foo =~ ab*c ]] # bash 3, matches abbbbcde, or ac, etc. }}} If you are programming in the BourneShell instead of Bash, there is a more portable (but less pretty) syntax: {{{ case "$foo" in *bar*) .... ;; esac }}} This should allow you to match variables against globbing-style patterns. if you need a portable way to match variables against regular expressions, use {{{grep}}} or {{{egrep}}}. {{{ if echo "$foo" | egrep some-regex >/dev/null; then ... }}} [[Anchor(faq42)]] == How can I find out if a process is still running? == The {{{kill}}} command is used to send signals to a running process. As a convenience function, the signal "0", which does not exist, can be used to find out if a process is still running: {{{ myprog & # Start program in the background daemonpid=$! # ...and save its process id while sleep 60 do if kill -0 $daemonpid # Is the process still alive? then echo >&2 "OK - process is still running" else echo >&2 "ERROR - process $daemonpid is no longer running!" break fi done}}} This is one of those questions that usually masks a much deeper issue. It's rare that someone wants to know whether a process is still running simply to display a red or green light to an operator. More often, there's some ulterior motive, such as the desire to ensure that some daemon which is known to crash frequently is still running, or to ensure mutually exclusive access to a resource, etc. For much better discussion of these issues, see ProcessManagement or [#faq33 FAQ #33]. [[Anchor(faq43)]] == How can I use array variables? == BASH and KornShell already have one-dimensional arrays indexed by a numerical expression, e.g. {{{ host[0]="micky" host[1]="minnie" host[2]="goofy" i=0 while (($i < ${#host[@]} )) do echo "host number $i is ${host[i++]}" done}}} The awkward experssion {{{ ${#host[@]} }}} returns the number of elements for the array {{{host}}}. It's possible to assign multiple values to an array at once, but the syntax differs from BASH to KornShell: {{{ # BASH array=(one two three four) # KornShell set -A array -- one two three four}}} [[Anchor(faq44)]] == How can I use associative arrays or variable variables? == Sometimes it's convenient to have associative arrays, arrays indexed by a string. Perl calls them "hashes". KornShell93 already supports this kind of array: {{{ # KornShell93 script - does not work with BASH typeset -A homedir # Declare KornShell93 associative array homedir[jim]=/home/jim homedir[silvia]=/home/silvia homedir[alex]=/home/alex for user in ${!homedir[@]} # Enumerate all indices (user names) do echo "Home directory of user $user is ${homedir[$user]}" done}}} BASH (including version 3.x) does not (yet) support them. However, we could simulate this kind of array by dynamically creating variables like in the following example: {{{ for user in jim silvia alex do eval homedir_$user=/home/$user done}}} This creates the variables {{{ homedir_jim=/home/jim homedir_silvia=/home/silvia homedir_alex=/home/alex}}} with the corresponding content. Note the use of the {{{eval}}} command, which interprets a command line not just one time like the shell usually does, but '''twice'''. In the first step, the shell uses the input {{{homedir_$user=/home/$user}}} to create a new line {{{homedir_jim=/home/jim}}}. In the second step, caused by {{{eval}}}, this variable assignment is executed, actually creating the variable. Print the variables using {{{ for user in jim silvia alex do varname=homedir_$user # e.g. "homedir_jim" eval varcontent='$'$varname # e.g. "/home/jim" echo "home directory of $user is $varcontent" done}}} The {{{eval}}} line needs some explanation. In a first step the command substitution is run: {{{ eval varcontent='$'$varname}}} becomes {{{ eval varcontent=$homedir_jim}}} In a second step the {{{eval}}} re-evaluates the line, and converts this to {{{ varcontent=/home/jim}}} Before starting to use dynamically created variables, think again of a simpler approach. If it still seems to be the best thing to do, have a look at the following disadvantages: 1. it's hard to read and to maintain 1. the variable names must match the regular expression ^[a-zA-Z_][a-zA-Z_0-9]* , i.e. a variable name cannot contain arbitrary characters but only letters, digits, and underscores. In the example above we e.g. could not have processed the home directory of a user named {{{hong-hu}}}, because a dash '-' can be no valid part of a user name. 1. Quoting is hard to get right. If a content (not variable name) string can contain whitespace characters, it's hard to quote it right to preserve it. Here is the summary. "{{{var}}}" is a constant prefix, "{{{$index}}}" contains index string, "{{{$content}}}" is the string to store. Note that quoting is absolutely essential here. A missing backslash \ or a wrong type of quote (e.g. apostrophes '...' instead of quotation marks "...") can (and probably will) cause the examples to fail: * Set variables {{{ eval "var$index=\"$content\"" # index must only contain characters from [a-zA-Z0-9_]}}} * Print variable content {{{ eval "echo \"var$index=\$$varname\""}}} * Check if a variable is empty {{{ if eval "[ -z "\$var$index\" ]" then echo "variable is empty: $var$index" fi}}} You've seen the examples. Now maybe you can go a step back and consider using AWK associative arrays, or a multi-line environment variable instead of dynamically created variables. [[Anchor(faq45)]] == How can I ensure that only one instance of a script is running at a time (mutual exclusion)? == We need some means of '''mutual exclusion'''. One easy way is to use a "lock": any number of processes can try to acquire the lock simultaneously, but only one of them will succeed. How can we implement this using shell scripts? Some people suggest creating a lock file, and checking for its presence: {{{ # locking example -- WRONG lockfile=/tmp/myscript.lock if [ -f "$lockfile" ] then # lock is already held echo >&2 "cannot acquire lock, giving up: $lockfile" exit 0 else # nobody owns the lock > "$lockfile" # create the file #...continue script fi}}} This example '''does not work''', because there is a time window between checking and creating the file. Assume two processes are running the code at the same time. Both check if the lockfile exists, and both get the result that it does not exist. Now both processes assume they have acquired the lock -- a disaster waiting to happen. We need an atomic check-and-create operation, and fortunately there is one: {{{mkdir}}}, the command to create a directory: {{{ # locking example -- CORRECT lockdir=/tmp/myscript.lock if mkdir "$lockdir" then # directory did not exist, but was created successfully echo >&2 "successfully acquired lock: $lockdir" # continue script else echo >&2 "cannot acquire lock, giving up on $lockdir" exit 0 fi}}} The advantage over using a lock file is, that even when two processes call {{{mkdir}}} at the same time, only one process can succeed at most. This atomicity of check-and-create is ensured at the operating system kernel level. Note that we cannot use "mkdir -p" to automatically create missing path components: "mkdir -p" does not return an error if the directory exists already, but that's the feature we rely upon to ensure mutual exclusion. Now let's spice up this example by automatically removing the lock when the script finishes: {{{ lockdir=/tmp/myscript.lock if mkdir "$lockdir" then echo >&2 "successfully acquired lock" # Remove lockdir when the script finishes, or when it receives a signal trap 'rm -rf "$lockdir"' 0 # remove directory when script finishes trap "exit 2" 1 2 3 15 # terminate script when receiving signal # Optionally create temporary files in this directory, because # they will be removed automatically: tmpfile=$lockdir/filelist else echo >&2 "cannot acquire lock, giving up on $lockdir" exit 0 fi}}} This example provides reliable mutual exclusion. There is still the disadvantage that a ''stale'' lock file could remain when the script is terminated with a signal not caught (or signal 9, SIGKILL), but it's a good step towards reliable mutual exclusion. Instead of using {{{mkdir}}} we could also have used the program to create a symbolic link, {{{ln -s}}}. For more discussion on these issues, see ProcessManagement. [[Anchor(faq46)]] == I want to check to see whether a word is in a list (or an element is a member of a set). == Let's suppose you have your "list" stored as a big string of words, with spaces in between them. (That's the most common case when people are asking this one.) What you actually want to do is determine whether the string " foo " (note the spaces around it) appears in the list. But since your list may not have leading/trailing spaces, you have to add them as well. So, here's the most portable way to do it: {{{ if echo " $list " | grep " foo " >/dev/null; then ....}}} GNU grep seems to have a special {{{-w}}} extension which lets you avoid the spaces: {{{ if echo "$list" | GNUgrep -q -w "foo"; then ....}}} Finally, if you want to use Bash builtins, you can do it thus: {{{ if [[ " $list " = *\ foo\ * ]]; then ....}}} This is basically the same as the original grep -- we surround both the list and the word (foo) with spaces, and then do a simple text matching. [[Anchor(faq47)]] == How can I redirect stderr to a pipe? == A pipe can only carry stdout of a program. To pipe stderr through it, you need to redirect stderr to the same destination as stdout. Optionally you can close stdout or redirect it to /dev/null to only get stderr. Some sample code: {{{ # - 'myprog' is an example for a program that outputs both, stdout and # stderr # - after the pipe I will just use a 'cat', of course you can put there # what you want # version 1: redirect stderr towards the pipe while stdout survives (both come # mixed) myprog 2>&1 | cat # version 2: redirect stderr towards the pipe without getting stdout (it's # redirected to /dev/null) myprog 2>&1 >/dev/null | cat #Note that '>/dev/null' comes after '2>&1', otherwise the stderr will also be directed to /dev/null # version 3: redirect stderr towards the pipe while the "original" stdout gets # closed myprog 2>&1 >&- | cat }}} One may also pipe stderr only but keep stdout intact (without ''a priori'' knowledge of where the script's output is going). This is a bit trickier. This has an obvious application with eg. dialog, which draws (using ncurses) windows onto the screen to stdout, and returns output to stderr. This may be a little inconvenient, because it may lead to a necessary temporary file which we may like to evade. (Although this is not necessary -- see [#faq40 FAQ #40] for more examples of using dialog specifically!) On [http://www.tldp.org/LDP/abs/html/io-redirection.html TLDP], I've found following trick: {{{ # Redirecting only stderr to a pipe. exec 3>&1 # Save current "value" of stdout. ls -l 2>&1 >&3 3>&- | grep bad 3>&- # Close fd 3 for 'grep' (but not 'ls'). # ^^^^ ^^^^ exec 3>&- # Now close it for the remainder of the script. # Thanks, S.C. }}} To show it as a dialog one-liner: {{{ exec 3>&1 dialog --menu Title 0 0 0 FirstItem FirstDescription 2>&1 >&3 3>&- | sed 's/First/Only/' exec 3>&- }}} This will have the dialog window working properly, yet it will be the output of dialog (returned to stderr) being altered by the sed. Cheers. [[Anchor(faq48)]] == Why should I never use eval? == "eval" is a common misspelling of "evil". The section dealing with spaces in file names used to include the following quote "helpful tool (which is probably not as safe as the \0 technique)", end quote. {{{ Syntax : nasty_find_all [path] [command] <maxdepth> }}} {{{ #This code is evil and must never be used export IFS=" " [ -z "$3" ] && set -- "$1" "$2" 1 FILES=`find "$1" -maxdepth "$3" -type f -printf "\"%p\" "` #warning, evilness eval FILES=($FILES) for ((I=0; I < ${#FILES[@]}; I++)) do eval "$2 \"${FILES[I]}\"" done unset IFS }}} This script is supposed to recursively search for files with newlines and/or spaces in them, arguing that {{{find -print0 | xargs -0}}} was unsuitable for some purposes such as multiple commands. It was followed by an instructional description on all the lines involved, which we'll skip. To its defense, it works: {{{ $ ls -lR .: total 8 drwxr-xr-x 2 vidar users 4096 Nov 12 21:51 dir with spaces -rwxr-xr-x 1 vidar users 248 Nov 12 21:50 nasty_find_all ./dir with spaces: total 0 -rw-r--r-- 1 vidar users 0 Nov 12 21:51 file?with newlines $ ./nasty_find_all . echo 3 ./nasty_find_all ./dir with spaces/file with newlines $ }}} But consider this: {{{ $ touch "\"); ls -l $'\x2F'; #" }}} You just created a file called {{{ "); ls -l $'\x2F'; #}}} Now FILES will contain {{{ ""); ls -l $'\x2F'; #}}}. When we do {{{eval FILES=($FILES)}}}, it becomes {{{ FILES=(""); ls -l $'\x2F'; #" }}} Which becomes the two statements {{{ FILES=(""); }}} and {{{ ls -l / }}}. Congratulations, you just allowed execution of arbitrary commands. {{{ $ touch "\"); ls -l $'\x2F'; #" $ ./nasty_find_all . echo 3 total 1052 -rw-r--r-- 1 root root 1018530 Apr 6 2005 System.map drwxr-xr-x 2 root root 4096 Oct 26 22:05 bin drwxr-xr-x 3 root root 4096 Oct 26 22:05 boot drwxr-xr-x 17 root root 29500 Nov 12 20:52 dev drwxr-xr-x 68 root root 4096 Nov 12 20:54 etc drwxr-xr-x 9 root root 4096 Oct 5 11:37 home drwxr-xr-x 10 root root 4096 Oct 26 22:05 lib drwxr-xr-x 2 root root 4096 Nov 4 00:14 lost+found drwxr-xr-x 6 root root 4096 Nov 4 18:22 mnt drwxr-xr-x 11 root root 4096 Oct 26 22:05 opt dr-xr-xr-x 82 root root 0 Nov 4 00:41 proc drwx------ 26 root root 4096 Oct 26 22:05 root drwxr-xr-x 2 root root 4096 Nov 4 00:34 sbin drwxr-xr-x 9 root root 0 Nov 4 00:41 sys drwxrwxrwt 8 root root 4096 Nov 12 21:55 tmp drwxr-xr-x 15 root root 4096 Oct 26 22:05 usr drwxr-xr-x 13 root root 4096 Oct 26 22:05 var ./nasty_find_all ./dir with spaces/file with newlines ./ $ }}} It doesn't take much imagination to replace {{{ ls -l }}} with {{{ rm -rf }}} or worse. One might think these circumstances are obscure, but one should not be tricked by this. All it takes is one malicious user, or perhaps more likely, a benign user who left the terminal unlocked when going to the bathroom, wrote a funny php uploading script that doesn't sanity check file names or who made the same mistake as oneself in allowing arbitrary code execution (now instead of being limited to the www-user, an attacker can use {{{nasty_find_all}}} to traverse chroot jails and/or gain additional privileges), uses an IRC or IM client that's too liberal in the filenames it accepts for file transfers or conversation logs, etc. [[Anchor(faq49)]] == How can I view periodic updates/appends to a file? (ex: growing log file) == {{{tail -f}}} will show you the growing log file. On some systems (e.g. OpenBSD), this will automatically track a rotated log file to the new file with the same name (which is usually what you want). To get the equivalent functionality on GNU systems, use {{{tail --follow=name}}} instead. This is helpful if you need to view only the updates to the file after your last view. {{{ # Start by setting n=1 tail -n $n testfile; n="+$(( $(wc -l < testfile) + 1 ))" }}} Every invocation of this gives the update to the file from where we stopped last. If you know the line number from where you want to start, set n to that. [[Anchor(faq50)]] == I'm trying to construct a command dynamically, but I can't figure out how to deal with quoted multi-word arguments. == Some people attempt to do things like this: {{{ # Non-working example args="-s 'The subject' $address" mail $args < $body }}} This fails because of word-splitting. When {{{$args}}} is evaluated, it becomes four words: {{{'The}}} is the second word, and {{{subject'}}} is the third word. What's needed is a way to maintain each word as a separate item, even if that word contains multiple spaces. Quotes won't do it, but an array will. {{{ # Working example args=(-s "The subject" "$address") mail "${args[@]}" < $body }}} Usually, this question arises when someone is trying to use {{{dialog}}} to construct a menu on the fly. For an example of how to do this properly, see [#faq40 FAQ #40] above. [[Anchor(faq51)]] == I want history-search just like in tcsh. How can I bind it to the up and down keys? == Just add the following to /etc/inputrc or your ~/.inputrc {{{ "\e[A":history-search-backward "\e[B":history-search-forward }}} [[Anchor(faq52)]] == How do I convert a file in DOS format to UNIX format. ( Remove CRLF line terminators ) == All these are from the sed one-liners page {{{ sed 's/.$//' dosfile # assumes that all lines end with CR/LF sed 's/^M$//' dosfile # in bash/tcsh, press Ctrl-V then Ctrl-M sed 's/\x0D$//' dosfile }}} Some distributions have ''dos2unix'' command which can do this. In vim, you can use '':set fileformat=unix'' [[Anchor(faq53)]] == I have a fancy prompt with colors, and now bash doesn't seem to know how wide my terminal is. Lines wrap around incorrectly. == You must put {{{\[}}} and {{{\]}}} around any non-printing escape sequences in your prompt. Thus: {{{ BLUE=$(tput setaf 4) PURPLE=$(tput setaf 5) BLACK=$(tput setaf 0) PS1='\[$BLUE\]\h:\[$PURPLE\]\w\[$BLACK\]\$ ' }}} Without the {{{\[ \]}}}, bash will think the bytes which constitute the escape sequences for the color codes will actually take up space on the screen, so bash won't be able to know where the cursor actually is. [[Anchor(faq54)]] == How can I tell whether a variable contains a valid number? == First, you have to define what you mean by "number". The most common case seems to be that, when people ask this, they actually mean "a non-negative integer, with no leading + sign". {{{ if [[ $foo = *[^0-9]* ]]; then echo "'$foo' has a non-digit somewhere in it" else echo "'$foo' is strictly numeric" fi }}} This can be done in Korn and legacy Bourne shells as well, using {{{case}}}: {{{ case "$foo" in *[!0-9]*) echo "'$foo' has a non-digit somewhere in it" ;; *) echo "'$foo' is strictly numeric" ;; esac }}} If what you actually mean is "a valid floating-point number" or something else more complex, then you might prefer to use a regular expression. Bash version 3 and above have regular expression support in the [[ command: {{{ if [[ $foo =~ ^[-+]?[0-9]+\(\.[0-9]+\)?$ ]]; then echo "'$foo' looks rather like a number" else echo "'$foo' doesn't look particularly numeric to me" fi }}} If you don't have bash version 3, then you would use {{{egrep}}}: {{{ if echo "$foo" | egrep '^[-+]?[0-9]+(\.[0-9]+)?$' >/dev/null; then echo "'$foo' might be a number" else echo "'$foo' might not be a number" fi }}} Note that the parentheses in the {{{egrep}}} regular expression don't require backslashes in front of them, whereas the ones in the bash3 command do. [[Anchor(faq55)]] == Tell me all about 2>&1 -- what's the difference between 2>&1 >foo and >foo 2>&1, and when do I use which? == Bash processes all redirections from left to right, in order. And the order is significant. Moving them around within a command may change the results of that command. For newbies who've somehow managed to miss the previous hundred or so examples, here's what you want: {{{ foo >file 2>&1 # Sends both stdout and stderr to file. }}} Now for the rest of you, here's a simple demonstration of what's happening: {{{ foo() { echo "This is stdout" echo "This is stderr" 1>&2 } foo >/dev/null 2>&1 # produces no output foo 2>&1 >/dev/null # writes "This is stderr" on the screen }}} Why do the results differ? In the first case, {{{>/dev/null}}} is performed first, and therefore the standard output of the command is sent to {{{/dev/null}}}. Then, the {{{2>&1}}} is performed, which causes standard error to be sent to the same place that standard output is ''already'' going. So both of them are discarded. In the second example, {{{2>&1}}} is performed first. This means standard error is sent to wherever standard output happens to be going -- in this case, the user's terminal. Then, standard output is sent to {{{/dev/null}}} and is therefore discarded. So when we run {{{foo}}} the second time, we see only its standard error, not its standard output. There are times when we really do want {{{2>&1}}} to appear first -- for one example of this, see [#faq40 FAQ 40]. There are other times when we may use {{{2>&1}}} without any other redirections. Consider: {{{ find ... 2>&1 | grep "some error" }}} In this example, we want to search {{{find}}}'s standard error (as well as its standard output) for the string "some error". The {{{2>&1}}} in the piped command forces standard error to go into the pipe along with standard output. (When pipes and redirections are mixed in this way, remember: the pipe is done ''first'', before any redirections. So {{{find}}}'s standard output is already set to point to the pipe before we process the {{{2>&1}}} redirection.) If we wanted to read ''only'' standard error in the pipe, and discard standard output, we could do it like this: {{{ find ... 2>&1 >/dev/null | grep "some error" }}} The redirections in that example are processed thus: 1. First, the pipe is created. {{{find}}}'s output is sent to it. 1. Next, {{{2>&1}}} causes {{{find}}}'s standard error to go to the pipe as well. 1. Finally, {{{>/dev/null}}} causes {{{find}}}'s standard output to be discarded, leaving only stderr going into the pipe. A related question is [#faq47 FAQ #47], which discusses how to send stderr to a pipeline. [[Anchor(faq56)]] == How can I untar or unzip multiple tarballs at once? == As the {{{tar}}} command was originally designed to read from and write to tape devices (tar - Tape ARchiver), you can specify only filenames to put inside an archive or to extract out of an archive (e.g. {{{tar x myfileonthe.tape}}}). There is an option to tell {{{tar}}} that the archive is not on some tape, but in a file: {{{-f}}}. This option takes exactly one argument: the filename of the file containing the archive. All other (following) filenames are taken to be archive members: {{{ tar -x -f backup.tar myfile.txt # OR (more common syntax IMHO) tar xf backup.tar myfile.txt }}} Now here's a common mistake -- imagine a directory containing the following archive-files you want to extract all at once: {{{ $ ls backup1.tar backup2.tar backup3.tar }}} Maybe you think of {{{tar xf *.tar}}}. Let's see: {{{ $ tar xf *.tar tar: backup2.tar: Not found in archive tar: backup3.tar: Not found in archive tar: Error exit delayed from previous errors }}} What happened? The shell replaced your *.tar by the matching filenames. You really wrote: {{{ tar xf backup1.tar backup2.tar backup3.tar }}} And as we saw earlier, it means: "extract the files backup2.tar and backup3.tar from the archive backup1.tar", which will of course only succeed when there are such filenames stored in the archive. The solution is relatively easy: extract the contents of all archives '''one at a time'''. As we use a UNIX shell and we are lazy, we do that with a loop: {{{ for tarname in *.tar; do tar xf "$tarname" done }}} What happens? The for-loop will iterate through all filenames matching {{{*.tar}}} and call {{{tar xf}}} for each of them. That way you extract all archives one-by-one and you even do it automagically. The second common archive type in these days is ZIP. The command to extract contents from a ZIP file is {{{unzip}}} (who would have guessed that!). The problem here is the very same: {{{unzip}}} takes only one option specifying the ZIP-file. So, you solve it the very same way: {{{ for zipfile in *.zip; do unzip "$zipfile" done }}} Not enough? Ok. There's another option with {{{unzip}}}: it can take shell-like patterns to specify the ZIP-file names. And to avoid interpretion of those patterns by the shell, you need to quote them. {{{unzip}}} itself and '''not''' the shell will interpret {{{*.zip}}} in this case: {{{ unzip "*.zip" # OR, to make more clear what we do: unzip \*.zip }}} (This feature of {{{unzip}}} derives mainly from its origins as an MS-DOS program. MS-DOS's command interpreter does not perform glob expansions, so every MS-DOS program must be able to expand wildcards into a list of filenames. This feature was left in the Unix version, and as we just demonstrated, it can occasionally be useful.) [[Anchor(faq57)]] == How can group entries (in a file by common prefixes)? == as in, convert: {{{ foo: entry1 bar: entry2 foo: entry3 baz: entry4 }}} to {{{ foo: entry1 entry3 bar: entry2 baz: entry4 }}} there are two simple general methods for this: a. sort the file, and then iterate over it, collectin entries until the prefix changes, and then print the collected entries with the previous prefix b iterate over the file, collect entries for each prefix in an array indexed by the prefix a basic implementation of a) in bash: {{{ old=xxx ; stuff= (sort file ; echo xxx) | while read prefix line ; do if [[ $prefix = $old ]] ; then stuff="$stuff $line" else echo "$old: $stuff" old="$prefix" stuff= fi done }}} and a basic implementation of b) in awk: {{{ { a[$1] = a[$1] " " $2 } END{ for (x in a) print x, a[x] } }}} usage: {{{ awk '{a[$1] = a[$1] " " $2}END{for (x in a) print x, a[x]}' file }}} [[Anchor(faq58)]] == Can bash handle binary data? == the answer is, basically no... while bash won't have as much problems with it as older shells, it still can't process arbitrary binary data, and more specifically, shell variables are not 100% binary clean, so you can't store binary files in them. one instance where such would sometimes be handy is for example storing small temporary bitmaps while working with netpbm... here i resorted to adding an extra pnmnoraw to the pipe, creating (larger) ascii files that bash has no problems storing) if you are feeling adventurous, consider this experiment: {{{ # bindec.bash, attempt to decode binary data to ascii decimals IFS= while read -n1 x ;do case "$x" in '') echo empty ;; # insert the 256 lines generated by the following oneliner here: # for x in $(seq 0 255) ;do echo " $'\\$(printf %o $x)') echo $x;;" ;done esac done }}} and then pipe binary data into it, maybe like so: {{{ for x in $(seq 0 255) ;do echo -ne "\\$(printf %o $x)" ;done | bash bindec.bash | nl | less }}} this suggests that a the 0 character is skipped entirely, because we can't create it with the input generation, enough to conveniently corrupt most binary files we try to process (note that this refers to storing them in variables... moving data between programs using pipes is always binary clean) [[Anchor(faq59)]] == How can I remove the last character of a line? == Using bash and ksh extended parameter substitution: {{{ var=${var%?} }}} Remember that ${var%foo} removes foo from the end, and ${var#foo} removes foo from the beginning, of {{{var}}}. As a mnemonic, # appears to the left of % on the keyboard (US keyboards, at least). More portable, but slower: {{{ var=`expr "$var" : '\(.*\).'` }}} or (using {{{sed}}}): {{{ var=`echo "$var" | sed 's/.$//'` }}} [[Anchor(faq60)]] == I'm trying to write a script that will change directory (or set a variable), but after the script finishes, I'm back where I started (or my variable isn't set)! == Consider this: {{{ #!/bin/sh cd /tmp }}} If one executes this simple script, what happens? Bash forks, and the parent waits. The child executes the script, including the {{{chdir(2)}}} system call, and then exits. The parent, which was waiting for the child, harvests the child's exit status (presumably 0 for success), and then bash carries on with the next command. Since the {{{chdir}}} was done by a child process, it has no effect on the parent. Moreover, there is '''no conceivable way''' you can ''ever'' have a child process affect ''any'' part of the parent's environment, which includes its variables as well as its current working directory. So, how does one go about it? You can still have the {{{cd}}} command in an external file, but you can't ''run it'' as a script. Instead, you must {{{source}}} it (or "dot it in", using the {{{.}}} command, which is a synonym for {{{source}}}). {{{ echo 'cd /tmp' > $HOME/mycd source $HOME/mycd pwd # Now, we're in /tmp }}} The same thing applies to setting variables. {{{source}}} the file that contains the commands; don't try to run it. [[Anchor(faq61)]] == Is there a list of which features were added to specific releases of Bash? == * [http://cnswww.cns.cwru.edu/~chet/bash/NEWS NEWS]: a file tersely listing the notable changes between the current and previous versions * [http://cnswww.cns.cwru.edu/~chet/bash/CHANGES CHANGES]: a complete bash change history * [http://cnswww.cns.cwru.edu/~chet/bash/COMPAT COMPAT]: compatibility issues between bash3 and previous versions Here's a ''partial'' list of the changes, in a more compact format: ||'''Feature'''||'''Added in version'''|| ||x+=string||3.1-alpha1|| ||{x..y}||3.0-alpha|| ||${!array[@]}||3.0-alpha|| ||[[ =~||3.0-alpha|| ||<<<||2.05b-alpha1|| ||i++||2.04-devel|| ||for ((;;))||2.04-devel|| ||/dev/fd/N, /dev/tcp/host/port, etc.||2.04-devel|| ||a=(*.txt) file expansion||2.03-alpha|| ||extglob||2.02-alpha1|| ||[[||2.02-alpha1|| ||builtin printf||2.02-alpha1|| ||$(< filename)||2.02-alpha1|| ||** (exponentiation)||2.02-alpha1|| ||\xNNN||2.02-alpha1|| ||(( ))||2.0-beta2|| [[Anchor(faq62)]] == How do I create a temporary file in a secure manner? == Good question. To be filled in later. (Interim hints: {{{tempfile}}} is not portable. {{{mktemp}}} exists more widely, but it may require a {{{-c}}} switch to create the file in advance; or it may create the file by default and barf if {{{-c}}} is supplied. There does not appear to be any single command that simply ''works'' everywhere, without testing various arguments.) [[Anchor(faq63)]] == My ssh client hangs when I try to run a remote background job! == The following will not do what you expect: {{{ ssh me@remotehost 'sleep 120 &' # Client hangs for 120 seconds }}} This is a "feature" of [http://www.openssh.org/ OpenSSH]. The client will not close the connection as long as the remote end's terminal still is still in use -- and in the case of {{{sleep 120 &}}}, stdout and stderr are still connected to the terminal. The immediate answer to your question -- "How do I get the client to disconnect so I can get my shell back?" -- is to kill the ssh client. You can do this with the {{{kill}}} or {{{pkill}}} commands, of course; or by sending the INT signal (usually Ctrl-C) for a non-interactive ssh session (as above); or by pressing '''<Enter><~><.>''' (Enter, Tilde, Period) in the client's terminal window for an interactive remote shell. The long-term workaround for this is to ensure that all the file descriptors are redirected to a log file (or {{{/dev/null}}}) on the remote side: {{{ ssh me@remotehost 'sleep 120 >/dev/null 2>&1 &' # Client should return immediately }}} This also applies to restarting daemons on some legacy Unix systems. {{{ ssh root@hp-ux-box # Interactive shell ... # Discover that the problem is stale NFS handles /sbin/init.d/nfs.client stop # autofs is managed by this script and /sbin/init.d/nfs.client start # killing it on HP-UX is OK (unlike Linux) exit # Client hangs -- use Enter ~ . to kill it. }}} The legacy Unix {{{/sbin/init.d/nfs.client}}} script runs daemons in the background but leaves their stdout and stderr attached to the terminal (and they don't fully self-daemonize). The solution is either to fix the Unix vendor's broken init script, or to kill the ssh client process after this happens. The author of this article uses the latter approach. [[Anchor(faq64)]] == Why is it so hard to get an answer to the question that I asked in #bash ? == * #bash aphorism #1 "The questioner's first description of the problem/question will be misleading." * corollary 1.1 "The questioner's second description of the problem/question will also be misleading" * corollary 1.2 "The questioner is never precise" ex: will say "print the file" when they mean print the file's name, rather than printing the file itself." * #bash aphorism #2, "The questioner will keep changing their original question until it drives the helpers in the channel insane." * #bash aphorism #3, "The data is never formatted in the way that makes it easiest to manipulate :-)" * #bash aphorism #4, "30 to 40 percent of the conversations in #bash will be about aphorisms #1 and #2" [[Anchor(faq65)]] == Is there a "PAUSE" command in bash like there is in MSDOS batch scripts? == No, but you can use these: {{{ echo press enter to continue; read }}} {{{ echo press any key to continue; read -n 1 }}} [[Anchor(faq66)]] == I want to check if [[ $var == foo || $var == bar || $var = more ]] without repeating $var n times. == {{{ case $var in foo|bar|more) ... ;; esac }}} [[Anchor(faq67)]] == How can I trim leading/trailing white space from one of my variables? == There are a few ways to do this -- none of them elegant. First, the most portable way would be to use sed: {{{ x=$(echo "$x" | sed -e 's/^ *//' -e 's/ *$//') # Note: this only removes spaces. For tabs too: x=$(echo "$x" | sed -e $'s/^[ \t]*//' -e $'s/[ \t]*$//') # Or possibly, with some systems: x=$(echo "$x" | sed -e 's/^[[:space:]]\+//' -e 's/[[:space:]]\+$//') }}} One can achieve the goal using builtins, although at the moment I'm not sure which shells the following syntax supports: {{{ # Remove leading whitespace: while [[ $x = [$' \t\n']* ]]; do x=${x#[$' \t\n']}; done # And now trailing: while [[ $x = *[$' \t\n'] ]]; do x=${x%[$' \t\n']}; done }}} Of course, the preceding example is pretty slow, because it removes one character at a time, in a loop (although it's good enough in practice for most purposes). If you want something a bit fancier, there's a bash-only solution using extglob: {{{ shopt -s extglob x=${x##*([$' \t\n'])}; x=${x%%*([$' \t\n'])} shopt -u extglob }}} There are many, many other ways to do this. These are not necessarily the most efficient, but they're known to work. [[Anchor(faq68)]] == How do I run a command, and have it abort (timeout) after N seconds? == There are two C programs that can do this: [http://pilcrow.madison.wi.us/ doalarm], and [http://www.porcupine.org/forensics/tct.html timeout]. (Compiling them is beyond the scope of this document; suffice to say, it'll be trivial on GNU/Linux systems, easy on most BSDs, and painful on anything else....) If you don't have or don't want one of the above two programs, you can use a perl one-liner to set an ALRM and then exec the program you want to run under a time limit. In any case, you must understand what your program does with SIGALRM. {{{ function doalarm () { perl -e 'alarm shift; exec @ARGV' "$@" ; } doalarm ${NUMBER_OF_SECONDS_BEFORE_ALRMING} program arg arg ... }}} If you can't or won't install one of these programs (which ''really'' should have been included with the basic core Unix utilities 30 years ago!), then the best you can do is an ugly hack like: {{{ command & pid=$!; { sleep 10 && kill $pid; } & }}} This will, as you will soon discover, produce quite a mess regardless of whether the timeout condition kicked in or not. Cleaning it up is not something worth my time -- just use {{{doalarm}}} or {{{timeout}}} instead. Really. |
---- CategoryShell |
BASH Frequently Asked Questions
Note: The FAQ was split into individual pages for easier editing. Also, for faster loading of this page, the answers are no longer presented here in their entirety. |
These are answers to frequently asked questions on channel #bash on the freenode IRC network. These answers are contributed by the regular members of the channel (originally heiner, and then others including greycat and r00t), and by users like you. If you find something inaccurate or simply misspelled, please feel free to correct it!
All the information here is presented without any warranty or guarantee of accuracy. Use it at your own risk. When in doubt, please consult the man pages or the GNU info pages as the authoritative references.
BASH is a BourneShell compatible shell, which adds many new features to its ancestor. Most of them are available in the KornShell, too. The answers given in this FAQ may be slanted toward Bash, or they may be slanted toward the lowest common denominator Bourne shell, depending on who wrote the answer. In most cases, an effort is made to provide both a portable (Bourne) and an efficient (Bash, where appropriate) answer. If a question is not strictly shell specific, but rather related to Unix, it may be in the UnixFaq.
This FAQ assumes a certain level of familiarity with basic shell script syntax. If you're completely new to Bash or to the Bourne family of shells, you may wish to start with the BashGuide.
More advanced users may wish to read BashPitfalls and BashProgramming.
If you want to help, you can add new questions with answers here, or try to answer one of the BashOpenQuestions.
Chet Ramey's official Bash FAQ contains many technical questions not covered here.
Contents
- How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?
- How can I store the return value and/or output of a command in a variable?
- How can I sort or compare files based on some metadata attribute (most recently modified, size, etc)?
- How can I check whether a directory is empty or not? How do I check for any *.mpg files, or count how many there are?
- How can I use array variables?
- How can I use variable variables (indirect variables, pointers, references) or associative arrays?
- Is there a function to return the length of a string?
- How can I recursively search all files for a string?
- What is buffering? Or, why does my command line produce no output: tail -f logfile | grep 'foo bar' | awk ...
- How can I recreate a directory hierarchy structure, without the files?
- How can I print the n'th line of a file?
- How do I invoke a shell command from a non-shell application?
- How can I concatenate two variables? How do I append a string to a variable?
- How can I redirect the output of multiple commands at once?
- How can I run a command on all files with the extension .gz?
- How can I use a logical AND/OR/NOT in a shell pattern (glob)?
- How can I group expressions in an if statement, e.g. if (A AND B) OR C?
- How can I use numbers with leading zeros in a loop, e.g. 01, 02?
- How can I split a file into line ranges, e.g. lines 1-10, 11-20, 21-30?
- How can I find and safely handle file names containing newlines, spaces or both?
- How can I replace a string with another string in a variable, a stream, a file, or in all the files in a directory?
- How can I calculate with floating point numbers instead of just integers?
- I want to launch an interactive shell that has special aliases and functions, not the ones in the user's ~/.bashrc.
- I set variables in a loop that's in a pipeline. Why do they disappear after the loop terminates? Or, why can't I pipe data to read?
- How can I access positional parameters after $9?
- How can I randomize (shuffle) the order of lines in a file? Or select a random line from a file, or select a random file from a directory?
- How can two unrelated processes communicate?
- How do I determine the location of my script? I want to read some config files from the same place.
- How can I display the target of a symbolic link?
- How can I rename all my *.foo files to *.bar, or convert spaces to underscores, or convert upper-case file names to lower case?
- What is the difference between test, [ and [[ ?
- How can I redirect the output of 'time' to a variable or file?
- How can I find a process ID for a process given its name?
- Can I do a spinner in Bash?
- How can I handle command-line options and arguments in my script easily?
- How can I get all lines that are: in both of two files (set intersection) or in only one of two files (set subtraction).
- How can I print text in various colors?
- How do Unix file permissions work?
- What are all the dot-files that bash reads?
- How do I use dialog to get input from the user?
- How do I determine whether a variable contains a substring?
- How can I find out if a process is still running?
- Why does my crontab job fail? 0 0 * * * some command > /var/log/mylog.`date +%Y%m%d`
- How do I create a progress bar? How do I see a progress indicator when copying/moving files?
- How can I ensure that only one instance of a script is running at a time (mutual exclusion, locking)?
- I want to check to see whether a word is in a list (or an element is a member of a set).
- How can I redirect stderr to a pipe?
- Eval command and security issues
- How can I view periodic updates/appends to a file? (ex: growing log file)
- I'm trying to put a command in a variable, but the complex cases always fail!
- I want history-search just like in tcsh. How can I bind it to the up and down keys?
- How do I convert a file from DOS format to UNIX format (remove CRs from CR-LF line terminators)?
- I have a fancy prompt with colors, and now bash doesn't seem to know how wide my terminal is. Lines wrap around incorrectly.
- How can I tell whether a variable contains a valid number?
- Tell me all about 2>&1 -- what's the difference between 2>&1 >foo and >foo 2>&1, and when do I use which?
- How can I untar (or unzip) multiple tarballs at once?
- How can I group entries (in a file by common prefixes)?
- Can bash handle binary data?
- I saw this command somewhere: :(){ :|:& } (fork bomb). How does it work?
- I'm trying to write a script that will change directory (or set a variable), but after the script finishes, I'm back where I started (or my variable isn't set)!
- Is there a list of which features were added to specific releases (versions) of Bash?
- How do I create a temporary file in a secure manner?
- My ssh client hangs when I try to logout after running a remote background job!
- Why is it so hard to get an answer to the question that I asked in #bash?
- Is there a "PAUSE" command in bash like there is in MSDOS batch scripts? To prompt the user to press any key to continue?
- I want to check if [[ $var == foo || $var == bar || $var == more ]] without repeating $var n times.
- How can I trim leading/trailing white space from one of my variables?
- How do I run a command, and have it abort (timeout) after N seconds?
- I want to automate an ssh (or scp, or sftp) connection, but I don't know how to send the password....
- How do I convert Unix (epoch) times to human-readable values?
- How do I convert an ASCII character to its decimal (or hexadecimal) value and back? How do I do URL encoding or URL decoding?
- How can I ensure my environment is configured for cron, batch, and at jobs?
- How can I use parameter expansion? How can I get substrings? How can I get a file without its extension, or get just a file's extension? What are some good ways to do basename and dirname?
- How do I get the effects of those nifty Bash Parameter Expansions in older shells?
- How do I use 'find'? I can't understand the man page at all!
- How do I get the sum of all the numbers in a column?
- How do I log history or "secure" bash against history removal?
- I want to set a user's password using the Unix passwd command, but how do I script that? It doesn't read standard input!
- How can I grep for lines containing foo AND bar, foo OR bar? Or for files containing foo AND bar, possibly on separate lines? Or files containing foo but NOT bar?
- How can I make an alias that takes an argument?
- How can I determine whether a command exists anywhere in my PATH?
- Why is $(...) preferred over `...` (backticks)?
- How do I determine whether a variable is already defined? Or a function?
- How do I return a string (or large number, or negative number) from a function? "return" only lets me give a number from 0 to 255.
- How to write several times to a fifo without having to reopen it?
- How to ignore aliases, functions, or builtins when running a command?
- How can I get a file's permissions (or other metadata) without parsing ls -l output?
- How can I avoid losing any history lines?
- I'm reading a file line by line and running ssh or ffmpeg, only the first line gets processed!
- How do I prepend a text to a file (the opposite of >>)?
- I'm trying to get the number of columns or lines of my terminal but the variables COLUMNS / LINES are always empty.
- How do I write a CGI script that accepts parameters?
- How can I set the contents of my terminal's title bar?
- I want to get an alert when my disk is full (parsing df output).
- I'm getting "Argument list too long". How can I process a large list in chunks?
- ssh eats my word boundaries! I can't do ssh remotehost make CFLAGS="-g -O"!
- How do I determine whether a symlink is dangling (broken)?
- How to add localization support to your bash scripts
- How can I get the newest (or oldest) file from a directory?
- How do I do string manipulations in bash?
- Common utility functions (warn, die)
- How to get the difference between two dates
- How do I check whether my file was modified in a certain month or date range?
- Why doesn't foo=bar echo "$foo" print bar?
- Why doesn't set -e (or set -o errexit, or trap ERR) do what I expected?
- Logging! I want to send all of my script's output to a log file. But I want to do it from inside the script. And I want to see it on the terminal too!
- How do I add a timestamp to every line of a stream?
- How do I wait for several spawned processes?
- How can I tell whether my script was sourced (dotted in) or executed?
- How do I copy a file to a remote system, and specify a remote name which may contain spaces?
- What is the Shellshock vulnerability in Bash?
- What are the advantages and disadvantages of using set -u (or set -o nounset)?
- How do I extract data from an HTML or XML file?
- How do I operate on IP addresses and netmasks?
- How do I make a menu?
- I have two files. The first one contains bad IP addresses (plus other fields). I want to remove all of these bad addresses from a second file.
- I have a pipeline where a long-running command feeds into a filter. If the filter finds "foo", I want the long-running command to die.
- How do I print the contents of an array in reverse order, or reverse an array?
- What's the difference between "cmd < file" and "cat file | cmd"? What is a UUOC?
- How can I find out where this strange variable in my interactive shell came from?
- What does value too great for base mean? (Octal values in arithmetic.)
- How do I print a horizontal line of characters like ----?
1. How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?
Don't try to use "for". Use a while loop and the read command. Here is the basic template; there are many variations to discuss:
2. How can I store the return value and/or output of a command in a variable?
Well, that depends on whether you want to store the command's output (either stdout, or stdout + stderr) or its exit status (0 to 255, with 0 typically meaning "success").
3. How can I sort or compare files based on some metadata attribute (most recently modified, size, etc)?
The tempting solution is to use ls to output sorted filenames and operate on the results using e.g. awk. As usual, the ls approach cannot be made robust and should never be used in scripts due in part to the possibility of arbitrary characters (including newlines) present in filenames. Therefore, we need some other way to compare file metadata.
4. How can I check whether a directory is empty or not? How do I check for any *.mpg files, or count how many there are?
In Bash, you can count files safely and easily with the nullglob and dotglob options (which change the behaviour of globbing), and an array:
5. How can I use array variables?
This answer assumes you have a basic understanding of what arrays are. If you're new to this kind of programming, you may wish to start with the guide's explanation. This page is more thorough. See links at the bottom for more resources.
6. How can I use variable variables (indirect variables, pointers, references) or associative arrays?
This is a complex page, because it's a complex topic. It's been divided into roughly four parts: associative arrays, name references, evaluating indirect variables, and assigning indirect variables. There are discussions of programming issues and concepts scattered throughout.
7. Is there a function to return the length of a string?
The fastest way, not requiring external programs (but not usable in Bourne shells):
8. How can I recursively search all files for a string?
If you are on a typical GNU or BSD system, all you need is one of these:
9. What is buffering? Or, why does my command line produce no output: tail -f logfile | grep 'foo bar' | awk ...
Most standard Unix commands buffer their output when used non-interactively. This means that they don't write each character (or even each line) immediately, but instead collect a larger number of characters (often 4 kilobytes) before printing anything at all. In the case above, the grep command buffers its output, and therefore awk only gets its input in large chunks.
10. How can I recreate a directory hierarchy structure, without the files?
11. How can I print the n'th line of a file?
One dirty (but not quick) way is:
12. How do I invoke a shell command from a non-shell application?
You can use the shell's -c option to run the shell with the sole purpose of executing a short bit of script:
13. How can I concatenate two variables? How do I append a string to a variable?
There is no (explicit) concatenation operator for strings (either literal or variable dereferences) in the shell; you just write them adjacent to each other:
14. How can I redirect the output of multiple commands at once?
Redirecting the standard output of a single command is as easy as:
15. How can I run a command on all files with the extension .gz?
Often a command already accepts several files as arguments, e.g.
16. How can I use a logical AND/OR/NOT in a shell pattern (glob)?
"Globs" are simple patterns that can be used to match filenames or strings. They're generally not very powerful. If you need more power, there are a few options available.
17. How can I group expressions in an if statement, e.g. if (A AND B) OR C?
The portable (POSIX or Bourne) way is to use multiple test (or [) commands:
18. How can I use numbers with leading zeros in a loop, e.g. 01, 02?
As always, there are many different ways to solve the problem, each with its own advantages and disadvantages. The most important considerations are which shell you're using, whether the start/end numbers are constants, and how many times the loop is going to iterate.
19. How can I split a file into line ranges, e.g. lines 1-10, 11-20, 21-30?
POSIX specifies the split utility, which can be used for this purpose:
20. How can I find and safely handle file names containing newlines, spaces or both?
First and foremost, to understand why you're having trouble, read Arguments to get a grasp on how the shell understands the statements you give it. It is vital that you grasp this matter well if you're going to be doing anything with the shell.
21. How can I replace a string with another string in a variable, a stream, a file, or in all the files in a directory?
There are a number of techniques for this. Which one to use depends on many factors, the biggest of which is what we're editing. This page also contains contradictory advice from multiple authors. This is a deeply ugly topic, and there are no universally right answers (but plenty of universally wrong ones).
22. How can I calculate with floating point numbers instead of just integers?
BASH's builtin arithmetic uses integers only:
$ printf '%s\n' "$((10 / 3))" 3
23. I want to launch an interactive shell that has special aliases and functions, not the ones in the user's ~/.bashrc.
When starting bash in non-POSIX mode, specify a different start-up file with --rcfile:
24. I set variables in a loop that's in a pipeline. Why do they disappear after the loop terminates? Or, why can't I pipe data to read?
In most shells, each command of a pipeline is executed in a separate SubShell. Non-working example:
25. How can I access positional parameters after $9?
Use ${10} instead of $10. This works for BASH and KornShell, but not for older BourneShell implementations. Another way to access arbitrary positional parameters after $9 is to use for, e.g. to get the last parameter:
26. How can I randomize (shuffle) the order of lines in a file? Or select a random line from a file, or select a random file from a directory?
To randomize the lines of a file, here is one approach. This one involves generating a random number, which is prefixed to each line; then sorting the resulting lines, and removing the numbers.
27. How can two unrelated processes communicate?
Two unrelated processes cannot use the arguments, the environment or stdin/stdout to communicate; some form of inter-process communication (IPC) is required.
28. How do I determine the location of my script? I want to read some config files from the same place.
There are two prime reasons why this issue comes up: either you want to externalize data or configuration of your script and need a way to find these external resources, or your script is intended to act upon a bundle of some sort (eg. a build script), and needs to find the resources to act upon.
29. How can I display the target of a symbolic link?
The nonstandard external command readlink(1) can be used to display the target of a symbolic link:
30. How can I rename all my *.foo files to *.bar, or convert spaces to underscores, or convert upper-case file names to lower case?
There are a bunch of different ways to do this, depending on which nonstandard tools you have available. Even with just standard POSIX tools, you can still perform most of the simple cases. We'll show the portable tool examples first.
31. What is the difference between test, [ and [[ ?
The open square bracket [ command (aka test command) and the [[ ... ]] test construct are used to evaluate expressions. [[ ... ]] works only in the Korn shell (where it originates), Bash, Zsh, and recent versions of Yash and busybox sh (if enabled at compilation time, and still very limited there especially in the hush-based variant), and is more powerful; [ and test are POSIX utilities (generally builtin). POSIX doesn't specify the [[ ... ]] construct (which has a specific syntax with significant variations between implementations) though allows shells to treat [[ as a keyword. Here are some examples:
32. How can I redirect the output of 'time' to a variable or file?
Bash's time keyword uses special trickery, so that you can do things like
33. How can I find a process ID for a process given its name?
Usually a process is referred to using its process ID (PID), and the ps(1) command can display the information for any process given its process ID, e.g.
34. Can I do a spinner in Bash?
Sure!
35. How can I handle command-line options and arguments in my script easily?
Well, that depends a great deal on what you want to do with them. There are two standard approaches, each with its strengths and weaknesses.
36. How can I get all lines that are: in both of two files (set intersection) or in only one of two files (set subtraction).
Use the comm(1) command:
37. How can I print text in various colors?
Do not hard-code ANSI color escape sequences in your program! The tput command lets you interact with the terminal database in a sane way:
40. How do I use dialog to get input from the user?
Here is an example:
42. How can I find out if a process is still running?
The kill command is used to send signals to a running process. As a convenience function, the signal "0", which does not exist, can be used to find out if a process is still running:
43. Why does my crontab job fail? 0 0 * * * some command > /var/log/mylog.`date +%Y%m%d`
In many versions of crontab, the percent sign (%) is treated specially, and therefore must be escaped with backslashes:
44. How do I create a progress bar? How do I see a progress indicator when copying/moving files?
The easiest way to add a progress bar to your own script is to use dialog --gauge. Here is an example, which relies heavily on BASH features:
45. How can I ensure that only one instance of a script is running at a time (mutual exclusion, locking)?
We need some means of mutual exclusion. One way is to use a "lock": any number of processes can try to acquire the lock simultaneously, but only one of them will succeed.
46. I want to check to see whether a word is in a list (or an element is a member of a set).
If your real question is How do I check whether one of my parameters was -v? then see FAQ #35 instead. Otherwise, read on…
47. How can I redirect stderr to a pipe?
A pipe can only carry standard output (stdout) of a program. To pipe standard error (stderr) through it, you need to redirect stderr to the same destination as stdout. Optionally you can close stdout or redirect it to /dev/null to only get stderr. Some sample code:
48. Eval command and security issues
The eval command is extremely powerful and extremely easy to abuse.
49. How can I view periodic updates/appends to a file? (ex: growing log file)
tail -f will show you the growing log file. On some systems (e.g. OpenBSD), this will automatically track a rotated log file to the new file with the same name (which is usually what you want). To get the equivalent functionality on GNU systems, use tail -F instead.
50. I'm trying to put a command in a variable, but the complex cases always fail!
Variables hold data. Functions hold code. Don't put code inside variables! There are many situations in which people try to shove commands, or command arguments, into variables and then run them. Each case needs to be handled separately.
51. I want history-search just like in tcsh. How can I bind it to the up and down keys?
Just add the following to /etc/inputrc or your ~/.inputrc:
52. How do I convert a file from DOS format to UNIX format (remove CRs from CR-LF line terminators)?
Carriage return (CR) characters are used in line ending markers on some systems. There are three different kinds of line endings in common use:
53. I have a fancy prompt with colors, and now bash doesn't seem to know how wide my terminal is. Lines wrap around incorrectly.
53.1. Escape the colors with \[ \]
You must put \[ and \] around any non-printing escape sequences in your prompt. Thus:
54. How can I tell whether a variable contains a valid number?
First, you have to define what you mean by "number". The most common case when people ask this seems to be "a non-negative integer, with no leading + sign". Or in other words, a string of all digits. Other times, people want to validate a floating-point input, with optional sign and optional decimal point.
55. Tell me all about 2>&1 -- what's the difference between 2>&1 >foo and >foo 2>&1, and when do I use which?
Bash processes all redirections from left to right, in order. And the order is significant. Moving them around within a command may change the results of that command.
56. How can I untar (or unzip) multiple tarballs at once?
As the tar command was originally designed to read from and write to tape devices (tar - Tape ARchiver), you can specify only filenames to put inside an archive (write to tape) or to extract out of an archive (read from tape).
57. How can I group entries (in a file by common prefixes)?
As in, one wants to convert:
58. Can bash handle binary data?
The answer is, basically, no....
59. I saw this command somewhere: :(){ :|:& } (fork bomb). How does it work?
This is a potentially dangerous command. Don't run it! The "trigger" is omitted from the question above, leaving only the part that sets up the function.
60. I'm trying to write a script that will change directory (or set a variable), but after the script finishes, I'm back where I started (or my variable isn't set)!
Consider this:
#!/bin/sh cd /tmp
61. Is there a list of which features were added to specific releases (versions) of Bash?
Here are some links to official Bash documentation:
62. How do I create a temporary file in a secure manner?
There does not appear to be any single command that simply works everywhere. tempfile is not portable. mktemp exists more widely (but still not ubiquitously), but it may require a -c switch to create the file in advance; or it may create the file by default and barf if -c is supplied. Some systems don't have either command (Solaris, POSIX). POSIX systems are supposed to have m4 which has the ability to create a temporary file, but some systems may not install m4 by default, or their implementation of m4 may be missing this feature.
63. My ssh client hangs when I try to logout after running a remote background job!
The following will not do what you expect:
ssh me@remotehost 'sleep 120 &' # Client hangs for 120 seconds
64. Why is it so hard to get an answer to the question that I asked in #bash?
Maybe nobody knows the answer (or the people who know the answer are busy). Maybe you haven't given enough detail about the problem, or you haven't presented the problem clearly. Maybe the question you asked is answered in this FAQ, or in BashPitfalls, or in the BashGuide.
65. Is there a "PAUSE" command in bash like there is in MSDOS batch scripts? To prompt the user to press any key to continue?
Use the following to wait until the user presses enter:
66. I want to check if [[ $var == foo || $var == bar || $var == more ]] without repeating $var n times.
The portable solution uses case:
67. How can I trim leading/trailing white space from one of my variables?
There are a few ways to do this. Some involve special tricks that only work with whitespace. Others are more general, and can be used to strip leading zeroes, etc.
68. How do I run a command, and have it abort (timeout) after N seconds?
FIRST check whether the command you're running can be told to timeout directly. The methods described here are "hacky" workarounds to force a command to terminate after a certain time has elapsed. Configuring your command properly is always preferable to the alternatives below.
69. I want to automate an ssh (or scp, or sftp) connection, but I don't know how to send the password....
When dealing with authentication in a shell script, please bear in mind the following points:
70. How do I convert Unix (epoch) times to human-readable values?
The only sane way to handle time values within a program is to convert them into a linear scale. You can't store "January 17, 2005 at 5:37 PM" in a variable and expect to do anything with it....
71. How do I convert an ASCII character to its decimal (or hexadecimal) value and back? How do I do URL encoding or URL decoding?
If you have a known octal or hexadecimal value (at script-writing time), you can just use printf:
72. How can I ensure my environment is configured for cron, batch, and at jobs?
If a shell or other script calling shell commands runs fine interactively but fails due to environment configurations (say: a complex $PATH) when run noninteractively, you'll need to force your environment to be properly configured.
73. How can I use parameter expansion? How can I get substrings? How can I get a file without its extension, or get just a file's extension? What are some good ways to do basename and dirname?
Parameter expansion is an important subject. This page contains a concise overview of parameter expansion.
74. How do I get the effects of those nifty Bash Parameter Expansions in older shells?
Most of the extended forms of parameter expansion do not work with the older BourneShell. If your code needs to be portable to that shell as well, sed and expr can often be used.
76. How do I get the sum of all the numbers in a column?
This and all similar questions are best answered with an AWK one-liner.
77. How do I log history or "secure" bash against history removal?
If you're a shell user who wants to record your own activities, see FAQ #88 instead. If you're a system administrator who wants to know how to find out what a user had executed when they unset or /dev/nulled their shell history, there are several problems with this....
78. I want to set a user's password using the Unix passwd command, but how do I script that? It doesn't read standard input!
OK, first of all, I know there are going to be some people reading this, right now, who don't even understand the question. Here, this does not work:
{ echo oldpass; echo newpass; echo newpass; } | passwd
# This DOES NOT WORK!
79. How can I grep for lines containing foo AND bar, foo OR bar? Or for files containing foo AND bar, possibly on separate lines? Or files containing foo but NOT bar?
This is really four different questions, so we'll break this answer into parts.
80. How can I make an alias that takes an argument?
You can't. Aliases in bash are extremely rudimentary, and not really suitable to any serious purpose. The bash man page even says so explicitly:
81. How can I determine whether a command exists anywhere in my PATH?
POSIX specifies a shell builtin called command which can be used for this purpose:
82. Why is $(...) preferred over `...` (backticks)?
`...` is the legacy syntax for command substitution, required by only the very oldest of non-POSIX-compatible Bourne shells. There are several reasons to always prefer the $(...) syntax:
83. How do I determine whether a variable is already defined? Or a function?
There are several ways to test these things, depending on the exact requirements. Most of the time, the desired test is whether a variable has a non-empty value. In this case, we may simply use:
84. How do I return a string (or large number, or negative number) from a function? "return" only lets me give a number from 0 to 255.
Functions in Bash (as well as all the other Bourne-family shells) work like commands: that is, they only "return" an exit status, which is an integer from 0 to 255 inclusive. This is intended to be used only for signaling errors, not for returning the results of computations, or other data.
85. How to write several times to a fifo without having to reopen it?
In the general case, you'll open a new FileDescriptor (FD) pointing to the fifo, and write through that. For simple cases, it may be possible to skip that step.
86. How to ignore aliases, functions, or builtins when running a command?
functions, builtins, external utilities, and aliases can all be defined with the same name at once. It's sometimes necessary specify which of these the shell should resolve while bypassing the others.
87. How can I get a file's permissions (or other metadata) without parsing ls -l output?
There are several potential ways, most of which are system-specific. They also depend on precisely why you want the information; in most cases, there will be some other way to accomplish your real goal. You don't want to parse ls's output if there's any possible way to avoid doing so.
88. How can I avoid losing any history lines?
89. I'm reading a file line by line and running ssh or ffmpeg, only the first line gets processed!
When reading a file line by line, if a command inside the loop also reads stdin, it can exhaust the input file. For example:
90. How do I prepend a text to a file (the opposite of >>)?
You cannot do it with bash redirections alone; the opposite of >> does not exist....
91. I'm trying to get the number of columns or lines of my terminal but the variables COLUMNS / LINES are always empty.
COLUMNS and LINES are set by BASH in interactive mode; they are not available by default in a script. On most systems, you can try to query the terminal yourself:
92. How do I write a CGI script that accepts parameters?
There are always circumstances beyond our control that drive us to do things that we would never choose to do on our own. This FAQ entry describes one of those situations.
93. How can I set the contents of my terminal's title bar?
If you have a terminal that understands xterm-compatible escape sequences, and you just want to set the title one time, you can use a function like this:
94. I want to get an alert when my disk is full (parsing df output).
Sadly, parsing the output of df really is the most reliable way to determine how full a disk is, on most operating systems. However, please note that this is a "least bad" answer, not a "best" answer. Parsing any command-line reporting tool's output in a program is never pretty. The purpose of this FAQ is to try to describe all the problems this approach is known to encounter, and work around them.
95. I'm getting "Argument list too long". How can I process a large list in chunks?
First, let's review some background material. When a process wants to run another process, it fork()s a child, and the child calls one of the exec* family of system calls (e.g. execve()), giving the name or path of the new process's program file; the name of the new process; the list of arguments for the new process; and, in some cases, a set of environment variables. Thus:
96. ssh eats my word boundaries! I can't do ssh remotehost make CFLAGS="-g -O"!
ssh emulates the behavior of the Unix remote shell command (rsh or remsh), including this bug. There are a few ways to work around it, depending on exactly what you need.
97. How do I determine whether a symlink is dangling (broken)?
The documentation on this is fuzzy, but it turns out you can do this with shell builtins:
98. How to add localization support to your bash scripts
Looking for examples of how to add simple localization to your bash scripts, and how to do testing? This is probably what you need....
99. How can I get the newest (or oldest) file from a directory?
This page should be merged with BashFAQ/003
100. How do I do string manipulations in bash?
Bash can do string operations. LOTS of string operations. This is an introduction to bash string manipulations and related techniques. It overlaps with the Parameter Expansion question, but the information here is presented in a more beginner-friendly manner (we hope).
101. Common utility functions (warn, die)
(If you were looking for option processing, see BashFAQ/035.) Bash and sh don't offer a die builtin command like Perl does, but it's common to use a die function in scripts. You just have to write one yourself. Most people who write a die function like to keep it simple. There are two common varieties: one that only takes a message to print, and one that takes a message and an exit status value.
102. How to get the difference between two dates
It's best if you work with timestamps throughout your code, and then only convert timestamps to human-readable formats for output. If you must handle human-readable dates as input, then you will need something that can parse them.
103. How do I check whether my file was modified in a certain month or date range?
Doing date-related math in Bash is hard because Bash has no builtins in place for doing math with dates or getting metadata such as modification time from files.
104. Why doesn't foo=bar echo "$foo" print bar?
This is subtle, and has to do with the exact order in which the BashParser performs each step.
105. Why doesn't set -e (or set -o errexit, or trap ERR) do what I expected?
set -e was an attempt to add "automatic error detection" to the shell. Its goal was to cause the shell to abort any time an error occurred, so you don't have to put || exit 1 after each important command. This does not work well in practice.
106. Logging! I want to send all of my script's output to a log file. But I want to do it from inside the script. And I want to see it on the terminal too!
Normally, if you want to run a script and send its output to a logfile, you'd simply use Redirection: myscript >log 2>&1. Or to see the output on the screen and also redirect to a file: myscript 2>&1 | tee log (or better still, run your script within the script(1) command if your system has it; script is intended for use with interactive shell sessions.). If you want to insert commands into a script that cause it to do this kind of logging internally, without altering your invocation, then it gets trickier.
107. How do I add a timestamp to every line of a stream?
Adding timestamps to a stream is a challenge, because there aren't any standard tools to do it. You either have to install something specifically for it (e.g. ts from moreutils, or multilog from daemontools), or write a filter in some programming language. Ideally, you do not want to fork a date(1) command for every line of input that you're logging, because it's too slow. You want to use builtins. Older versions of bash cannot do this. You need at least Bash 4.2 for the printf %(...)T option. Otherwise, you could write something in Perl, Python, Tcl, etc. to read lines and write them out with timestamps.
108. How do I wait for several spawned processes?
There are numerous ways to do this, but all of them are either limited by the available tools. I have come up with the following solutions.
109. How can I tell whether my script was sourced (dotted in) or executed?
There seem to be two reasons why people ask this: either they're trying to detect user errors and provide a friendly message, or they're Python programmers who want to use one of Python's most idiosyncratic features in bash.
110. How do I copy a file to a remote system, and specify a remote name which may contain spaces?
All of the common tools for copying files to a remote system (ssh, scp, rsync) send the filename as part of a shell command, which the remote system interprets. This makes the issue extremely complex, because the remote shell will often mangle the filename. There are at least three ways to deal with the problem: NFS, careful encoding of the filename, or submission of the filename as part of the data stream.
111. What is the Shellshock vulnerability in Bash?
"Shellshock" refers to two remotely-exploitable vulnerabilities in Bash, discovered in September 2014. The first vulnerability exploits the mechanism that Bash used to export and import functions, and allowed arbitrary command execution. The second vulnerability exploits a parser bug and allowed local files to be created.
112. What are the advantages and disadvantages of using set -u (or set -o nounset)?
Bash (like all other Bourne shell derivatives) has a feature activated by the command set -u (or set -o nounset). When this feature is in effect, any command which attempts to expand an unset variable will cause a fatal error (the shell immediately exits, unless it is interactive).
113. How do I extract data from an HTML or XML file?
Do not attempt this with sed, awk, grep, and so on (it leads to undesired results). In many cases, your best option is to write in a language that has support for XML data. If you have to use a shell script, there are a few HTML- and XML-specific tools available to parse these files for you.
114. How do I operate on IP addresses and netmasks?
IPv4 addresses are 32-bit unsigned integers. The "dotted quad" notation (192.168.1.2) is only one means of representing such an address. When applying netmasks, it's easier if we first convert the dotted quad format into a plain integer.
115. How do I make a menu?
Some people like to use select because it's simple. If your own needs are extremely simple, then this may be sufficient for you. If you want your own look and feel, you can simply write a menu yourself. There is also dialog, which we won't cover on this page.
116. I have two files. The first one contains bad IP addresses (plus other fields). I want to remove all of these bad addresses from a second file.
This is a more generalized form of one of the questions from FAQ 36 (where the entire line is significant in each file). In this form, we're only using part of each line as a key. We're going to show how to approach this kind of problem using an associative array.
117. I have a pipeline where a long-running command feeds into a filter. If the filter finds "foo", I want the long-running command to die.
In general this is not possible, because sibling processes (two children of the same parent) do not have any knowledge of each other. But consider the following example and answers:
118. How do I print the contents of an array in reverse order, or reverse an array?
First note that the concept of order applies only to indexed arrays, not associative arrays. The answers would be simpler if there were no sparse arrays, but bash's arrays can be sparse (non-sequential indices). So we have to introduce an extra step.
119. What's the difference between "cmd < file" and "cat file | cmd"? What is a UUOC?
Most of the time, these commands do the same thing, but the second one is less efficient, and it also breaks in certain rare circumstances.
120. How can I find out where this strange variable in my interactive shell came from?
Some variables are set by programs that run before bash, and this FAQ can't help you with those. For the variables that are set by bash reading a dot file, if you are not root, you can use bash's trace mode:
121. What does value too great for base mean? (Octal values in arithmetic.)
When reading numbers from files or commands and then performing arithmetic with them, leading zeroes may cause a problem:
$ echo $((09)) bash: 09: value too great for base (error token is "09")
122. How do I print a horizontal line of characters like ----?
There are many different ways, depending on how fancy you'd like to be. This page is a restoration of content from the now-defunct bash-hackers wiki, with some additional text.