1. How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?

Don't try to use "for". Use a while loop and the read command. Here is the basic template; there are many variations to discuss:

2. How can I store the return value and/or output of a command in a variable?

Well, that depends on whether you want to store the command's output (either stdout, or stdout + stderr) or its exit status (0 to 255, with 0 typically meaning "success").

3. How can I sort or compare files based on some metadata attribute (most recently modified, size, etc)?

The tempting solution is to use ls to output sorted filenames and operate on the results using e.g. awk. As usual, the ls approach cannot be made robust and should never be used in scripts due in part to the possibility of arbitrary characters (including newlines) present in filenames. Therefore, we need some other way to compare file metadata.

4. How can I check whether a directory is empty or not? How do I check for any *.mpg files, or count how many there are?

In Bash, you can count files safely and easily with the nullglob and dotglob options (which change the behaviour of globbing), and an array:

5. How can I use array variables?

This answer assumes you have a basic understanding of what arrays are. If you're new to this kind of programming, you may wish to start with the guide's explanation. This page is more thorough. See links at the bottom for more resources.

6. How can I use variable variables (indirect variables, pointers, references) or associative arrays?

This is a complex page, because it's a complex topic. It's been divided into roughly three parts: associative arrays, evaluating indirect variables, and assigning indirect variables. There are discussions of programming issues and concepts scattered throughout.

7. Is there a function to return the length of a string?

The fastest way, not requiring external programs (but not usable in Bourne shells):

8. How can I recursively search all files for a string?

If you are on a typical GNU or BSD system, all you need is one of these:

9. What is buffering? Or, why does my command line produce no output: tail -f logfile | grep 'foo bar' | awk ...

Most standard Unix commands buffer their output when used non-interactively. This means that they don't write each character (or even each line) immediately, but instead collect a larger number of characters (often 4 kilobytes) before printing anything at all. In the case above, the grep command buffers its output, and therefore awk only gets its input in large chunks.

10. How can I recreate a directory hierarchy structure, without the files?

11. How can I print the n'th line of a file?

One dirty (but not quick) way is:

12. How do I invoke a shell command from a non-shell application?

You can use the shell's -c option to run the shell with the sole purpose of executing a short bit of script:

13. How can I concatenate two variables? How do I append a string to a variable?

There is no (explicit) concatenation operator for strings (either literal or variable dereferences) in the shell; you just write them adjacent to each other:

14. How can I redirect the output of multiple commands at once?

Redirecting the standard output of a single command is as easy as:

15. How can I run a command on all files with the extension .gz?

Often a command already accepts several files as arguments, e.g.

16. How can I use a logical AND/OR/NOT in a shell pattern (glob)?

"Globs" are simple patterns that can be used to match filenames or strings. They're generally not very powerful. If you need more power, there are a few options available.

17. How can I group expressions in an if statement, e.g. if (A AND B) OR C?

The portable (POSIX or Bourne) way is to use multiple test (or [) commands:

18. How can I use numbers with leading zeros in a loop, e.g. 01, 02?

As always, there are many different ways to solve the problem, each with its own advantages and disadvantages. The most important considerations are which shell you're using, whether the start/end numbers are constants, and how many times the loop is going to iterate.

19. How can I split a file into line ranges, e.g. lines 1-10, 11-20, 21-30?

POSIX specifies the split utility, which can be used for this purpose:

20. How can I find and safely handle file names containing newlines, spaces or both?

First and foremost, to understand why you're having trouble, read Arguments to get a grasp on how the shell understands the statements you give it. It is vital that you grasp this matter well if you're going to be doing anything with the shell.

21. How can I replace a string with another string in a variable, a stream, a file, or in all the files in a directory?

There are a number of techniques for this. Which one to use depends on many factors, the biggest of which is what we're editing. This page also contains contradictory advice from multiple authors. This is a deeply ugly topic, and there are no universally right answers (but plenty of universally wrong ones).

22. How can I calculate with floating point numbers instead of just integers?

BASH's builtin arithmetic uses integers only:

$ printf '%s\n' "$((10 / 3))"
3

23. I want to launch an interactive shell that has special aliases and functions, not the ones in the user's ~/.bashrc.

When starting bash in non-POSIX mode, specify a different start-up file with --rcfile:

24. I set variables in a loop that's in a pipeline. Why do they disappear after the loop terminates? Or, why can't I pipe data to read?

In most shells, each command of a pipeline is executed in a separate SubShell. Non-working example:

25. How can I access positional parameters after $9?

Use ${10} instead of $10. This works for BASH and KornShell, but not for older BourneShell implementations. Another way to access arbitrary positional parameters after $9 is to use for, e.g. to get the last parameter:

26. How can I randomize (shuffle) the order of lines in a file? Or select a random line from a file, or select a random file from a directory?

To randomize the lines of a file, here is one approach. This one involves generating a random number, which is prefixed to each line; then sorting the resulting lines, and removing the numbers.

27. How can two unrelated processes communicate?

Two unrelated processes cannot use the arguments, the environment or stdin/stdout to communicate; some form of inter-process communication (IPC) is required.

28. How do I determine the location of my script? I want to read some config files from the same place.

There are two prime reasons why this issue comes up: either you want to externalize data or configuration of your script and need a way to find these external resources, or your script is intended to act upon a bundle of some sort (eg. a build script), and needs to find the resources to act upon.

29. How can I display the target of a symbolic link?

The nonstandard external command readlink(1) can be used to display the target of a symbolic link:

30. How can I rename all my *.foo files to *.bar, or convert spaces to underscores, or convert upper-case file names to lower case?

There are a bunch of different ways to do this, depending on which nonstandard tools you have available. Even with just standard POSIX tools, you can still perform most of the simple cases. We'll show the portable tool examples first.

31. What is the difference between test, [ and [[ ?

The open square bracket [ command (aka test command) and the [[ ... ]] test construct are used to evaluate expressions. [[ ... ]] works only in the Korn shell (where it originates), Bash, Zsh, and recent versions of Yash and busybox sh (if enabled at compilation time, and still very limited there especially in the hush-based variant), and is more powerful; [ and test are POSIX utilities (generally builtin). POSIX doesn't specify the [[ ... ]] construct (which has a specific syntax with significant variations between implementations) though allows shells to treat [[ as a keyword. Here are some examples:

32. How can I redirect the output of 'time' to a variable or file?

Bash's time keyword uses special trickery, so that you can do things like

33. How can I find a process ID for a process given its name?

Usually a process is referred to using its process ID (PID), and the ps(1) command can display the information for any process given its process ID, e.g.

34. Can I do a spinner in Bash?

Sure!

35. How can I handle command-line options and arguments in my script easily?

Well, that depends a great deal on what you want to do with them. There are two standard approaches, each with its strengths and weaknesses.

36. How can I get all lines that are: in both of two files (set intersection) or in only one of two files (set subtraction).

Use the comm(1) command:

37. How can I print text in various colors?

Do not hard-code ANSI color escape sequences in your program! The tput command lets you interact with the terminal database in a sane way:

38. How do Unix file permissions work?

See Permissions.

39. What are all the dot-files that bash reads?

See DotFiles.

40. How do I use dialog to get input from the user?

Here is an example:

41. How do I determine whether a variable contains a substring?

In BASH:

42. How can I find out if a process is still running?

The kill command is used to send signals to a running process. As a convenience function, the signal "0", which does not exist, can be used to find out if a process is still running:

43. Why does my crontab job fail? 0 0 * * * some command > /var/log/mylog.`date +%Y%m%d`

In many versions of crontab, the percent sign (%) is treated specially, and therefore must be escaped with backslashes:

44. How do I create a progress bar? How do I see a progress indicator when copying/moving files?

The easiest way to add a progress bar to your own script is to use dialog --gauge. Here is an example, which relies heavily on BASH features:

45. How can I ensure that only one instance of a script is running at a time (mutual exclusion, locking)?

We need some means of mutual exclusion. One way is to use a "lock": any number of processes can try to acquire the lock simultaneously, but only one of them will succeed.

This example was contributed by Charles Duffy (who is now more than a little embarassed by same, and suggests using flock instead). It has been separated from the parent page because the code has several issues that make it dubious.

46. I want to check to see whether a word is in a list (or an element is a member of a set).

If your real question is How do I check whether one of my parameters was -v? then see FAQ #35 instead. Otherwise, read on…

47. How can I redirect stderr to a pipe?

A pipe can only carry standard output (stdout) of a program. To pipe standard error (stderr) through it, you need to redirect stderr to the same destination as stdout. Optionally you can close stdout or redirect it to /dev/null to only get stderr. Some sample code:

48. Eval command and security issues

The eval command is extremely powerful and extremely easy to abuse.

49. How can I view periodic updates/appends to a file? (ex: growing log file)

tail -f will show you the growing log file. On some systems (e.g. OpenBSD), this will automatically track a rotated log file to the new file with the same name (which is usually what you want). To get the equivalent functionality on GNU systems, use tail -F instead.

50. I'm trying to put a command in a variable, but the complex cases always fail!

Variables hold data. Functions hold code. Don't put code inside variables! There are many situations in which people try to shove commands, or command arguments, into variables and then run them. Each case needs to be handled separately.

51. I want history-search just like in tcsh. How can I bind it to the up and down keys?

Just add the following to /etc/inputrc or your ~/.inputrc:

52. How do I convert a file from DOS format to UNIX format (remove CRs from CR-LF line terminators)?

Carriage return (CR) characters are used in line ending markers on some systems. There are three different kinds of line endings in common use:

53. I have a fancy prompt with colors, and now bash doesn't seem to know how wide my terminal is. Lines wrap around incorrectly.

53.1. Escape the colors with \[ \]

You must put \[ and \] around any non-printing escape sequences in your prompt. Thus:

54. How can I tell whether a variable contains a valid number?

First, you have to define what you mean by "number". The most common case when people ask this seems to be "a non-negative integer, with no leading + sign". Or in other words, a string of all digits. Other times, people want to validate a floating-point input, with optional sign and optional decimal point.

55. Tell me all about 2>&1 -- what's the difference between 2>&1 >foo and >foo 2>&1, and when do I use which?

Bash processes all redirections from left to right, in order. And the order is significant. Moving them around within a command may change the results of that command.

56. How can I untar (or unzip) multiple tarballs at once?

As the tar command was originally designed to read from and write to tape devices (tar - Tape ARchiver), you can specify only filenames to put inside an archive (write to tape) or to extract out of an archive (read from tape).

57. How can I group entries (in a file by common prefixes)?

As in, one wants to convert:

58. Can bash handle binary data?

The answer is, basically, no....

59. I saw this command somewhere: :(){ :|:& } (fork bomb). How does it work?

This is a potentially dangerous command. Don't run it! The "trigger" is omitted from the question above, leaving only the part that sets up the function.

60. I'm trying to write a script that will change directory (or set a variable), but after the script finishes, I'm back where I started (or my variable isn't set)!

Consider this:

   #!/bin/sh
   cd /tmp

61. Is there a list of which features were added to specific releases (versions) of Bash?

Here are some links to official Bash documentation:

62. How do I create a temporary file in a secure manner?

There does not appear to be any single command that simply works everywhere. tempfile is not portable. mktemp exists more widely (but still not ubiquitously), but it may require a -c switch to create the file in advance; or it may create the file by default and barf if -c is supplied. Some systems don't have either command (Solaris, POSIX). POSIX systems are supposed to have m4 which has the ability to create a temporary file, but some systems may not install m4 by default, or their implementation of m4 may be missing this feature.

63. My ssh client hangs when I try to logout after running a remote background job!

The following will not do what you expect:

   ssh me@remotehost 'sleep 120 &'
   # Client hangs for 120 seconds

64. Why is it so hard to get an answer to the question that I asked in #bash?

Maybe nobody knows the answer (or the people who know the answer are busy). Maybe you haven't given enough detail about the problem, or you haven't presented the problem clearly. Maybe the question you asked is answered in this FAQ, or in BashPitfalls, or in the BashGuide.

Don't believe us? Here, just look. This is an actual homework question that was inflicted upon someone in #bash.

65. Is there a "PAUSE" command in bash like there is in MSDOS batch scripts? To prompt the user to press any key to continue?

Use the following to wait until the user presses enter:

66. I want to check if [[ $var == foo || $var == bar || $var == more ]] without repeating $var n times.

The portable solution uses case:

67. How can I trim leading/trailing white space from one of my variables?

There are a few ways to do this. Some involve special tricks that only work with whitespace. Others are more general, and can be used to strip leading zeroes, etc.

68. How do I run a command, and have it abort (timeout) after N seconds?

FIRST check whether the command you're running can be told to timeout directly. The methods described here are "hacky" workarounds to force a command to terminate after a certain time has elapsed. Configuring your command properly is always preferable to the alternatives below.

69. I want to automate an ssh (or scp, or sftp) connection, but I don't know how to send the password....

When dealing with authentication in a shell script, please bear in mind the following points:

70. How do I convert Unix (epoch) times to human-readable values?

The only sane way to handle time values within a program is to convert them into a linear scale. You can't store "January 17, 2005 at 5:37 PM" in a variable and expect to do anything with it....

71. How do I convert an ASCII character to its decimal (or hexadecimal) value and back? How do I do URL encoding or URL decoding?

If you have a known octal or hexadecimal value (at script-writing time), you can just use printf:

72. How can I ensure my environment is configured for cron, batch, and at jobs?

If a shell or other script calling shell commands runs fine interactively but fails due to environment configurations (say: a complex $PATH) when run noninteractively, you'll need to force your environment to be properly configured.

73. How can I use parameter expansion? How can I get substrings? How can I get a file without its extension, or get just a file's extension? What are some good ways to do basename and dirname?

Parameter expansion is an important subject. This page contains a concise overview of parameter expansion.

74. How do I get the effects of those nifty Bash Parameter Expansions in older shells?

Most of the extended forms of parameter expansion do not work with the older BourneShell. If your code needs to be portable to that shell as well, sed and expr can often be used.

75. How do I use 'find'? I can't understand the man page at all!

See UsingFind.

76. How do I get the sum of all the numbers in a column?

This and all similar questions are best answered with an AWK one-liner.

77. How do I log history or "secure" bash against history removal?

If you're a shell user who wants to record your own activities, see FAQ #88 instead. If you're a system administrator who wants to know how to find out what a user had executed when they unset or /dev/nulled their shell history, there are several problems with this....

78. I want to set a user's password using the Unix passwd command, but how do I script that? It doesn't read standard input!

OK, first of all, I know there are going to be some people reading this, right now, who don't even understand the question. Here, this does not work:

{ echo oldpass; echo newpass; echo newpass; } | passwd
# This DOES NOT WORK!

79. How can I grep for lines containing foo AND bar, foo OR bar? Or for files containing foo AND bar, possibly on separate lines? Or files containing foo but NOT bar?

This is really four different questions, so we'll break this answer into parts.

80. How can I make an alias that takes an argument?

You can't. Aliases in bash are extremely rudimentary, and not really suitable to any serious purpose. The bash man page even says so explicitly:

81. How can I determine whether a command exists anywhere in my PATH?

POSIX specifies a shell builtin called command which can be used for this purpose:

82. Why is $(...) preferred over `...` (backticks)?

`...` is the legacy syntax for command substitution, required by only the very oldest of non-POSIX-compatible Bourne shells. There are several reasons to always prefer the $(...) syntax:

83. How do I determine whether a variable is already defined? Or a function?

There are several ways to test these things, depending on the exact requirements. Most of the time, the desired test is whether a variable has a non-empty value. In this case, we may simply use:

84. How do I return a string (or large number, or negative number) from a function? "return" only lets me give a number from 0 to 255.

Functions in Bash (as well as all the other Bourne-family shells) work like commands: that is, they only "return" an exit status, which is an integer from 0 to 255 inclusive. This is intended to be used only for signaling errors, not for returning the results of computations, or other data.

85. How to write several times to a fifo without having to reopen it?

In the general case, you'll open a new FileDescriptor (FD) pointing to the fifo, and write through that. For simple cases, it may be possible to skip that step.

86. How to ignore aliases, functions, or builtins when running a command?

functions, builtins, external utilities, and aliases can all be defined with the same name at once. It's sometimes necessary specify which of these the shell should resolve while bypassing the others.

87. How can I get a file's permissions (or other metadata) without parsing ls -l output?

There are several potential ways, most of which are system-specific. They also depend on precisely why you want the information; in most cases, there will be some other way to accomplish your real goal. You don't want to parse ls's output if there's any possible way to avoid doing so.

88. How can I avoid losing any history lines?

89. I'm reading a file line by line and running ssh or ffmpeg, only the first line gets processed!

When reading a file line by line, if a command inside the loop also reads stdin, it can exhaust the input file. For example:

90. How do I prepend a text to a file (the opposite of >>)?

You cannot do it with bash redirections alone; the opposite of >> does not exist....

91. I'm trying to get the number of columns or lines of my terminal but the variables COLUMNS / LINES are always empty.

COLUMNS and LINES are set by BASH in interactive mode; they are not available by default in a script. On most systems, you can try to query the terminal yourself:

92. How do I write a CGI script that accepts parameters?

There are always circumstances beyond our control that drive us to do things that we would never choose to do on our own. This FAQ entry describes one of those situations.

93. How can I set the contents of my terminal's title bar?

If you have a terminal that understands xterm-compatible escape sequences, and you just want to set the title one time, you can use a function like this:

94. I want to get an alert when my disk is full (parsing df output).

Sadly, parsing the output of df really is the most reliable way to determine how full a disk is, on most operating systems. However, please note that this is a "least bad" answer, not a "best" answer. Parsing any command-line reporting tool's output in a program is never pretty. The purpose of this FAQ is to try to describe all the problems this approach is known to encounter, and work around them.

95. I'm getting "Argument list too long". How can I process a large list in chunks?

First, let's review some background material. When a process wants to run another process, it fork()s a child, and the child calls one of the exec* family of system calls (e.g. execve()), giving the name or path of the new process's program file; the name of the new process; the list of arguments for the new process; and, in some cases, a set of environment variables. Thus:

96. ssh eats my word boundaries! I can't do ssh remotehost make CFLAGS="-g -O"!

ssh emulates the behavior of the Unix remote shell command (rsh or remsh), including this bug. There are a few ways to work around it, depending on exactly what you need.

97. How do I determine whether a symlink is dangling (broken)?

The documentation on this is fuzzy, but it turns out you can do this with shell builtins:

98. How to add localization support to your bash scripts

Looking for examples of how to add simple localization to your bash scripts, and how to do testing? This is probably what you need....

99. How can I get the newest (or oldest) file from a directory?

This page should be merged with BashFAQ/003

100. How do I do string manipulations in bash?

Bash can do string operations. LOTS of string operations. This is an introduction to bash string manipulations and related techniques. It overlaps with the Parameter Expansion question, but the information here is presented in a more beginner-friendly manner (we hope).

101. Common utility functions (warn, die)

(If you were looking for option processing, see BashFAQ/035.) Bash and sh don't offer a die builtin command like Perl does, but it's common to use a die function in scripts. You just have to write one yourself. Most people who write a die function like to keep it simple. There are two common varieties: one that only takes a message to print, and one that takes a message and an exit status value.

102. How to get the difference between two dates

It's best if you work with timestamps throughout your code, and then only convert timestamps to human-readable formats for output. If you must handle human-readable dates as input, then you will need something that can parse them.

103. How do I check whether my file was modified in a certain month or date range?

Doing date-related math in Bash is hard because Bash has no builtins in place for doing math with dates or getting metadata such as modification time from files.

104. Why doesn't foo=bar echo "$foo" print bar?

This is subtle, and has to do with the exact order in which the BashParser performs each step.

105. Why doesn't set -e (or set -o errexit, or trap ERR) do what I expected?

set -e was an attempt to add "automatic error detection" to the shell. Its goal was to cause the shell to abort any time an error occurred, so you don't have to put || exit 1 after each important command. This does not work well in practice.

The correct answer to every exercise is actually "because set -e is crap".

106. Logging! I want to send all of my script's output to a log file. But I want to do it from inside the script. And I want to see it on the terminal too!

Normally, if you want to run a script and send its output to a logfile, you'd simply use Redirection: myscript >log 2>&1. Or to see the output on the screen and also redirect to a file: myscript 2>&1 | tee log (or better still, run your script within the script(1) command if your system has it; script is intended for use with interactive shell sessions.). If you want to insert commands into a script that cause it to do this kind of logging internally, without altering your invocation, then it gets trickier.

107. How do I add a timestamp to every line of a stream?

Adding timestamps to a stream is a challenge, because there aren't any standard tools to do it. You either have to install something specifically for it (e.g. ts from moreutils, or multilog from daemontools), or write a filter in some programming language. Ideally, you do not want to fork a date(1) command for every line of input that you're logging, because it's too slow. You want to use builtins. Older versions of bash cannot do this. You need at least Bash 4.2 for the printf %(...)T option. Otherwise, you could write something in Perl, Python, Tcl, etc. to read lines and write them out with timestamps.

108. How do I wait for several spawned processes?

There are numerous ways to do this, but all of them are either limited by the available tools. I have come up with the following solutions.

109. How can I tell whether my script was sourced (dotted in) or executed?

There seem to be two reasons why people ask this: either they're trying to detect user errors and provide a friendly message, or they're Python programmers who want to use one of Python's most idiosyncratic features in bash.

110. How do I copy a file to a remote system, and specify a remote name which may contain spaces?

All of the common tools for copying files to a remote system (ssh, scp, rsync) send the filename as part of a shell command, which the remote system interprets. This makes the issue extremely complex, because the remote shell will often mangle the filename. There are at least three ways to deal with the problem: NFS, careful encoding of the filename, or submission of the filename as part of the data stream.

111. What is the Shellshock vulnerability in Bash?

"Shellshock" refers to two remotely-exploitable vulnerabilities in Bash, discovered in September 2014. The first vulnerability exploits the mechanism that Bash used to export and import functions, and allowed arbitrary command execution. The second vulnerability exploits a parser bug and allowed local files to be created.

112. What are the advantages and disadvantages of using set -u (or set -o nounset)?

Bash (like all other Bourne shell derivatives) has a feature activated by the command set -u (or set -o nounset). When this feature is in effect, any command which attempts to expand an unset variable will cause a fatal error (the shell immediately exits, unless it is interactive).

113. How do I extract data from an HTML or XML file?

Do not attempt this with sed, awk, grep, and so on (it leads to undesired results). In many cases, your best option is to write in a language that has support for XML data. If you have to use a shell script, there are a few HTML- and XML-specific tools available to parse these files for you.

114. How do I operate on IP addresses and netmasks?

IPv4 addresses are 32-bit unsigned integers. The "dotted quad" notation (192.168.1.2) is only one means of representing such an address. When applying netmasks, it's easier if we first convert the dotted quad format into a plain integer.

115. How do I make a menu?

Some people like to use select because it's simple. If your own needs are extremely simple, then this may be sufficient for you. If you want your own look and feel, you can simply write a menu yourself. There is also dialog, which we won't cover on this page.

116. I have two files. The first one contains bad IP addresses (plus other fields). I want to remove all of these bad addresses from a second file.

This is a more generalized form of one of the questions from FAQ 36 (where the entire line is significant in each file). In this form, we're only using part of each line as a key. We're going to show how to approach this kind of problem using an associative array.

117. I have a pipeline where a long-running command feeds into a filter. If the filter finds "foo", I want the long-running command to die.

In general this is not possible, because sibling processes (two children of the same parent) do not have any knowledge of each other. But consider the following example and answers:

118. How do I print the contents of an array in reverse order, or reverse an array?

First note that the concept of order applies only to indexed arrays, not associative arrays. The answers would be simpler if there were no sparse arrays, but bash's arrays can be sparse (non-sequential indices). So we have to introduce an extra step.

119. What's the difference between "cmd < file" and "cat file | cmd"? What is a UUOC?

Most of the time, these commands do the same thing, but the second one is less efficient, and it also breaks in certain rare circumstances.

120. How can I find out where this strange variable in my interactive shell came from?

Some variables are set by programs that run before bash, and this FAQ can't help you with those. For the variables that are set by bash reading a dot file, if you are not root, you can use bash's trace mode:

121. What does value too great for base mean? (Octal values in arithmetic.)

When reading numbers from files or commands and then performing arithmetic with them, leading zeroes may cause a problem:

$ echo $((09))
bash: 09: value too great for base (error token is "09")

122. How do I print a horizontal line of characters like ----?

There are many different ways, depending on how fancy you'd like to be. This page is a restoration of content from the now-defunct bash-hackers wiki, with some additional text.