|
Size: 8733
Comment: mention about trailing characters after last line
|
Size: 9529
Comment: That's only for space and tab (if present in IFS, or IFS is unset), that special treatment doesn't apply to other blank/whitespace characters.
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 3: | Line 3: |
| Use a `while` loop and the `read` command: {{{ while read -r line do echo "$line" done < "$file" }}} The `-r` option to `read` prevents backslash interpretation (usually used as a backslash newline pair, to continue over multiple lines). Without this option, any backslashes in the input will be discarded. You should always use the `-r` option with read. |
[[DontReadLinesWithFor|Don't try to use "for"]]. Use a `while` loop and the `read` command: {{{#!highlight bash while IFS= read -r line; do printf '%s\n' "$line" done < "$file" }}} The `-r` option to `read` prevents backslash interpretation (usually used as a backslash newline pair, to continue over multiple lines or to escape the delimiters). Without this option, any unescaped backslashes in the input will be discarded. You should almost always use the `-r` option with read. The most common exception to this rule is when -e is used, which uses Readline to obtain the line from an interactive shell. In that case, tab completion will add backslashes to escape spaces and such, and you do not want them to be literally included in the variable. This would never be used when reading anything line-by-line, though, and -r should always be used when doing so. In the scenario above `IFS=` prevents [[#Trimming|trimming of leading and trailing whitespace]]. Remove it if you want this effect. |
| Line 15: | Line 18: |
| If avoiding comments starting with `#` is desired {{{ while read -r line do [[ $line = \#* ]] && continue echo "$line" done < "$file" }}} If your input source is the script's standard input, then you don't need any redirection at all. |
The [[BashGuide/InputAndOutput#Redirection|redirection]] `< "$file"` tells the `while` loop to read from the file whose name is in the variable `file`. If you would prefer to use a literal pathname instead of a variable, you may do that as well. If your input source is the script's standard input, then you don't need any redirection at all. |
| Line 27: | Line 22: |
| {{{ while read -r line; do echo "$line" done <<< "$var" }}} The same can be done in any Bourne-type shell by using a "here document": {{{ while read -r line; do echo "$line" done <<EOF |
{{{#!highlight bash while IFS= read -r line; do printf '%s\n' "$line" done <<< "$var" }}} The same can be done in any Bourne-type shell by using a "here document" (although `read -r` is POSIX, not Bourne): {{{#!highlight bash while IFS= read -r line; do printf '%s\n' "$line" done <<EOF |
| Line 43: | Line 38: |
| If avoiding comments starting with `#` is desired, you can simply skip them inside the loop: {{{#!highlight bash # Bash while read -r line; do [[ $line = \#* ]] && continue printf '%s\n' "$line" done < "$file" }}} |
|
| Line 45: | Line 49: |
| {{{ # Input file has 3 columns separated by white space. while read -r first_name last_name phone; do ... done < "$file" |
{{{#!highlight bash # Input file has 3 columns separated by white space (space or tab characters only). while read -r first_name last_name phone; do # Only print the last name (second column) printf '%s\n' "$last_name" done < "$file" |
| Line 54: | Line 59: |
| {{{ while IFS=: read -r user pass uid gid gecos home shell; do ... done < /etc/passwd }}} For TAB delimited files, use [[Quotes|IFS=$'\t']]. Also, please note that you do ''not'' necessarily need to know how many fields each line of input contains. If you supply more variables than there are fields, the extra variables will be empty. If you supply fewer, the last variable gets "all the rest" of the fields after the preceding ones are satisfied. For example, {{{ while read -r first_name last_name junk; do ... done <<< 'Bob Smith 123 Main Street Elk Grove Iowa 123-555-6789' # Inside the loop, first_name will contain "Bob", and # last_name will contain "Smith". The variable "junk" holds # everything else. |
{{{#!highlight bash # Extract the username and its shell from /etc/passwd: while IFS=: read -r user pass uid gid gecos home shell; do printf '%s: %s\n' "$user" "$shell" done < /etc/passwd }}} For tab-delimited files, use [[Quotes|IFS=$'\t']] though beware that multiple tab characters in the input will be considered as '''one''' delimiter (and the Ksh93/Zsh `IFS=$'\t\t'` workaround won't work in Bash). You do ''not'' necessarily need to know how many fields each line of input contains. If you supply more variables than there are fields, the extra variables will be empty. If you supply fewer, the last variable gets "all the rest" of the fields after the preceding ones are satisfied. For example, {{{#!highlight bash read -r first last junk <<< 'Bob Smith 123 Main Street Elk Grove Iowa 123-555-6789' # first will contain "Bob", and last will contain "Smith". # junk holds everything else. |
| Line 75: | Line 78: |
| {{{ read -r _ _ first middle last _ <<< "$record" # Remember, the final _ can absorb any number of fields. # It doesn't need to be repeated there. }}} The {{{read}}} command modifies each line read, e.g. by default it removes all leading whitespace characters (blanks, tab characters, ... -- any whitespace characters present in [[IFS]]). If that is not desired, the {{{IFS}}} variable has to be cleared: {{{ while IFS= read -r line do echo "$line" done < "$file" }}} Also note that, if there are some characters after the last line in the file (or to put it differently, if the last line is not terminated by a newline character), then read will read it but return false. You can still process it by doing: {{{ while IFS= read -r line do printf '%s\n' "$line" done < "$file" [ -n "$line" ] && printf %s "$line" |
{{{#!highlight bash read -r _ _ first middle last _ <<< "$record" # We skip the first two fields, then read the next three. # Remember, the final _ can absorb any number of fields. # It doesn't need to be repeated there. }}} Note that this usage of `_` is only guaranteed to work in Bash. Many other shells use `_` for other purposes that will at best cause this to not have the desired effect, and can break the script entirely. It is better to choose a unique variable that isn't used elsewhere in the script, even though `_` is a common Bash convention. <<Anchor(Trimming)>>The {{{read}}} command modifies each line read; by default it [[BashFAQ/067|removes all leading and trailing whitespace]] characters (spaces and tabs, if present in [[IFS]]). If that is not desired, the {{{IFS}}} variable has to be cleared: {{{#!highlight bash # Exact lines, no trimming while IFS= read -r line; do printf '%s\n' "$line" done < "$file" |
| Line 103: | Line 99: |
| {{{ some command | while read -r line; do other commands done |
{{{#!highlight bash some command | while IFS= read -r line; do printf '%s\n' "$line" done |
| Line 110: | Line 106: |
| {{{ find . -print0 | while IFS= read -r -d $'\0' file; do mv "$file" "${file// /_}" done }}} This reads one filename at a time from the find command and renames the file, replacing spaces with underscores. Note the usage of {{{-print0}}} in the find command, which uses NUL bytes as filename delimiters, and {{{-d $'\0'}}} in the read command to instruct it to read all text into the file variable until it finds a NUL byte. By default, `find` and `read` delimit their input with newlines; however, since filenames can potentially contain newlines themselves, this default behaviour will split up those filenames with newlines and cause the loop body to fail. Additionally it is necessary to set IFS to an empty string, because otherwise read would strip trailing whitespace. See [[BashFAQ/020|FAQ #20]] for more details. Using a pipe to send find's output into a while loop places the loop in a SubShell and may therefore cause problems later on if the commands inside the body of the loop attempt to set variables which need to be used outside the loop; in that case, see [[BashFAQ/024|FAQ 24]], or use ProcessSubstitution like: {{{ while read -r line; do other commands done < <(some command) }}} If you want to read a file into an [[BashFAQ/005|array]], see [[BashFAQ/005|FAQ 5]]. If the the input source lacks a trailing newline (such as {{{/proc/$$/cmdline}}} on Linux), the line will not be processed by a {{{while read ...}}} loop, as {{{read}}} returns a failure that aborts the while loop, thus failing to process the last line. It does, however, store the contents of the partial line in the variable, so you can test whether there was such an unterminated line by checking whether the variable is non-empty at the end of the loop: {{{ # This does not work: printf 'line 1\ntruncated line 2' | while read -r line; do echo $line; done # This does not work either: printf 'line 1\ntruncated line 2' | while read -r line; do echo "$line"; done; [[ $line ]] && echo -n "$line" # This works: printf 'line 1\ntruncated line 2' | (while read -r line; do echo "$line"; done; [[ $line ]] && echo "$line") }}} |
{{{#!highlight bash find . -type f -print0 | while IFS= read -r -d '' file; do mv "$file" "${file// /_}" done }}} This reads one filename at a time from the `find` command and [[BashFAQ/030|renames the file]], replacing spaces with underscores. Note the usage of `-print0` in the `find` command, which uses NUL bytes as filename delimiters; and {{{-d ''}}} in the `read` command to instruct it to read all text into the `file` variable until it finds a NUL byte. By default, `find` and `read` delimit their input with newlines; however, since filenames can potentially contain newlines themselves, this default behaviour will split up those filenames at the newlines and cause the loop body to fail. Additionally it is necessary to set `IFS` to an empty string, because otherwise `read` would still strip leading and trailing whitespace. See [[BashFAQ/020|FAQ #20]] for more details. Using a pipe to send `find`'s output into a while loop places the loop in a SubShell and may therefore cause problems later on if the commands inside the body of the loop attempt to set variables which need to be used after the loop; in that case, see [[BashFAQ/024|FAQ 24]], or use a ProcessSubstitution like: {{{#!highlight bash while IFS= read -r line; do printf '%s\n' "$line" done < <(some command) }}} If you want to read lines from a file into an [[BashFAQ/005|array]], see [[BashFAQ/005|FAQ 5]]. === My text files are broken! They lack their final newlines! === If there are some characters after the last line in the file (or to put it differently, if the last line is not terminated by a newline character), then `read` will read it but return false, leaving the broken partial line in the `read` variable(s). You can process this after the loop: {{{#!highlight bash # Emulate cat while IFS= read -r line; do printf '%s\n' "$line" done < "$file" [[ -n $line ]] && printf %s "$line" }}} Or: {{{#!highlight bash # This does not work: printf 'line 1\ntruncated line 2' | while read -r line; do echo $line; done # This does not work either: printf 'line 1\ntruncated line 2' | while read -r line; do echo "$line"; done; [[ $line ]] && echo -n "$line" # This works: printf 'line 1\ntruncated line 2' | { while read -r line; do echo "$line"; done; [[ $line ]] && echo "$line"; } }}} The first example, beyond missing the after-loop test, is also missing quotes. See [[Quotes|Quotes]] or [[Arguments|Arguments]] for an explanation why. The [[Arguments|Arguments]] page is an especially important read. |
| Line 142: | Line 152: |
Alternatively, you can simply add a logical OR to the while test: {{{#!highlight bash while IFS= read -r line || [[ -n $line ]]; do printf '%s\n' "$line" done < "$file" printf 'line 1\ntruncated line 2' | while read -r line || [[ -n $line ]]; do echo "$line"; done }}} |
|
| Line 145: | Line 164: |
| {{{ while read -r line do cat > ignoredfile echo "$line" done < "$file" |
{{{#!highlight bash while read -r line; do cat > ignoredfile printf '%s\n' "$line" done < "$file" |
| Line 154: | Line 172: |
| One workaround is to use a numeric file descriptor rather than standard input: {{{ # Bash exec 9< "$file" while read -r -u9 line do cat > ignoredfile echo "$line" done |
One workaround is to use a numeric FileDescriptor rather than standard input: {{{#!highlight bash # Bash while IFS= read -r -u 9 line; do cat > ignoredfile printf '%s\n' "$line" done 9< "$file" # Note that read -u is not portable to every shell. Use a redirect to ensure it works in any POSIX compliant shell: while IFS= read -r line <&9; do cat > ignoredfile printf '%s\n' "$line" done 9< "$file" }}} Or: {{{#!highlight bash exec 9< "$file" while IFS= read -r line <&9; do cat > ignoredfile printf '%s\n' "$line" done exec 9<&- |
| Line 167: | Line 200: |
| You might need this, for example, with {{{mencoder}}} which will accept user input if there is any, but will continue silently if there isn't. Other commands that act this way include `ssh` and `ffmpeg`. Additional workarounds for this are discussed in [[BashFAQ/089|FAQ #89]]. === Why you don't use "for" for this === Using {{{for}}} instead of {{{while read ...}}} is generally less efficient and suffers a number of unexpected side-effects. {{{ $ cat afile ef gh * $ while read -r i ; do echo "$i" ; done < afile ef gh * $ for i in $(<afile) ; do echo "$i" ; done ef gh afile # the glob was expanded, and it looped per word. #workaround: $ oIFS=$IFS IFS=$'\n' ;set -f ; for i in $(<afile) ; do echo "$i" ; done; IFS=$oIFS ef gh * }}} Notice that the syntax to get this "right" is more verbose. All-in-all, (ab)using {{{for}}} this way is a more dangerous, less intuitive (you don't get what you expect out of a normal {{{for}}}!) and not any more useful method. Also, as discussed in [[BashFAQ/005|FAQ #5]], the use of `IFS=$'\n'` (or any other "whitespace" in `IFS`) causes the shell to consolidate all consecutive instances of the whitespace delimiter into one. In other words, it skips over blank lines. Did you see the blank line in the input file, and how it was missing in the output? It's easier to spot if it's not at the end: {{{ ~$ cat foo line one line three ~$ IFS=$'\n'; set -f; for line in $(<foo); do echo "$line"; done; unset IFS; set +f line one line three ~$ }}} All that setting and unsetting, and we ''still'' couldn't even mimic a simple `cat`. If preservation of blank lines is important, just go back to using `while read`. |
You might need this, for example, with `mencoder` which will accept user input if there is any, but will continue silently if there isn't. Other commands that act this way include `ssh` and `ffmpeg`. Additional workarounds for this are discussed in [[BashFAQ/089|FAQ #89]]. |
| Line 212: | Line 203: |
| CategoryShell | CategoryShell CategoryBashguide |
How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?
Don't try to use "for". Use a while loop and the read command:
The -r option to read prevents backslash interpretation (usually used as a backslash newline pair, to continue over multiple lines or to escape the delimiters). Without this option, any unescaped backslashes in the input will be discarded. You should almost always use the -r option with read.
The most common exception to this rule is when -e is used, which uses Readline to obtain the line from an interactive shell. In that case, tab completion will add backslashes to escape spaces and such, and you do not want them to be literally included in the variable. This would never be used when reading anything line-by-line, though, and -r should always be used when doing so.
In the scenario above IFS= prevents trimming of leading and trailing whitespace. Remove it if you want this effect.
line is a variable name, chosen by you. You can use any valid shell variable name there.
The redirection < "$file" tells the while loop to read from the file whose name is in the variable file. If you would prefer to use a literal pathname instead of a variable, you may do that as well. If your input source is the script's standard input, then you don't need any redirection at all.
If your input source is the contents of a variable/parameter, BASH can iterate over its lines using a "here string":
The same can be done in any Bourne-type shell by using a "here document" (although read -r is POSIX, not Bourne):
If avoiding comments starting with # is desired, you can simply skip them inside the loop:
If you want to operate on individual fields within each line, you may supply additional variables to read:
If the field delimiters are not whitespace, you can set IFS (internal field separator):
For tab-delimited files, use IFS=$'\t' though beware that multiple tab characters in the input will be considered as one delimiter (and the Ksh93/Zsh IFS=$'\t\t' workaround won't work in Bash).
You do not necessarily need to know how many fields each line of input contains. If you supply more variables than there are fields, the extra variables will be empty. If you supply fewer, the last variable gets "all the rest" of the fields after the preceding ones are satisfied. For example,
Some people use the throwaway variable _ as a "junk variable" to ignore fields. It (or indeed any variable) can also be used more than once in a single read command, if we don't care what goes into it:
Note that this usage of _ is only guaranteed to work in Bash. Many other shells use _ for other purposes that will at best cause this to not have the desired effect, and can break the script entirely. It is better to choose a unique variable that isn't used elsewhere in the script, even though _ is a common Bash convention.
The read command modifies each line read; by default it removes all leading and trailing whitespace characters (spaces and tabs, if present in IFS). If that is not desired, the IFS variable has to be cleared:
One may also read from a command instead of a regular file:
This method is especially useful for processing the output of find with a block of commands:
This reads one filename at a time from the find command and renames the file, replacing spaces with underscores.
Note the usage of -print0 in the find command, which uses NUL bytes as filename delimiters; and -d '' in the read command to instruct it to read all text into the file variable until it finds a NUL byte. By default, find and read delimit their input with newlines; however, since filenames can potentially contain newlines themselves, this default behaviour will split up those filenames at the newlines and cause the loop body to fail. Additionally it is necessary to set IFS to an empty string, because otherwise read would still strip leading and trailing whitespace. See FAQ #20 for more details.
Using a pipe to send find's output into a while loop places the loop in a SubShell and may therefore cause problems later on if the commands inside the body of the loop attempt to set variables which need to be used after the loop; in that case, see FAQ 24, or use a ProcessSubstitution like:
If you want to read lines from a file into an array, see FAQ 5.
My text files are broken! They lack their final newlines!
If there are some characters after the last line in the file (or to put it differently, if the last line is not terminated by a newline character), then read will read it but return false, leaving the broken partial line in the read variable(s). You can process this after the loop:
Or:
1 # This does not work:
2 printf 'line 1\ntruncated line 2' | while read -r line; do echo $line; done
3
4 # This does not work either:
5 printf 'line 1\ntruncated line 2' | while read -r line; do echo "$line"; done; [[ $line ]] && echo -n "$line"
6
7 # This works:
8 printf 'line 1\ntruncated line 2' | { while read -r line; do echo "$line"; done; [[ $line ]] && echo "$line"; }
The first example, beyond missing the after-loop test, is also missing quotes. See Quotes or Arguments for an explanation why. The Arguments page is an especially important read.
For a discussion of why the second example above does not work as expected, see FAQ #24.
Alternatively, you can simply add a logical OR to the while test:
How to keep other commands from "eating" the input
Some commands greedily eat up all available data on standard input. The examples above do not take precautions against such programs. For example,
will only print the contents of the first line, with the remaining contents going to "ignoredfile", as cat slurps up all available input.
One workaround is to use a numeric FileDescriptor rather than standard input:
1 # Bash
2 while IFS= read -r -u 9 line; do
3 cat > ignoredfile
4 printf '%s\n' "$line"
5 done 9< "$file"
6
7 # Note that read -u is not portable to every shell. Use a redirect to ensure it works in any POSIX compliant shell:
8 while IFS= read -r line <&9; do
9 cat > ignoredfile
10 printf '%s\n' "$line"
11 done 9< "$file"
Or:
This example will wait for the user to type something into the file ignoredfile at each iteration instead of eating up the loop input.
You might need this, for example, with mencoder which will accept user input if there is any, but will continue silently if there isn't. Other commands that act this way include ssh and ffmpeg. Additional workarounds for this are discussed in FAQ #89.