Differences between revisions 57 and 69 (spanning 12 versions)
Revision 57 as of 2013-08-29 16:14:56
Size: 8957
Editor: e36freak
Comment: Make all examples use a consistent style, mention not using -r with read, and use printf instead of echo
Revision 69 as of 2018-03-22 15:43:22
Size: 9937
Editor: GreyCat
Comment: Organize the huge mess of random examples into two sections. Add ToC.
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
[[DontReadLinesWithFor|Don't try to use "for"]]. Use a `while` loop and the `read` command:

{{{
    
while IFS= read -r line; do
      printf '%s\n' "$line"
    done < "$file"
}}}
The `-r` option to `read` prevents backslash interpretation (usually used as a backslash newline pair, to continue over multiple lines). Without this option, any backslashes in the input will be discarded. You should almost always use the `-r` option with read.
[[DontReadLinesWithFor|Don't try to use "for"]]. Use a `while` loop and the `read` command. Here is the basic template; there are many variations to discuss:

{{{#!highlight bash
while IFS= read -r line; do
  printf '%s\n' "$line"
done < "$file"
}}}

`line` is a variable name, chosen by you. You can use any valid shell variable name(s) there; see [[#trimming|field splitting]] below.

`< "$file"` redirects the loop's input from a file whose name is stored in a variable; see [[#source|source selection]] below.

If you want to read lines from a file into an [[BashFAQ/005|array]], see [[BashFAQ/005|FAQ 5]].

<<
TableOfContents>>

<<Anchor(trimming)>>
=== Field splitting, whitespace trimming, and other input processing ===

T
he `-r` option to `read` prevents backslash interpretation (usually used as a backslash newline pair, to continue over multiple lines or to escape the delimiters). Without this option, any unescaped backslashes in the input will be discarded. You should almost always use the `-r` option with read.
Line 13: Line 26:
In the scenario above `IFS=` prevents [[#Trimming|trimming of leading and trailing whitespace]]. Remove it if you want this effect.

`line` is a variable name, chosen by you. You can use any valid shell variable name there.
By default, `read` modifies each line read, by [[BashFAQ/067|removing all leading and trailing whitespace]] characters (spaces and tabs, if present in [[IFS]]). If that is not desired, the `IFS` variable may be cleared, as in the example above. If you want the trimming, leave `IFS` alone:

{{{#!highlight bash
# Leading/trailing whitespace trimming.
while read -r line; do
  printf '%s\n' "$line"
done < "$file"
}}}

If you want to operate on individual fields within each line, you may supply additional variables to {{{read}}}:

{{{#!highlight bash
# Input file has 3 columns separated by white space (space or tab characters only).
while read -r first_name last_name phone; do
  # Only print the last name (second column)
  printf '%s\n' "$last_name"
done < "$file"
}}}

If the field delimiters are not whitespace, you can set [[IFS|IFS (internal field separator)]]:

{{{#!highlight bash
# Extract the username and its shell from /etc/passwd:
while IFS=: read -r user pass uid gid gecos home shell; do
  printf '%s: %s\n' "$user" "$shell"
done < /etc/passwd
}}}
For tab-delimited files, use [[Quotes|IFS=$'\t']] though beware that multiple tab characters in the input will be considered as '''one''' delimiter (and the Ksh93/Zsh `IFS=$'\t\t'` workaround won't work in Bash).

You do ''not'' necessarily need to know how many fields each line of input contains. If you supply more variables than there are fields, the extra variables will be empty. If you supply fewer, the last variable gets "all the rest" of the fields after the preceding ones are satisfied. For example,

{{{#!highlight bash
read -r first last junk <<< 'Bob Smith 123 Main Street Elk Grove Iowa 123-555-6789'

# first will contain "Bob", and last will contain "Smith".
# junk holds everything else.
}}}

Some people use the throwaway variable `_` as a "junk variable" to ignore fields. It (or indeed any variable) can also be used more than once in a single `read` command, if we don't care what goes into it:

{{{#!highlight bash
read -r _ _ first middle last _ <<< "$record"

# We skip the first two fields, then read the next three.
# Remember, the final _ can absorb any number of fields.
# It doesn't need to be repeated there.
}}}

Note that this usage of `_` is only guaranteed to work in Bash. Many other shells use `_` for other purposes that will at best cause this to not have the desired effect, and can break the script entirely. It is better to choose a unique variable that isn't used elsewhere in the script, even though `_` is a common Bash convention.

If avoiding comments starting with `#` is desired, you can simply skip them inside the loop:
{{{#!highlight bash
# Bash
while read -r line; do
  [[ $line = \#* ]] && continue
  printf '%s\n' "$line"
done < "$file"
}}}

<<Anchor(source)>>
=== Input source selection ===
Line 19: Line 90:
If your input source is the contents of a variable/parameter, [[BASH]] can iterate over its lines using a "here string":

{{{
    
while read -r line; do
     printf '%s\n' "$line"
    done <<< "$var"
If your input source is the contents of a variable/parameter, bash can iterate over its lines using a [[BashGuide/InputAndOutput#Heredocs_And_Herestrings|here string]]:

{{{#!highlight bash
while IFS= read -r line; do
  printf '%s\n' "$line"
done <<< "$var"
Line 29: Line 100:
{{{
    while read -r line; do
     printf '%s\n' "$line"
    done <<EOF
{{{#!highlight bash
while IFS= read -r line; do
  printf '%s\n' "$line"
done <<EOF
Line 37: Line 108:
If avoiding comments starting with `#` is desired, you can simply skip them inside the loop:
{{{
    # Bash
    while read -r line; do
      [[ $line = \#* ]] && continue
      printf '%s\n' "$line"
    done < "$file"
}}}

If you want to operate on individual fields within each line, you may supply additional variables to {{{read}}}:

{{{
    # Input file has 3 columns separated by white space.
    while read -r first_name last_name phone; do
      # Only print the last name (second column)
      printf '%s\n' "$last_name"
    done < "$file"
}}}

If the field delimiters are not whitespace, you can set [[IFS|IFS (internal field separator)]]:

{{{
    # Extract the username and its shell from /etc/passwd:
    while IFS=: read -r user pass uid gid gecos home shell; do
      printf '%s: %s\n' "$user" "$shell"
    done < /etc/passwd
}}}
For tab-delimited files, use [[Quotes|IFS=$'\t']].

You do ''not'' necessarily need to know how many fields each line of input contains. If you supply more variables than there are fields, the extra variables will be empty. If you supply fewer, the last variable gets "all the rest" of the fields after the preceding ones are satisfied. For example,

{{{
    read -r first last junk <<< 'Bob Smith 123 Main Street Elk Grove Iowa 123-555-6789'

    # first will contain "Bob", and last will contain "Smith".
    # junk holds everything else.
}}}

Some people use the throwaway variable `_` as a "junk variable" to ignore fields. It (or indeed any variable) can also be used more than once in a single `read` command, if we don't care what goes into it:

{{{
    read -r _ _ first middle last _ <<< "$record"

    # We skip the first two fields, then read the next three.
    # Remember, the final _ can absorb any number of fields.
    # It doesn't need to be repeated there.
}}}

<<Anchor(Trimming)>>The {{{read}}} command modifies each line read; by default it [[BashFAQ/067|removes all leading and trailing whitespace]] characters (spaces and tabs, or any whitespace characters present in [[IFS]]). If that is not desired, the {{{IFS}}} variable has to be cleared:

{{{
    # Exact lines, no trimming
    while IFS= read -r line; do
      printf '%s\n' "$line"
    done < "$file"
}}}
Line 96: Line 110:
{{{
    some command | while read -r line; do
     printf '%s\n' "$line"
    done
}}}
{{{#!highlight bash
some command | while IFS= read -r line; do
  printf '%s\n' "$line"
done
}}}
Line 103: Line 118:
{{{
    find . -type f -print0 | while IFS= read -r -d '' file; do
        mv "$file" "${file// /_}"
    done
}}}
{{{#!highlight bash
find . -type f -print0 | while IFS= read -r -d '' file; do
    mv "$file" "${file// /_}"
done
}}}
Line 112: Line 128:
Using a pipe to send `find`'s output into a while loop places the loop in a SubShell and may therefore cause problems later on if the commands inside the body of the loop attempt to set variables which need to be used after the loop; in that case, see [[BashFAQ/024|FAQ 24]], or use a ProcessSubstitution like:

{{{
    while read -r line; do
      printf '%s\n' "$line"
    done < <(some command)
}}}

If you want to read lines from a file into an [[BashFAQ/005|array]], see [[BashFAQ/005|FAQ 5]].
Using a pipe to send `find`'s output into a `while` loop places the loop in a SubShell, which means any state changes you make (changing variables, `cd`, opening and closing [[FileDescriptor|files]], etc.) will be lost when the loop finishes. To avoid that, you may use a ProcessSubstitution:

{{{#!highlight bash
while IFS= read -r line; do
  printf '%s\n' "$line"
done < <(some command)
}}}

See [[BashFAQ/024|FAQ 24]] for more discussion.
Line 126: Line 142:
{{{
    # Emulate cat
    while IFS= read -r line; do
      printf '%s\n' "$line"
    done < "$file"
    [[ -n $line ]] && printf %s "$line"
{{{#!highlight bash
# Emulate cat
while IFS= read -r line; do
  printf '%s\n' "$line"
done < "$file"
[[ -n $line ]] && printf %s "$line"
Line 136: Line 152:
{{{
    
# This does not work:
    printf 'line 1\ntruncated line 2' | while read -r line; do echo $line; done

    # This does not work either:
    printf 'line 1\ntruncated line 2' | while read -r line; do echo "$line"; done; [[ $line ]] && echo -n "$line"

    # This works:
    printf 'line 1\ntruncated line 2' | { while read -r line; do echo "$line"; done; [[ $line ]] && echo "$line"; }
{{{#!highlight bash
# This does not work:
printf 'line 1\ntruncated line 2' | while read -r line; do echo $line; done

# This does not work either:
printf 'line 1\ntruncated line 2' | while read -r line; do echo "$line"; done; [[ $line ]] && echo -n "$line"

# This works:
printf 'line 1\ntruncated line 2' | { while read -r line; do echo "$line"; done; [[ $line ]] && echo "$line"; }
Line 151: Line 167:
{{{
    
while IFS= read -r line || [[ $line ]]; do
     printf '%s\n' "$line"
    done < "$file"

    printf 'line 1\ntruncated line 2' | while read -r line || [[ $line ]]; do echo "$line"; done
{{{#!highlight bash
while IFS= read -r line || [[ -n $line ]]; do
  printf '%s\n' "$line"
done < "$file"

printf 'line 1\ntruncated line 2' | while read -r line || [[ -n $line ]]; do echo "$line"; done
Line 161: Line 177:
{{{
    while read -r line; do
      cat > ignoredfile
     printf '%s\n' "$line"
    done < "$file"
{{{#!highlight bash
while read -r line; do
  cat > ignoredfile
  printf '%s\n' "$line"
done < "$file"
Line 170: Line 186:
{{{
    
# Bash
    while read -r -u 9 line; do
      cat > ignoredfile
     printf '%s\n' "$line"
    done 9< "$file"

    # Note that read -u is not portable, and pretty much useless when you can use simple redirections instead:
    while read -r line <&9; do
      cat > ignoredfile
     printf '%s\n' "$line"
    done 9< "$file"
{{{#!highlight bash
#
Bash
while IFS= read -r -u 9 line; do
  cat > ignoredfile
  printf '%s\n' "$line"
done 9< "$file"

# Note that read -u is not portable to every shell. Use a redirect to ensure it works in any POSIX compliant shell:
while IFS= read -r line <&9; do
  cat > ignoredfile
  printf '%s\n' "$line"
done 9< "$file"
Line 186: Line 202:
{{{
    # Bourne
    
exec 9< "$file"
    while read -r line <&9; do
      cat > ignoredfile
     printf '%s\n' "$line"
    done
    exec 9<&-
{{{#!highlight bash
exec 9< "$file"
while IFS= read -r line <&9; do
  cat > ignoredfile
  printf '%s\n' "$line"
done
exec 9<&-
Line 201: Line 216:
CategoryShell CategoryShell CategoryBashguide

How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?

Don't try to use "for". Use a while loop and the read command. Here is the basic template; there are many variations to discuss:

   1 while IFS= read -r line; do
   2   printf '%s\n' "$line"
   3 done < "$file"

line is a variable name, chosen by you. You can use any valid shell variable name(s) there; see field splitting below.

< "$file" redirects the loop's input from a file whose name is stored in a variable; see source selection below.

If you want to read lines from a file into an array, see FAQ 5.

Field splitting, whitespace trimming, and other input processing

The -r option to read prevents backslash interpretation (usually used as a backslash newline pair, to continue over multiple lines or to escape the delimiters). Without this option, any unescaped backslashes in the input will be discarded. You should almost always use the -r option with read.

The most common exception to this rule is when -e is used, which uses Readline to obtain the line from an interactive shell. In that case, tab completion will add backslashes to escape spaces and such, and you do not want them to be literally included in the variable. This would never be used when reading anything line-by-line, though, and -r should always be used when doing so.

By default, read modifies each line read, by removing all leading and trailing whitespace characters (spaces and tabs, if present in IFS). If that is not desired, the IFS variable may be cleared, as in the example above. If you want the trimming, leave IFS alone:

   1 # Leading/trailing whitespace trimming.
   2 while read -r line; do
   3   printf '%s\n' "$line"
   4 done < "$file"

If you want to operate on individual fields within each line, you may supply additional variables to read:

   1 # Input file has 3 columns separated by white space (space or tab characters only).
   2 while read -r first_name last_name phone; do
   3   # Only print the last name (second column)
   4   printf '%s\n' "$last_name"
   5 done < "$file"

If the field delimiters are not whitespace, you can set IFS (internal field separator):

   1 # Extract the username and its shell from /etc/passwd:
   2 while IFS=: read -r user pass uid gid gecos home shell; do
   3   printf '%s: %s\n' "$user" "$shell"
   4 done < /etc/passwd

For tab-delimited files, use IFS=$'\t' though beware that multiple tab characters in the input will be considered as one delimiter (and the Ksh93/Zsh IFS=$'\t\t' workaround won't work in Bash).

You do not necessarily need to know how many fields each line of input contains. If you supply more variables than there are fields, the extra variables will be empty. If you supply fewer, the last variable gets "all the rest" of the fields after the preceding ones are satisfied. For example,

   1 read -r first last junk <<< 'Bob Smith 123 Main Street Elk Grove Iowa 123-555-6789'
   2 
   3 # first will contain "Bob", and last will contain "Smith".
   4 # junk holds everything else.

Some people use the throwaway variable _ as a "junk variable" to ignore fields. It (or indeed any variable) can also be used more than once in a single read command, if we don't care what goes into it:

   1 read -r _ _ first middle last _ <<< "$record"
   2 
   3 # We skip the first two fields, then read the next three.
   4 # Remember, the final _ can absorb any number of fields.
   5 # It doesn't need to be repeated there.

Note that this usage of _ is only guaranteed to work in Bash. Many other shells use _ for other purposes that will at best cause this to not have the desired effect, and can break the script entirely. It is better to choose a unique variable that isn't used elsewhere in the script, even though _ is a common Bash convention.

If avoiding comments starting with # is desired, you can simply skip them inside the loop:

   1 # Bash
   2 while read -r line; do
   3   [[ $line = \#* ]] && continue
   4   printf '%s\n' "$line"
   5 done < "$file"

Input source selection

The redirection < "$file" tells the while loop to read from the file whose name is in the variable file. If you would prefer to use a literal pathname instead of a variable, you may do that as well. If your input source is the script's standard input, then you don't need any redirection at all.

If your input source is the contents of a variable/parameter, bash can iterate over its lines using a here string:

   1 while IFS= read -r line; do
   2   printf '%s\n' "$line"
   3 done <<< "$var"

The same can be done in any Bourne-type shell by using a "here document" (although read -r is POSIX, not Bourne):

   1 while IFS= read -r line; do
   2   printf '%s\n' "$line"
   3 done <<EOF
   4 $var
   5 EOF

One may also read from a command instead of a regular file:

   1 some command | while IFS= read -r line; do
   2   printf '%s\n' "$line"
   3 done

This method is especially useful for processing the output of find with a block of commands:

   1 find . -type f -print0 | while IFS= read -r -d '' file; do
   2     mv "$file" "${file// /_}"
   3 done

This reads one filename at a time from the find command and renames the file, replacing spaces with underscores.

Note the usage of -print0 in the find command, which uses NUL bytes as filename delimiters; and -d '' in the read command to instruct it to read all text into the file variable until it finds a NUL byte. By default, find and read delimit their input with newlines; however, since filenames can potentially contain newlines themselves, this default behaviour will split up those filenames at the newlines and cause the loop body to fail. Additionally it is necessary to set IFS to an empty string, because otherwise read would still strip leading and trailing whitespace. See FAQ #20 for more details.

Using a pipe to send find's output into a while loop places the loop in a SubShell, which means any state changes you make (changing variables, cd, opening and closing files, etc.) will be lost when the loop finishes. To avoid that, you may use a ProcessSubstitution:

   1 while IFS= read -r line; do
   2   printf '%s\n' "$line"
   3 done < <(some command)

See FAQ 24 for more discussion.

My text files are broken! They lack their final newlines!

If there are some characters after the last line in the file (or to put it differently, if the last line is not terminated by a newline character), then read will read it but return false, leaving the broken partial line in the read variable(s). You can process this after the loop:

   1 # Emulate cat
   2 while IFS= read -r line; do
   3   printf '%s\n' "$line"
   4 done < "$file"
   5 [[ -n $line ]] && printf %s "$line"

Or:

   1 # This does not work:
   2 printf 'line 1\ntruncated line 2' | while read -r line; do echo $line; done
   3 
   4 # This does not work either:
   5 printf 'line 1\ntruncated line 2' | while read -r line; do echo "$line"; done; [[ $line ]] && echo -n "$line"
   6 
   7 # This works:
   8 printf 'line 1\ntruncated line 2' | { while read -r line; do echo "$line"; done; [[ $line ]] && echo "$line"; }

The first example, beyond missing the after-loop test, is also missing quotes. See Quotes or Arguments for an explanation why. The Arguments page is an especially important read.

For a discussion of why the second example above does not work as expected, see FAQ #24.

Alternatively, you can simply add a logical OR to the while test:

   1 while IFS= read -r line || [[ -n $line ]]; do
   2   printf '%s\n' "$line"
   3 done < "$file"
   4 
   5 printf 'line 1\ntruncated line 2' | while read -r line || [[ -n $line ]]; do echo "$line"; done

How to keep other commands from "eating" the input

Some commands greedily eat up all available data on standard input. The examples above do not take precautions against such programs. For example,

   1 while read -r line; do
   2   cat > ignoredfile
   3   printf '%s\n' "$line"
   4 done < "$file"

will only print the contents of the first line, with the remaining contents going to "ignoredfile", as cat slurps up all available input.

One workaround is to use a numeric FileDescriptor rather than standard input:

   1 # Bash
   2 while IFS= read -r -u 9 line; do
   3   cat > ignoredfile
   4   printf '%s\n' "$line"
   5 done 9< "$file"
   6 
   7 # Note that read -u is not portable to every shell. Use a redirect to ensure it works in any POSIX compliant shell:
   8 while IFS= read -r line <&9; do
   9   cat > ignoredfile
  10   printf '%s\n' "$line"
  11 done 9< "$file"

Or:

   1 exec 9< "$file"
   2 while IFS= read -r line <&9; do
   3   cat > ignoredfile
   4   printf '%s\n' "$line"
   5 done
   6 exec 9<&-

This example will wait for the user to type something into the file ignoredfile at each iteration instead of eating up the loop input.

You might need this, for example, with mencoder which will accept user input if there is any, but will continue silently if there isn't. Other commands that act this way include ssh and ffmpeg. Additional workarounds for this are discussed in FAQ #89.


CategoryShell CategoryBashguide

BashFAQ/001 (last edited 2023-06-28 01:53:29 by larryv)