Differences between revisions 1 and 4 (spanning 3 versions)
Revision 1 as of 2007-05-02 18:44:03
Size: 4150
Editor: redondos
Comment: creation
Revision 4 as of 2007-05-02 22:15:15
Size: 667
Editor: redondos
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
[[Anchor(head-5b201b72c451ac6de9f7e30e0387eae860815182)]]
Line 26: Line 27:

Also, please note that you do ''not'' necessarily need to know how many fields each line of input contains. If you supply more variables than there are fields, the extra variables will be empty. If you supply fewer, the last variable gets "all the rest" of the fields after the preceding ones are satisfied. For example,

{{{
    while read first_name last_name junk; do
      ...
    done <<< 'Bob Smith 123 Main Street Elk Grove Iowa 123-555-6789'
    # Inside the loop, first_name will contain "Bob", and
    # last_name will contain "Smith". The variable "junk" holds
    # everything else.
}}}

The {{{read}}} command modifies each line read, e.g. by default it removes all leading whitespace characters (blanks, tab characters, ... -- basically any leading characters present in IFS). If that is not desired, the {{{IFS}}} variable has to be cleared:

{{{
    OIFS=$IFS; IFS=
    while read line
    do
        echo "$line"
    done < "$file"
    IFS=$OIFS
}}}

As a feature, the {{{read}}} command concatenates lines that end with a backslash '\' character to one single line. To disable this feature, KornShell and ["BASH"] have {{{read -r}}}:

{{{
    OIFS=$IFS; IFS=
    while read -r line
    do
        echo "$line"
    done < "$file"
    IFS=$OIFS
}}}

'''Note that reading a file line by line this way is ''very slow'' for large files. Consider using e.g. ["AWK"] instead if you get performance problems.'''


One may also read from a command instead of a regular file:

{{{
    some command | while read line; do
       other commands
    done
}}}

This method is especially useful for processing the output of ''find'' with a block of commands:

{{{
    find . -print0 | while read -d $'\0' file; do
        mv "$file" "${file// /_}"
    done
}}}

This command reads one filename at a time from the file command and renames the file so that its spaces are replaced by underscores.

Note the usage of {{{-print0}}} in the find command, which uses NUL bytes as filename delimiters, and {{{-d $'\0'}}} in the read command to instruct it to read all text into the file variable until it finds a NUL byte. By default, find and read delimit their input with newlines; however, since filenames can potentially contain newlines themselves, this default behaviour will split those filenames with newlines up and cause the command block to fail. See [#faq20 FAQ #20] for more details.

Using a pipe to send find's output into a while loop places the loop in a ''subshell'' and may therefore cause problems later on if the commands inside the body of the loop attempt to set variables which need to be used outside the loop; in that case, see [#faq24 FAQ 24], or use process substitution like:

{{{
    while read line; do
        other commands
    done < <(some command)
}}}

Sometimes it's useful to read a file into an array, one array element per line. You can do that with the following example:

{{{
    O=$IFS IFS=$'\n' arr=($(< myfile)) IFS=$O
}}}

This temporarily changes the Input Field Separator to a newline, so that each line will be considered one field by read. Then it populates the array {{{arr}}} with the fields. Then it sets the {{{IFS}}} back to what it was before.

This same trick works on a stream of data as well as a file:

{{{
    O=$IFS IFS=$'\n' arr=($(find . -type f)) IFS=$O
}}}

Of course, this will blow up in your face if the filenames contain newlines; see [020 FAQ 20] for hints on dealing with such filenames.

Anchor(head-5b201b72c451ac6de9f7e30e0387eae860815182) Anchor(faq1)

How can I read a file line-by-line?

    while read line
    do
        echo "$line"
    done < "$file"

If you want to operate on individual fields within each line, you may supply additional variables to read:

    # Input file has 3 columns separated by white space.
    while read first_name last_name phone; do
      ...
    done < "$file"

If the field delimiters are not whitespace, you can set IFS (input field separator):

    while IFS=: read user pass uid gid gecos home shell; do
      ...
    done < /etc/passwd

BashFAQ/001 (last edited 2023-06-28 01:53:29 by larryv)