<> == How can I read a file (data stream, variable) line-by-line (and/or field-by-field)? == [[DontReadLinesWithFor|Don't try to use "for"]]. Use a `while` loop and the `read` command. Here is the basic template; there are many variations to discuss: {{{#!highlight bash while IFS= read -r line; do printf '%s\n' "$line" done < "$file" }}} `line` is a variable name, chosen by you. You can use any valid shell variable name(s) there; see [[#trimming|field splitting]] below. `< "$file"` redirects the loop's input from a file whose name is stored in a variable; see [[#source|source selection]] below. If you want to read lines from a file into an [[BashFAQ/005|array]], see [[BashFAQ/005|FAQ 5]]. <> <> === Field splitting, whitespace trimming, and other input processing === The `-r` option to `read` prevents backslash interpretation (usually used as a backslash newline pair, to continue over multiple lines or to escape the delimiters). Without this option, any unescaped backslashes in the input will be discarded. You should almost always use the `-r` option with read. The most common exception to this rule is when -e is used, which uses Readline to obtain the line from an interactive shell. In that case, tab completion will add backslashes to escape spaces and such, and you do not want them to be literally included in the variable. This would never be used when reading anything line-by-line, though, and -r should always be used when doing so. By default, `read` modifies each line read, by [[BashFAQ/067|removing all leading and trailing whitespace]] characters (spaces and tabs, if present in [[IFS]]). If that is not desired, the `IFS` variable may be cleared, as in the example above. If you want the trimming, leave `IFS` alone: {{{#!highlight bash # Leading/trailing whitespace trimming. while read -r line; do printf '%s\n' "$line" done < "$file" }}} If you want to operate on individual fields within each line, you may supply additional variables to {{{read}}}: {{{#!highlight bash # Input file has 3 columns separated by white space (space or tab characters only). while read -r first_name last_name phone; do # Only print the last name (second column) printf '%s\n' "$last_name" done < "$file" }}} If the field delimiters are not whitespace, you can set [[IFS|IFS (internal field separator)]]: {{{#!highlight bash # Extract the username and its shell from /etc/passwd: while IFS=: read -r user pass uid gid gecos home shell; do printf '%s: %s\n' "$user" "$shell" done < /etc/passwd }}} For tab-delimited files, use [[Quotes|IFS=$'\t']] though beware that multiple tab characters in the input will be considered as '''one''' delimiter (and the Ksh93/Zsh `IFS=$'\t\t'` workaround won't work in Bash). You do ''not'' necessarily need to know how many fields each line of input contains. If you supply more variables than there are fields, the extra variables will be empty. If you supply fewer, the last variable gets "all the rest" of the fields after the preceding ones are satisfied. For example, {{{#!highlight bash # Bash read -r first last junk <<< 'Bob Smith 123 Main Street Elk Grove Iowa 123-555-6789' # first will contain "Bob", and last will contain "Smith". # junk holds everything else. }}} Some people use the throwaway variable `_` as a "junk variable" to ignore fields. It (or indeed any variable) can also be used more than once in a single `read` command, if we don't care what goes into it: {{{#!highlight bash # Bash read -r _ _ first middle last _ <<< "$record" # We skip the first two fields, then read the next three. # Remember, the final _ can absorb any number of fields. # It doesn't need to be repeated there. }}} Note that this usage of `_` is only guaranteed to work in Bash. Many other shells use `_` for other purposes that will at best cause this to not have the desired effect, and can break the script entirely. It is better to choose a unique variable that isn't used elsewhere in the script, even though `_` is a common Bash convention. If avoiding comments starting with `#` is desired, you can simply skip them inside the loop: {{{#!highlight bash # Bash/ while read -r line; do [[ $line = \#* ]] && continue printf '%s\n' "$line" done < "$file" }}} Above `read` removes leading and trailing spaces or tabs (assuming `IFS` hasn't been modified from its default value) so we just need to look for a `#` at the start of the (now trimmed) line. To preserve the spacing: {{{#!highlight bash # Bash while IFS= read -r line; do [[ $line = *([[:blank:]])\#* ]] && continue printf '%s\n' "$line" done < "$file" }}} In older versions of Bash, you'd need `shopt -s extglob` for the `*(...)` [[glob#extglob|extended glob operator]] to be available. In newer versions they are always available for the `=`/`==`/`!=` pattern matching operator of the `[[ ... ]]` construct. <> === Input source selection === The [[BashGuide/InputAndOutput#Redirection|redirection]] `< "$file"` tells the `while` loop to read from the file whose name is in the variable `file`. If you would prefer to use a literal pathname instead of a variable, you may do that as well. If your input source is the script's standard input, then you don't need any redirection at all. If your input source is the contents of a variable/parameter, bash can iterate over its lines using a [[BashGuide/InputAndOutput#Heredocs_And_Herestrings|here string]]: {{{#!highlight bash while IFS= read -r line; do printf '%s\n' "$line" done <<< "$var" }}} The same can be done in any Bourne-type shell by using a "here document" (although `read -r` is POSIX, not Bourne): {{{#!highlight bash while IFS= read -r line; do printf '%s\n' "$line" done < ignoredfile printf '%s\n' "$line" done < "$file" }}} will only print the contents of the first line, with the remaining contents going to "ignoredfile", as `cat` slurps up all available input. One workaround is to use a numeric FileDescriptor rather than standard input: {{{#!highlight bash # Bash while IFS= read -r -u 9 line; do cat > ignoredfile printf '%s\n' "$line" done 9< "$file" # Note that read -u is not portable to every shell. # Use a redirect to ensure it works in any POSIX compliant shell: while IFS= read -r line <&9; do cat > ignoredfile printf '%s\n' "$line" done 9< "$file" }}} Or: {{{#!highlight bash exec 9< "$file" while IFS= read -r line <&9; do cat > ignoredfile printf '%s\n' "$line" done exec 9<&- }}} This example will wait for the user to type something into the file {{{ignoredfile}}} at each iteration instead of eating up the loop input. You might need this, for example, with `mencoder` which will accept user input if there is any, but will continue silently if there isn't. Other commands that act this way include `ssh` and `ffmpeg`. Additional workarounds for this are discussed in [[BashFAQ/089|FAQ #89]]. ---- CategoryShell CategoryBashguide