Differences between revisions 1 and 42 (spanning 41 versions)
Revision 1 as of 2007-05-02 23:11:07
Size: 3986
Editor: redondos
Comment:
Revision 42 as of 2022-03-10 13:02:31
Size: 6587
Editor: 46
Comment: POSIX read requires a var param, usually I would pick _ to indicate that the var isn't going to be used, but in accordance with the other examples on this page, I used "line"
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
[[Anchor(faq24)]]
== I set variables in a loop. Why do they suddenly disappear after the loop terminates? ==

The following command always prints "total number of lines: 0", although the variable {{{linecnt}}} has a larger value in the {{{while}}} loop:
<<Anchor(faq24)>>
== I set variables in a loop that's in a pipeline. Why do they disappear after the loop terminates? Or, why can't I pipe data to read? ==
In most shells, each command of a pipeline is executed in a separate SubShell. Non-working example:
Line 7: Line 6:
    linecnt=0
    cat /etc/passwd | while read line
    do
        linecnt=`expr $linecnt + 1`
    done
    echo "total number of lines: $linecnt"
# Works only in ksh88/ksh93, or zsh or bash 4.2 with lastpipe enabled
# In other shells, this will print 0
linecount=0

printf '%s\n' foo bar |
while IFS= read -r line
do
    linecount=$((linecount + 1))
done

echo "total number of lines: $linecount"
Line 15: Line 19:
The reason for this surprising behaviour is that a {{{while/for/until}}} loop runs in a subshell when its input or output is redirected from a pipeline. For the {{{while}}} loop above, a new subshell with its own copy of the variable {{{linecnt}}} is created (initial value, taken from the parent shell: "0"). This copy then is used for counting. When the {{{while}}} loop is finished, the subshell copy is discarded, and the original variable {{{linecnt}}} of the parent (whose value has not changed) is used in the {{{echo}}} command. The reason for this potentially surprising behaviour, as described above, is that each SubShell introduces a new variable context and environment. The `while` loop above is executed in a new subshell with its own copy of the variable `linecount` created with the initial value of '0' taken from the parent shell. This copy then is used for counting. When the `while` loop is finished, the subshell copy is discarded, and the original variable `linecount` of the parent (whose value hasn't changed) is used in the `echo` command.
Line 17: Line 21:
It's hard to tell when shell would create a new process for a loop:
 * BourneShell creates it when the input or output is redirected, either by using a pipeline or by a redirection operator ('<', '>').
 * ["BASH"] creates a new process only if the loop is part of a pipeline
 * KornShell creates it only if the loop is part of a pipeline, but ''not'' if the loop is the last part of it.
Different shells exhibit different behaviors in this situation:
 * BourneShell creates a subshell when the input or output of anything (loops, case etc..) but a simple command is redirected, either by using a pipeline or by a redirection operator ('<', '>').
 * [[BASH]], Yash and PDKsh-derived shells create a new process only if the loop is part of a pipeline.
 * KornShell and Zsh creates it only if the loop is part of a pipeline, but ''not'' if the loop is the last part of it. The read example above actually ''works'' in ksh88, ksh93, zsh! (but not MKsh or other PDKsh-derived shells)
 * POSIX specifies the bash behaviour, but as an extension allows any or all of the parts of the pipeline to run without a subshell (thus permitting the KornShell behaviour, as well).
Line 22: Line 27:
To solve this, either use a method that works without a subshell (shown below), or make sure you do all processing inside that subshell (a bit of a kludge, but easier to work with): More broken stuff:
{{{
# Bash 4
# The problem also occurs without a loop
printf '%s\n' foo bar | mapfile -t line
printf 'total number of lines: %s\n' "${#line[@]}" # prints 0
}}}
Line 25: Line 36:
    linecnt=0
    cat /etc/passwd |
    (
        while read line ; do
                linecnt="$((linecnt+1))"
        done
        echo "total number of lines: $linecnt"
    )
f() {
    if [[ -t 0 ]]; then
        echo "$1"
    else
        read -r var
    fi
}

f 'hello' | f
echo "$var" # prints nothing
Line 35: Line 48:
To avoid the subshell completely (not easily possible if the other part of the pipe is a command!), use redirection, which does not have this problem at least for ["BASH"] and KornShell (but still for BourneShell): Again, in both cases the pipeline causes `read` or some containing command to run in a subshell, so its effect is never witnessed in the parent process.
Line 37: Line 50:
{{{
    linecnt=0
    while read line ; do
        linecnt="$((linecnt+1))"
   done < /etc/passwd
   echo "total number of lines: $linecnt"
It should be stressed that this issue isn't specific to loops. It's a general property of all pipes, though the `while/read` loop might be considered the canonical example that crops up over and over when people read the help or manpage description of the `read` builtin and notice that it accepts data on stdin. They might recall that data redirected into a compound command is available throughout that command, but not understand why all the fancy process substitutions and redirects they run across in places like [[BashFAQ/001|FAQ #1]] are necessary. Naturally they proceed to put their funstuff directly into a pipeline, and confusion ensues.

=== Workarounds ===

 * If the input is a file, a simple redirect will suffice:

 {{{
# POSIX
while IFS= read -r line; do linecount=$((linecount + 1)); done < file
echo "$linecount"
Line 45: Line 62:
For ["BASH"], when the first part of the pipe is a command, you can use "process substitution". The command used here is a simple "echo -e $'a\nb\nc'" as a substitute for a command with a multiline output:  Unfortunately, this doesn't work with a Bourne shell; see [[http://heirloom.sourceforge.net/sh/sh.1.html#20|sh(1) from the Heirloom Bourne Shell]] for a workaround.
Line 47: Line 64:
{{{
    while read LINE; do
        echo "-> $LINE"
    done < <(echo -e $'a\nb\nc')
 * Use [[BashGuide/CompoundCommands#Command_grouping|command grouping]] and do everything in the subshell:

 {{{
# POSIX
linecount=0

cat /etc/passwd |
{
    while IFS= read -r line
    do
        linecount=$((linecount + 1))
    done

    echo "total number of lines: $linecount"
}
Line 53: Line 81:
A portable and common work-around is to redirect the input of the {{{read}}} command using {{{exec}}}:  This doesn't really change the subshell situation, but if nothing from the subshell is needed in the rest of your code then destroying the local environment after you're through with it could be just what you want anyway.
Line 55: Line 83:
{{{
    linecnt=0
    exec < /etc/passwd # redirect standard input from the file /etc/passwd
    while read line # "read" gets its input from the file /etc/passwd
    do
        linecnt=`expr $linecnt + 1`
    done
    echo "total number of lines: $linecnt"
 * Use ProcessSubstitution (Bash/Zsh/Ksh93 only):

 {{{
# Bash/Ksh93/Zsh
while IFS= read -r line
do
    ((linecount++))
done < <(grep PATH /etc/profile)

echo "total number of lines: $linecount"
Line 65: Line 95:
This works as expected, and prints a line count for the file /etc/passwd. But the input is redirected from that file permanently. What if we need to read the original standard input sometime later again? In that case we have to save a copy of the original standard input file descriptor, which we later can restore:  This is essentially identical to the first workaround above. We still redirect a file, only this time the file happens to be a named pipe temporarily created by our process substitution to transport the output of grep.
Line 67: Line 97:
{{{
    exec 3<&0 # save original standard input file descriptor "0" as FD "3"
    exec 0</etc/passwd # redirect standard input from the file /etc/passwd
 * Use a [[NamedPipes|named pipe]]:
Line 71: Line 99:
    linecnt=0
    while read line # "read" gets its input from the file /etc/passwd
    do
        linecnt=`expr $linecnt + 1`
    done
 {{{
# POSIX
mkfifo mypipe
grep PATH /etc/profile > mypipe &
Line 77: Line 104:
    exec 0<&3 # restore saved standard input (fd 0) from file descriptor "3"
    exec 3<&- # close the no longer needed file descriptor "3"
while IFS= read -r line
do
    linecount=$((linecount + 1))
done < mypipe
Line 80: Line 109:
    echo "total number of lines: $linecnt" echo "total number of lines: $linecount"
Line 83: Line 112:
Subsequent {{{exec}}} commands can be combined into one line, which is interpreted left-to-right:  * Use a [[http://wiki.bash-hackers.org/syntax/keywords/coproc|coprocess]] (ksh, even pdksh, oksh, mksh..):
Line 85: Line 114:
{{{
    exec 3<&0
    exec 0</etc/passwd
    _...read redirected standard input..._
    exec 0<&3
    exec 3<&-
 {{{
# ksh
grep PATH /etc/profile |&

while IFS= read -r -p line
do
    linecount=$((linecount + 1))
done

echo "total number of lines: $linecount"
Line 93: Line 126:
is equivalent to  {{{
# bash>4
coproc grep PATH /etc/profile
Line 95: Line 130:
{{{
    exec 3<&0 0</etc/passwd
    _...read redirected standard input..._
    exec 0<&3 3<&-
while IFS= read -r line
do
    linecount=$((linecount + 1))
done <&"${COPROC[0]}"

echo "total number of lines: $linecount"
Line 100: Line 137:

 * Use a HereString (Bash/Zsh/Ksh93 only, though the example uses the Bash-specific {{{read -a}}} (Ksh93 and Zsh using {{{read -A}}} instead)):

 {{{
# Options:
# -r Backslash does not act as an escape character for the word separators or line delimiter.
# -a The words are assigned to sequential indices of the array "words"

read -ra words <<< 'hi ho hum'
printf 'total number of words: %d\n' "${#words[@]}"
}}}

 The `<<<` operator is available in Bash (2.05b and later), Zsh (where it was first introduced inspired from a similar operator in the Unix port of the `rc` shell), Ksh93 and Yash.

 * With a POSIX shell, or for longer multi-line data, you can use a here document instead:

 {{{
# POSIX
linecount=0
while IFS= read -r line; do
    linecount=$((linecount+1))
done <<EOF
hi
ho
hum
EOF

printf 'total number of lines: %d\n' "$linecount"
}}}

 * Use lastpipe (Bash 4.2)

 {{{
# Bash 4.2
# +m: Disable monitor mode (job control). Background processes display their
# exit status upon completion when in monitor mode (we don't want that).
set +m
shopt -s lastpipe

x=0
printf '%s\n' hi{,,,,,} | while IFS= read -r "lines[x++]"; do :; done
printf 'total number of lines: %d\n' "${#lines[@]}"
}}}

 Bash 4.2 introduces the aforementioned ksh-like behavior to Bash. The one caveat is that job control must not be enabled, thereby limiting its usefulness in an interactive shell.

For more related examples of how to read input and break it into words, see [[BashFAQ/001|FAQ #1]].

----
CategoryShell

I set variables in a loop that's in a pipeline. Why do they disappear after the loop terminates? Or, why can't I pipe data to read?

In most shells, each command of a pipeline is executed in a separate SubShell. Non-working example:

# Works only in ksh88/ksh93, or zsh or bash 4.2 with lastpipe enabled
# In other shells, this will print 0
linecount=0

printf '%s\n' foo bar |
while IFS= read -r line
do
    linecount=$((linecount + 1))
done

echo "total number of lines: $linecount"

The reason for this potentially surprising behaviour, as described above, is that each SubShell introduces a new variable context and environment. The while loop above is executed in a new subshell with its own copy of the variable linecount created with the initial value of '0' taken from the parent shell. This copy then is used for counting. When the while loop is finished, the subshell copy is discarded, and the original variable linecount of the parent (whose value hasn't changed) is used in the echo command.

Different shells exhibit different behaviors in this situation:

  • BourneShell creates a subshell when the input or output of anything (loops, case etc..) but a simple command is redirected, either by using a pipeline or by a redirection operator ('<', '>').

  • BASH, Yash and PDKsh-derived shells create a new process only if the loop is part of a pipeline.

  • KornShell and Zsh creates it only if the loop is part of a pipeline, but not if the loop is the last part of it. The read example above actually works in ksh88, ksh93, zsh! (but not MKsh or other PDKsh-derived shells)

  • POSIX specifies the bash behaviour, but as an extension allows any or all of the parts of the pipeline to run without a subshell (thus permitting the KornShell behaviour, as well).

More broken stuff:

# Bash 4
# The problem also occurs without a loop
printf '%s\n' foo bar | mapfile -t line
printf 'total number of lines: %s\n' "${#line[@]}" # prints 0

f() {
    if [[ -t 0 ]]; then
        echo "$1"
    else
        read -r var
    fi
}

f 'hello' | f
echo "$var" # prints nothing

Again, in both cases the pipeline causes read or some containing command to run in a subshell, so its effect is never witnessed in the parent process.

It should be stressed that this issue isn't specific to loops. It's a general property of all pipes, though the while/read loop might be considered the canonical example that crops up over and over when people read the help or manpage description of the read builtin and notice that it accepts data on stdin. They might recall that data redirected into a compound command is available throughout that command, but not understand why all the fancy process substitutions and redirects they run across in places like FAQ #1 are necessary. Naturally they proceed to put their funstuff directly into a pipeline, and confusion ensues.

Workarounds

  • If the input is a file, a simple redirect will suffice:
    # POSIX
    while IFS= read -r line; do linecount=$((linecount + 1)); done < file
    echo "$linecount"

    Unfortunately, this doesn't work with a Bourne shell; see sh(1) from the Heirloom Bourne Shell for a workaround.

  • Use command grouping and do everything in the subshell:

    # POSIX
    linecount=0
    
    cat /etc/passwd |
    {
        while IFS= read -r line
        do
            linecount=$((linecount + 1))
        done
    
        echo "total number of lines: $linecount"
    }
    This doesn't really change the subshell situation, but if nothing from the subshell is needed in the rest of your code then destroying the local environment after you're through with it could be just what you want anyway.
  • Use ProcessSubstitution (Bash/Zsh/Ksh93 only):

    # Bash/Ksh93/Zsh
    while IFS= read -r line
    do
        ((linecount++))
    done < <(grep PATH /etc/profile)
    
    echo "total number of lines: $linecount"
    This is essentially identical to the first workaround above. We still redirect a file, only this time the file happens to be a named pipe temporarily created by our process substitution to transport the output of grep.
  • Use a named pipe:

    # POSIX
    mkfifo mypipe
    grep PATH /etc/profile > mypipe &
    
    while IFS= read -r line
    do
        linecount=$((linecount + 1))
    done < mypipe
    
    echo "total number of lines: $linecount"
  • Use a coprocess (ksh, even pdksh, oksh, mksh..):

    # ksh
    grep PATH /etc/profile |&
    
    while IFS= read -r -p line
    do
        linecount=$((linecount + 1))
    done
    
    echo "total number of lines: $linecount"
    # bash>4
    coproc grep PATH /etc/profile
    
    while IFS= read -r line
    do
        linecount=$((linecount + 1))
    done <&"${COPROC[0]}"
    
    echo "total number of lines: $linecount"
  • Use a HereString (Bash/Zsh/Ksh93 only, though the example uses the Bash-specific read -a (Ksh93 and Zsh using read -A instead)):

    # Options:
    # -r Backslash does not act as an escape character for the word separators or line delimiter.
    # -a The words are assigned to sequential indices of the array "words"
    
    read -ra words <<< 'hi ho hum'
    printf 'total number of words: %d\n' "${#words[@]}"

    The <<< operator is available in Bash (2.05b and later), Zsh (where it was first introduced inspired from a similar operator in the Unix port of the rc shell), Ksh93 and Yash.

  • With a POSIX shell, or for longer multi-line data, you can use a here document instead:
    # POSIX
    linecount=0
    while IFS= read -r line; do
        linecount=$((linecount+1))
    done <<EOF
    hi
    ho
    hum
    EOF
    
    printf 'total number of lines: %d\n' "$linecount"
  • Use lastpipe (Bash 4.2)
    # Bash 4.2
    # +m: Disable monitor mode (job control). Background processes display their
    #     exit status upon completion when in monitor mode (we don't want that).
    set +m
    shopt -s lastpipe
    
    x=0
    printf '%s\n' hi{,,,,,} | while IFS= read -r "lines[x++]"; do :; done
    printf 'total number of lines: %d\n' "${#lines[@]}"
    Bash 4.2 introduces the aforementioned ksh-like behavior to Bash. The one caveat is that job control must not be enabled, thereby limiting its usefulness in an interactive shell.

For more related examples of how to read input and break it into words, see FAQ #1.


CategoryShell

BashFAQ/024 (last edited 2023-12-12 13:15:33 by 195)