Diff for "BashFAQ/024"

I set variables in a loop. Why do they suddenly disappear after the loop terminates? Or, why can't I pipe data to read?

The problem

In most shells, each command of a pipeline is executed in a separate SubShell.

    # Non-working example (except in ksh88/ksh93)
    linecnt=0
    printf "%s\n" foo bar  | while read -r line
    do
        linecnt=$((linecnt+1))
    done
    echo "total number of lines: $linecnt" # prints 0

    # the problem also occurs without a loop
    var=0
    echo 2 | read -r var
    echo $var # also prints 0

The reason for this surprising behaviour is that a while/for/until loop runs in a SubShell when it's part of a pipeline. For the while loop above, a new subshell with its own copy of the variable linecnt is created (initial value, taken from the parent shell: "0"). This copy then is used for counting. When the while loop is finished, the subshell copy is discarded, and the original variable linecnt of the parent (whose value has not changed) is used in the echo command.

Different shells behave differently when using redirection or pipes with a loop:

• BourneShell creates a subshell when the input or output of anything (loops, case etc..) but a simple command is redirected, either by using a pipeline or by a redirection operator ('<', '>').

• BASH creates a new process only if the loop is part of a pipeline

• KornShell creates it only if the loop is part of a pipeline, but not if the loop is the last part of it. (The example above actually works in ksh88 and ksh93!)

• POSIX specifies the bash behaviour, but as an extension allows any or all of the parts of the pipeline to run without a subshell (thus permitting the KornShell behaviour, as well).

Workarounds

Several possibilities to avoid the subshell exist:

* If the input is a file, remove the useless use of cat

•   #POSIX
    while read -r line;do  linecnt=$((linecnt+1));done < file
    echo $linecnt

  Unfortunately this doesn't work with a Bourne shell see sh(1) from the Heirloom Bourne Shell for a workaround.

* Group the commands and do it all in the subshell

•     # POSIX
      linecnt=0
      cat /etc/passwd |
      {
          while read -r line ; do
             linecnt=$((linecnt+1))
          done
          echo "total number of lines: $linecnt"
      }

* Use process substitution (BASH only)

•     # Bash
      while read -r line; do
         linecnt=$((linecnt+1))
      done < <(grep PATH /etc/profile)
      echo "total number of lines: $linecnt"

  See also FAQ #1 and ProcessSubstitution

* Use a named pipe (POSIX)

•    # POSIX
     mkfifo mypipe
     grep PATH /etc/profile > mypipe &
     while read -r line;do
         linecnt=$((linecnt+1))
     done < mypipe
     echo "total number of lines: $linecnt"

• For more information see NamedPipes

* Use a coprocess (ksh, even pdksh, oksh, mksh..)

•   #ksh
    grep PATH /etc/profile |&
    while read -r -p line; do
      linecnt=$((linecnt+1))
    done
    echo "total number of lines: $linecnt"

* Another useful trick (using Bash/ksh93 syntax) is breaking a variable into words using read:

•     # Bash
      echo "$foo" | read -r a b c      # this doesn't work
      read -r a b c <<< "$foo"         # but this does

  Again, the pipeline causes the read command in the first example to run in a subshell, so its effect is never witnessed in the parent process. The second example does not create any subshells, so it works as we expect. The <<< operator is specific to bash (2.05b and later), and the input which follows it is usually called a "here string".

For more examples of how to break input into words, see FAQ #1.

* With a Posix shell you can use a here document instead:

•    #Posix
     read -r a b c << EOF
     $foo
     EOF