Differences between revisions 10 and 11
Revision 10 as of 2010-07-02 18:49:59
Size: 3990
Editor: GreyCat
Comment:
Revision 11 as of 2010-07-30 14:43:25
Size: 4013
Editor: GreyCat
Comment: category
Deletions are marked like this. Additions are marked like this.
Line 100: Line 100:

----
CategoryShell

Process Substitution

Process substitution is a very useful BASH extension. It is similar to awk's "command" | getline and is especially important to get round subshells caused by pipelines.

Process substitution comes in two forms: <(some command) and >(some command). Each form either causes a FIFO to be created under /tmp or /var/tmp, or uses a file descriptor special device (/dev/fd/*), depending on the operating system. The substitution syntax is replaced by the name of the FIFO or FD, and the command inside it is run in the background. The substitution is peformed at the same time as parameter expansion and command substitution.

One of the most common uses of this feature is to avoid the creation of temporary files, e.g. when using diff(1):

diff <(sort list1) <(sort list2)

This is (roughly) equivalent to:

mkfifo /var/tmp/fifo1
mkfifo /var/tmp/fifo2
sort list1 >/var/tmp/fifo1 &
sort list2 >/var/tmp/fifo2 &
diff /var/tmp/fifo1 /var/tmp/fifo2
rm /var/tmp/fifo1 /var/tmp/fifo2

Note that the diff command actually receives two filename arguments.

Another common use is avoiding the loss of variables inside a loop that is part of a pipeline. For example, this will fail:

# This example will fail, unless run in ksh88/ksh93
i=0
sort list1 | while read line; do
  i=$(($i + 1))
  ...
done
echo "$i lines processed"   # FAILS

But this works:

# Working example, using bash syntax.
i=0
while read line; do
  ((i++))
  ...
done < <(sort list1)
echo "$i lines processed"

The difference between <(...) and >(...) is merely which way the redirections are done. With <(...) one is expected to read from the substitution, and the command is set up to write to it. With >(...) one is expected to write to the substitution, and the command inside is set up to read from it.

>(...) is used less frequently; the most common situation is in conjunction with tee(1).

exec > >(tee logfile)

# Rest of script goes here
# Stdout of everything is logged, and also falls through to real stdout.
# Beware of buffering issues, especially if you also have stderr (e.g.
# prompts for user input may appear before the previous line of stdout).

Here's a more complicated example:

hasFile='Note: the (top-|highly )?secret plans are backed up at:(.*)'
criticalFile=
while read -r line
do
    [[ $line ]] || continue
    case $line in
        '!!! '*)
            errMsg "${line#'!!! '}" ;;
        *important* )
            echo "$line" ;;
        * )
            if [[ $line =~ $hasFile ]]; then
                criticalFile=${BASH_REMATCH[2]}
                warn "File at $criticalFile"
            else
                spin
            fi
            ;;
    esac
done < <(command $options "${param[@]}" 2>&1 | tee "$logfile")
[[ $criticalFile ]] || abort 'File not found'

Piping the command to a while loop would mean any variables set would be lost. Note that the actual command can be a pipeline. In fact you can continue to type a whole script in that side as well. Be aware that this is running in a subshell, and also that it will continue to run when your script exits (unless you manage your child processes.)

In the above example the regex could as easily be done with a case:

        'Note: the '*'secret plans are backed up at:'*) criticalFile=${line#*'secret plans are backed up at:'}

Process substitution where the external is an awk command, is particularly powerful and flexible.

Portability

Process substitution is definitely not portable. You may use NamedPipes to accomplish the same things.

zsh and ksh93 support process substitution, but, in the case of ksh93 you cannot use it with redirection, ie cat <(echo foo) works but not cat < <(echo foo). zsh ksh and bash4 also offer coprocess which can sometimes be used instead of process substitution.


CategoryShell

ProcessSubstitution (last edited 2016-09-22 12:53:39 by geirha)