Differences between revisions 6 and 7
Revision 6 as of 2009-05-26 20:29:42
Size: 3247
Editor: GreyCat
Comment: add more examples and explanation
Revision 7 as of 2009-06-19 19:03:16
Size: 3330
Editor: localhost
Comment: mention time of substitution, and remove unnecessary 3>&1 3>&- stuff
Deletions are marked like this. Additions are marked like this.
Line 5: Line 5:
Process substitution comes in two forms: `<(some command)` and `>(some command)`. Each form either causes a [[NamedPipes|FIFO]] to be created under `/tmp`, or uses a file descriptor special device, depending on the operating system. The substitution syntax is replaced by the name of the FIFO or FD, and the command inside it is run in the background. Process substitution comes in two forms: `<(some command)` and `>(some command)`. Each form either causes a [[NamedPipes|FIFO]] to be created under `/tmp`, or uses a file descriptor special device, depending on the operating system. The substitution syntax is replaced by the name of the FIFO or FD, and the command inside it is run in the background.  The substitution is peformed at the same time as parameter expansion and command substitution.
Line 42: Line 42:
exec 3>&1 > >(tee logfile >&3) 3>&- exec > >(tee logfile)
Line 45: Line 45:
# Stdout of everything is logged, and also falls through to real stdout # Stdout of everything is logged, and also falls through to real stdout.

Process Substitution

Process substitution is a very useful BASH extension. It is similar to awk's "command" | getline and is especially important to get round subshells caused by pipelines.

Process substitution comes in two forms: <(some command) and >(some command). Each form either causes a FIFO to be created under /tmp, or uses a file descriptor special device, depending on the operating system. The substitution syntax is replaced by the name of the FIFO or FD, and the command inside it is run in the background. The substitution is peformed at the same time as parameter expansion and command substitution.

One of the most common uses of this feature is to avoid the creation of temporary files, e.g. when using diff(1):

diff <(sort < list1) <(sort < list2)

Another common use is avoiding the loss of variables inside a loop that is part of a pipeline. For example, this will fail:

# This example will fail, unless run in ksh88/ksh93
i=0
sort list1 | while read line; do
  i=$(($i + 1))
  ...
done
echo "$i lines processed"   # FAILS

But this works:

# Working example, using bash syntax.
i=0
while read line; do
  ((i++))
  ...
done < <(sort list1)
echo "$i lines processed"

The difference between <(...) and >(...) is merely which way the redirections are done. With <(...) one is expected to read from the substitution, and the command is set up to write to it. With >(...) one is expected to write to the substitution, and the command inside is set up to read from it.

>(...) is used less frequently; the most common situation is in conjunction with tee(1).

exec > >(tee logfile)

# Rest of script goes here
# Stdout of everything is logged, and also falls through to real stdout.
# Beware of buffering issues, especially if you also have stderr (e.g.
# prompts for user input may appear before the previous line of stdout).

Here's a more complicated example:

hasFile='Note: the (top-|highly )?secret plans are backed up at:(.*)'
criticalFile=
while read -r line
do
    [[ $line ]] || continue
    case $line in
        '!!! '*)
            errMsg "${line#'!!! '}" ;;
        *important* )
            echo "$line" ;;
        * )
            if [[ $line =~ $hasFile ]]; then
                criticalFile=${BASH_REMATCH[2]}
                warn "File at $criticalFile"
            else
                spin
            fi
            ;;
    esac
done < <(command $options "${param[@]}" 2>&1 | tee "$logfile")
[[ $criticalFile ]] || abort 'File not found'

Piping the command to a while loop would mean any variables set would be lost. Note that the actual command can be a pipeline. In fact you can continue to type a whole script in that side as well. Be aware that this is running in a subshell, and also that it will continue to run when your script exits (unless you manage your child processes.)

In the above example the regex could as easily be done with a case:

        'Note: the '*'secret plans are backed up at:'*) criticalFile=${line#*'secret plans are backed up at:'}

Process substitution where the external is an awk command, is particularly powerful and flexible.

Portability

Process substitution is definitely not portable.

ProcessSubstitution (last edited 2016-09-22 12:53:39 by geirha)