Size: 1402
Comment: minor: quoting fix
|
Size: 4085
Comment: link to BashFAQ/106 (forthcoming)
|
Deletions are marked like this. | Additions are marked like this. |
Line 3: | Line 3: |
Process Substitution is a very useful BASH extension. It is similar to awk's {{{"command" | getline}}} and is especially important to get round subshell restrictions in pipes, eg: | Process substitution is a very useful BASH extension. It is similar to awk's {{{"command" | getline}}} and is especially important to get round [[SubShell|subshells]] caused by pipelines. Process substitution comes in two forms: `<(some command)` and `>(some command)`. Each form either causes a [[NamedPipes|FIFO]] to be created under `/tmp` or `/var/tmp`, or uses a file descriptor special device (`/dev/fd/*`), depending on the operating system. The substitution syntax is replaced by the name of the FIFO or FD, and the command inside it is run in the background. The substitution is peformed at the same time as parameter expansion and command substitution. One of the most common uses of this feature is to avoid the creation of temporary files, e.g. when using `diff(1)`: {{{ diff <(sort list1) <(sort list2) }}} This is (roughly) equivalent to: {{{ mkfifo /var/tmp/fifo1 mkfifo /var/tmp/fifo2 sort list1 >/var/tmp/fifo1 & sort list2 >/var/tmp/fifo2 & diff /var/tmp/fifo1 /var/tmp/fifo2 rm /var/tmp/fifo1 /var/tmp/fifo2 }}} Note that the `diff` command actually receives two filename arguments. Another common use is avoiding the [[BashFAQ/024|loss of variables inside a loop that is part of a pipeline]]. For example, this will fail: {{{ # This example will fail, unless run in ksh88/ksh93 i=0 sort list1 | while read line; do i=$(($i + 1)) ... done echo "$i lines processed" # FAILS }}} But this works: {{{ # Working example, using bash syntax. i=0 while read line; do ((i++)) ... done < <(sort list1) echo "$i lines processed" }}} The difference between `<(...)` and `>(...)` is merely which way the redirections are done. With `<(...)` one is expected to ''read'' from the substitution, and the command is set up to write to it. With `>(...)` one is expected to ''write'' to the substitution, and the command inside is set up to read from it. `>(...)` is used less frequently; the most common situation is in conjunction with `tee(1)`. {{{ exec > >(tee logfile) # Rest of script goes here # Stdout of everything is logged, and also falls through to real stdout. # Beware of buffering issues, especially if you also have stderr (e.g. # prompts for user input may appear before the previous line of stdout). }}} See [[BashFAQ/106|Bash FAQ #106]] for more discussion of that usage. Here's a more complicated example: |
Line 9: | Line 70: |
do [[ $line ]] || continue case $line in '!!! '*) errMsg "${line#'!!! '}" ;; *important* ) echo "$line" ;; * ) if [[ $line =~ $hasFile ]]; then |
do [[ $line ]] || continue case $line in '!!! '*) errMsg "${line#'!!! '}" ;; *important* ) echo "$line" ;; * ) if [[ $line =~ $hasFile ]]; then |
Line 16: | Line 81: |
else spin | else spin |
Line 18: | Line 84: |
;; esac done < <(command $options "${param[@]}" 2>&1|tee "$logfile") |
;; esac done < <(command $options "${param[@]}" 2>&1 | tee "$logfile") |
Line 32: | Line 99: |
Process substitution is definitely '''not''' portable. | Process substitution is definitely '''not''' portable. You may use NamedPipes to accomplish the same things. zsh and ksh93 support process substitution, but, in the case of ksh93 you cannot use it with redirection, ie ''cat <(echo foo)'' works but not ''cat < <(echo foo)''. zsh ksh and bash4 also offer coprocess which can sometimes be used instead of process substitution. ---- CategoryShell |
Process Substitution
Process substitution is a very useful BASH extension. It is similar to awk's "command" | getline and is especially important to get round subshells caused by pipelines.
Process substitution comes in two forms: <(some command) and >(some command). Each form either causes a FIFO to be created under /tmp or /var/tmp, or uses a file descriptor special device (/dev/fd/*), depending on the operating system. The substitution syntax is replaced by the name of the FIFO or FD, and the command inside it is run in the background. The substitution is peformed at the same time as parameter expansion and command substitution.
One of the most common uses of this feature is to avoid the creation of temporary files, e.g. when using diff(1):
diff <(sort list1) <(sort list2)
This is (roughly) equivalent to:
mkfifo /var/tmp/fifo1 mkfifo /var/tmp/fifo2 sort list1 >/var/tmp/fifo1 & sort list2 >/var/tmp/fifo2 & diff /var/tmp/fifo1 /var/tmp/fifo2 rm /var/tmp/fifo1 /var/tmp/fifo2
Note that the diff command actually receives two filename arguments.
Another common use is avoiding the loss of variables inside a loop that is part of a pipeline. For example, this will fail:
# This example will fail, unless run in ksh88/ksh93 i=0 sort list1 | while read line; do i=$(($i + 1)) ... done echo "$i lines processed" # FAILS
But this works:
# Working example, using bash syntax. i=0 while read line; do ((i++)) ... done < <(sort list1) echo "$i lines processed"
The difference between <(...) and >(...) is merely which way the redirections are done. With <(...) one is expected to read from the substitution, and the command is set up to write to it. With >(...) one is expected to write to the substitution, and the command inside is set up to read from it.
>(...) is used less frequently; the most common situation is in conjunction with tee(1).
exec > >(tee logfile) # Rest of script goes here # Stdout of everything is logged, and also falls through to real stdout. # Beware of buffering issues, especially if you also have stderr (e.g. # prompts for user input may appear before the previous line of stdout).
See Bash FAQ #106 for more discussion of that usage.
Here's a more complicated example:
hasFile='Note: the (top-|highly )?secret plans are backed up at:(.*)' criticalFile= while read -r line do [[ $line ]] || continue case $line in '!!! '*) errMsg "${line#'!!! '}" ;; *important* ) echo "$line" ;; * ) if [[ $line =~ $hasFile ]]; then criticalFile=${BASH_REMATCH[2]} warn "File at $criticalFile" else spin fi ;; esac done < <(command $options "${param[@]}" 2>&1 | tee "$logfile") [[ $criticalFile ]] || abort 'File not found'
Piping the command to a while loop would mean any variables set would be lost. Note that the actual command can be a pipeline. In fact you can continue to type a whole script in that side as well. Be aware that this is running in a subshell, and also that it will continue to run when your script exits (unless you manage your child processes.)
In the above example the regex could as easily be done with a case:
'Note: the '*'secret plans are backed up at:'*) criticalFile=${line#*'secret plans are backed up at:'}
Process substitution where the external is an awk command, is particularly powerful and flexible.
Portability
Process substitution is definitely not portable. You may use NamedPipes to accomplish the same things.
zsh and ksh93 support process substitution, but, in the case of ksh93 you cannot use it with redirection, ie cat <(echo foo) works but not cat < <(echo foo). zsh ksh and bash4 also offer coprocess which can sometimes be used instead of process substitution.