Diff for "BashFAQ/089"

Differences between revisions 1 and 20 (spanning 19 versions)

I'm reading a file line by line and running ssh or ffmpeg, only the first line gets processed!

When reading a file line by line, if a command inside the loop also reads stdin, it can exhaust the input file. For example:

   1 # Non-working example
   2 while IFS= read -r file; do
   3   ffmpeg -i "$file" -c:v libx264 -c:a aac "${file%.avi}".mkv
   4 done < <(find . -name '*.avi')

   1 # Non-working example
   2 while read host; do
   3   ssh "$host" some command
   4 done <hostslist

What's happening here? Let's take the first example. read reads a line from standard input (FD 0), puts it in the file parameter, and then ffmpeg is executed. Like any program you execute from BASH, ffmpeg inherits standard input, which for some reason it reads. I don't know why. But in any case, when ffmpeg reads stdin, it sucks up all the input from the find command, starving the loop.

Use the -nostdin global option in ffmpeg to disable interaction on standard input:

   1 while IFS= read -r file; do
   2   ffmpeg -nostdin -i "$file" -c:v libx264 -c:a aac "${file%.avi}".mkv
   3 done < <(find . -name '*.avi')

Alternatively you could use redirection at the end of the ffmpeg line: </dev/null. The ssh example can be fixed the same way, or with the -n switch (at least with OpenSSH).

Sometimes with large loops it might be difficult to work out what's reading from stdin, or a program might change its behaviour when you add </dev/null to it. In this case you can make read use a different FileDescriptor that a random program is less likely to read from:

   1 while read -r line <&3; do
   2   ...
   3 done 3<file

In bash, the read builtin can also be told to read directly from an fd (-u fd) without redirection, and since bash 4.1, an available fd can be assigned ({var}<file) instead of hard coding a file descriptor.

   1 # bash 4.1+
   2 while read -r -u "$fd" line; do
   3   ...
   4 done {fd}<file

-  ⇤ ← Revision 1 as of 2008-04-08 23:25:13 → 
  Size: 1196
  Editor: 82-71-12-170
  Comment:
+   ← Revision 20 as of 2020-02-25 20:51:21 → ⇥
  Size: 2124
  Editor: 10-29-237-24
  Comment: mention -nostdin ffmpeg option. modernize ffmpeg example options (libfaac doesn't exist anymore).
-Deletions are marked like this.
+Additions are marked like this.
 Line 1:
-[[Anchor(faq89)]]
== I'm using a loop which runs once per line of input but it only seems to run once; everything after the first line is ignored? ==
+<<Anchor(faq89)>>
== I'm reading a file line by line and running ssh or ffmpeg, only the first line gets processed! ==
When [[BashFAQ/001|reading a file line by line]], if a command inside the loop also reads stdin, it can exhaust the input file.  For example:
-Line 4:
+Line 5:
-Typically you'll see this behaviour in situations like these:
{{{
  while IFS= read -r file; do
    ffmpeg -i "$file" -vcodec libxvid -acodec libfaac -ar 32000 "${file%.avi}".mkv
  done < <(find . -name '*.avi')

  while read host
  do
    ssh "$host" command
  done <hostslist
+{{{#!highlight bash
# Non-working example
while IFS= read -r file; do
  ffmpeg -i "$file" -c:v libx264 -c:a aac "${file%.avi}".mkv
done < <(find . -name '*.avi')
-Line 16:
+Line 12:
-What's happening here?  Let's take the first example.  `read` reads a line from standard input (FD 0), puts it in the file parameter, and then ffmpeg is executed.  Like any program you execute from BASH, ffmpeg inherits standard input, which for some reason it reads.  I don't know why.  Here's how you make it work:
{{{
  while IFS= read -r file; do
    ffmpeg </dev/null -i "$file" -vcodec libxvid -acodec libfaac -ar 32000 "${file%.avi}".mkv
  done < <(find . -name '*.avi')
+{{{#!highlight bash
# Non-working example
while read host; do
  ssh "$host" some command
done <hostslist
-Line 23:
+Line 19:
-Notice the redirection on the ffmpeg line: `</dev/null`.  See the [:BashGuide#Redirection:redirection section] of the BASH Guide for more information on this.
+What's happening here?  Let's take the first example.  `read` reads a line from standard input (FD 0), puts it in the file parameter, and then `ffmpeg` is executed.  Like any program you execute from BASH, `ffmpeg` inherits standard input, which for some reason it reads.  I don't know why.  But in any case, when `ffmpeg` reads stdin, it sucks up all the input from the `find` command, starving the loop.
-Line 25:
+Line 21:
-The ssh example can be fixed the same way, or with the `-n` switch (at least with `openssh`).
+Use the `-nostdin` global option in `ffmpeg` to disable interaction on standard input:
{{{#!highlight bash
while IFS= read -r file; do
  ffmpeg -nostdin -i "$file" -c:v libx264 -c:a aac "${file%.avi}".mkv
done < <(find . -name '*.avi')
}}}
Alternatively you could use [[BashGuide/InputAndOutput#Redirection|redirection]] at the end of the ffmpeg line: `</dev/null`. The ssh example can be fixed the same way, or with the `-n` switch (at least with [[http://www.openssh.org/|OpenSSH]]).

Sometimes with large loops it might be difficult to work out what's reading from stdin, or a program might change its behaviour when you add `</dev/null` to it.  In this case you can make read use a different FileDescriptor that a random program is less likely to read from:
{{{#!highlight bash
while read -r line <&3; do
  ...
done 3<file
}}}

In bash, the `read` builtin can also be told to read directly from an fd (`-u fd`) without redirection, and since bash 4.1, an available fd can be assigned (`{var}<file`) instead of hard coding a file descriptor.

{{{#!highlight bash
# bash 4.1+
while read -r -u "$fd" line; do
  ...
done {fd}<file
}}}