Diff for "BashFAQ/089"

Differences between revisions 16 and 23 (spanning 7 versions)

I'm reading a file line by line and running ssh or ffmpeg, only the first line gets processed!

When reading a file line by line, if a command inside the loop also reads stdin, it can exhaust the input file. For example:

   1 # Non-working example
   2 while IFS= read -r file; do
   3   ffmpeg -i "$file" -c:v libx264 -c:a aac "${file%.avi}".mkv
   4 done < <(find . -name '*.avi')

   1 # Non-working example
   2 while read host; do
   3   ssh "$host" some command
   4 done < hostslist

What's happening here? Let's take the first example. read reads a line from standard input (FD 0), puts it in the file parameter, and then ffmpeg is executed. Like any program you execute from BASH, ffmpeg inherits standard input. However, ffmpeg additionally uses standard input to detect quit commands indicated by user input of q, thus sucking up all the input from the find command and starving the loop.

Use the -nostdin global option in ffmpeg to disable interaction on standard input:

   1 while IFS= read -r file; do
   2   ffmpeg -nostdin -i "$file" -c:v libx264 -c:a aac "${file%.avi}".mkv
   3 done < <(find . -name '*.avi')

Alternatively you could use redirection at the end of the ffmpeg line: </dev/null. The ssh example can be fixed the same way, or with the -n switch (at least with OpenSSH).

Sometimes with large loops it might be difficult to work out what's reading from stdin, or a program might change its behaviour when you add </dev/null to it. In this case you can make read use a different FileDescriptor that a random program is less likely to read from:

   1 while IFS= read -r line <&3; do
   2   ...
   3 done 3< file

In bash, the read builtin can also be told to read directly from an fd (-u fd) without redirection, and since bash 4.1, an available fd can be assigned ({var}<file) instead of hard coding a file descriptor.

   1 # bash 4.1+
   2 while IFS= read -r -u "$fd" line; do
   3   ...
   4 done {fd}< file
   5 exec {fd}<&-

-  ⇤ ← Revision 16 as of 2015-01-07 14:51:17 → 
  Size: 1705
  Editor: GreyCat
  Comment: read -u 3 is dumb.  don't even bother showing it.
+   ← Revision 23 as of 2024-04-13 21:56:35 → ⇥
  Size: 2170
  Editor: Reg
  Comment: Why ffmpeg sucks up stdin.
-Deletions are marked like this.
+Additions are marked like this.
 Line 5:
-{{{
+{{{#!highlight bash
 Line 8:
-  ffmpeg -i "$file" -vcodec libxvid -acodec libfaac -ar 32000 "${file%.avi}".mkv
+  ffmpeg -i "$file" -c:v libx264 -c:a aac "${file%.avi}".mkv
 Line 12:
-{{{
+{{{#!highlight bash
 Line 16:
-done <hostslist
+done < hostslist
 Line 19:
-What's happening here?  Let's take the first example.  `read` reads a line from standard input (FD 0), puts it in the file parameter, and then `ffmpeg` is executed.  Like any program you execute from BASH, `ffmpeg` inherits standard input, which for some reason it reads.  I don't know why.  But in any case, when `ffmpeg` reads stdin, it sucks up all the input from the `find` command, starving the loop.
+What's happening here?  Let's take the first example.  `read` reads a line from standard input (FD 0), puts it in the file parameter, and then `ffmpeg` is executed.  Like any program you execute from BASH, `ffmpeg` inherits standard input.  However, `ffmpeg` additionally uses standard input to detect quit commands indicated by user input of `q`, thus sucking up all the input from the `find` command and starving the loop.
 Line 21:
-Here's one way to make it work:
{{{
+Use the `-nostdin` global option in `ffmpeg` to disable interaction on standard input:
{{{#!highlight bash
 Line 24:
-  ffmpeg -i "$file" -vcodec libxvid -acodec libfaac -ar 32000 "${file%.avi}".mkv </dev/null
+  ffmpeg -nostdin -i "$file" -c:v libx264 -c:a aac "${file%.avi}".mkv
 Line 27:
-Notice the [[BashGuide/InputAndOutput#Redirection|redirection]] on the ffmpeg line: `</dev/null`. The ssh example can be fixed the same way, or with the `-n` switch (at least with [[http://www.openssh.org/|OpenSSH]]).
+Alternatively you could use [[BashGuide/InputAndOutput#Redirection|redirection]] at the end of the ffmpeg line: `</dev/null`. The ssh example can be fixed the same way, or with the `-n` switch (at least with [[http://www.openssh.org/|OpenSSH]]).
-Line 31:
+Line 30:
-{{{
while read <&3 line; do
+{{{#!highlight bash
while IFS= read -r line <&3; do
-Line 34:
+Line 33:
-done 3<file
+done 3< file
-Line 36:
+Line 35:
+In bash, the `read` builtin can also be told to read directly from an fd (`-u fd`) without redirection, and since bash 4.1, an available fd can be assigned (`{var}<file`) instead of hard coding a file descriptor.

{{{#!highlight bash
# bash 4.1+
while IFS= read -r -u "$fd" line; do
  ...
done {fd}< file
exec {fd}<&-
}}}