Diff for "BashFAQ/009"

What is buffering? Or, Why does my command line produce no output: tail -f logfile | grep 'foo bar'

Most standard Unix commands buffer their output when used non-interactively. This means that they don't write each character (or even each line) as the input arrives, but instead collect a larger number of characters (often 4 kilobytes) before printing anything at all. In the case above, the tail command buffers its output, and therefore grep only gets its input in e.g. 4K blocks.

Buffering greatly increases the efficiency of I/O operations, and it's usually done in a way that doesn't visibly affect the user. A simple tail -f from an interactive terminal session works just fine, but when commands are in scripts, functions, or part of a complicated set of pipes, the command might not recognize that the final output is needed in (near-)real-time. Fortunately, there are several techniques available for controlling I/O buffering behavior.

Your command may already support unbuffered output

Some programs provide special command line options specifically for this sort of problem:

unbuffer

The expect package has an unbuffer program which effectively tricks other programs into always behaving as if they were being used interactively (which may often disable buffering). Here's a simple example:

    unbuffer tail -f logfile | grep 'foo bar'

expect and unbuffer are not standard POSIX tools, but they may already be installed on your system.

tee

At least the GNU version of tee appears to produce unbuffered output. For example:

   $ program | tee -a program.log

   In another window:
   $ tail -f program.log | grep whatever

This has only been tested on GNU tee, so YMMV.

less

If you simply wanted to highlight the search term, rather than filter out non-matching lines, you can use the less program instead of Bash:

   $ less program.log

• Inside less, start a search with the '/' command (similar to searching in vi).

• This should highlight any instances of the search term.

• Now put less into "follow" mode, which by default is bound to shift+f.

• You should get an unfiltered tail of the specified file, with the search term highlighted.

"follow" mode is stopped with an interrupt, which is probably control+c on your system. The '/' command accepts regular expressions, so you could do things like highlight the entire line on which a term appears. For details, consult man less.

coproc

If you're using ksh or Bash 4.0+, whatever you're really trying to do with tail -f might benefit from using coproc and fflush() to create a coprocess. Note well that coproc does not itself address buffering issues (in fact it's prone to buffering problems -- hence the reference to fflush). coproc is only mentioned here because whenever someone is trying to continuously monitor and react to a still-growing file (or pipe), they might be trying to do something which would benefit from coprocesses.

CategoryShell