|
Size: 2005
Comment:
|
Size: 3598
Comment: Added a reference to 'coproc', and rephrased some of the text
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| [[Anchor(faq9)]] | <<Anchor(faq9)>> |
| Line 3: | Line 3: |
| Most standard Unix commands buffer their output if used non-interactively. This means, that they don't write each character (or even each line) as they are ready, but collect a larger number (e.g. 4 kilobytes) before printing it. In the case above, the {{{tail}}} command buffers its output, and therefore {{{grep}}} only gets its input in e.g. 4K blocks. |
Most standard Unix commands buffer their output when used non-interactively. This means that they don't write each character (or even each line) as the input arrives, but instead collect a larger number of characters (often 4 kilobytes) before printing anything at all. In the case above, the {{{tail}}} command buffers its output, and therefore {{{grep}}} only gets its input in e.g. 4K blocks. |
| Line 6: | Line 10: |
| Unfortunately there's no easy solution to this, because the behaviour of the standard programs would need to be changed. *See bottom of section before taking 'no easy solution' to heart* | Buffering greatly increases the efficiency of I/O operations, and it's usually done in a way that doesn't visibly affect the user. A simple "tail -f" from an interactive terminal session works just fine, but when commands are in scripts, functions, or part of a complicated set of pipes, the command might not recognize that the final output is being used interactively. Fortunately, there are several techniques available for controlling I/O buffering behavior. |
| Line 8: | Line 17: |
| Some programs provide special command line options for this purpose, e.g. | ==== Your command may already support unbuffered output ==== Some programs provide special command line options specifically for this sort of problem: |
| Line 15: | Line 25: |
| The {{{expect}}} package (http://expect.nist.gov/) has an {{{unbuffer}}} example program, which can help here. It disables buffering for the output of a program. Example usage: |
==== unbuffer ==== The {{{expect}}} package has an [[http://expect.nist.gov/example/unbuffer.man.html|unbuffer]] program which effectively tricks other programs into always behaving as if they were being used interactively (which should disable buffering). Here's a simple example: |
| Line 22: | Line 34: |
There is another option when you have more control over the creation of the log file. If you would like to {{{grep}}} the real-time log of a text interface program which does buffered session logging by default (or you were using {{{script}}} to make a session log), then try this instead: |
{{{expect}}} and {{{unbuffer}}} may already be installed on your system. If not, the {{{expect}}} package can be found at: http://expect.nist.gov/ ==== tee ==== At least the GNU version of {{{tee}}} appears to produce unbuffered output. For example: |
| Line 31: | Line 44: |
| This has only been tested on GNU {{{tee}}}, so [[http://en.wiktionary.org/wiki/your_mileage_may_vary|YMMV]]. | |
| Line 32: | Line 46: |
| Apparently this works because {{{tee}}} produces unbuffered output. This has only been tested on GNU {{{tee}}}, YMMV. A solution to this is to use the 'less' command in follow mode. This is simple to do! |
==== less ==== If you simply wanted to highlight the search term, rather than filter out non-matching lines, you can use the {{{less}}} program instead of Bash: |
| Line 38: | Line 52: |
| Then enter your search pattern (/ is search in less, like vi) /foo bar |
* Inside {{{less}}}, start a search with the '/' command (similar to searching in vi). * This should highlight any instances of the search term. * Now put {{{less}}} into "follow" mode, which by default is bound to shift+f. * You should get an unfiltered tail of the specified file, with the search term highlighted. |
| Line 41: | Line 57: |
| Next, put less into follow mode by issuing shift+f | "follow" mode is stopped with an interrupt, which is probably control+c on your system. The '/' command accepts regular expressions, so you could do things like highlight the entire line on which a term appears. For details, consult {{{man less}}}. |
| Line 43: | Line 62: |
| Thats all there is to it! | ==== coproc ==== If you're using ksh or Bash 4.0+, whatever you're really trying to do with {{{tail -f}}} might benefit from using [[http://bash-hackers.org/wiki/doku.php/syntax/keywords/coproc|coproc]] and fflush() to create a coprocess. Note well that {{{coproc}}} does '''not''' itself address buffering issues (in fact it's prone to buffering problems -- hence the reference to fflush). {{{coproc}}} is only mentioned here because whenever someone is trying to continuously monitor and react to a still-growing file (or pipe), they might be trying to do something which would benefit from coprocesses. ---- CategoryShell |
My command line produces no output: tail -f logfile | grep 'foo bar'
Most standard Unix commands buffer their output when used non-interactively. This means that they don't write each character (or even each line) as the input arrives, but instead collect a larger number of characters (often 4 kilobytes) before printing anything at all. In the case above, the tail command buffers its output, and therefore grep only gets its input in e.g. 4K blocks.
Buffering greatly increases the efficiency of I/O operations, and it's usually done in a way that doesn't visibly affect the user. A simple "tail -f" from an interactive terminal session works just fine, but when commands are in scripts, functions, or part of a complicated set of pipes, the command might not recognize that the final output is being used interactively. Fortunately, there are several techniques available for controlling I/O buffering behavior.
Your command may already support unbuffered output
Some programs provide special command line options specifically for this sort of problem:
grep (e.g. GNU version 2.5.1) |
--line-buffered |
sed (e.g. GNU version 4.0.6) |
-u,--unbuffered |
awk (some GNU versions) |
-W interactive, or use the fflush() function |
tcpdump, tethereal |
-l |
unbuffer
The expect package has an unbuffer program which effectively tricks other programs into always behaving as if they were being used interactively (which should disable buffering). Here's a simple example:
unbuffer tail -f logfile | grep 'foo bar'
expect and unbuffer may already be installed on your system. If not, the expect package can be found at: http://expect.nist.gov/
tee
At least the GNU version of tee appears to produce unbuffered output. For example:
$ program | tee -a program.log In another window: $ tail -f program.log | grep whatever
This has only been tested on GNU tee, so YMMV.
less
If you simply wanted to highlight the search term, rather than filter out non-matching lines, you can use the less program instead of Bash:
$ less program.log
Inside less, start a search with the '/' command (similar to searching in vi).
- This should highlight any instances of the search term.
Now put less into "follow" mode, which by default is bound to shift+f.
- You should get an unfiltered tail of the specified file, with the search term highlighted.
"follow" mode is stopped with an interrupt, which is probably control+c on your system. The '/' command accepts regular expressions, so you could do things like highlight the entire line on which a term appears. For details, consult man less.
coproc
If you're using ksh or Bash 4.0+, whatever you're really trying to do with tail -f might benefit from using coproc and fflush() to create a coprocess. Note well that coproc does not itself address buffering issues (in fact it's prone to buffering problems -- hence the reference to fflush). coproc is only mentioned here because whenever someone is trying to continuously monitor and react to a still-growing file (or pipe), they might be trying to do something which would benefit from coprocesses.