Diff for "BashFAQ/036"

Anchor(faq36)

How can I get all lines that are: in both of two files (set intersection) or in only one of two files (set subtraction).

Use the comm(1) command.

  # intersection of file1 and file2
  comm -12 <(sort file1) <(sort file2)
  # subtraction of file1 from file2
  comm -13 <(sort file1) <(sort file2)

Read the comm(1) manpage for details.

If for some reason you lack the core comm(1) program, you can use these other methods:

1. An amazingly simple and fast implementation, that took just 20 seconds to match a 30k line file against a 400k line file for me.
   • It has grep read one of the sets as a pattern list from a file (-f), and interpret the patterns as plain strings not regexps (-F), matching only whole lines (-x).
   • Note that the file specified with -f is will be loaded into ram, so it doesn't scale for very large files.
   • It should work with any POSIX grep; on older systems you may need to use fgrep rather than grep -F.

     # intersection of file1 and file2
     grep -xF -f file1 file2
     # subtraction of file1 from file2
     grep -vxF -f file1 file2

2. An implementation using sort and uniq

     # intersection of file1 and file2
     sort file1 file2 | uniq -d  (Assuming each of file1 or file2 does not have repeated content)
     # file1-file2 (Subtraction)
     sort file1 file2 file2 | uniq -u
     # same way for file2 - file1, change last file2 to file1
     sort file1 file2 file1 | uniq -u

3. Another implementation of subtraction:

     cat file1 file1 file2 | sort | uniq -c |
     awk '{ if ($1 == 2) { $1 = ""; print; } }'

   • This may introduce an extra space at the start of the line; if that's a problem, just strip it away.
   • Also, this approach assumes that neither file1 nor file2 has any duplicates in it.
   • Finally, it sorts the output for you. If that's a problem, then you'll have to abandon this approach altogether. Perhaps you could use awk's associative arrays (or perl's hashes or tcl's arrays) instead.

See also: http://www.pixelbeat.org/cmdline.html#sets