I'm looking for a way to list all files in a directory that contain the full set of keywords I'm seeking, anywhere in the file.So, the keywords need not to appear on the same line.One way to do this would be:grep -l one $(grep -l two $(grep -l three *))Three keywords is just an example, it could just as well be two, or four, and so on.A second way I can think of is:grep -l one * | xargs grep -l two | xargs grep -l threeA third method, that appeared in another question, would be:find . -type f \  -exec grep -q one {} \; -a \  -exec grep -q two {} \; -a \  -exec grep -q three {} \; -a -printBut that's definitely not the direction I'm going here. I want something that requires less typing, and possibly just one call to grep, awk, perl or similar.For example, I like how awk lets you match lines that contain all keywords, like:awk '/one/ && /two/ && /three/' *Or, print just the file names:awk '/one/ && /two/ && /three/ { print FILENAME ; nextfile }' *But I want to find files where the keywords may be anywhere in the file, not necessarily on the same line.Preferred solutions would be gzip friendly, for example grep has the zgrep variant that works on compressed files. Why I mention this, is that some solutions may not work well given this constraint. For example, in the awk example of printing matching files, you can't just do:zcat * | awk '/pattern/ {print FILENAME; nextfile}'You need to significantly change the command, to something like:for f in *; do zcat $f | awk -v F=$f '/pattern/ { print F; nextfile }'; doneSo, because of the constraint, you need to call awk many times, even though you could do it only once with uncompressed files. And certainly, it would be nicer to just do zawk '/pattern/ {print FILENAME; nextfile}' * and get the same effect, so I would prefer solutions that allow this.
Find files that contain multiple keywords anywhere in the file
shell;text processing;awk;grep;find
Of all the solutions proposed so far, my original solution using grep is the fastest one, finishing in 25 seconds. It's drawback is that it's tedious to add and remove keywords. So I came up with a script (dubbed multi) that simulates the behavior, but allows to change the syntax:#!/bin/bash# Usage: multi [z]grep PATTERNS -- FILEScommand=$1# first two arguments constitute the first commandcommand_head=$1 -le '$2'shift 2# arguments before double-dash are keywords to be piped with xargswhile (($#)) && [ $1 != -- ] ; do  command_tail+=| xargs $command -le '$1'   shiftdoneshift# remaining arguments are fileseval $command_head $@ $command_tailSo now, writing multi grep one two three -- * is equivalent to to my original proposal and runs in the same time. I can also easily use it on compressed files by using zgrep as the first argument instead.Other solutionsI also experimented with a Python script using two strategies: searching for all keywords line by line, and searching in whole file keyword by keyword. The second strategy was faster in my case. But it was slower than just using grep, finishing in 33 seconds. Line by line keyword matching finished in 60 seconds.#!/usr/bin/python3import gzip, sysi = sys.argv.index('--')patterns = sys.argv[1:i]files = sys.argv[i+1:]for f in files:  with (gzip.open if f.endswith('.gz') else open)(f, 'rt') as s:    txt = s.read()    if all(p in txt for p in patterns):      print(f)The script given by terdon finished in 54 seconds. Actually it took 39 seconds of wall time, because my processor is dual core. Which is interesting, because my Python script took 49 seconds of wall time (and grep was 29 seconds).The script by cas failed to terminate in reasonable time, even on a smaller number of files that were processed with grep under 4 seconds, so I had to kill it.But his original awk proposal, even though it's slower than grep as is, has potential advantage. In some cases, at least in my experience, it's possible to expect that all the keywords should all appear somewhere in the head of the file if they are in the file at all. This gives this solution a dramatic boost in performance:for f in *; do  zcat $f | awk -v F=$f \    'NR>100 {exit} /one/ {a++} /two/ {b++} /three/ {c++} a&&b&&c {print F; exit}'doneFinishes in a quarter of a second, as opposed to 25 seconds.Of course, we may not have the advantage of searching for keywords that are known to occur near the beginning of the files. In such case, solution without NR>100 {exit} takes 63 seconds (50s of wall time).Uncompressed filesThere's no significant difference in running time between my grep solution and cas' awk proposal, both take a fraction of a second to execute.Note that the variable initialization FNR == 1 { f1=f2=f3=0; } is mandatory in such case to reset the counters for every subsequent processed file. As such, this solution requires editing the command in three places if you want to change a keyword or add new ones. On the other hand, with grep you can just append | xargs grep -l four or edit the keyword you want.A disadvantage of grep solution that uses command substitution, is that it will hang if anywhere in the chain, before the last step, there are no matching files. This doesn't affect the xargs variant because the pipe will be aborted once grep returns a non-zero status. I've updated my script to use xargs so I don't have to handle this myself, making the script simpler.