I often need to tail -f apache access logs for websites to troubleshoot issues- one thing that makes it annoying is that anyone loading a page once may cause 12+ lines to get written to the log, and since they're long lines each one wraps multiple lines in my terminal.tail -f seems to play nicely with piping to grep and awk, and I came up with a pretty simple solution to filter out duplicates when one IP address makes many requests in a particular second (as well as trim it to the particular info I usually need)-tail -f log.file | awk ' { print $1   $4   $9}' | uniqThe problem is, this doesn't work. I just get no output at all, even when I know there should be tons of lines printed.I've tried some troubleshooting, but haven't been able to get things to really work-tail -f log.file | awk ' { print $1   $4   $9}' This works exactly as I think it should, and prints the lines as they happen (but with many duplicates) like so:12.34.56.78 [10/May/2016:18:42:01 20012.34.56.78 [10/May/2016:18:42:02 30412.34.56.78 [10/May/2016:18:42:02 30412.34.56.78 [10/May/2016:18:42:02 30412.34.56.78 [10/May/2016:18:42:02 30412.34.56.78 [10/May/2016:18:42:02 30412.34.56.78 [10/May/2016:18:42:02 30412.34.56.78 [10/May/2016:18:42:02 304tail log.file | awk ' { print $1   $4    $9}' | uniqThis also works exactly as I think it should, and filters out any duplicate lines. But for my troubleshooting I really need the real time updates of tail -fHow can I make tail -f filter out duplicate lines?
Filter out duplicate lines of tail -f
awk;pipe;tail;buffer;uniq
As a pure awk solution, try:tail -f log.file | awk ' $0!=last{ print $1   $4   $9} {last=$0}'This one prints a new output line only if the input line is different from the previous input line.As a slight variation, this one prints a new output line only if this output line differs from the previous output line:tail -f log.file | awk '{$0=$1 $4 $9} last!=$0{print} {last=$0}'ExampleLet's try this test file:$ cat logfile1 2 3 4 5 6 7 8 91 2 3 4 5 6 7 8 9 101 2 3 4 5 6 7 8 9 111 2 3 4 5 6 7 8 191 2 3 4 5 6 7 8 19 121 2 3 4 5 6 7 8 19 131 2 3 4 5 6 7 8 191 2 3 4 5 6 7 8 29awk filters out the duplicate output lines:$ cat logfile | awk '{$0=$1 $4 $9} last!=$0{print} {last=$0}' 1 4 91 4 191 4 29