HylaFax keeps a history of faxes sent for about a week in its spool folder. I have a script that can go through the files, extract the user and page counts and output it to a log.What's the most efficient way to process the files in the folder on a daily basis without resulting in duplicate outputs in the log file and without moving/altering the processed files?Description of Folder:# ls /var/spool/hylafax/doneq/q10000 q10001 ... q10010Script#!/bin/bashfor i in /var/spool/hylafax/doneq/q*  do   user=$(cat $i | grep mailaddr | sed 's/mailaddr://g')   pgs=$(cat $i | grep npages | sed 's/npages://g')      echo $i $user - $pgs pages >> /scripts/log_output.txtdone
Bash Script: Parsing a Folder of Text Files Daily without Duplicating Output
bash;shell script
You askedWhat's the most efficient way to process the files in the folder on a daily basis without resulting in duplicate outputs in the log file and without moving/altering the processed files?One way to achieve this is to store a sorted list of the files that have been processed. Using comm against the list of the candidate files for processing will remove duplicates.As an illustration of the technique, something like this could be used as a basis for handling for well-formed filenames such as those produced by HylaFax:find * -print | sort > /tmp/current_filestest -f /tmp/previous_files || { echo Come back tomorrow; exit 0; }comm -13 /tmp/previous_files /tmp/current_files > /tmp/new_files# ... Process entries in /tmp/new_files ...mv -f /tmp/current_files /tmp/previous_files