I need to iterate over between 120k and 500k files. find handles this well.find $PWD -type f -path fragments/*.pdbqtFor some reason I want to list the same set of files again, in the same order. I don't mind if the list is sorted alpha-numerically or by fstat calls or whatever. Using sort is not an option because it needs to know the full find output before proceeding, slowing down the script execution. Do you know some way to accomplish this?Update.I described my problem poorly. While my script is processing this large number of files, a light outage might occur, and the UPS could go down. My script should continue from where it left off. How can I process the remaining files?
Dealing with script interruption
shell script;scripting;find
The find command returns files in directory order. Directory order is not stable; in some filesystems, if you add or remove a file, it can affect the order of other files in the same directory.I question your assertion that sort is not an option. Unless your processing of each file is extremely fast, building the list of files is pretty fast. Save it to a file so that it can be reused if the job is interrupted.You'll need to know where you left off anyway, and resume from there. Resuming from an already-generated list of files will be easier than resuming some complex enumeration in the middle.