I thought about whether this question is suitable for SE or not, I hope you agree it is.Some time ago I asked on SE how to find text in files and leave the file with only the matching lines that contain the text I was searching for. The question is here:  How to find text in files and only keep the respective matching lines using the terminal on OS X?While the answer worked perfectly I now wonder, how come sed is so fast? In my use case, I had quite a lot of files which in total were about 30 Gb in size. The sed command ran in about 12 seconds which I never would have believed (working with a normal HDD). Within 12 seconds the command read through 30 Gb of text, truncating each file to only keep the respective lines I was filtering for. How does this work? (or: what is this sorcery?)The actual command was:find . -type f -exec sed -i'' '/\B\/foobar\b/!d' {} \;
How come sed is so fast?
files;sed;performance
The likely answer is that:The 30Gb file was not fragmented (or had very little fragmentation): all hard drives perform much better with sequential access (SSDs included) because they are able to cache large chunks of the file. This allows them to approach their maximum performance. Sequential access will help with all levels of caches.sed is a stream editor; it only processes one line at a time. This means that its memory footprint is minuscule. Unlike a text editor such as emacs or vim, it does not need to maintain the entire copy of the file in memory.You are editing the file in-place (with -i) which (as demonstrated by @Ramesh and also stated on wikipedia page) creates a temporary files which then becomes the old file.All of this means that sed is able to perform almost the bare minimum of file operations: each line of the original file is read once, and only the lines that are matched are written.Your choice of regular expressions also impacts performance, sometimes in very bad ways.: coding horror blog.