Q:

Large directories slow processing

We have a java process that does some analysis on files and then writes those files to a directory.   Recently we noticed the process slowing dramatically.   The culprit was the output directory which had grown to over 1 million files.  When we emptied the directory, the speed of the process sped up dramatically.    Why would this be the case?   This is a Windows Server 2012 machine.   No processing is done on the files in the directory (other than writing to the directory after the process is completed).  

A:

You will need to use a new data structure which sorts your files to subdirectories, or use faster hardware(look into SSDs).  Currently, I believe what is occurring is that when writing a new file you are checking all other files.  

