EDIT: You were right, the problem is not checking if a file with the same name exists but the connection. I made a mistake measuring the time with StopWatch and included a part which used the network connection. Original Question:I am writing a software which analysis data from a database. In the database are many textfiles - could be 100GB - ranging from 100KB to some MB (<10MB). My software should run an analysis task every night. So I want to implement a file cache which stores the files and only downloads them again if they have changed. My problem is, the more files I have inside my storage folder, the slower the whole process of checking if the file exists becomes. I am using C#.My ideas until now:Lazy loading - just download the file if it could not be accessedGroup the files in folder so that I have less files in one directoryCreating a Dictionary with filename as key and File object as value for all files in one directory before starting analysis of one folder (assuming finding files in a dictionary should be much faster than on the disk)I invested already some time and before I reimplement it I want to know:Is there something else (idea) which I could try?In C# is there a best practice regarding file handling which leads to the best performance?Just as side information, if there is a new file I have to analyse all lines of it. Is there a faster way than StreamReader if I know upfront that it is a tabulator seperated file and I have to read for sure all lines of it and parse the information?We are talking about an Oracle Database. The files are stored as BLOB in the database. The database is organized as tree and elements have children which are files that contain the state of the elements. Those files I have to download to be able to analyze them. I can get the last modify date so I am able to download only the files I need (which have changed or are new). There are about 300.000 files in the database but I don't need all of them. I have write access to the folders where I want to store them create them with my program on the local disk. This folder structure where I want to store them I call file cache.
Improving performance with caching of files in C#
c#;database;performance;caching;files
Saving stuff to files isn't normally a good way of creating a cache. If in your case its the downloading of large files over the network from a remote machine which is slow. I would create another identical local database to store the files once you have downloaded them.If this is not possible, create a small database in which you just store the id, filename and directory of the downloaded files. You can use this to check for the existence of the file and then read it off disk as you do now.Rather than reading all the lines in a file to see if it's the same as the new one, create a checksum when you download it and add this to the database table.