Q:

What's the best strategy to crawl very big filesystem?

I need to get the owner for about 100,000,000 windows network files distributed amongst several shares. I'am using advapi32 GetNamedSecurityInfo function from within C# code, but it's a long running process. I'm currently crawling several shares in parallel. And for each share I'm querying every files and folders sequentially. My question, what would be the best approach to minimise the crawl/collection time ?
Thanks for your answers

A:

Ultimately the fastest solution is to query the information locally instead of over the network.  Granted, that might not be an option.
File ownership is going to be queried over the network via the SMB protocol.  Performance will be very sensitive to latency because lots of requests and responses are going back and forth, and your code is always waiting for a response before sending the next request.  Although more recent versions of the SMB protocol support pipelining, your current method of sequential querying per network share means that the pipeline won't be used anyway.
You can probably improve performance by splitting a single network share's requests into multiple threads.  I don't know how many concurrent connections are allowed for your particular network shares, but I would test with 8 concurrent requests per server to see how performance is impacted.

