Q:

Process items in parallel

I have a FileSystemWatcher which looks for files in a folder. Moment the file is created a Created event is fired. I am adding each created file name to a queue.
void Main(){
            FileSystemWatcher fsw = new FileSystemWatcher();
            fsw.Path = System.Configuration.ConfigurationManager.AppSettings["PathToDataFolder"];
            //fsw.Filter = "*.docx";
            fsw.EnableRaisingEvents = true;
            fsw.IncludeSubdirectories = true;
            fsw.Created += new FileSystemEventHandler(fsw_Created);
}
private void fsw_Created(object sender, FileSystemEventArgs e)
{
            queue.Enqueue(e.FullPath);    
}

Overtime the files will increase and the queue will become large. So the queue is dynamic. I want to process each file in parallel. But I dont want to process a lot of files at once because that is quite resource intensive. Once a file is processed i want to dequeue it and pick another file to process.
How can i achieve this in C#?

A:

You can use an ActionBlock with a configurable degree of parallelism. By default, an ActionBlock uses only one Task to process incoming messages. You can make it use multiple tasks to process files in parallel. The FSW's event should post paths directly to the block :
ActionBlock<string> _block;

void Main()
{

    ...
    var options= new ExecutionDataflowBlockOptions
     {
        MaxDegreeOfParallelism = 4
     };

    _block=new ActionBlock<string>(path=>MyPathProcessingFunction(path), options);

   //Configure the FSW as before
}

private void fsw_Created(object sender, FileSystemEventArgs e)
{
    _block.Post(e.FullPath);
}

