Q:

How does dask manage tasks when writing data to HDF5 files in parallel?

I want to parallelise a computation over a large array, using python. Neither the input array nor the output array fits in memory. Furthermore, writing the output file is a potential bottleneck (because e.g. HDF5 generally does not support parallel writes). 
dask.array.map_blocks(func).store(...)

While waiting to write out the generated chunks, how does dask decide how many new chunks to bring into memory (i.e. read in and start computing)? Presumably, if it tries to keep the CPU fully occupied, it risks causing python to hit a memory limit. Does a relevant configuration setting exist, or is it better to avoid dask and implement some other kind of task queue?

A:

Dask generally tries to pipeline things when possible to avoid keeping a lot of data in memory at once.  It handles writing to HDF5 files by calling many store tasks in parallel, but using a lock to control access.  We've found that restricting the CPU in this way tends to be most robust in most situations.

