I'm working on a code that we eventually plan to scale up to 10's or 100's of thousands of processors (using MPI). I have a single HDF5 file (mesh nodes and connectivity, etc) that needs to be read and distributed to every process with identical data. This is only done once during the setup part of the code. The current solution is to have each process independently read the file, but my collaborator is worried this will not scale well and suggests having process 0 read the file then MPI distribute the data to the other processors. So I wanted to get an outside opinion from someone with more extensive HPC experience. Is this actually going to be a significant bottleneck?
Many processors reading a single HDF5 file
hpc;io
It will not scale! Use the parallel HDF5 functionality to read it in parallel with a relatively small number of readers, or do what your colleague suggests. There will be tradeoffs with both methods, and you will have to do some tuning. It depends pretty strongly on how big the file is, how many tasks you have reading it, what the underlying filesystem is (GPFS, Lustre, etc), how the file is striped, and what the underlying disk infrastructure is. 