There is no best ratio of files per node, or TiB per CPU. The common response to the question will be "it depends on the workflow and data layout". You may have billions of files in an archive, or transient data in scratch space where Isilon operates just fine. Or a much smaller data set that is problematic. you have correctly identified the real challenge in the Job Engine which is serialized (runs one job at a time) and jobs are often paused by higher priority jobs starting. The results is that very useful and nesessary jobs do not complete in a timely fashion, some times not at all. Scaling CPU can help by scaling Job Engine threads and SSD can assist in the metadata operations of the Job Engine, however the solution (we hope) will be the improvements in the Job Engine itself.  OneFS 7.1 will include Job Engine 2.0 which can run mutiple concurrent jobs (3) thereby improve effeciency and effectiveness of internal operations.
Looking for recommendations on number of files per node. I seem to have numerous existing customers that are having massive challenges with jobs completing. In some cases certain jobs have never successfully run. I have them all engaged with technical support but the one trend that I've noticed is that my predecessor didn't necessarily take into consideration the amount of files needed to support rather focused on the capacity required and on a hunch we've remedied one of these accounts by adding additional nodes to the cluster. Obviously, this isn't a fix I'd like to offer up often so I'd like to know if there is any good estimate of files supported per node. Or rather, where should I look at leveraging smaller drives to manipulate the CPU/TiB ratio?
File Count Per Node Recommendations