I am using xargs to call a python script to process about 30 million small files. I hope to use xargs to parallelize the process. The command I am using is:find ./data -name *.json -print0 |  xargs -0 -I{} -P 40 python Convert.py {} > log.txtBasically, Convert.py will read in a small json file (4kb), do some processing and write to another 4kb file. I am running on a server with 40 CPU cores. And no other CPU-intense process is running on this server.By monitoring htop (btw, is there any other good way to monitor the CPU performance?), I find that -P 40 is not as fast as expected. Sometimes all cores will freeze and decrease almost to zero for 3-4 seconds, then will recover to 60-70%. Then I try to decrease the number of parallel processes to -P 20-30, but it's still not very fast. The ideal behavior should be linear speed-up. Any suggestions for the parallel usage of xargs ?
correct xargs parallel usage
xargs;parallelism