I'm setting up a small Linux cluster for my laboratory. Current we have an login node for job submit and 3 computational node, and they are connected by Gigabit LAN (sadly no infiniband).At first I simply used NFS to export shared storage (home folder) for cluster users. But soon we found out that NFS doesn't perform well under real-world payload.It turned out that it's very common for my cluster users to read and write GBs of small picture files (one is <100KB) in shared storage. NFS performs very badly for small file IO.I have a simple test of NFS/CIFS/SSHFS performance for small files writing. The times to untar 20000 small files (about 600MB total) on each filesystem are:CIFS: 25~30sSSHFS: 45~55sNFS(v3/v4): so slow that I didn't wait (estimation is 10mins)CIFS/SSHFS performs very well comparing to NFS. But it's not a general practice to use CIFS/SSHFS for Linux home folders in a computer cluster scenario (and CIFS isn't a really posix-compliant filesystem AFAIK).What's your opinion on a network filesystem choice for small files payload? Should I replace NFS with other cluster/distributed filesystems? Or is there a way to optimize NFS for small files?
Cluster filesystem choice for reading/writing large number of small files
samba;nfs;cifs;cluster;sshfs