I have large batches of bash processes.Each bash script invokes executeables which have their stdout redirected to distinct log files.About 5% of the runs end up with:sh: [name of log]: Resource temporarily unavailableI tried to reduce amount of jobs running in parallel, but still the error persisted on some of the bash scripts.Additional info:Ubuntu 14.04 LTS running on VM using ESXiHappens on a new partition, allocated with gparted and LVM (new logical volume consisting of the entire partition)The LV is exported using nfs-kernel-serverThe LV is also shared to windows using SambaThe LV is formatted using ext4I have admin rights on this machineMore detailed infoEverything is run in a cluster, using Sun-Grid-EngineThere are 4 virtual machines: m1, m2, m3, m4m1 runs sge master, sge exec, and ldap serverm2, m3, m4 run sge execm3 runs nfs-kernel-server, exporting a home folder sitting in logical volume (using LVM) that uses a partition on a local disk, to m1, m2, m4m3 has a soft link to the home folderm1, m2, m4 mount the home folder through fstab, so all machines end up pointing to the same home folderm3, m2, m4 run ldap clients, connecting to m1All jobs are submitted to the cluster through m1 (configured as a submission host)Jobs fail exclusively on m3 (which exports the disk). Most of the jobs on m3 are passing though. Failures are random, but consistently on m3 alone.m3 also shares the home via samba to windows clientsAny help would be greatly appreciated :) (how to debug, which logs are relevant, how to get more info out of the system, etc...)Thank you in advance!
stdout redirect. sh: resource temporarily unavailable
lvm;stdout;gridengine
Thanks everyone who tried to help!The issue got solved by mounting the logical volume on m3 using nfs exactly the same way as done on the rest of the machines m1/m2/m4 which are nfs clients, instead of having soft link on m3 to the logical volume. Simply add the following line to /etc/fstab: <nfs server>:/ /mnt nfs auto 0 0 and then invoke sudo mount -a.The hint was in the fact that failures were happening consistently on m3 which is the nfs server, and automatic resubmit of failed jobs solved the issue as well. Never were there failures on m1/m2/m4 (which are the nfs clients). Remember that m3 is nfs server, which has a simple soft link to the logical volume, while all clients use nfs to connect this logical volume.In the back of my head I had the feeling that probably nfs protects it's clients from having these issues, but I thought that the filesystem on the logical-volume shouldn't fail, and if it does then I have a real problem that I must root cause. This might be still the case btw.If you have insights about the issue, and the solution - please write. I don't want to mask issues if they are real.