Current scripts create and delete multiple files over NFS. I have been told that this is the problem, so was thinking if the run directory the script ran in looked like a single file to NFS then I might alleviate the problem.I should add that we have hundreds of these scripts working in their own run directories but all over NFS at any one time, and we have a bottleneck due to the creation and deletion of many tens of files for each run directory.I have since found http://code.google.com/p/fuse-zip/ that I will investigate. Could someone share their experiences with this?
Is there a way to create a directory structure and files inside a file on a 'base' filesystem'?
filesystems
Ok, from the comments, it seems like it'll fall into one of these suggestions:You're making and deleting a bunch of temporary files, taking relatively little disk space. Suggestion: Mount a tmpfs somewhere on the local machine, and use that. (You may already have one under /tmp anyway)Most of them are temporary files, but a few need to be keptSuggestion: tmpfs again, but before unmounting the tmpfs at the end (thus losing its contents), copy the few files you need off it.They're temporary, but they're not small.Suggestion: Local storage.You need to access these many small files from multiple machines, concurrently.Suggestion: Beefier NFS server and/or network. Dedicated NFS server for just this load. Distributed filesystems. I'm having a hard time coming up with a scenario where the approach in your question is the best idea. But if you do want to do it...Suggestion: Make a single file on the NFS server. Use losetup to map it to a loop device, then mkfs that, and mount it. (Depending on which filesystem you're using, you make be able to mkfs the file directly, that'll work with mke2fs for example). Probably, mount it with the async flag (or similar). This will perform much better than a zip file.Another thing I'll note, depending on what you're making and deleting all these temporary files for, files may not be the best approach. E.g., maybe they should be rows in a database of some sort.