Re: even more questions about the Richmond cluster Greetings, Just for variety, I'll answer the questions from easy to hard order :-) /var/spool/mqueue filling: var/spool/mqueue is where outgoing mail goes for sendmail to process (even if it's going from and to a local user).  Currently, sendmail isn't running on pscm1, so the mail piles up in mqueue rather than being delivered to user mailbox files in /var/spool/mail. /var/spool/at is where the batch system holds job output until completion. Upon completion, it will mail the output to the job's owner. Probably, sendmail should be running on the master. To avoid overfilling /var, /var/spool can either be symlinked to a larger volume (/usr or to one of the RAID volumes) or have your batch scripts redirect the output to a scratch file and then move (or bpcp) it to the user's home directory when the run is complete. I am currently tracking down a bug in the batch queue that pushes jobs off to the b queue and leaves them there when they can't be immediatly dispatched. I believe there is a workaround for that, I'll dig that up and send it to you. The ultimate solution will be an update to the at rpm. Changing the loadaverage threshold won't do anything here. The bproc mods to the batch queue system use a 'slot allocation' rather than loadavg. The slot allocator assumes that the best allocation is 1 job per CPU on a compute node. That is done for three reasons. First, that's generally a better allocation for any scientific computation while loadavg is better suited for general purpose servers. Second, loadavg is more expensive to compute in that it will generate extra net traffic and disrupt the cache and take extra locks on the compute nodes (which will hurt performance). Compute jobs often have bursts of I/O where the job will sleep waiting, followed by long periods of pure computation. That can cause a brief dip in apparent loadavg and lead to bad allocations. G'day, sjames Quoting gilfoyle : > Hi Steven, >  >    Happy New Year and yet another question about the Richmond cluster.  > I have been experimenting with different ways of running the cluster  > and I have run into a problem with the batch system. I'm submitting  > jobs in two different ways; one uses the beomap command to allocate  > slave nodes and the other just picks the slave nodes `by hand'. I'm  > using this second method because the limiting factor now is the  > ability to transfer the data files to the slave nodes. I was thinking  > that I could transfer the data on the first pass and leave it there  > for later passes to speed things up. The problem now is that after  > many jobs are submitted (from 60-100 of so) the remaining jobs get  > sent to the `b' batch queue and never run. This has happened even when  > the /var/spool area is not full. My thoughts are the following. >  > 1. Can we reset the average cpu load with the 'atd -l' command? I've  > tried this and it seems to have little effect. >  > 2. Can we restart the jobs in the queue? Now they just sit there and  > never get started. >  > 3. In some of the recent analysis runs, the /var/spool/mqueue area has  > filled up and hung things up. Before it was the /var/spool/at or  > /var/spool/mail areas. Do you have any idea what would cause that?  > Should we make a link for /var/spool/mqueue to one of RAID disks so  > there is plenty of space? >  > Let me know what you think? >  > Thanks-in-advance, >  > jerry >  > --  > Dr. Gerard P. Gilfoyle > Physics Department                e-mail: ggilfoyl@richmond.edu > University of Richmond, VA 23173  phone:  804-289-8255 > USA                               fax:    804-289-8482 >  ----------------------------steven james, director of research, linux labs LinuxBIOS Cluster Solutions                   230 peachtree st nw ste 2705 High-Speed Colocation, Hosting,                        atlanta.ga.us 30303 Linux Hardware, Development & Support             http://www.linuxlabs.com * Visit us at SuperComputing 2002, Booth 1441 *   office/fax 404.577.7747/3 --------------------------------------------------------------------------