Hi Damal  this is roughly how I use these commands to  investigate performance issues in new situations:  > isi statistics pstat  Bottom half: Overall view to see what's going on. Compare Network -- Filesystem -- Disk  throughputs to see wether they are consistent with what is expected for the workflow as far as it is known. In a typical NAS workflow, network throughput should match filesystem througput. Write throughput should match disk write throughput (mind the protection overhead). Read throughput matches disk read throughput for uncached workflows, or can be much higher which good caching.  High disk activity without network or filesystem activity indicates some internal job is running (restripes etc).  High CPU without any network/filesystem/disk activity would be very strange, for example (some process running wild, etc)  Just illustrating how things can be learned from pstat.  Top half of pstat is protocol specific, might need to run separately for NFS and SMB. Again, quick check for consistency:  Does the observed mix of reads/writes/ etc. make sense in the light of the assumed workflow?   > isi statistics system --nodes --top  Quick check wether the load is well balanced across the cluster, and how the protocols are used (SMB vs NFS etc). Can also indicate where physical network bandwidth hits the max.  > isi statistics client --orderby=Ops --top  Who is causing the load? --orderby=In or Out or TimeAvg for throughputs or latencies resp.  > isi statistics client --orderby=Ops --top --long  With --long there are InAvg and OutAvg, which denote the request ("block") sizes (NOT the average of In and Out rates!!).  Small request sizes often indicate suboptimal configs on the clients side.  > isi statistics drive -nall -t --long --orderby=OpsOut  Do the disk activities max out? Also --orderby=OpsIn or TimeInQ  Do the disk activities match the assumed workload: Small SizeIn and SizeOut request size indicate metadata ops or small random IO ops; pure streaming reads/write are usually upto 64k.  Disclaimer: Those are just by personal favorites (plus isi statistics heat), and I might err on interpretations; I am not aiming at convincing anyone [IMAGE]  Cheers  -- Peter
Hi all, We have setup of four X 200 storage nodes, where each node has 6 Gig RAM,12 disks of 7200 RPM SATA (except storage node #4 , which has 11 disks), total 47 disk. We are accessing storage cluster using 4 compute nodes via NFS protocol.Front end networking from compute node to storage cluster is through 1Gbps switch (each compute node can read and write data to storage cluster using 1Gbps bandwidth) ,backend networking is done using QDR infiband.  We did performance evaluation to get Maximum random read IOPS for 128K chunk read(we choose 128K , because our elasticsearch(application) read data in 128 K chunk). We have use FIO tool to check performance. Our observations for the same is : 1.With 20 FIO jobs(20 parallel reading thread) , running on single compute node, gives us max 850 IOPS of 128 K chunk (max out network utilization of 1 Gbps) , when we run same FIO job for 2 compute nodes in parallel(total 40 reading thread, 20 on each)  , performance drops to 673 & 691 respectively , total 1364 IOPS. When we run same FIO job for 4 compute nodes in parallel(total 80 reading thread, 20 on each) , performance further drops to 252,255,567,550,total 1624 IOPS. Whereas with 8K random read chunk from 4 compute nodes(80 reading threads,20 on each compute node) it was giving 4602 IOPS.  My question : Why we are stuck at max 1624 IOPS in case of 128 K chunk random read whereas total 47 disk can give much higher IOPS(as with 8k random read chunk ,it is giving 4602 IOPS) I understand , when we read 128K chunk , we effectively reading more, but in this case : neither network from compute to storage nor CPU on storage and compute node was bottleneck. In all above cases, we used 8K disk block size on isilon ,random access pattern to optimize IOPS , protection was +2:1. We also tried to increase disk block size in isilon but with increase in disk block size , the performance was decreasing.  It will be nice if you people will recommend tuning parameters to optimize IOPS for 128 K chunk.  Regards, Prakash     
Isilon storage performance issue