Firoz,  I took a look at the NAR and it looks like the beginning of the NAR starts about 7:15am (15 minutes in to the 7-8am time frame you are concerned about).  Can you confirm that?   Assuming that's true, here's what I see..  From 7-8am there is a batch job, backup, or some other activity occuring on Meta180 that drives utilization up to about 10MB/sec of writes, with write IO sizes around 200-300KB.  Response time climbs up over 300ms, which is likely contributing to the IO waits on the host.  Looking at write cache stats, it seems these writes are random, ie: very little locality.  FASTCache hit % is low during that time and the SP Write Cache rehit ratio is near zero.  The queue length on LUN 180, it's component LUNs, and the underlying disks are all reasonable which points to a bottleneck elsewhere.  I suspect two possible causes right now...  1.) Copy on First Write activity related to MV/A during the 7-8am window is causing high response times.  It looks like MV/A updates about every 30 minutes and in this NAR there is an update finishing around 7:20 and another one starts around 7:45.  During the actual active updates, any writes to a block that is part of the update requires a COFW to the Reserve LUN Pool.  Since these writes are large, a single 256KB write requires 4 chucks to be transferred to the RLP, which actually translates to 16 IO's since the RLP is RAID5.  The RLP LUN (621) shows as 100% busy during the high response time period.  Also, since the writes have low locality of reference, there is likelihood of large amounts of RLP copies during that time.      a.) Moving the RLP LUNs to RAID10 could help      As a test, see if you can disable the automatic MV/A updates for an evening and let the 7-8am period run through with no MV/A update.  If there is no active update, there will not be any COFW activity and the Reserve LUN Pool will be out of the picture.  If the response time improves you have a place to focus.  2.) It's also possible that the host needs more disk IO queues.  If the process they are running is highly multi-threaded, you could be bottlenecking at the host IO queue side.  The wait to solve this is to present more than one array LUN to the host.  Currently, even though you have the MetaLUN with two components across 16 disks, the host sees just one LUN and one queue.  If you did away with the Meta, and just created two LUNs, one in each RG, assigned one to SPA and the other to SPB, and then had the host stripe across those LUNs with whatever filesystem it has (ASM, VxFS, ZFS, etc) you'd double the number of IO queues at the host, balance the load across the two SPs, and the second LUN would get it's own RLP LUN as well.  If the IO queue is the bottleneck, this would improve performance, even though you are using the same two 4+4 RGs.      BIG CAVEAT:  If you use two LUNs instead of a single MetaLUN, you MUST put the two LUNs into a consistency group for Mirrorview or you won't be able to recover the data on the target.  [IMAGE]   Separately from the meta LUN performance, you are also seeing Queue Full's pretty much all day on some of the front end ports.  Ports 5 and 6 on both SPs are the worst, with B5 and B6 being the standouts.  It would be good to spend some time checking HBA execution throttles and numbers of initiators per port to see if you can adjust the throttles down and alleviate the port queuing.  Each front end port has an effective queue depth of 1600 so if many hosts are sharing the same port and they all have high execution throttles, the port queue can get filled up.  Queue Fulls result in periodic slowdowns at the host side.  Hope that helps a bit.  Richard
Hi All  I am facing issue in analyzing the problem with one of the 1000 VMs on our CX4-960. We got ESX4.0 with 1000+ VMs in multiple clusters. Linux RHEL 4.0 on an ESX 4.0 connected to CX4-960 4.30.xx.xx.525, This VM is mounted on a 6 different luns with different raidgroups on the array. The VM has an Oracle DB running on it. The problem statement is they see high IO wait only during 7:00 to 08:00 am GMT & thats the time when 2 different activity happens (1) Users login in (2) Oracle schedule jobs are run. The rest of the day it is running perfectly fine with no issues. Server team claims CPU/Memory of the server to be under control & only bottleneck they see is IOwait. So I ran NAR files for couple of days and this is what I see. Out of the 6 luns their are 2 luns of RAID 10. One of the lun is on a 4+4 RG where as other one is a metalun spread on a two different RGs of (4+4 i.e. 16 spindles) I have analyzed all the 6 luns and found only one 350 GB luns which is a metalun residing on a 2 x 8 spindles RAID 10 RG  This is a striped metalun of 175 GB each carved on a (4+4) 2 different RG R1_0 (FC 15krpm drives)  Below is the output i see on the NAR files Lun utilization during the slowness period is 100% Lun total throughput is 160 IOPS r/w ratios is 80:80 iops though read bandwidth is hardly an mbps but writes are big chunks upto 8-10 mbps with filesize over 128kbps Lun doesn't show SP forced flush not even once SP utilization during this period is hardly 45% (both SPs with sp cache being within 60-80 watermark) no flush occuring Lun queue length is between 8 to 14 same as ABQL Lun response time seems to be around 300 ms but i dont think this is any problem as its been processed by cache  I drilled down to the Raidgroups/disks and found it to be very optimal and underutilized Disk utilization of both the RGs is under 10% Disk queue length of both RGs is 0.50 to 0.80 Disk response tims is ranging from 15 ms to 69 ms (this i feel is an issue) each disks in both the RGs are transfering @ the rate of 1-2mbps IOPS of each drive is not above 40 IOPS write files size on each drive seems higher than 96 kbps. read & write bandwidth both are below 1 mbps ABQL is between 8 to 12  I verified all the other luns in the two RGs are very quite during the problem period basically nails down to this lun & its component lun in the other RGs  How would I judge if  this lun really has issues in getting IOs processed. Though my Disk respone time is higher but i still dont see a single spike on my SP Cache. Write file size are bigger and so the bandwidth too is higher on the lun but the same is low in my drives.   Is their any other parameter which can give me a clue here? its been 3 days since the problem is appearing & its hitting hard in my nerves.   I am not knowledgeable on Linux stuff and the server owner loaded me with the IO outputs when asked and was only able to drill down to the devices SDC/SDC1. I have attached the output in the text format to this discussion.  This lun is in mirrorview/A relationship  any advice at the earliest to this will be highly appreciated  thanks  Firoz
Application slowness issue