So if RR made it worse, then the issue isn't MEM.   It really sounds like the switch can't handle the additional traffic.  The 28xx series is one I've worked with in the past and it tends to fail at higher loads.  If it is, if you get the latest FW for it, there's a setting you can enable called "qos-passthrough-mode one-queue'"   That will provide you the max buffer / port.  By default it splits the available mem (which is small to start) into four QoS queues.   So iSCSI only has access to 1/4 of the buffers.     With HP switches you are looking for switches that have the "Provision" chipset.  That allows both Jumbo Frames and Flowcontrol.   Also they tend to have more buffers as well.    Regards, 
Three Poweredge r710s running vSphere 5, software ISCSI on Intel NICs, running through two HP switches (lower end switches) physically separated from all other network traffic and lagged together.  Note that the hp switches have an issues running jumbo frames and flow control so I am using flow control without jumbo frames. The SAN is a PS4000 with 600 GB SAS (10 or 15k I forget) in two raid 5 LUNs  with dual controllers. After installing MEM the throughput plummets to about 1/3rd of single ISCSI connection as measured on my backup servers speed/network utilization.  I have been using MEM for quite sometime with ESX 4i so it is not a settings issue (not to mention the script runs you through the install) and have verified multiple paths are "active". As soon as I change the setup back to a single active path the backup process and network utilization triples in speed. Anyone having any luck with the new MEM and vSphere 5? Thanks,Bob
MEM is slow with vSphere 5?