I finally made headway with this issue by using sndcwnd = 65535 in my environment.  Doing a read test over NFS from a Linux client attached via 1Gbps (VNX has 10GbE), the results looked like this:  sndcwnd = 0 (default): ~13MB/sec sndcwnd = 300000: ~13MB/sec sndcwnd = 131071: ~13MB/sec sndcwnd = 75000: 111MB/sec with occasional drop to 68MB/sec sndcwnd = 65536: 117MB/sec extremely consistent sndcwnd = 65535: 117MB/sec extremely consistent sndcwnd = 32768: 117MB/sec extremely consistent  Our environment involves Cisco 7000 series with FEX switches; the VNX is attached to the core switches while the 1Gbps servers are attached to FEX.  We found that performance was much better when a client was attached at 1Gbps directly to the core, but consistently poor when attached to the FEX until this sndcwnd parameter was modified.  edit: This is a server_param parameter that requires a datamover reboot.  e.g.:  server_param server_2 -facility tcp -modify sndcwnd -value 65535 server_standby server_2 -activate mover ... server_standby server_2 -restore mover
Running into issue where we are migrating off of NetApp arrays with 1Gb connections to a Celerra with 10Gb connections and the old NetApp is handily outperforming the Celerra on large I/O.  Tracing things down in the network it looks like we overrunning buffers like crazy.  We have a Cisco Nexus 5000 core with Cisco 2xxx FEX links that the Celerra is connected to.  On one of the NAS heads I've got >2% retransmit rate, playing with the sndcwnd, fastRTO, std_slowstart I'm able to get speeds back up to resonable levels but the window size is tiny (64k bascially no window scaling) which then affects replication and high-latency WAN clients.    The network group is telling me the switch is sending pause frames constantly (not sure if they are to the Celerra or the FEX ISL's), my understanding is that the Broadcom bcmXG 10Gig chips don't support flow control so even if they are sent the Celerra ignores it.  I'd have expected TCP window scaling to kick in and prevent it from being such a big problem (if I disable window scaling on local clients performance is good as well).  Is there any reason that TCP windowing is not preventing such large amount of drops from occuring?  Are there any specific network switch configuations that should be done etc. for a mixed speed network?  Clients are all HP servers (DL360 or blades) running Redhat 4 or 5 with broadcom chips as well.  Cisco TAC is having us play with some buffer settings and moving links around to better distribute across ASIC's but not sure how much runway that will get me when I'm dropping so much already.  I'm not pumping high aggregate bandwidth (maybe 2-3Gbit) but I'm running over the switch buffers like crazy.  I'm a bit at a loss, almost thinking of dumping 10Gb connections and doing a 4x 1Gb link aggregation channel but there has got to be a way to do it as everybody I talk to don't seem to have the problem I'm having.  Celerra NS960: Each DM has 2x 10Gb connections in LACP configuration to Nexus 2xxx fabric extender with FEX links to Nexus 5000 switch
Best way to deal with mixed speed network (10Gb & 1Gb)