Turns out you need to start multiple iperf processes from separate SSH sessions to max out the 25GbE pipe. esxcli network firewall set --enabled false SSH session 1 - host 1 # /usr/lib/vmware/vsan/bin/iperf3.copy -s -B 10.10.10.10 -p 5201 SSH session 2 - host 1 # /usr/lib/vmware/vsan/bin/iperf3.copy -s -B 10.10.10.10 -p 5202 SSH session 3 - host 1 # /usr/lib/vmware/vsan/bin/iperf3.copy -s -B 10.10.10.10 -p 5202 SSH session 1 - host 2 # /usr/lib/vmware/vsan/bin/iperf3.copy -P 4 -M -t 60 -c 10.10.10.10 -p 5201 SSH session 2 - host 2 # /usr/lib/vmware/vsan/bin/iperf3.copy -P 4 -M -t 60 -c 10.10.10.10 -p 5202 SSH session 3 - host 2 # /usr/lib/vmware/vsan/bin/iperf3.copy -P 4 -M -t 60 -c 10.10.10.10 -p 5202 esxcli network firewall set –enabled true
Customer has 5 vSAN ready nodes all PE7525 AMD EPYC processors. Setup: 2x S5248F-ON firmware 10.5.1.4. 2x Mellanox ConnectX-5 dual port - firmware 16.27.61.06 ESXi 7.0 - Mellanox driver - Mellanox-nmlx5_4.19.70.1-1OEM.700.1.0.15525992_16253686-package iperf between hosts we can only get about 15-16Gbps. I wasn't expecting line rate, but at least 20Gbps+ Host1: esxcli network firewall set --enabled false/usr/lib/vmware/vsan/bin/iperf3.copy -s 10.0.0.1 Host2: esxcli network firewall set --enabled false/usr/lib/vmware/vsan/bin/iperf3.copy -c 10.0.0.1 -i 1 -t 10 -w 16M Tuning we have tried, but these make no difference. esxcfg-advcfg -s 4 /Net/TcpipRxDispatchQueuesesxcfg-advcfg -s 65535 /Net/VmxnetLROMaxLengthesxcli system settings kernel set -s netMaxPktsToProcess -v 128esxcli system settings kernel set -s intrBalancingEnabled -v falseesxcli network nic ring current set -r 4096 -n vmnic2esxcli network nic coalesce set -a false -n vmnic2 Any thoughts please, Dell support won't help?
Mellanox ConnectX-5 not achieving 25GbE with vSphere 7.0