On multiple intel servers we got issues with connectivity between Virtual machines on ESXI 6.7
Part Number: MCX4121A-ACA_Ax
Description: ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
Its like 100k packet retransmitted in iperf3 test during 15 second test. On host level and via live cd everything works fine.
We can lower packet retransmitted to about 1k by changing bufferes. Anyone got similar issues ?
You can check if packet are retransmitted by iperf3 on linux or by netstat -s
esxcli network nic ring current set -n vmnicx -t 8192 -r 8192
I would make sure latest driver/FW are installed on the Hypervisor(s).
I would make sure based on the VM’s OS that Mellanox driver are installed.
I would recommend to use iperf2, why?
We recommend using iperf and iperf2 and not iperf3. iperf3 lacks several features found in iperf2, for example multicast tests, bidirectional tests, multi-threading, and official Windows support. For testing our high throughput adapters (100GbE), we recommend to use iperf2 (2.0.x) in Linux and NTTTCP in Windows.
- Are you getting any drops/discards or buffer errors (ie: rx_out_of_buffer)?
(VM’s and/or Switch(s) as applicable).
If you reverse client/server on the VM’s, are you observing the same issue?
Do you have flow control configure on the host and switch (if switch applicable).
If switch applicable, did you test b2b accordingly? (no switch).
I am not sure what you meant by “via live cd”.
What is the iperf CLI options you are using on the client/server?
What are the ring buffers on the VM’s?
- Yes we have Firmware Version: 14.27.1016 , Driver Version: 220.127.116.11 (tried also .3)
- Mellanox driver inside VM ?! Why ? We use vmxnet3 driver . Anyway we got error "current mlnx-en is intended for debian 10.0)
- Yes rxSwWqeErr
8192 buffer size - retransmitted 9071 from 42793460 = 0.021197 Bandwitch 10 Gb
1024 default - retransmitted 3004664 from 444775670 = 0.67554 . Bandwitch 20 Gb
Carrier transitions: 103, Errors: 0, Drops: 13588, Collisions: 0, Aged packets: 0, FIFO errors: 0,
Queue counters: Queued packets Transmitted packets Dropped packets
0 0 31412819106 13588
3 0 0 0
4 0 0 0
7 0 1415459 0
8 0 74232 0
Nope on switch Ethernet-Switching Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled
Dont know how to check on host but
Pause Autonegotiate: false
Pause RX: true
Pause TX: true
7 not enabled
- we booted server from ISO debian-live-10.4.0-amd64-kde.iso
9_ iperf3 -P12 -t 15
10.) default - changes without effect - only if we change on host level not inside vm
rx mini 128
rx jumbo 256