Packet retransmissions ESXI 6.7 ConnectX4 LX

Infrastructure & Networking Software And Drivers
1

On multiple intel servers we got issues with connectivity between Virtual machines on ESXI 6.7

Part Number: MCX4121A-ACA_Ax

Description: ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6

PSID: MT_2420110034

Its like 100k packet retransmitted in iperf3 test during 15 second test. On host level and via live cd everything works fine.

We can lower packet retransmitted to about 1k by changing bufferes. Anyone got similar issues ?

You can check if packet are retransmitted by iperf3 on linux or by netstat -s

esxcli network nic ring current set -n vmnicx -t 8192 -r 8192

2

Hello,

  1. I would make sure latest driver/FW are installed on the Hypervisor(s).

  2. I would make sure based on the VM’s OS that Mellanox driver are installed.

  3. I would recommend to use iperf2, why?

We recommend using iperf and iperf2 and not iperf3. iperf3 lacks several features found in iperf2, for example multicast tests, bidirectional tests, multi-threading, and official Windows support. For testing our high throughput adapters (100GbE), we recommend to use iperf2 (2.0.x) in Linux and NTTTCP in Windows.

  1. Are you getting any drops/discards or buffer errors (ie: rx_out_of_buffer)?

(VM’s and/or Switch(s) as applicable).

  1. If you reverse client/server on the VM’s, are you observing the same issue?

  2. Do you have flow control configure on the host and switch (if switch applicable).

  3. If switch applicable, did you test b2b accordingly? (no switch).

  4. I am not sure what you meant by “via live cd”.

  5. What is the iperf CLI options you are using on the client/server?

  6. What are the ring buffers on the VM’s?

Sophie.

3
  1. Yes we have Firmware Version: 14.27.1016 , Driver Version: 4.17.70.1 (tried also .3)
  2. Mellanox driver inside VM ?! Why ? We use vmxnet3 driver . Anyway we got error "current mlnx-en is intended for debian 10.0)
  4. Yes rxSwWqeErr

8192 buffer size - retransmitted 9071 from 42793460 = 0.021197 Bandwitch 10 Gb

1024 default - retransmitted 3004664 from 444775670 = 0.67554 . Bandwitch 20 Gb

Output errors:

Carrier transitions: 103, Errors: 0, Drops: 13588, Collisions: 0, Aged packets: 0, FIFO errors: 0,

Queue counters: Queued packets Transmitted packets Dropped packets

0 0 31412819106 13588

3 0 0 0

4 0 0 0

7 0 1415459 0

8 0 74232 0

  1. YES

  2. Nope on switch Ethernet-Switching Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled

Dont know how to check on host but

Pause Autonegotiate: false

Pause RX: true

Pause TX: true

7 not enabled

  1. we booted server from ISO debian-live-10.4.0-amd64-kde.iso

9_ iperf3 -P12 -t 15

10.) default - changes without effect - only if we change on host level not inside vm

rx 1024

rx mini 128

rx jumbo 256

tx 512

4

Hello,

The Inbox driver embedded via the OS is supported by the OS vendor, Mellanox support is via our driver.

Also, Mellanox has not tested Debian 10.4.

The vmxnet3 driver is supported from Vmware (could be the bottleneck).

Reference links that might help: