ESXI 6.7 nmlx drivers

Infrastructure & Networking Software And Drivers Virtualization For Infiniband And Ethernet
1

Hi,

im wondering whats the difference between my hosts and drivers. Since i have massive speed problems when using storage vmotion and im trying to find the best driver/software version but i cant figure out the source of the problem

3 HP DL380 G9 with Mellanox Technologies MT27700 Family [ConnectX-4]

Remove old net-mlx4 when using 5 ? Do i need both ? Why has the current 6.7 stock driver version 1.9.7 when the latest mlx driver package from homepage has 4.17 ?

Is the VMWare 6.7 known issue the problem ?:

The Mellanox ConnectX-4/ConnectX-5 native ESXi driver might exhibit performance degradation when its Default Queue Receive Side Scaling (DRSS) feature is turned on

Receive Side Scaling (RSS) technology distributes incoming network traffic across several hardware-based receive queues, allowing inbound traffic to be processed by multiple CPUs. In Default Queue Receive Side Scaling (DRSS) mode, the entire device is in RSS mode. The driver presents a single logical queue to OS and is backed by several hardware queues.

The native nmlx5_core driver for the Mellanox ConnectX-4 and ConnectX-5 adapter cards enables the DRSS functionality by default. While DRSS helps to improve performance for many workloads, it could lead to possible performance degradation with certain multi-VM and multi-vCPU workloads.

Workaround: If significant performance degradation is observed, you can disable the DRSS functionality.

  1. Run the esxcli system module parameters set -m nmlx5_core -p DRSS=0 RSS=0 command.
  2. Reboot the host.

Host1 (working fast)

NetAppNasPlugin 1.1.0-0 NetApp VMwareAccepted 2016-12-06

net-mlx4-core 1.9.7.0-1vmw.670.0.0.8169922 VMW VMwareCertified 2018-04-27

net-mlx4-en 1.9.7.0-1vmw.670.0.0.8169922 VMW VMwareCertified 2018-04-27

Host2 SLOW

NetAppNasPlugin 1.1.0-0 NetApp VMwareAccepted 2016-12-05

nmlx4-core 3.17.9.12-1vmw.670.0.0.8169922 VMW VMwareCertified 2018-04-27

nmlx4-en 3.17.9.12-1vmw.670.0.0.8169922 VMW VMwareCertified 2018-04-27

nmlx4-rdma 3.17.9.12-1vmw.670.0.0.8169922 VMW VMwareCertified 2018-04-27

nmlx5-core 4.17.9.12-1vmw.670.0.0.8169922 VMW VMwareCertified 2018-04-27

nmlx5-rdma 4.17.9.12-1vmw.670.0.0.8169922 VMW VMwareCertified 2018-04-27

Host 3 (Partly slow)

nmst 4.7.0.42-1OEM.650.0.0.4598673 MEL PartnerSupported 2017-08-09

mft 4.7.0.42-0 Mellanox PartnerSupported 2017-08-09

NetAppNasPlugin 1.1.2-3 NetApp VMwareAccepted 2017-08-18

nmlx4-core 3.17.9.12-1vmw.670.0.0.8169922 VMW VMwareCertified 2018-04-27

nmlx4-en 3.17.9.12-1vmw.670.0.0.8169922 VMW VMwareCertified 2018-04-27

nmlx4-rdma 3.17.9.12-1vmw.670.0.0.8169922 VMW VMwareCertified 2018-04-27

nmlx5-core 4.17.9.12-1vmw.670.0.0.8169922 VMW VMwareCertified 2018-04-27

nmlx5-rdma 4.17.9.12-1vmw.670.0.0.8169922 VMW VMwareCertified 2018-04-27

thanks for the help

2

Hi Tim.

  1. ESXi 6.7 has Mellanox driver as inbox. ( in your case : nmlx5-core + nmlx5-rdma modules for ConnectX-4, ConnectX-5 family)

  2. nmlx4-* modules are of native driver for ConnectX-3 NIC family (in your case can be removed)

  3. net-mlx4-* modules are of vmkinux driver (old API) for ConnectX-3 NIC family (in your case can be removed)

  4. In general, it’s always advised to install latest native driver from (when available) : http://www.mellanox.com/page/products_dyn?product_family=29 http://www.mellanox.com/page/products_dyn?product_family=29

  5. About performance there is too little information about configuration and actual bandwidth, kindly elaborate.