VMware ESXi 6.0 virtual ib_ipoib interfaces

1

Newbie question: What is the correct syntax (if at all possible with version of VMware ESXi 6.0 U1 and OFED 2.4.0.0) for creating sub/virtual interfaces? From what I can gather in the documentation would it be something like the following:

esxcli system module parameters set -m mlx4_core -p set_4k_mtu=1,msi_x=1,num_vfs=16,probe_vf=16

After rebooting I have not be able to see any additional interfaces prompting this query on the forum. Please advise.

2

Hi David,

  1. Driver version 2.4.0 is an Ethernet driver only so if you have a VPI card, please make sure the port protocol is configured as 2 which is Ethernet.

#/opt/mellanox/bin/mlxconfig -d /dev/mt4099_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

  1. Enable SRIOV with the MFT tool (Mellanox FW tool) as follow:

#/opt/mellanox/bin/mlxconfig -d /dev/mt4099_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=16

  1. Confirm BIOS setting supports SRIOV and/or virtualization.

  2. Set the driver module parameters for VF:

#esxcli system module parameters set -m mlx4_core -p ‘num_vfs=<VFs over Port1, VFs over port2, 0> port_type_array=2’

This is all documented in the driver UM.

Thank you,

Sophie.

3

Driver 2.4.0 include IPoIB driver and works!

Absolutlely! driver 2.4.0 located in ethernet driver section, but why this driver 2.4.0 include IPoIB driver & works?

4

Hi Sopie

we don’t want to use it as Ethernet card but as IB over IP. That’s the problem that I can’t get it to work as a 40GB IBoIP card in ESX 6 anymore, which worked perfectly with ESX 5.5 and the 1.8.2.4 drivers (which are not supported anymore).

Would you please also link to the driver UM you are referring too?

5

Hi david,

Can you confirm what type of HCA card(s) installed within the ESXI server?

What is the port protocol configured as IB or Ethernet?

Thank you,

Sophie.

6

I have a similar problem: i cant get IP over IB working.

MT 27500 ConnectX-3 (PSID MT_1100120019) on ESX 6 (latest Updates 2 days ago). In the BIOS (SuperMicro) I set in the PCIe Configuration Page “Above 4G Decoding” as “Disabled” since otherwise the card was not recognized in older SuperMicro hardware.

First installed the 3.15.5.5 and also updated the firmware to 5150

Then had a vmnic2 with up to 40000, but link was down.

Then I uninstalled and installed OFED 2.4.0.0

Then had a vmnic2 with up to 1000GB, link down.

I also tried to install both drivers, same issue: link down, card is recognized as Ethernet and not IPoIB.

I already opened a support ticket with VMWare but the technician had no solution.

It seems the last know driver to work is 1.8.2.4 (see here VSphere 6.0 Infrastructure & Networking - NVIDIA Developer Forums ) but this driver is not supported in 6.x

By the way: when you select now OFED 2.4.0.0. you are redirected to 2.3.3.1 for download (http://www.mellanox.com/page/products_dyn?product_family=36&mtag=vmware_drivers http://www.mellanox.com/page/products_dyn?product_family=36&mtag=vmware_drivers )

I had no problem with the same cards in similar (older) hardware and ESX 5.5.