SR-IOV on ESXi

Hello !

We are trying to deploy scale out converged software defined storage in vSphere and are limited by the number of physical PCIe slots in some hosts. Therefore we have a need for SR-IOV.

We have had a nightmare and spent a tremendous amount of time trying to figure some questions out but have been unsuccessful so far.

The questions are as follows and concern mainly ConnectX-3 VPI (MCX354A-FCBT) and ConnectIB (MCB194A-FCAT) adapters:

  1. Which ESXi version exactly allows SR-IOV with these adapters (5.5 / 6.0 / 6.0u2 etc…) ?
  2. Which matching drivers exactly should be used (Native / OFED) for each version of ESXi ?
  3. Is there a specific recommendation on which firmware to use ?
  4. Where can we find precise documentation and procedure on how to configure SR-IOV on the ESXi hosts ?
  5. In a physical host running ESXi 6.0u2 (or eventually lower), is it possible to mix ConnectIB, ConnectX-3 and ConnectX-2 adapters where some adapters are passed-through entirely to one VM ?
  6. Is there by any chance a script to clean up an ESXi host of all previous Mellanox and OFED drivers completely… ? Indeed, each installation requires a manual removal of each module, which is very time consuming. This is especially true when updating the ESXi hosts and the drivers having to be uninstalled before the update and reinstalled after the update due to the strict acceptance level of the ESXi update process.

Thanks a lot for your insight.

Hi

Answers are inline:

[Q] - Which ESXi version exactly allows SR-IOV with these adapters (5.5 / 6.0 / 6.0u2 etc…) ?

[A] - In Esxi5.5Ux & Esxi6.0Ux - Mellanox supports SR-IOV over Connectx-3 VPI (not over ConnectX-IB) with the following restrictions:

  1. ESXi Hypervisor (Vsphere) - SR-IOV Ethernet (binded to Linux)

  2. ESXi Hypervisor (Vsphere) - SR-IOV Infiniband (binded to Windows)

[Q] - Which matching drivers exactly should be used (Native / OFED) for each version of ESXi ?

[A] - Matching drivers:

  1. Inbox driver v3.0.0.0 is for Esxi6.0.(ethernet interface only )

2.Native driver v3.2.0 is for Esxi6.0.(ethernet interface only )

  1. mlnx_ofed_esxi v2.3.3 is for Esxi5.5 and supports both IB & ETH

  2. mlnx_ofed_esxi v2.40 is for Esxi5.5 & esxi6.0 and supports both IB & ETH

[Q] - Is there a specific recommendation on which firmware to use ?

[A] - yes. It is recommended to install fw ver. 2.36.5000 for Connectx-3 VPI (can be grabbed from Mellanox web site)

[Q] - Where can we find precise documentation and procedure on how to configure SR-IOV on the ESXi hosts ?

[A] - In chapter 3.3 ofMellanox_OFED_ESXi_User_Manual_v2.4.0 you’ll find a detailed description on how to set SR-IOV on the esxi[

[Q] - In a physical host running ESXi 6.0u2 (or eventually lower), is it possible to mix ConnectIB, ConnectX-3 and ConnectX-2 adapters where some adapters are passed-through entirely to one VM ?

[A] - Basically it is feasible but it is not recommended at all to do do. the reason is because it involves burning new & old fw on the different old & new adapters, and the other drawback is that you’ll have one precisely mach the proper/common mlnx_ofed driver that can handle all the “mixed” firmwares.

[Q] - Is there by any chance a script to clean up an ESXi host of all previous Mellanox and OFED drivers completely… ? Indeed, each installation requires a manual removal of each module, which is very time consuming. This is especially true when updating the ESXi hosts and the drivers having to be uninstalled before the update and reinstalled after the update due to the strict acceptance level of the ESXi update process.

[A] I certainly understand what you mean, but I’m not aware of any script that we use to clean up the older modules in one shut…we selectively remove them prior installing the new driver…this is the securest way to ensure all “leftovers” are removed

I don’t know why but I had not seen mlnx_ofed_esxi v2.40 until now.

Was unsuccessfully trying with v2.3.3.1 on ESXi 6.0.

So, just to confirm if I understood correctly:

  • According to the restrictions you mentioned, it is not possible to assign an Infiniband VF function to a Linux guest in ESXi 6 (hence no IPoIB) ?

  • SR-IOV is not available for Connect-IB in ESXi ?

  • There is no storage protocol (iSER / SRP) available in OFED 2.4 ?

Many thanks for the detailed information.

Hi!

[Q] Is there by any chance a script to clean up an ESXi host of all previous Mellanox and OFED drivers completely… ? Indeed, each installation requires a manual removal of each module, which is very time consuming. This is especially true when updating the ESXi hosts and the drivers having to be uninstalled before the update and reinstalled after the update due to the strict acceptance level of the ESXi update process.

[A] Did you try using VMware Power CLI that based on Windows PowerShell?

I made a custom image for my R610 ESXi hosts.

This image removed original Mellanox inbox drivers then I can install vSphere OFED after ESXi host installed…:)

Hi Jae-Hoon,

I use ESXi-Customizer-PS to inject the drivers but it does not help when the ESXi is already installed on the host.

We have variety of hosts and adapter types (Connect-X2, ConnectX-3, Connect-IB and probably soon ConnectX-4 or 5), and the manual cleaning up of drivers and VIBs are painful and time consuming because the list of items to remove is different from host to host, which makes a simple list impossible to apply to “esxcli software vib remove…”

Any ideas are welcome.

Thanks for your contribution.

Hi!

I pray these scrips to help you.

  1. Uninstall Mellanox vSphere OFED 1.8.2.4 & 1.8.3 from ESXi host

1st you must check your installed driver list with esxcli software vib list | grep Mellanox

esxcli software vib remove -n scsi-ib-srp

esxcli software vib remove -n net-ib-ipoib

esxcli software vib remove -n net-mlx4-ib

esxcli software vib remove -n scsi-ib-iser

esxcli software vib remove -n net-rdma-cm

esxcli software vib remove -n net-ib-addr

esxcli software vib remove -n net-ib-cm

esxcli software vib remove -n net-ib-umad

esxcli software vib remove -n net-ib-sa

esxcli software vib remove -n net-ib-mad

esxcli software vib remove -n net-ib-core

esxcli software vib remove -n net-mlx4-en

esxcli software vib remove -n net-mlx4-core

esxcli software vib remove -n net-mlx-compat

esxcli software vib remove -n net-memtrack

then reboot your ESXi host

  1. Uninstall Mellanox vSphere OFED 2.3.3.1 & 2.4.0 from ESXi host

1st you must check your installed driver list with esxcli software vib list | grep MEL

esxcli software vib remove -n net-ib-ipoib

esxcli software vib remove -n net-mlx4-ib

esxcli software vib remove -n net-ib-sa

esxcli software vib remove -n net-ib-mad

esxcli software vib remove -n net-ib-core

esxcli software vib remove -n net-mlx4-en

esxcli software vib remove -n net-mlx4-core

esxcli software vib remove -n net-mlx-compat

reboot your ESXi host

You’re correct.

This is a time consuming work, but there is no choice…

AND If you want to upgrade ESXi host to new version, you must remove inbox driver from original ESXi image.

If you want build a custom image for your ESXi host, you can build with ESXi offline bundle and vSphere PowerCLI.

I have a 10 of Dell PE R610 ESXi host.

I’m always remove Mellanox inbox driver from Dell Customized offline bundle then create custom ESXi ISO image for my hosts.

This custom ESXi ISO image - removed inbox driver - can upgrade ESXi host to new version directly.

I’m also build a custom ESXi ISO image that include Mellanox vSphere OFED driver 1.8.2.4.

  • also remove inbox driver then inject driver 1.8.2.4

All was successfully work for my ESXi hosts.

If you have standard your host configuration and driver version, you can build a custom images for your host.

I think build a custom image will help you.