I'm trying to enable sriov with the adapter MCX512A-ACAT in a esxi 7.0, but no luck so far. I have followed the instructions of this link https://docs.nvidia.com/networking/pages/releaseview.action?pageId=15053024. What am I missing?

Infrastructure & Networking Adapters and Cables
1

The blade is a Supermicro Super Server and sriov is enabled in the BIOS.

In the kernel logs I see the following:

ESC[7m2022-02-18T01:53:51.528Z cpu2:1049177)WARNING: PCI: 870: 0000:17:00.0: unable to allocate 0x800000 bytes in prefetch mmio VF-BAR[0]ESC[0m

2022-02-18T01:53:51.528Z cpu2:1049177)PCI: 344: 0000:16:00.0 P2P bridge resources:

2022-02-18T01:53:51.528Z cpu2:1049177)PCI: 345: IO: 0x0 - 0x0

2022-02-18T01:53:51.528Z cpu2:1049177)PCI: 346: Mem: 0xc5d00000 - 0xc5efffff

2022-02-18T01:53:51.528Z cpu2:1049177)PCI: 347: PrefetchMem: 0xc0000000 - 0xc3ffffff

2022-02-18T01:53:51.528Z cpu2:1049177)PCI: 955: Failed to allocate and program VF BARs on 0000:17:00.0

ESC[7m2022-02-18T01:53:51.528Z cpu2:1049177)WARNING: PCI: 1558: Enable VFs failed for device @0000:17:00.0, Please make sure the system has proper BIOS installed and enabled for SRIOV.ESC[0m

2022-02-18T01:53:51.528Z cpu2:1049177)<NMLX_ERR> nmlx5_core: 0000:17:00.0: nmlx5_CoreQueryMaxVfs - (nmlx5_core_main.c:719) vmk_PCIEnableVFs failed - Failure

2022-02-18T01:53:51.528Z cpu2:1049177)<NMLX_ERR> nmlx5_core: 0000:17:00.0: nmlx5_core_EnableVFs - (nmlx5_core_main.c:848) done, status: Failure

2022-02-18T01:53:51.528Z cpu2:1049177)<NMLX_INF> nmlx5_core: nmlx5_en_Attach - (nmlx5_core_en_main.c:4976) called

2022-02-18T01:53:51.528Z cpu2:1049177)<NMLX_INF> nmlx5_core: nmlx5_en_CheckHCACaps - (nmlx5_core_en_main.c:2169) wqeInlineMode = 2

2022-02-18T01:53:51.528Z cpu2:1049177)<NMLX_INF> nmlx5_core: nmlx5_en_CheckHCACaps - (nmlx5_core_en_main.c:2177) etsSupported = 1 pfcSupported = 1

2022-02-18T01:53:51.528Z cpu2:1049177)<NMLX_INF> nmlx5_core: nmlx5_en_CheckHCACaps - (nmlx5_core_en_main.c:2183) HCA_CAP.dcbx = 0

2022-02-18T01:53:51.528Z cpu2:1049177)<NMLX_INF> nmlx5_core: nmlx5_en_CheckHCACaps - (nmlx5_core_en_main.c:2189) vxlanSupported = 1 swpSupported = 1

2022-02-18T01:53:51.528Z cpu2:1049177)<NMLX_INF> nmlx5_core: nmlx5_en_CheckHCACaps - (nmlx5_core_en_main.c:2195) geneveSupported = 1 geneveSwLimitOptLenEn = 0

2022-02-18T01:53:51.529Z cpu2:1049177)<NMLX_INF> nmlx5_core: 0000:17:00.0: nmlx5_en_Attach - (nmlx5_core_en_main.c:5059) Device 0x15fa4305a5a071b5 is not configured as ENS uplink, mode: ENS disabled

2022-02-18T01:53:51.529Z cpu2:1049177)<NMLX_DBG> nmlx5_core: 0000:17:00.0: nmlx5_en_Attach - (nmlx5_core_en_main.c:5079) vmk_EnsIsDeviceConfigured query failed for device 0x15fa4305a5a071b5

2022-02-18T01:53:51.529Z cpu2:1049177)<NMLX_INF> nmlx5_core: 0000:17:00.0: nmlx5_en_Attach - (nmlx5_core_en_main.c:5088) Device 0x15fa4305a5a071b5 is not configured as ENS uplink

2022-02-18T01:53:51.529Z cpu2:1049177)<NMLX_INF> nmlx5_core: nmlx5_en_DCBInitCurrMode - (nmlx5_core_en_dcb.c:812) No HW DCBX Capability!

2

It worked after reducing NUM_OF_VFS from 8 to 2. VF_LOG_BAR_SIZE is already set to 0, so I don’t think I can lower it further. I suppose the limitation comes from the chipset I’m using. Is there a way to decrease the BAR size of VFs in the BIOS?

3

Hello Marcelo,

Thank you for posting your inquiry on the NVIDIA Networking Community.

Based on the information provided, and several similar tickets we had in the past for this particular message, it is system platform related. We recommend to update the BIOS of your platform to the latest version available and reach out to your system vendor and inquire regarding to the limitation for SR-IOV on your platform.

Be aware, that when configuring SR-IOV in ESXi, you need to configure the VF you want to use on the OS + 1 on the f/w of the adapter.

So if you want to use 16 VFs in ESXi, the NUM_OF_VFS should be 17.

Thank you and regards,

~NVIDIA Networking Technical Support