ConnectX-3 EN Failed to query device current configuration

Infrastructure & Networking Adapters and Cables
1

There have been multiple reports of mlxconfig/mstconfig failing to query the configuration on ConnectX-3 cards. I myself have two MCX311A-XCAT cards flashed with FW 2.42.5000 / FlexBoot 3.4.752 that I cannot enable SR-IOV.

I have tried reflashing the FW, I tried FW 2.40.7000, I tried removing the FlexBoot rom, I tried resetting the configuration through mlxconfig/mstconfig. I also reflashed one card in livefish mode.

I have tried with MFT/mstflint versions 4.11, 4.8, 4.7 and MFT 4.5 with OFED and EN driver versions 4.5-1.0.1.0, 4.1-1.0.2 and 3.4-1.0.0.

I have tried with Centos 7.6, Fedora 29, Fedora 28, Ubuntu 16.04 and Windows 10 1803 and 1809 versions.

Other reports:

Infrastructure & Networking - NVIDIA Developer Forums Yes I tried the solution mentioned here.

I cannot set SR-IOV in FlexBoot either as it spits out an ipxe permission denied error with an http link that is dead. This is also mentioned in one of the community threads above.

These are $200+ cards that supposedly support SR-IOV but we cannot enable it because your tools are not working or your firmware is corrupted. Please help.

2

Hello Sotir -

Could you run the sysinfo-snaphot utility and attach the results?

Also, you could open a case with Mellanox Support: Contact NVIDIA Networks | NVIDIA

Many thanks -

~Steve

3

Hello Stephen, sorry for my quite abrupt post up there. I just have been working on this non-stop for the last three days to no avail.

I did email your support team but since I don’t have a support plan, I did not look into it further. Case number assigned is # 00569187.

I attach the sysinfo tarball. This is from one of many machines I used in the process.

Thank you for your reply.

4

Hello Sotir -

  • Have you swapped a know good NIC into this server? Does it work?
  • Are all your ConnectX-3 EN NICs exhibiting the same issue? Or, just this one?
  • Did you reboot the server after you updated the FW?
  • Was the NIC working before the upgrade?
  • Did it ever work for this NIC? All NICs?
  • Does this bad NIC work in another server?

Thanks - steve

5

Hello Stephen,

Both of my two NICs that show this behaviour. Unfortunately I don’t have any other NICs to test, just these two.

The NICs work fine for accessing our network, our File shares etc etc. I had not tried to this point to change the configuration, a recent change in the setup led me to try and enable SR-IOV on one card and since that was not working I tried the second card but that did not work either. The cards are in separate servers, I just swapped them to try the other one. The servers do not have any other Mellanox or other party networking cards in them, just the onboard Ethernet controllers.

I did reboot the server after upgrading the FW. My procedure was as such:

  1. Try to query the configuration with mlxconfig/mstconfig either using mst and /dev/mst/mt4099_pci_cr0 or _pciconf0 or through the pcie address ( 03:00.0, 000:03:00.0) etc and have that fail.
  2. Flash the FW, either 2.42.5000 or 2.40.7000 with MFT (flint) or mstflint. I tried flashing through /dev/mst/mt4099_pci_cr0, /dev/mst/mt4099_pciconf0 or pci hardware address (e.g. 03:00.0). Process completes without errors, I also checksum the image on the flash with the image on the card to make sure.
  3. Reboot server and again try to query the config as step 1 and have it fail again.
  4. Reboot and reset the config with mlxconfig or mstconfig. Again I tried each time resetting either only pci_cr0, only pciconf0 or only 03:00.0 and as a last resort I tried resetting them all at the same time before rebooting.
  5. Reboot, run checksum again on the image on the server and on the image on the card, same checksums.
  6. Try to query the configuration, end up with the same error: -E- Failed to query device current configuration.

Again, the NICs work fine and perform as expected for browsing the network, Samba and CIFS file sharing, I just can’t access the configuration on the cards to enable SR-IOV and I have confirmed that it is disabled.

6

Hello Sotir -

Lets try this first:

Could you install the NIC in a PCIe 3.0 slot or enable PCIe 3.0 in the BIOS for the slot

  • Desired PCIe Generation: 3

  • Current PCIe Generation: 2

Thank you -

~Steve

7

OK, I moved the Centos drive and the card to another system, here is the sysinfo. I still do have the same problem.

8

Hello Sotir

This is an ASRock Mother Board and unfortunately, Mellanox does not provide custom built images.

Thank you -

~Steve

9

For anyone looking to enable SR-IOV for the ConnectX-3 and cannot query the config: