Unable to set Mellanox ConnectX-3 to Ethernet (Failed to query device current configuration)

I have three Mellanox ConnectX-3 cards, that I’m trying to setup with Proxmox (Proxmox Installer does not see Mellanox ConnectX-3 card at all? | Proxmox Support Forum https://forum.proxmox.com/threads/proxmox-installer-does-not-see-mellanox-connectx-3-card-at-all.48722/#post-228189 )

I need to change them from Infiniband mode to Ethernet mode.

I was able to install the Mellanox Management Tools, and they can see my card:

root@gcc-proxmox:~/mft-4.10.0-104-x86_64-deb# mst start Starting MST (Mellanox Software Tools) driver set Loading MST PCI module - Success Loading MST PCI configuration module - Success Create devices root@gcc-proxmox:~/mft-4.10.0-104-x86_64-deb# mst status MST modules: ------------ MST PCI module loaded MST PCI configuration module loaded MST devices: ------------ /dev/mst/mt4099_pciconf0 - PCI configuration cycles access. domain:bus:dev.fn=0000:41:00.0 addr.reg=88 data.reg=92 Chip revision is: 01 /dev/mst/mt4099_pci_cr0 - PCI direct access. domain:bus:dev.fn=0000:41:00.0 bar=0xd4f00000 size=0x100000 Chip revision is: 01

However, when I tried to query the current config, it complains about the firmware version being too old.

root@gcc-proxmox:~/mft-4.10.0-104-x86_64-deb# mlxconfig -d /dev/mst/mt4099_pciconf0 q -E- Failed to open device: /dev/mst/mt4099_pciconf0. Unsupported FW (version 2.31.5000 or above required for CX3/PRO)

So I updated the firmware:

root@gcc-proxmox:~# flint -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX311A-XCA_Ax-FlexBoot-3.4.752.bin burn Current FW version on flash: 2.10.4290 New FW version: 2.42.5000 Burn process will not be failsafe. No checks will be performed. ALL flash, including the Invariant Sector will be overwritten. If this process fails, computer may remain in an inoperable state. Do you want to continue ? (y/n) [n] : y Burning FS2 FW image without signatures - OK Restoring signature - OK

But now when I try to read the config - I get a new error:

root@gcc-proxmox:~# mlxconfig -d /dev/mst/mt4099_pciconf0 q Device #1: ---------- Device type: ConnectX3 Device: /dev/mst/mt4099_pciconf0 Configurations: Next Boot -E- Failed to query device current configuration

Any ideas what’s going on, or how to get these cards working in permanent Ethernet mode?

I have the same issue that mlxconfig cannot read the current device configuration ( Iwant to set sr-iov, my cards are already in ethernet mode). I also noticed that the bios screen cannot save the configuration for SR-IOV claiming “access denied”. Any help would be much appreciated

/Louis

Hi Victor,

  • Did you reset the configuration by running:

#mlxconfig -d /dev/mst/mt4099_pciconf0 reset

#reboot

#flint -d /dev/mst/mt4099_pciconf0 q

#mlxconfig -d /dev/mst/mt4099_pciconf0 q

Note: If this still does not work, you can reflash the FW:

#flint -d /dev/mst/mt4099_pciconf0 -i b

#reboot

  • Regarding the port type change (permanent) to ethernet , use the following:

mlxconfig -d /dev/mst/mt4115_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

Note : (2) ETH (1) IB

Thanks,

Samer