Unable to set Mellanox ConnectX-3 to Ethernet (Failed to query device current configuration)

Infrastructure & Networking Adapters and Cables Ethernet Adapter Cards
, ,
1

I have three Mellanox ConnectX-3 cards, that I’m trying to setup with Proxmox (Proxmox Installer does not see Mellanox ConnectX-3 card at all? | Proxmox Support Forum Proxmox Installer does not see Mellanox ConnectX-3 card at all? | Proxmox Support Forum )

I need to change them from Infiniband mode to Ethernet mode.

I was able to install the Mellanox Management Tools, and they can see my card:

root@gcc-proxmox:~/mft-4.10.0-104-x86_64-deb# mst start Starting MST (Mellanox Software Tools) driver set Loading MST PCI module - Success Loading MST PCI configuration module - Success Create devices root@gcc-proxmox:~/mft-4.10.0-104-x86_64-deb# mst status MST modules: ------------ MST PCI module loaded MST PCI configuration module loaded MST devices: ------------ /dev/mst/mt4099_pciconf0 - PCI configuration cycles access. domain:bus:dev.fn=0000:41:00.0 addr.reg=88 data.reg=92 Chip revision is: 01 /dev/mst/mt4099_pci_cr0 - PCI direct access. domain:bus:dev.fn=0000:41:00.0 bar=0xd4f00000 size=0x100000 Chip revision is: 01

However, when I tried to query the current config, it complains about the firmware version being too old.

root@gcc-proxmox:~/mft-4.10.0-104-x86_64-deb# mlxconfig -d /dev/mst/mt4099_pciconf0 q -E- Failed to open device: /dev/mst/mt4099_pciconf0. Unsupported FW (version 2.31.5000 or above required for CX3/PRO)

So I updated the firmware:

root@gcc-proxmox:~# flint -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX311A-XCA_Ax-FlexBoot-3.4.752.bin burn Current FW version on flash: 2.10.4290 New FW version: 2.42.5000 Burn process will not be failsafe. No checks will be performed. ALL flash, including the Invariant Sector will be overwritten. If this process fails, computer may remain in an inoperable state. Do you want to continue ? (y/n) [n] : y Burning FS2 FW image without signatures - OK Restoring signature - OK

But now when I try to read the config - I get a new error:

root@gcc-proxmox:~# mlxconfig -d /dev/mst/mt4099_pciconf0 q Device #1: ---------- Device type: ConnectX3 Device: /dev/mst/mt4099_pciconf0 Configurations: Next Boot -E- Failed to query device current configuration

Any ideas what’s going on, or how to get these cards working in permanent Ethernet mode?

2

I have the same issue that mlxconfig cannot read the current device configuration ( Iwant to set sr-iov, my cards are already in ethernet mode). I also noticed that the bios screen cannot save the configuration for SR-IOV claiming “access denied”. Any help would be much appreciated

/Louis

3

Hi Victor,

  • Did you reset the configuration by running:

#mlxconfig -d /dev/mst/mt4099_pciconf0 reset

#reboot

#flint -d /dev/mst/mt4099_pciconf0 q

#mlxconfig -d /dev/mst/mt4099_pciconf0 q

Note: If this still does not work, you can reflash the FW:

#flint -d /dev/mst/mt4099_pciconf0 -i b

#reboot

  • Regarding the port type change (permanent) to ethernet , use the following:

mlxconfig -d /dev/mst/mt4115_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

Note : (2) ETH (1) IB

Thanks,

Samer

4

I’ve got same issue.
reflash twice with flint, latest stock FW from NVIDIA website, and still same issue.

mst start

mst status → device not found

mst start --with_unknown

mst status → mst device ok.

…

mlxconfig -d /dev/mst/mt4099_pciconf0 reset
Are you sure? Y

  • E - Unsupported device

root@test01:~# flint -d /dev/mst/mt4099_pciconf0 q
Image type: FS2
FW Version: 2.42.5000
FW Release Date: 5.9.2017
Product Version: 02.42.50.00
Rom Info: type=PXE version=3.4.752
Device ID: 4099
Description: Node Port1 Port2 Sys image
GUIDs: 506b4b03005edf70 506b4b03005edf71 506b4b03005edf72 506b4b03005edf73
MACs: 506b4b5edf71 506b4b5edf72
VSD:
PSID: MT_1090110018

Debian12

Any ideas?

Thanks

5

Try to use port_type_array parameter for mlx4_core module ( applicable for ConnectX-3 device only)
parm: port_type_array:Array of port types: HW_DEFAULT (0) is default 1 for IB, 2 for Ethernet (array of int)