Non-Standard ConnectX-6 Dx Firmware

[shepard.siegel]

We purchased some ConnectX-6 Dx NICs (CX623106A) from a vendor on eBay. They look legit and appear to operate as the dual 100 GbE NICs they should be. But when we went to update the firmware, the firmware would not update as the cards have a non-standard PSID of CIS0000000008.

Is there a way to flash a standard NVIDIA firmware over this load?

Here is what we see currently:

shep@ar-9901k:~$ sudo flint -d /dev/mst/mt4125_pciconf0 query
Image type:            FS4
FW Version:            22.30.1004
FW Release Date:       29.3.2021
Product Version:       22.30.1004
Rom Info:              type=UEFI version=14.23.17 cpu=AMD64,AARCH64
                       type=PXE version=3.6.301 cpu=AMD64
Description:           UID                GuidsNumber
Base GUID:             1070fd030058XXXX        4
Base MAC:              1070fd58XXXX            4
Image VSD:             N/A
Device VSD:            N/A
PSID:                  CIS0000000008
Security Attributes:   secure-fw

Thanks in advance for any guidance.
Shep

[dwaxman]

These seem like Cisco/MLNX branded cards.

Upgrade versions should arrive from Cisco.

I think their latest release for this card is under:

https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/release/firmware_files/4_2/b_ucs_c-series-firmware-files_4_2/m_firmware_files_4_2_3.html

Cisco-MLNX MCX623106AS-CDAT 2x100GbE QSFP56 PCIe NIC

[shepard.siegel]

Thank you for that @dwaxman ! I would have guessed CIS ~= “cisco”, but wouldn’t have found the link you provided.

I grabbed the iso from the Cisco site and blew out the file system to see what is in the /firmware/Common/Mellanox_Technologies directories. I found three ConnectX6Dx firmwares as such:

-rw-r--r-- 1 shep shep 3578512 Jun  4 10:38 fw-ConnectX6Dx-rel-22_36_1010-30-100256-01_Ax-UEFI-14.29.14-FlexBoot-3.6.901.signed.bin
-rw-r--r-- 1 shep shep 3578496 Jun  4 10:38 fw-ConnectX6Dx-rel-22_36_1010-30-100260-01_Ax-UEFI-14.29.14-FlexBoot-3.6.901.signed.bin
-rw-r--r-- 1 shep shep 3579024 Jun  4 10:38 fw-ConnectX6Dx-rel-22_36_1010-30-100309-01_Ax-UEFI-14.29.14-FlexBoot-3.6.901.signed.bin

But when I try flint on any of these, I get a “Invalid Image signature” error.
Drats! I really dont want to recreate the whole Cisco update universe.

These boards seem to work fine; just a little uncomfortable not having the mainline Nvidia firmware (or even the latest Cisco-branded fw). Morale of the story - stick with Nvidia NICs bought as Nvidia NICs; as even a legit-looking board from eBay may have other firmware.

[dwaxman]

It looks like a secured FW image. So I’m not sure it’ll allow you to burn any other image onto the card’s flash.

[nikola.borisof]

Hello, I have the same card also from ebay. I’m unable to make it work at 100Gb. It is able to connect at 40Gbps and the strangest thing is the switch shows the port as 100Gb.
lshw shows this:

product: MT2892 Family [ConnectX-6 Dx]
       vendor: Mellanox Technologies
       capacity: 40Gbit/s
       capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=23.04-1.1.3 duplex=full firmware=22.30.1004 (CIS0000000008) ip=x.x.x. latency=0 link=yes multicast=yes

From ethtool i can see it also shows speed 40Gbps.
I installed the latest driver 23.04-1.1.3 Any advice on why the card is not able to do 100G?

[chris520]

Try flashing to the NVIDIA OEM firmware for the MCX623106AS.

I was able to flash the eBay CIS0000000008 cards to MT_00000000437 by going into “livefish”
(flash firmware recovery) mode.

I did the following using the open source flint tools:

1. use a thin wire to jumper JP7 holes together, but don’t short out to any other points on the board
2. power up the machine
3. you’ll know the jumper is seen by the card because an orange LED will stay lit
4. get the latest MCX623106AS-CDAT firmware. At this time, the file I got from NVIDIA was fw-ConnectX6Dx-rel-22_39_2048-MCX623106AS-CDA_Ax-UEFI-14.32.17-FlexBoot-3.7.300.signed.bin
5. figure out the pci address for the card, mine was pci0:1:0:0
6. mstflint -d pci0:1:0:0 --psid MT_0000000437 -i fw-ConnectX6Dx-rel-22_39_2048-MCX623106AS-CDA_Ax-UEFI-14.32.17-FlexBoot-3.7.300.signed.bin -ocr --nofs --allow_psid_change --yes burn
7. when finished, turn off the system and remove the jumper wire

now when you boot up, you’ll have the latest NVIDIA signed firmware. the unsigned firmware for MCX623106AN does not work on this card, you won’t be able to boot the card firmware. the crypto firmware for MCX623106AC is reported to work on these cards. (reports from Flashing the Cisco CX623106A NIC to OEM firmware | ServeTheHome Forums)

[chris520]

As I’m sitting here flashing these eBay cards, I’m finding the latest NVIDIA mft version of ‘flint’ works consistently, unlike the mstflint command from the open-source package. The arguments are identical.

Download the mft package from NVIDIA for your Linux/FreeBSD/whatever and use that. Also, one way to find your PCI device string (to use in the -d flag) is to use the ‘mst status’ command (from the NVIDIA mft.)

The FreeBSD version of NVIDIA mft requires that you install bash (pkg install bash) and ln -s /usr/local/bin/bash /bin/bash (or fix the scripts themselves.) The scripts are quite sloppy in this way, since FreeBSD does not typically install bash as /bin/bash. It would be nice if someone at NVIDIA would fix them.