ConnectX-8: Inconsistent visibility of NUM_OF_PLANES_P1 across identical setups

[gauravg]

I’m working with NVIDIA ConnectX-8 adapters and observing inconsistent visiblity of the NUM_OF_PLANES_P1 configuraiton across devices that otherwise appear identical.

Setup:

1. Adapter: NVIDIA ConnectX-8 (900-9X81E-00EX-ST0_Ax)
2. Firmware version: 40.46.3048
3. PSID: MT_0000001167
4. Secure FW enabled
5. OS: Linux

Observation:

I’m using the following command to list available configuration parameters:

$ sudo mlxconfig -d /dev/mst/mt4131_pciconfX q

On one of the CX8s I was able to see:

NUM_OF_PLANES_P1    1

However, on another CX8 in the same system (same FW version & PSID), this parameter was not visible.

Impact:

By default enumerates as 4 separate PCIe devices (x4 each), which I’m assuming is because the CX8 is configured in 4x200G links mode by default. I want to work in single x16 device and hence the requirement to change the NUM_OF_PLANES_P1 configuration variable to 1.

Logs:

mlxconfig dump where NUM_OF_PLANES_P1 is visible:

$ sudo flint -d /dev/mst/mt4131_pciconf0 q full
Image type:            FS5
FW Version:            40.46.3048
FW Release Date:       31.8.2025
Part Number:           900-9X81E-00EX-ST0_Ax
Description:           NVIDIA ConnectX-8 C8180L HHHL SuperNIC; 800Gbs XDR IB (default mode) / 2x400GbE; Single-cage OSFP; PCIe 6 x16 with x16 PCIe Socket Direct Extension option; Crypto Enabled; Secure Boot Enabled
Product Version:       40.46.3048
Rom Info:              type=UEFI version=14.39.14 cpu=AMD64,AARCH64
                       type=PXE version=3.8.100 cpu=AMD64
Base GUID:             605e65030098afca        32
Base MAC:              605e6598afca            32
System GUID:           605e65030098afca
Node GUID:             605e65030098afca
Port GUID:             605e65030098afca
Allocated GUID:        0000000000000000
Image VSD:             N/A
Device VSD:            N/A
PSID:                  MT_0000001167
Security Attributes:   secure-fw
Default Update Method: fw_ctrl
Life cycle:            GA SECURED
Secure Boot Capable:   Enabled
EFUSE Security Ver:    0
Image Security Ver:    0
Security Ver Program:  Manually ; Disabled
Encryption:            Enabled
PCIe switch mode only: Disabled

$ sudo mlxconfig -d /dev/mst/mt4131_pciconf0 q | grep PLANES
        NUM_OF_PLANES_P1    1

mlxconfig dump where NUM_OF_PLANES_P1 is not visible:

$ sudo flint -d /dev/mst/mt4131_pciconf0 q full
Image type:            FS4
FW Version:            40.46.3048
FW Release Date:       31.8.2025
Part Number:           900-9X81E-00EX-ST0_Ax
Description:           NVIDIA ConnectX-8 C8180L HHHL SuperNIC; 800Gbs XDR IB (default mode) / 2x400GbE; Single-cage OSFP; PCIe 6 x16 with x16 PCIe Socket Direct Extension option; Crypto Enabled; Secure Boot Enabled
Product Version:       40.46.3048
Rom Info:              type=UEFI version=14.39.14 cpu=AMD64,AARCH64
                       type=PXE version=3.8.100 cpu=AMD64
Base GUID:             605e65030098b58a        32
Base MAC:              605e6598b58a            32
System GUID:           605e65030098b58a
Node GUID:             605e65030098b58a
Port GUID:             605e65030098b58a
Allocated GUID:        0000000000000000
Image VSD:             N/A
Device VSD:            N/A
PSID:                  MT_0000001167
Security Attributes:   secure-fw
Default Update Method: fw_ctrl
Life cycle:            GA SECURED
Secure Boot Capable:   Enabled
Encryption:            Enabled

$ sudo mlxconfig -d /dev/mst/mt4131_pciconf0 q | grep PLANES
# (no output)

The only difference in the above log I see is Image Type. But I’m not sure if that will create an issue with the visiblity of a configuration.

Questions:

1. What conditions determine whether NUM_OF_PLANES_P1 is exposed via mlxconfig?
2. What is the supported and persistent method to configure ConnectX-8 to enumerate as a single x16 PCIe endpoint?
3. Are there recommended firmware images, modes, or procedures to ensure this configuration remains stable across reboots?

[hyungkwangc]

Hi

It’s interesting. Based on your triage, it looks like related with Image type: FS4 vs FS5.

1.Can you try with the same Image type and check?

2.Could you please move it to other good working server and check?

3.Server DOCA/OFED verion is the same? as good working VS not working server’s ?

If it’s still not resolved, please open a case with sysinfo-snapshot.

/HyungKwang

[gauravg]

Hi,

Thanks for your response. Find my answers below:

1. I’m not sure if there’s any way I can force the Image type to be FS4 or FS5. If there’s any way you know, please let me know. But I’m seeing that the NIC where NUM_OF_PLANES_P1 was listed, the image type for that is FS4 as well now.

2. I’ve connected 2 NICs on the same server, and one of them shows that config but the other doesn’t. Find the lspci dump where one NIC gets listed as single x16 device, and the other as 4 x4 devices:

   $ lspci | grep Mellanox
   ba:00.0 Ethernet controller: Mellanox Technologies CX8 Family [ConnectX-8]
   bb:00.0 Ethernet controller: Mellanox Technologies CX8 Family [ConnectX-8]
   bb:00.1 Ethernet controller: Mellanox Technologies CX8 Family [ConnectX-8]
   bb:00.2 Ethernet controller: Mellanox Technologies CX8 Family [ConnectX-8]
   bb:00.3 Ethernet controller: Mellanox Technologies CX8 Family [ConnectX-8]
   

• ba:00.0 is single x16 device. ba:00.[X] are 4 x4 devices.

3. The version is same, since both the NICs are being used on the same server.

I’m attaching the configuration dump for the non-working NIC.

Thanks

mlx_config_dump_non_working_cx8.txt (19.3 KB)

[gauravg]

Hi,

I updated my MFT tool version from 4.29.0 to 4.34.1 and I could see the NUM_OF_PLANES_P1 configuration visible in the configuration dump.

Earlier I had this:

Firmware Version : 40.46.3048 amBER Version : 3.3 MFT Version : mft 4.29.0-131

After updating:

Firmware Version : 40.46.3048 amBER Version : 5.75 MFT Version : mft 4.34.1-10

Above information is dumped using mlxlink command.

Can you let me know how MFT (or amBER) version is related to the visibility of the parameter?

P.S:

I checked with MFT 4.33.0 version as well (which is currently archived), and it seemed to work too.

Firmware Version : 40.46.3048 amBER Version : 5.75 MFT Version : mft 4.33.0-3002