BlueField-3 (900-9D3B6-00CN) boots to UEFI fine, but the ConnectX/NIC is dark on PCIe and the BMC sideband and rshim what's the recovery path?

Infrastructure & Networking BlueField
, ,
1

BF-3 B3240 · OPN 900-9D3B6-00CN-PA0 · S/N MT2515XZ06VE · BMC BF-24.10-17

TL;DR: The DPU’s Arm/UEFI firmware boots cleanly and repeatably (4.15.0) and the onboard BMC is fully responsive over Redfish + serial. But the ConnectX/NIC subsystem does not respond on any management interface: it is not enumerated on PCIe by any host (tested on two different servers), the BMC’s own NIC manager cannot reach it over NC-SI / MCTP / IPMB, and over rshim it reports OPN_STR N/A. The BMC’s Redfish inventory lists DPU_NIC as Enabled/OK but with an empty firmware version. I cannot reflash or reconfigure the NIC because (a) no host sees it to run mstflint/mlxconfig/mlxfwmanager, (b) this card has no USB rshim port, and (c) the NIC firmware for this specific SKU (00CN) is not distributed as a standalone flashable image. Full logs and everything I’ve already tried are below.

What works: Arm + UEFI boot cleanly

Captured on the BMC serial console (ssh -p 2200 / obmc-console) after a Redfish ComputerSystem.Reset. Every reset produces a full, clean boot to UEFI, then it idles at the boot manager (eMMC has no OS). So the Arm side, DDR, ATF, OP-TEE and UEFI are all healthy.

BMC serial console — cold boot to UEFI:

Nvidia BlueField-3 rev1 BL1 V1.0
INFO: psc supervisor init.
NOTICE:  Running as 9009D3B600CNAB system
NOTICE:  BL2: v2.2(release):4.15.0-4-g419fbf393
NOTICE:  BL2: Built : 20:31:39, May 14 2026
NOTICE:  Reset reason: cold
NOTICE:  DDR POST passed.
NOTICE:  BL31: v2.2(release):4.15.0-4-g419fbf393
I/TC: OP-TEE version: 4.1.0-35-ge909cab5e (gcc version 8.3.0 (GCC)) aarch64
I/TC: Primary CPU switching to normal world boot
UEFI firmware (version BlueField:4.15.0-19-g37c6f5adb2-BId13977 built at 20:31:59 on May 14 2026)

Note: an earlier symptom was CRITICAL ERROR: ATX power not detected! Halting system!! at UEFI — that was a missing 8-pin PCIe aux power connector and is resolved; aux power is connected and the card now boots as above. Boot firmware was also upgraded 4.13.0 → 4.15.0 by a bf-bundle boot-image capsule during a recovery attempt, which applied successfully.

The fault: the NIC is unresponsive on every channel

1. rshim — the NIC does not identify itself

Read from the BMC. Note OPN_STR N/A, the NIC is not reporting its part number/identity. (BF_MODE is DPU mode; the rshim is only present during early SoC boot and drops once UEFI initializes.)

BF_MODE         DPU mode
BOOT_MODE       1 (0:rshim, 1:emmc, 2:emmc-boot-swap)
DROP_MODE       0 (0:normal, 1:drop)
DEV_INFO        BlueField-3(Rev 1)
OPN_STR         N/A
SECURE_NIC_MODE 0 (0:no, 1:yes)

2. BMC sideband — NC-SI / MCTP / IPMB to the NIC all fail

The BMC’s own “DPU Nic Manager” service cannot reach the NIC. This repeats continuously in the BMC journal:

BMC journalctl (dpu-bmc):

bf-dpu-eth[1081]: Error in sendRequest to ipmbRequestData
systemd[1]: nic-mctp-ctl.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: nic-mctp-ctl.service: Failed with result 'exit-code'.
host_state_monitor[1434]: Error in ncsi command to NIC, Retrying
systemd[1]: xyz.openbmc_project.Settings.connectx.service: Control process exited, code=exited, status=1/FAILURE
systemd[1]: xyz.openbmc_project.Settings.connectx.service: Failed with result 'exit-code'.
systemd[1]: Failed to start DPU Nic Manager.

3. Redfish FirmwareInventory — NIC version cannot be read

All DPU components exist and report State=Enabled/Health=OK, but their Version fields are empty, the BMC cannot read the NIC/DPU firmware. Only BMC_Firmware and CEC report versions.

Redfish firmware versions (via OobUpdate --show_all_versions):

       BMC :  BF-24.10-17
       CEC :  00.02.0195.0000_n02
       ATF :  (empty)
     BOARD :  (empty)
       BSP :  (empty)
       NIC :  (empty)
        OS :  (empty)
      UEFI :  (empty)
CONF_IMAGE :  version not available

FirmwareInventory members (note golden_image_nic is present):

BMC_Firmware   Bluefield_FW_ERoT   DPU_ATF   DPU_BOARD   DPU_BSP
DPU_NIC        DPU_NODE            DPU_OFED  DPU_OS      DPU_SYS_IMAGE
DPU_UEFI       golden_image_arm    golden_image_config  golden_image_nic

DPU_NIC -> { "Version": "",  "Status": { "State": "Enabled", "Health": "OK" } }

4. Host PCIe — no host enumerates the card

With the card seated and powered (8-pin PCIe aux connected), no host sees it on the bus. Tested on two different machines (an AMD/AM5 workstation and a Supermicro server); lspci shows zero devices with vendor ID 15b3, even after a full host reboot and a rescan. No /dev/rshim ever appears on the host.

$ lspci -nn | grep -i 15b3
(no output)

$ echo 1 | sudo tee /sys/bus/pci/rescan && lspci -nn | grep -i 15b3
(still no output — 15b3 device count: 0)

5. Redfish Oem/Nvidia Mode.Set returns 500

The BMC exposes a NIC/DPU mode switch, but it fails because the underlying NIC manager (above) can’t reach the NIC. A BMC reboot (Manager.Reset GracefulRestart) does not change this.

POST /redfish/v1/Systems/Bluefield/Oem/Nvidia/Actions/Mode.Set   {"Mode":"NicMode"}
-> HTTP 500
   "code": "Base.1.18.1.InternalError",
   "message": "The request failed due to an internal service error. The service is still operational."

What I’ve already tried (and why each is blocked)

  • OS install over the BMC’s remote rshim (bf-bundle 3.4.0, Ubuntu 24.04) — both a raw cat <bfb> | ssh root@bmc "cat > /dev/rshim0/boot" and NVIDIA’s own bfb-install (remote scp mode) freeze at exactly 15,073,280 bytes. The bundle’s boot-image capsule warm-resets the SoC mid-stream, and a remote rshim can’t re-feed the boot stream across that reset (the bf-reg register step isn’t available remotely). A local rshim would be required — which brings us back to the fact that no host sees the card.
  • BMC firmware push — bf-fwbundle 3.4.0 contains NIC firmware .pldm for 00CV / 00SV / 9D3C6-00CV / 9D3C6-00SV / SN37B36732 / 8217991, but not 900-9D3B6-00CN. This SKU’s NIC firmware appears only embedded inside mlxfwmanager, which flashes by running on the DPU during a completed install — not as a BMC-pushable image. Pushing the wrong-SKU .pldm returns No Matching Devices.
  • Direct NIC tools (mstflint / mlxconfig / mlxfwmanager / mlxfwreset) — not reachable: no host enumerates the card on PCIe, and this card has no USB rshim port, so there is no path to the NIC.
  • BMC reboot (Manager.Reset GracefulRestart) — completes cleanly, does not restore NIC communication.

Questions

  1. What is the supported way to recover/reflash the ConnectX NIC firmware when the NIC is unresponsive on PCIe, NC-SI, MCTP and rshim? In particular — can golden_image_nic (visible in Redfish FirmwareInventory) be triggered from the BMC to restore the NIC, and if so, what is the exact Redfish/ipmitool call?
  2. Is the standalone NIC firmware for OPN 900-9D3B6-00CN-PA0 available (and what is its PSID)? Or is it only ever applied on-target via mlxfwmanager during an install?
  3. When driving a bf-bundle install over a remote (BMC) rshim, the boot-image capsule resets the SoC mid-stream and the push freezes at 15,073,280 bytes. Is there a supported way to complete an install/flash over a remote/BMC rshim across that capsule reset (a BOOT_RESET_SKIP / bf-reg equivalent exposed on the BMC side)?
  4. Is “Arm+UEFI boot normally, NIC dark on every interface, firmware versions unreadable” a known/recoverable failure mode — or does it indicate a hardware fault that should be RMA’d?