BlueField-3 (DPU mode) stuck in "FW pre-initializing" — no host netdevs; is NIC-mode switch or BFB install the right fix?

Infrastructure & Networking BlueField
1

Hardware: NVIDIA BlueField-3 B3210 P-Series FHHL, 100GbE, PN 900-9D3B6-00CC-AA0, PSID MT_0000001024, firmware 32.41.0084.
Host: Ubuntu 24.04 (kernel 7.0.0-28-generic), DOCA-OFED 3.4.1 installed cleanly, Secure Boot disabled. PCIe link is Gen5 x16 (504 Gb/s), card enumerates on the bus.

Symptom: On boot, mlx5_core loads but never binds the ports:

mlx5_core 0000:01:00.0: firmware version: 32.41.84
mlx5_core 0000:01:00.0: Waiting for FW pre-initializing, timeout abort in 100s
mlx5_core 0000:01:00.0: Firmware over 120000 MS in pre-initializing state, aborting
mlx5_core 0000:01:00.0: mlx5_init_one failed with error code -110

Both PFs (01:00.0 and 01:00.1) time out the same way. ibv_devices is empty and no 100GbE host netdevs appear. tmfifo_net0 is present and rshim attaches successfully (“rshim0 attached”).

mlxconfig query shows the card is in DPU mode:

INTERNAL_CPU_MODEL EMBEDDED_CPU(1)
INTERNAL_CPU_RSHIM ENABLED(0)
LINK_TYPE_P1/P2 ETH(2)

My understanding is that in DPU (embedded-CPU) mode the host PFs only initialize after the Arm-side OS boots and releases the ports, and that this “FW pre-initializing” timeout is consistent with there being no BlueField OS installed on the Arm side yet.

Questions:

  1. Is that interpretation correct — is the host-side “FW pre-initializing” timeout expected when the DPU has no OS booted in embedded-CPU mode?
  2. To bring the ports up, is the right path (a) installing a BlueField bundle / BFB via rshim to boot the Arm side, or (b) switching the card to NIC/separated-host mode via mlxconfig so the ports present directly to the host? What are the trade-offs?
  3. For firmware 32.41.0084 / PSID MT_0000001024 on this card, which BFB / BlueField bundle version is the correct/validated one to use?

I want to confirm the correct and safe procedure before taking any irreversible step.

Thank you.

Vinayak V

2

Resolved. Two independent faults, and the more important one wasn’t the software.

To answer my own question in the title: neither the NIC-mode switch nor the BFB install was “the right fix” on its own. Both were needed, and a third thing I hadn’t considered turned out to matter most.

1. No auxiliary power (the primary cause). The B3210 halts in the bootloader with:

ERR[BL31]: ATX missing - initiating halt sequence…

ASSERT[BL31]: CRITICAL ERROR: ATX power not detected! Halting system!!

The card draws up to 150 W and the PCIe slot supplies roughly 66 W, so the 8-pin PCIe auxiliary power cable is required to activate the card — and it does not ship in the box. Without it the Arm cores never reach the OS, so there are no host netdevs no matter how the card is configured. Use a PCIe auxiliary (GPU-style) rail. Not a CPU/EPS12V cable — the pinout differs and it can damage the card. It should seat without force.

This is worth checking first, because: it produces exactly the same -110/ FW pre-initializing symptom as a mode misconfiguration, and it stays invisible during an rshim BFB install — that path never reaches the power-capping check, so the install completes successfully and the card still won’t boot.

I spent a long time on (2) below because the install kept succeeding.

2. Obsolete DPU NIC mode. The card was in the BlueField-2 style DPU NIC mode (INTERNAL_CPU_OFFLOAD_ENGINE = DISABLED(1)), which BlueField-3 firmware obsoleted in v32.38.1002. Its next-boot configuration also contradicted itself: the ownership parameters had reverted to `ECPF(0)` while the offload engine stayed disabled, so the firmware waited indefinitely on an Arm side configured never to boot.

sudo mst start

sudo mlxconfig -d /dev/mst/mt41692_pciconf0 s INTERNAL_CPU_OFFLOAD_ENGINE=0

Then a full AC power cycle — a warm reboot is not enough. Verify that Default, Current and Next Boot all agree before going further:

sudo mlxconfig -d /dev/mst/mt41692_pciconf0 -e q | grep -i cpu

A trap for anyone searching:

INTERNAL_CPU_MODEL reads EMBEDDED_CPU(1) in both modes on BlueField-3 and will mislead you into thinking you are already in DPU mode. INTERNAL_CPU_OFFLOAD_ENGINE is the field that tells you which mode you are actually in.

## Confirmation

After the mode fix, BF-Bundle 3.4.0-92, and the aux cable connected:

BF_MODE DPU mode

UP_TIME 1668(s)

Firmware 32.49.1014, `DDR POST passed`, Ubuntu running from eMMC.

On the host:

mlx5_0, mlx5_1 in ibv_devices

enp1s0f0np0, enp1s0f1np1 UP host netdevs

On the Arm:

p0, p1 uplinks

pf0hpf, pf1hpf host PF representors

ovsbr1, ovsbr2 default OVS bridges

One last note in case it saves someone a diagnostic detour: on the host, `enp1s0f0np0` shows `Link detected: yes` with `Speed: Unknown!`. That is normal in DPU mode — the host PF is a virtual link into the embedded switch, not a physical port, so its carrier says nothing about whether anything is plugged into the QSFP cages. `ethtool` on the host cannot tell you: the `p0`/`p1` netdevs exist only on the Arm. From the host, query the firmware directly instead:

sudo mlxlink -d /dev/mst/mt41692_pciconf0 -p 1 # p0

sudo mlxlink -d /dev/mst/mt41692_pciconf0 -p 2 -m # p1, with module info

Thanks to everyone who read the original post. It collected a fair number of views and no replies, which I take to mean the symptom is genuinely hard to diagnose from the outside — `-110` on both PFs looks identical whether the cause is configuration or power, and nothing in the logs points at the power rail until the card gets far enough to check it. So I am writing up the whole thing rather than just closing the thread, in the hope it saves the next person the detour. Happy to answer questions if anyone is stuck on the same symptom.