BlueField-3 stuck at PSC BL1 START / HW watchdog disabled, RShim BFB transfer stalls after a few KiB, flint reports DRAM not available

Infrastructure & Networking BlueField
1

Hi,

I am trying to recover a BlueField-3 DPU. My current suspicion is that some early firmware component may be corrupted or in an inconsistent state, but I would like to understand what the most likely root cause is and what the supported recovery path should be.

The most important detail is that I tried the BMC golden image reprovisioning script, but it did not actually start the recovery flow. It aborted at the precondition check because the PCIe slot was not in reset:

root@dpu-bmc:~# sh -x /usr/bin/dpu_golden_image_reprovisioning.sh 15 60 2 2>&1 | tee /tmp/golden-run.log
...
[2025-05-28-06:09:40 GOLDEN-IMAGE-RECOVERY] Checking pcie slot is in reset
+ gpiofind perst
+ gpioget gpiochip0 138
perst_val=1
[2025-05-28-06:09:40 GOLDEN-IMAGE-RECOVERY] PCIE is not in reset, try again in 10 seconds
...
+ gpioget gpiochip0 138
perst_val=1
[2025-05-28-06:09:50 GOLDEN-IMAGE-RECOVERY] ERROR: aborting process! PCIE is not in reset.
exit 1

So the script did not proceed to:

Read golden images from flash
Set FNP to 0
Starting ATF/UEFI golden image update
Starting NIC FW golden image update
Booting BFB from network

It looks like the script expects the PERST# GPIO to read 0, but my board returns 1. I am not sure what the correct platform procedure is to put the PCIe slot into reset while keeping the BMC powered.

Device / environment:

Device: NVIDIA BlueField-3 DPU
Architecture / ASIC: BF3 / mt41692
Host OS: Linux
BMC: BlueField BMC with local RShim exposed as /dev/rshim0
RShim paths tested: Host PCIe RShim and BMC USB RShim
OPN / image string appears to include: 900-9D3B6
Current firmware on flash: 32.47.1088
Firmware image I tried to burn: 32.47.2682
Firmware image filename: fw-BlueField-3-rel-32_47_2682-900-9D3B6-O0SV-A_Ax.bin
MST device used: /dev/mst/mt41692_pciconf0

The host can enumerate the device on PCIe:

01:00.0 [15b3:a2dc]
01:00.1 [15b3:a2dc]
01:00.2 [15b3:c2d5]

In one boot log the PCIe link came up as Gen4 x16:

252.048 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x16 link
capable of 504.112 Gb/s with 32.0 GT/s PCIe x16 link

However, the host mlx5 driver cannot initialize the device. Both PFs stay in firmware pre-initializing state and eventually time out:

mlx5_core 0000:01:00.0: firmware version: 32.47.1088
mlx5_core 0000:01:00.0: wait_fw_init: Waiting for FW pre-initializing
mlx5_core 0000:01:00.0: Firmware over 120000 MS in pre-initializing state, aborting
mlx5_core 0000:01:00.0: mlx5_init_one failed with error code -110
mlx5_core 0000:01:00.0: probe with driver mlx5_core failed with error -110

The same happens for 01:00.1.

RShim can attach, but BFB installation over RShim does not progress correctly. When I write a BFB to /dev/rshim0/boot, the boot node only accepts a very small amount of data, roughly a few KiB, and then the transfer stalls until timeout.

The RShim journal usually shows only:

rshim0 boot open
rshim0 boot timeout
rshim0 boot close

or sometimes:

rshim0 boot open
boot write, waiting for re-probe
USB device leaving
USB partially disconnected
USB device detected
rshim0 attached

but the BFB stream still does not continue.

There are no later BL2, UEFI, or Linux messages.

When trying to boot the ARM OS normally, the console appears to stop after:

HW watchdog disabled

and there is no further progress.

I also tried using flint from the host. flint can read the current and new firmware version, but burning fails:

Current FW version on flash: 32.47.1088
New FW version: 32.47.2682
​
-W- BME is not set, DMA access is not supported, please make sure mst driver and mlx5 driver are loaded
-E- Burning FS4 image failed: DRAM not available

Questions:

  1. Do these symptoms indicate early firmware corruption or an inconsistent firmware bundle, for example PSC / BL1 / ATF / UEFI / NIC firmware mismatch? Is there any other problems on my DPU? What should I do to solve it?

  2. Does PSC BL1 START with no lat er BL2/UEFI output mean the DPU fails before the normal RShim BFB boot reader is available?

  3. Is flint expected to work in this state, or does DRAM not available mean the device is too early in initialization for normal FS4 burning?

  4. Is there a supported recovery mode for this exact state when:

    • ARM OS does not boot,

    • mlx5_core times out in firmware pre-initializing,

    • RShim BFB transfer only accepts a few KiB and then stalls,

    • flint fails with DRAM not available,

    • BMC golden image reprovisioning aborts because PCIe PERST# is not asserted?

  5. Should I try a specific flint mode such as -ocr / --no_fw_ctrl, or should I avoid further flint attempts and use only BMC golden image / factory recovery?

  6. Could a PCIe riser / signal integrity / PERST# sideband issue cause this state, or do the logs more strongly suggest a firmware boot-chain problem?

Any guidance on the correct recovery path would be appreciated.

2

Hi,

Thanks for your question.

In order to assist you, a new support case should be opened, and it will be handled according to the entitlement.

To Create a New Enterprise Support Case Please Use One of the Following Methods

  1. Log in and Create a Case in the Enterprise Support Portal
    AppHub UI
    This is recommended and will provide the fastest response

  2. Call your Regional Enterprise Support Number
    Find your regional phone numbers:
    NVIDIA Enterprise Customer Support

  3. Create a Case Without a Login
    EnterpriseSupport
    Can be used to check warranty status
    Recommended to reset login passwords
    Requires validation and may increase time to respond

Best Regards,

Anatoly