DGX Spark unexpected hard reset + GPU PCIe Gen1 x1 / root port Width x0 + mlx5 insufficient power + FieldDiag MODS-020000600139

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Hello,

We have been investigating unexpected hard resets on a DGX Spark system over the last few weeks and would appreciate guidance from the platform team.

System:

  • NVIDIA DGX Spark / GB10
  • DGX OS 7.5.0
  • Kernel: 6.17.0-1014-nvidia
  • Driver: 580.142
  • CUDA: 13.0
  • VBIOS: 9A.0B.1E.00.00

Main issue:
The system experienced an unexpected hard reset / unclean reboot without a normal shutdown sequence and without a clear OS-level software cause.

What we observed:

  • No clean shutdown in logs
  • No kernel panic
  • No OOM event near the crash
  • last -x marks the session as crash
  • System logs stop abruptly

Persistent platform symptoms (present before and after updates):

  1. Mellanox warnings on every boot:
    mlx5_core ... Detected insufficient power on the PCIe slot (27W)
    This appears on all 4 mlx5 devices.

  2. GPU PCIe link anomaly:
    Root port 000f:00:00.0:

    • LnkCap: Speed 32GT/s, Width x16
    • LnkSta: Speed unknown, Width x0

    GPU 000f:01:00.0:

    • LnkSta: Speed 2.5GT/s, Width x1 (downgraded)

  3. nvidia-smi -q reports:

    • PCIe Generation Current: 1
    • Link Width Current: 1x
    • Device Max: 5
    • Host Max: 5

  4. dmesg reports:
    pci 000f:01:00.0: 0.000 Gb/s available PCIe bandwidth, limited by Unknown x0 link at 000f:00:00.0

  5. We also see correctable PCIe/AER events such as:

    • PCIe Bus Error: severity=Correctable, type=Physical Layer
    • RxErr

Field Diagnostics:
Field Diagnostics fails with:

  • MODS-020000600139
  • Acceptable temperature limits exceeded or the thermal sensor is broken or miscalibrated

Additional context:

  • The machine can still appear functional in some workloads
  • nvidia-smi -q shows:
    • GPU C2C Mode: Enabled
    • Peer Type: Direct Connected

Because of that, we are unsure whether the PCIe state is:

  • expected / cosmetic due to the Blackwell C2C architecture,
  • or an actual hardware / firmware fault related to the instability.

What we already tried:

  • Updated firmware / kernel / driver stack
  • Power cycle
  • Rechecked logs and PCIe state
  • Symptoms persist after update

Questions:

  1. Is it expected on DGX Spark / GB10 for the GPU to show Gen1 x1 while the root port shows Width x0 because of the C2C architecture?
  2. Are the mlx5_core ... insufficient power on the PCIe slot (27W) messages expected on this platform?
  3. Could FieldDiag error MODS-020000600139 indicate a real thermal / sensor / platform issue related to the hard resets?
  4. What is the correct way to access BMC / SEL logs on DGX Spark if /dev/ipmi* is not exposed on the host?
  5. Should this case be treated as a possible hardware issue / RMA candidate?

Relevant command output excerpts:

journalctl -k -b 0 --no-pager | grep -i 'mlx5_core.*insufficient power'

  • 4 warnings, one for each mlx5 device, on every boot

lspci -vv -s 000f:00:00.0

  • LnkSta: Speed unknown, Width x0

lspci -vv -s 000f:01:00.0

  • LnkSta: Speed 2.5GT/s, Width x1 (downgraded)

nvidia-smi -q | sed -n '/GPU Link Info/,/Bridge Chip/p'

  • Current: 1
  • Width: 1x
  • Device Max: 5
  • Host Max: 5

dmesg | grep -i pcie

  • 0.000 Gb/s available PCIe bandwidth, limited by Unknown x0 link

Any guidance would be greatly appreciated.

Thank you.

2

The Mellanox and PCIe messages you are seeing are all expected and don’t point to any issue. However, the field diag results do indicate a hardware issue. Please contact consumer support and request an RMA

3

Today I made the suggested OS updates and since that, my spark switch off randomly after some minutes when it calculates some ZIT pictures in ComfyUI.

I have also this issue with “mlx5_core 0000:01:00.0: mlx5_pcie_event:326:(pid 12): Detected insufficient power on the PCIe slot (27W).”

sudo dmesg | grep -i “shutdown|power|temperature|thermal”
…
[ 0.208324] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input0
[ 0.208366] ACPI: button: Power Button [PWRB]
[ 0.221113] thermal LNXTHERM:00: registered as thermal_zone0
[ 0.221116] ACPI: thermal: Thermal Zone [TSOC] (60 C)
[ 0.221235] thermal LNXTHERM:01: registered as thermal_zone1
[ 0.221238] ACPI: thermal: Thermal Zone [TS0E] (58 C)
[ 0.221350] thermal LNXTHERM:02: registered as thermal_zone2
[ 0.221352] ACPI: thermal: Thermal Zone [TS0P] (59 C)
[ 0.221462] thermal LNXTHERM:03: registered as thermal_zone3
[ 0.221464] ACPI: thermal: Thermal Zone [TS1E] (58 C)
[ 0.221578] thermal LNXTHERM:04: registered as thermal_zone4
[ 0.221580] ACPI: thermal: Thermal Zone [TS1P] (59 C)
[ 0.221689] thermal LNXTHERM:05: registered as thermal_zone5
[ 0.221691] ACPI: thermal: Thermal Zone [TGPU] (60 C)
[ 0.221801] thermal LNXTHERM:06: registered as thermal_zone6
[ 0.221803] ACPI: thermal: Thermal Zone [TUNC] (59 C)
[ 0.447747] PM: genpd: Disabling unused power domains
[ 1.580440] mlx5_core 0000:01:00.0: mlx5_pcie_event:326:(pid 12): Detected insufficient power on the PCIe slot (27W).
[ 2.141108] mlx5_core 0000:01:00.1: mlx5_pcie_event:326:(pid 163): Detected insufficient power on the PCIe slot (27W).
[ 2.690173] mlx5_core 0002:01:00.0: mlx5_pcie_event:326:(pid 388): Detected insufficient power on the PCIe slot (27W).
[ 3.332909] mlx5_core 0002:01:00.1: mlx5_pcie_event:326:(pid 372): Detected insufficient power on the PCIe slot (27W).

4

As I said, you do not have to worry about those messages. Please go to this thread for further explanation PCIe power related and PCIe AER errors
Please run the Spark FD and share your results Get the Right Support for Your DGX Spark — DGX Spark User Guide