AER Storm on GX10

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

# ASUS Ascent GX10 (DGX Spark): repeated PCIe Surprise Down on both ConnectX-7 root ports → mlx5 recovery hang → hard reset (one unit only)

## Setup

- 3× ASUS Ascent GX10 (GB10). Two at a time run one vLLM tensor-parallel (TP=2) model over RoCEv2, using both ConnectX-7 PCIe functions (`rocep1s0f0` + `roceP2p1s0f0`), direct QSFP DAC, 200G.

- Ubuntu 24.04 / DGX OS, kernel **6.17.0-1031-nvidia** (pinned — see note 2 below), NVIDIA driver **580.173.02**

- BIOS `GX10DGX.0105.2026.0505.1153`, ConnectX-7 fw **28.45.4028** (PSID `NVD0000000087`, board “NVIDIA DGX Spark P4242”)

- EC `0x02000006`, SoC fw `0x03000007`, USB-C PD fw `0x516` on all units

- `iommu.passthrough=1` (see note 1)

## The problem

**Unit A** (always the same chassis) has had **11 events in 5 days**. **Units B and C**, on the same cable and workload, have **never logged a PCIe error**.

Signature, the same every time:

```

pcieport 0002:00:00.0: AER: Correctable error … [ 0] RxErr (First)

pcieport 0000:00:00.0: AER: Uncorrectable (Fatal) … Transaction Layer

pcieport 0000:00:00.0: device [10de:22ce] error status/mask=00000020/04400000

pcieport 0000:00:00.0: [ 5] SDES (First)

pcieport 0002:00:00.0: AER: Uncorrectable (Fatal) … [ 5] SDES

mlx5_core 0000:01:00.0: mlx5_pci_err_detected … pci channel state = 2

mlx5_core 0000:01:00.0: mlx5_error_sw_reset: PCI channel offline, stop waiting for NIC IFC

mlx5_core 0002:01:00.1: poll_health: Fatal error 1 detected / PCI slot is unavailable

mlx5_core 0000:01:00.0 enp1s0f0np0: NETDEV WATCHDOG … transmit queue timed out

watchdog: BUG: soft lockup - CPU#3 stuck for 130s! [irq/329-aerdrv]

… pci_find_ext_capability ← pci_save_state ← mlx5_pci_slot_reset ← report_slot_reset ← pcie_do_recovery ← aer_isr

```

- **Both** CX7 root ports report Surprise Down within ~50 ms. The two x4 links are separate, so the whole NIC is dropping off the bus.

- The AER recovery thread then spins on config reads to the missing device. After that come hundreds of soft lockups, and the box stays hung (up to 95 min) until it’s power-cycled.

- The peer node only logs QSFP `Link down` at the same instant. It has no PCIe or mlx5 errors.

- No AER has ever come from the NVMe, the 10G Realtek or the Wi-Fi domains.

**Event times (CDT):**

| Date | Times |

|—|—|

| Sep 21 | 18:25 |

| Sep 22 | 01:22, 02:22, 07:15, 08:20, 09:23, 10:51, 12:40, 15:44, 19:57 (TX timeout only) |

| Sep 25 | 13:57 |

**Observations:**

- **Startup timing:** 10 of 11 came within ~5 min of the vLLM container starting (NCCL/RDMA setup), often during restart loops. But Sep 25 hit while the engine was idle, blocked at rendezvous with essentially no NIC traffic.

- **Cabling:** every event was on a **direct DAC** link. Nothing happened in ~25 h through a switch, though that was light load.

- **Things that didn’t stop it:**

  • IOMMU translated or passthrough

  • PD firmware mismatched (`0x1` on the peer) or matched (`0x516`)

  • Different peer node

- **Cable ruled out as the failing part:** QSFP PHY counters (CRC/symbol/FEC/discards) are all 0, and the failure is on the PCIe side.

- **Slot power:** every unit logs `mlx5_pcie_event: Detected insufficient power on the PCIe slot (27W)` on every boot. `DISABLE_SLOT_POWER_LIMITER` is not supported on this firmware.

**Firmware error records (BERT)** appear after each reset. MediaTek vendor section `3c1e3f4b-1e1a-43df-af28-59820e958e3c` (“MTKID”), severity corrected. There are two variants:

```

…a451e0a4 c2964450 0ae9a1c7 0000fa95 00 a0 … 80 (after most storm hangs)

…a451e0a4 c2964450 0ae9a1c7 0080fa95 00 20 … 40 (other resets)

```

Linux can’t decode these.

**Unexplained reset:** Unit A also reset once while idle and healthy (Sep 25 04:06). There was nothing in the kernel log, no pstore dump and no button press. The peer saw the link down for 46 s, versus 12–15 s for a normal reboot.

**Current test:** Unit B has been replaced by Unit C on the same cable. It has been clean for ~21 h so far, but we’ve seen 66 h quiet stretches before.

## Questions

1. Is Surprise Down on both CX7 root ports (`10de:22ce`) a known issue on GX10 / DGX Spark, and is there a firmware or BIOS fix?

2. Can someone decode the MTKID BERT records above? We’d like to know whether they indicate a power or reset cause.

3. Is this RMA territory for Unit A? We can supply full logs, pstore dumps and BERT dumps.

4. The AER recovery path hangs the whole system instead of failing the NIC. Is that expected with mlx5 on this platform?

## Side findings that may help others

1. **`iommu.passthrough=0` (the stock `/etc/default/grub.d/iommu.cfg`) caps ConnectX-7 DMA at ~13 Gb/s per PCIe function.**

  • This showed up in loopback, cross-node, RDMA read/write/send and TCP.

  • The SMMU was not faulting.

  • With `iommu.passthrough=1` we get **~112 Gb/s per function and ~196 Gb/s across both** (`ib_write_bw`).

  • For TP=2 LLM serving this cut prefill time-to-first-token ~27% (32K: 23.5 s → 17.1 s).

  • Is the translated-mode cap expected?

2. **Kernel 7.0.0-1019-nvidia:** `ibv_reg_mr` on CUDA pinned host memory fails with ENOMEM ~40% of the time (4/10 runs, sometimes at the very first 4 MB MR). This breaks NCCL. 6.17.0-1031 had 0/10 failures.

3. **`dgx-spark-fieldiag` 2.0.4** depends on `ofed-scripts`, which isn’t in any DGX OS repo. It is in the DOCA host repo (`linux.mellanox.com/public/repo/doca/latest/ubuntu24.04/arm64-sbsa/`). Its signing key is the repo’s own `doca_keyring.gpg` (`DC726C5E41B9CC50`), not `GPG-KEY-Mellanox.pub`.

BERT RECORDS:
BERT (ACPI Boot Error Record Table) records — head gx10-3502 (ASUS GX10, BIOS GX10DGX.0105.2026.0505.1153)

Each record is reported by firmware at boot and describes an error from the PREVIOUS boot. Times US Central (CDT).

Generated 2026-09-25 21:40:21 CDT from the persistent systemd journal (kernel ring buffer at boot).

SUMMARY (head) — variant = differing bytes at payload offsets 0x2C-0x3D; firmware section GUID 3c1e3f4b-1e1a-43df-af28-59820e958e3c (MediaTek ‘MTKID’, not decoded by Linux)

Storm lines = SDES / mlx5_pci_err_detected / soft lockup / NETDEV WATCHDOG lines in the boot that ended.

reported at variant boot that ended (duration) storm lines note

Sep 20 19:15:29 A Sep 19 12:44–19:12 (1828 min) 0

Sep 21 16:26:27 A Sep 20 19:18–16:23 (1265 min) 0

Sep 22 02:19:27 B Sep 21 20:03–02:16 (373 min) 264

Sep 22 02:33:32 B Sep 22 02:21–02:30 (9 min) 43

Sep 22 07:07:39 A Sep 22 02:35–07:05 (269 min) 0

Sep 22 08:13:28 B Sep 22 07:11–08:09 (58 min) 249

Sep 22 08:27:40 A Sep 22 08:20–08:25 (5 min) 27

Sep 22 08:50:48 A Sep 22 08:41–08:47 (5 min) 0

Sep 22 09:27:12 A Sep 22 09:00–09:24 (24 min) 4

Sep 22 10:55:41 A Sep 22 09:36–10:53 (77 min) 5

Sep 22 11:03:32 A Sep 22 10:57–11:00 (2 min) 0

Sep 22 14:06:01 B Sep 22 11:25–14:03 (158 min) 386

Sep 22 16:54:10 B Sep 22 14:09–16:51 (162 min) 314

Sep 25 04:08:44 A Sep 23 23:22–04:06 (1723 min) 0 unexplained reset 04:06, lane loaded & idle, no pstore dump

Sep 25 14:03:01 B Sep 25 04:08–14:00 (592 min) 10

Worker gx10-efa8: one record only, reported Sep 11 22:19 CT (boot right after its fwupd SoC/EC firmware update at Sep 11 22:09 CT):

22:19:01.965229 gx10-efa8 kernel: ACPI: BERT 0x0000000087065D98 000030 (v01 MTKID MTKTABLE 00000001 CREA 00000001)

22:19:02.004372 gx10-efa8 kernel: BERT: Error records from previous boot:

22:19:02.004382 gx10-efa8 kernel: [Hardware Error]: It has been corrected by h/w and requires no further action

22:19:02.004393 gx10-efa8 kernel: [Hardware Error]: event severity: corrected

22:19:02.004402 gx10-efa8 kernel: [Hardware Error]: Error 0, type: corrected

22:19:02.004412 gx10-efa8 kernel: [Hardware Error]: section type: unknown, 3c1e3f4b-1e1a-43df-af28-59820e958e3c

22:19:02.004421 gx10-efa8 kernel: [Hardware Error]: section length: 0x3e

22:19:02.004430 gx10-efa8 kernel: [Hardware Error]: 00000000: 000d0000 544d0000 0044494b 00000000 …MTKID…

22:19:02.004440 gx10-efa8 kernel: [Hardware Error]: 00000010: 00000000 00000010 00000022 56190000 …"…V

22:19:02.004450 gx10-efa8 kernel: [Hardware Error]: 00000020: a451e0a4 c2964450 0ae9a1c7 0100fa95 ..Q.PD…

22:19:02.004459 gx10-efa8 kernel: [Hardware Error]: 00000030: 00 80 00 00 00 00 00 00 00 00 00 00 00 80 …

22:19:02.004468 gx10-efa8 kernel: BERT: Total records found: 1

(variant C: …0100fa95 00 80 … 80)

Observation: A and B do not separate storm crashes from non-storm resets — both follow storms. Decoding needs ASUS/MediaTek.

================================================================================

Reported at boot -64 starting Sep 20 19:15:29 | previous boot’s last journal entry: Sep 20 19:12:50 | variant A (…0080fa95 00 20 … 40)

================================================================================

19:15:29.398979 gx10-3502 kernel: ACPI: BERT 0x0000000087065D18 000030 (v01 MTKID MTKTABLE 00000001 CREA 00000001)

19:15:29.436040 gx10-3502 kernel: BERT: Error records from previous boot:

19:15:29.436050 gx10-3502 kernel: [Hardware Error]: It has been corrected by h/w and requires no further action

19:15:29.436060 gx10-3502 kernel: [Hardware Error]: event severity: corrected

19:15:29.436070 gx10-3502 kernel: [Hardware Error]: Error 0, type: corrected

19:15:29.436081 gx10-3502 kernel: [Hardware Error]: section type: unknown, 3c1e3f4b-1e1a-43df-af28-59820e958e3c

19:15:29.436091 gx10-3502 kernel: [Hardware Error]: section length: 0x3e

19:15:29.436103 gx10-3502 kernel: [Hardware Error]: 00000000: 000d0000 544d0000 0044494b 00000000 …MTKID…

19:15:29.436115 gx10-3502 kernel: [Hardware Error]: 00000010: 00000000 00000010 00000022 56190000 …"…V

19:15:29.436125 gx10-3502 kernel: [Hardware Error]: 00000020: a451e0a4 c2964450 0ae9a1c7 0080fa95 ..Q.PD…

19:15:29.436144 gx10-3502 kernel: [Hardware Error]: 00000030: 00 20 00 00 00 00 00 00 00 00 00 00 00 40 . …@

19:15:29.436153 gx10-3502 kernel: BERT: Total records found: 1

================================================================================

Reported at boot -61 starting Sep 21 16:26:27 | previous boot’s last journal entry: Sep 21 16:23:51 | variant A (…0080fa95 00 20 … 40)

================================================================================

16:26:27.719713 gx10-3502 kernel: ACPI: BERT 0x0000000087065D18 000030 (v01 MTKID MTKTABLE 00000001 CREA 00000001)

16:26:27.759364 gx10-3502 kernel: BERT: Error records from previous boot:

16:26:27.759373 gx10-3502 kernel: [Hardware Error]: It has been corrected by h/w and requires no further action

16:26:27.759383 gx10-3502 kernel: [Hardware Error]: event severity: corrected

16:26:27.759393 gx10-3502 kernel: [Hardware Error]: Error 0, type: corrected

16:26:27.759402 gx10-3502 kernel: [Hardware Error]: section type: unknown, 3c1e3f4b-1e1a-43df-af28-59820e958e3c

16:26:27.759412 gx10-3502 kernel: [Hardware Error]: section length: 0x3e

16:26:27.759422 gx10-3502 kernel: [Hardware Error]: 00000000: 000d0000 544d0000 0044494b 00000000 …MTKID…

16:26:27.759433 gx10-3502 kernel: [Hardware Error]: 00000010: 00000000 00000010 00000022 56190000 …"…V

16:26:27.759442 gx10-3502 kernel: [Hardware Error]: 00000020: a451e0a4 c2964450 0ae9a1c7 0080fa95 ..Q.PD…

16:26:27.759453 gx10-3502 kernel: [Hardware Error]: 00000030: 00 20 00 00 00 00 00 00 00 00 00 00 00 40 . …@

16:26:27.759462 gx10-3502 kernel: BERT: Total records found: 1

================================================================================

Reported at boot -55 starting Sep 22 02:19:27 | previous boot’s last journal entry: Sep 22 02:16:44 | variant B (…0000fa95 00 a0 … 80)

================================================================================

02:19:27.111556 gx10-3502 kernel: ACPI: BERT 0x0000000087065D18 000030 (v01 MTKID MTKTABLE 00000001 CREA 00000001)

02:19:27.153013 gx10-3502 kernel: BERT: Error records from previous boot:

02:19:27.153022 gx10-3502 kernel: [Hardware Error]: It has been corrected by h/w and requires no further action

02:19:27.153032 gx10-3502 kernel: [Hardware Error]: event severity: corrected

02:19:27.153042 gx10-3502 kernel: [Hardware Error]: Error 0, type: corrected

02:19:27.153052 gx10-3502 kernel: [Hardware Error]: section type: unknown, 3c1e3f4b-1e1a-43df-af28-59820e958e3c

02:19:27.153062 gx10-3502 kernel: [Hardware Error]: section length: 0x3e

02:19:27.153073 gx10-3502 kernel: [Hardware Error]: 00000000: 000d0000 544d0000 0044494b 00000000 …MTKID…

02:19:27.153085 gx10-3502 kernel: [Hardware Error]: 00000010: 00000000 00000010 00000022 56190000 …"…V

02:19:27.153095 gx10-3502 kernel: [Hardware Error]: 00000020: a451e0a4 c2964450 0ae9a1c7 0000fa95 ..Q.PD…

02:19:27.153105 gx10-3502 kernel: [Hardware Error]: 00000030: 00 a0 00 00 00 00 00 00 00 00 00 00 00 80 …

02:19:27.153114 gx10-3502 kernel: BERT: Total records found: 1

================================================================================

Reported at boot -53 starting Sep 22 02:33:32 | previous boot’s last journal entry: Sep 22 02:30:52 | variant B (…0000fa95 00 a0 … 80)

================================================================================

02:33:32.894030 gx10-3502 kernel: ACPI: BERT 0x0000000087065D18 000030 (v01 MTKID MTKTABLE 00000001 CREA 00000001)

02:33:32.933068 gx10-3502 kernel: BERT: Error records from previous boot:

02:33:32.933078 gx10-3502 kernel: [Hardware Error]: It has been corrected by h/w and requires no further action

02:33:32.933087 gx10-3502 kernel: [Hardware Error]: event severity: corrected

02:33:32.933097 gx10-3502 kernel: [Hardware Error]: Error 0, type: corrected

02:33:32.933106 gx10-3502 kernel: [Hardware Error]: section type: unknown, 3c1e3f4b-1e1a-43df-af28-59820e958e3c

02:33:32.933116 gx10-3502 kernel: [Hardware Error]: section length: 0x3e

02:33:32.933126 gx10-3502 kernel: [Hardware Error]: 00000000: 000d0000 544d0000 0044494b 00000000 …MTKID…

02:33:32.933137 gx10-3502 kernel: [Hardware Error]: 00000010: 00000000 00000010 00000022 56190000 …"…V

02:33:32.933147 gx10-3502 kernel: [Hardware Error]: 00000020: a451e0a4 c2964450 0ae9a1c7 0000fa95 ..Q.PD…

02:33:32.933157 gx10-3502 kernel: [Hardware Error]: 00000030: 00 a0 00 00 00 00 00 00 00 00 00 00 00 80 …

02:33:32.933167 gx10-3502 kernel: BERT: Total records found: 1

================================================================================

Reported at boot -51 starting Sep 22 07:07:39 | previous boot’s last journal entry: Sep 22 07:05:01 | variant A (…0080fa95 00 20 … 40)

================================================================================

07:07:39.876143 gx10-3502 kernel: ACPI: BERT 0x0000000087065D18 000030 (v01 MTKID MTKTABLE 00000001 CREA 00000001)

07:07:39.915864 gx10-3502 kernel: BERT: Error records from previous boot:

07:07:39.915874 gx10-3502 kernel: [Hardware Error]: It has been corrected by h/w and requires no further action

07:07:39.915884 gx10-3502 kernel: [Hardware Error]: event severity: corrected

07:07:39.915894 gx10-3502 kernel: [Hardware Error]: Error 0, type: corrected

07:07:39.915904 gx10-3502 kernel: [Hardware Error]: section type: unknown, 3c1e3f4b-1e1a-43df-af28-59820e958e3c

07:07:39.915914 gx10-3502 kernel: [Hardware Error]: section length: 0x3e

07:07:39.915924 gx10-3502 kernel: [Hardware Error]: 00000000: 000d0000 544d0000 0044494b 00000000 …MTKID…

07:07:39.915935 gx10-3502 kernel: [Hardware Error]: 00000010: 00000000 00000010 00000022 56190000 …"…V

07:07:39.915945 gx10-3502 kernel: [Hardware Error]: 00000020: a451e0a4 c2964450 0ae9a1c7 0080fa95 ..Q.PD…

07:07:39.915955 gx10-3502 kernel: [Hardware Error]: 00000030: 00 20 00 00 00 00 00 00 00 00 00 00 00 40 . …@

07:07:39.915964 gx10-3502 kernel: BERT: Total records found: 1

================================================================================

Reported at boot -49 starting Sep 22 08:13:28 | previous boot’s last journal entry: Sep 22 08:09:59 | variant B (…0000fa95 00 a0 … 80)

================================================================================

08:13:28.047227 gx10-3502 kernel: ACPI: BERT 0x0000000087065D18 000030 (v01 MTKID MTKTABLE 00000001 CREA 00000001)

08:13:28.086260 gx10-3502 kernel: BERT: Error records from previous boot:

08:13:28.086270 gx10-3502 kernel: [Hardware Error]: It has been corrected by h/w and requires no further action

08:13:28.086280 gx10-3502 kernel: [Hardware Error]: event severity: corrected

08:13:28.086289 gx10-3502 kernel: [Hardware Error]: Error 0, type: corrected

08:13:28.086299 gx10-3502 kernel: [Hardware Error]: section type: unknown, 3c1e3f4b-1e1a-43df-af28-59820e958e3c

08:13:28.086308 gx10-3502 kernel: [Hardware Error]: section length: 0x3e

08:13:28.086318 gx10-3502 kernel: [Hardware Error]: 00000000: 000d0000 544d0000 0044494b 00000000 …MTKID…

08:13:28.086329 gx10-3502 kernel: [Hardware Error]: 00000010: 00000000 00000010 00000022 56190000 …"…V

08:13:28.086339 gx10-3502 kernel: [Hardware Error]: 00000020: a451e0a4 c2964450 0ae9a1c7 0000fa95 ..Q.PD…

08:13:28.086350 gx10-3502 kernel: [Hardware Error]: 00000030: 00 a0 00 00 00 00 00 00 00 00 00 00 00 80 …

08:13:28.086360 gx10-3502 kernel: BERT: Total records found: 1

================================================================================

Reported at boot -46 starting Sep 22 08:27:40 | previous boot’s last journal entry: Sep 22 08:25:33 | variant A (…0080fa95 00 20 … 40)

================================================================================

08:27:40.146873 gx10-3502 kernel: ACPI: BERT 0x0000000087065D18 000030 (v01 MTKID MTKTABLE 00000001 CREA 00000001)

08:27:40.183091 gx10-3502 kernel: BERT: Error records from previous boot:

08:27:40.183101 gx10-3502 kernel: [Hardware Error]: It has been corrected by h/w and requires no further action

08:27:40.183110 gx10-3502 kernel: [Hardware Error]: event severity: corrected

08:27:40.183119 gx10-3502 kernel: [Hardware Error]: Error 0, type: corrected

08:27:40.183129 gx10-3502 kernel: [Hardware Error]: section type: unknown, 3c1e3f4b-1e1a-43df-af28-59820e958e3c

08:27:40.183138 gx10-3502 kernel: [Hardware Error]: section length: 0x3e

08:27:40.183148 gx10-3502 kernel: [Hardware Error]: 00000000: 000d0000 544d0000 0044494b 00000000 …MTKID…

08:27:40.183159 gx10-3502 kernel: [Hardware Error]: 00000010: 00000000 00000010 00000022 56190000 …"…V

08:27:40.183169 gx10-3502 kernel: [Hardware Error]: 00000020: a451e0a4 c2964450 0ae9a1c7 0080fa95 ..Q.PD…

08:27:40.183178 gx10-3502 kernel: [Hardware Error]: 00000030: 00 20 00 00 00 00 00 00 00 00 00 00 00 40 . …@

08:27:40.183188 gx10-3502 kernel: BERT: Total records found: 1

================================================================================

Reported at boot -37 starting Sep 22 08:50:48 | previous boot’s last journal entry: Sep 22 08:47:12 | variant A (…0080fa95 00 20 … 40)

================================================================================

08:50:48.066754 gx10-3502 kernel: ACPI: BERT 0x0000000087065D18 000030 (v01 MTKID MTKTABLE 00000001 CREA 00000001)

08:50:48.105649 gx10-3502 kernel: BERT: Error records from previous boot:

08:50:48.105658 gx10-3502 kernel: [Hardware Error]: It has been corrected by h/w and requires no further action

08:50:48.105668 gx10-3502 kernel: [Hardware Error]: event severity: corrected

08:50:48.105677 gx10-3502 kernel: [Hardware Error]: Error 0, type: corrected

08:50:48.105687 gx10-3502 kernel: [Hardware Error]: section type: unknown, 3c1e3f4b-1e1a-43df-af28-59820e958e3c

08:50:48.105696 gx10-3502 kernel: [Hardware Error]: section length: 0x3e

08:50:48.105706 gx10-3502 kernel: [Hardware Error]: 00000000: 000d0000 544d0000 0044494b 00000000 …MTKID…

08:50:48.105717 gx10-3502 kernel: [Hardware Error]: 00000010: 00000000 00000010 00000022 56190000 …"…V

08:50:48.105726 gx10-3502 kernel: [Hardware Error]: 00000020: a451e0a4 c2964450 0ae9a1c7 0080fa95 ..Q.PD…

08:50:48.105736 gx10-3502 kernel: [Hardware Error]: 00000030: 00 20 00 00 00 00 00 00 00 00 00 00 00 40 . …@

08:50:48.105745 gx10-3502 kernel: BERT: Total records found: 1

================================================================================

Reported at boot -31 starting Sep 22 09:27:12 | previous boot’s last journal entry: Sep 22 09:24:55 | variant A (…0080fa95 00 20 … 40)

================================================================================

09:27:12.354309 gx10-3502 kernel: ACPI: BERT 0x0000000087065D18 000030 (v01 MTKID MTKTABLE 00000001 CREA 00000001)

09:27:12.393463 gx10-3502 kernel: BERT: Error records from previous boot:

09:27:12.393473 gx10-3502 kernel: [Hardware Error]: It has been corrected by h/w and requires no further action

09:27:12.393483 gx10-3502 kernel: [Hardware Error]: event severity: corrected

09:27:12.393493 gx10-3502 kernel: [Hardware Error]: Error 0, type: corrected

09:27:12.393503 gx10-3502 kernel: [Hardware Error]: section type: unknown, 3c1e3f4b-1e1a-43df-af28-59820e958e3c

09:27:12.393512 gx10-3502 kernel: [Hardware Error]: section length: 0x3e

09:27:12.393522 gx10-3502 kernel: [Hardware Error]: 00000000: 000d0000 544d0000 0044494b 00000000 …MTKID…

09:27:12.393533 gx10-3502 kernel: [Hardware Error]: 00000010: 00000000 00000010 00000022 56190000 …"…V

09:27:12.393543 gx10-3502 kernel: [Hardware Error]: 00000020: a451e0a4 c2964450 0ae9a1c7 0080fa95 ..Q.PD…

09:27:12.393553 gx10-3502 kernel: [Hardware Error]: 00000030: 00 20 00 00 00 00 00 00 00 00 00 00 00 40 . …@

09:27:12.393563 gx10-3502 kernel: BERT: Total records found: 1

================================================================================

Reported at boot -25 starting Sep 22 10:55:41 | previous boot’s last journal entry: Sep 22 10:53:27 | variant A (…0080fa95 00 20 … 40)

================================================================================

10:55:41.225832 gx10-3502 kernel: ACPI: BERT 0x0000000087065D18 000030 (v01 MTKID MTKTABLE 00000001 CREA 00000001)

10:55:41.265113 gx10-3502 kernel: BERT: Error records from previous boot:

10:55:41.265123 gx10-3502 kernel: [Hardware Error]: It has been corrected by h/w and requires no further action

10:55:41.265133 gx10-3502 kernel: [Hardware Error]: event severity: corrected

10:55:41.265142 gx10-3502 kernel: [Hardware Error]: Error 0, type: corrected

10:55:41.265153 gx10-3502 kernel: [Hardware Error]: section type: unknown, 3c1e3f4b-1e1a-43df-af28-59820e958e3c

10:55:41.265164 gx10-3502 kernel: [Hardware Error]: section length: 0x3e

10:55:41.265174 gx10-3502 kernel: [Hardware Error]: 00000000: 000d0000 544d0000 0044494b 00000000 …MTKID…

10:55:41.265185 gx10-3502 kernel: [Hardware Error]: 00000010: 00000000 00000010 00000022 56190000 …"…V

10:55:41.265195 gx10-3502 kernel: [Hardware Error]: 00000020: a451e0a4 c2964450 0ae9a1c7 0080fa95 ..Q.PD…

10:55:41.265204 gx10-3502 kernel: [Hardware Error]: 00000030: 00 20 00 00 00 00 00 00 00 00 00 00 00 40 . …@

10:55:41.265214 gx10-3502 kernel: BERT: Total records found: 1

================================================================================

Reported at boot -23 starting Sep 22 11:03:32 | previous boot’s last journal entry: Sep 22 11:00:00 | variant A (…0080fa95 00 20 … 40)

================================================================================

11:03:32.036379 gx10-3502 kernel: ACPI: BERT 0x0000000087065D18 000030 (v01 MTKID MTKTABLE 00000001 CREA 00000001)

11:03:32.073010 gx10-3502 kernel: BERT: Error records from previous boot:

11:03:32.073019 gx10-3502 kernel: [Hardware Error]: It has been corrected by h/w and requires no further action

11:03:32.073029 gx10-3502 kernel: [Hardware Error]: event severity: corrected

11:03:32.073038 gx10-3502 kernel: [Hardware Error]: Error 0, type: corrected

11:03:32.073049 gx10-3502 kernel: [Hardware Error]: section type: unknown, 3c1e3f4b-1e1a-43df-af28-59820e958e3c

11:03:32.073058 gx10-3502 kernel: [Hardware Error]: section length: 0x3e

11:03:32.073068 gx10-3502 kernel: [Hardware Error]: 00000000: 000d0000 544d0000 0044494b 00000000 …MTKID…

11:03:32.073079 gx10-3502 kernel: [Hardware Error]: 00000010: 00000000 00000010 00000022 56190000 …"…V

11:03:32.073088 gx10-3502 kernel: [Hardware Error]: 00000020: a451e0a4 c2964450 0ae9a1c7 0080fa95 ..Q.PD…

11:03:32.073098 gx10-3502 kernel: [Hardware Error]: 00000030: 00 20 00 00 00 00 00 00 00 00 00 00 00 40 . …@

11:03:32.073107 gx10-3502 kernel: BERT: Total records found: 1

================================================================================

Reported at boot -11 starting Sep 22 14:06:01 | previous boot’s last journal entry: Sep 22 14:03:55 | variant B (…0000fa95 00 a0 … 80)

================================================================================

14:06:01.319239 gx10-3502 kernel: ACPI: BERT 0x0000000087065D18 000030 (v01 MTKID MTKTABLE 00000001 CREA 00000001)

14:06:01.359114 gx10-3502 kernel: BERT: Error records from previous boot:

14:06:01.359124 gx10-3502 kernel: [Hardware Error]: It has been corrected by h/w and requires no further action

14:06:01.359133 gx10-3502 kernel: [Hardware Error]: event severity: corrected

14:06:01.359142 gx10-3502 kernel: [Hardware Error]: Error 0, type: corrected

14:06:01.359152 gx10-3502 kernel: [Hardware Error]: section type: unknown, 3c1e3f4b-1e1a-43df-af28-59820e958e3c

14:06:01.359161 gx10-3502 kernel: [Hardware Error]: section length: 0x3e

14:06:01.359171 gx10-3502 kernel: [Hardware Error]: 00000000: 000d0000 544d0000 0044494b 00000000 …MTKID…

14:06:01.359182 gx10-3502 kernel: [Hardware Error]: 00000010: 00000000 00000010 00000022 56190000 …"…V

14:06:01.359192 gx10-3502 kernel: [Hardware Error]: 00000020: a451e0a4 c2964450 0ae9a1c7 0000fa95 ..Q.PD…

14:06:01.359201 gx10-3502 kernel: [Hardware Error]: 00000030: 00 a0 00 00 00 00 00 00 00 00 00 00 00 80 …

14:06:01.359211 gx10-3502 kernel: BERT: Total records found: 1

================================================================================

Reported at boot -9 starting Sep 22 16:54:10 | previous boot’s last journal entry: Sep 22 16:51:46 | variant B (…0000fa95 00 a0 … 80)

================================================================================

16:54:10.722777 gx10-3502 kernel: ACPI: BERT 0x0000000087065D18 000030 (v01 MTKID MTKTABLE 00000001 CREA 00000001)

16:54:10.762374 gx10-3502 kernel: BERT: Error records from previous boot:

16:54:10.762383 gx10-3502 kernel: [Hardware Error]: It has been corrected by h/w and requires no further action

16:54:10.762393 gx10-3502 kernel: [Hardware Error]: event severity: corrected

16:54:10.762403 gx10-3502 kernel: [Hardware Error]: Error 0, type: corrected

16:54:10.762412 gx10-3502 kernel: [Hardware Error]: section type: unknown, 3c1e3f4b-1e1a-43df-af28-59820e958e3c

16:54:10.762423 gx10-3502 kernel: [Hardware Error]: section length: 0x3e

16:54:10.762433 gx10-3502 kernel: [Hardware Error]: 00000000: 000d0000 544d0000 0044494b 00000000 …MTKID…

16:54:10.762444 gx10-3502 kernel: [Hardware Error]: 00000010: 00000000 00000010 00000022 56190000 …"…V

16:54:10.762453 gx10-3502 kernel: [Hardware Error]: 00000020: a451e0a4 c2964450 0ae9a1c7 0000fa95 ..Q.PD…

16:54:10.762463 gx10-3502 kernel: [Hardware Error]: 00000030: 00 a0 00 00 00 00 00 00 00 00 00 00 00 80 …

16:54:10.762473 gx10-3502 kernel: BERT: Total records found: 1

================================================================================

Reported at boot -2 starting Sep 25 04:08:44 | previous boot’s last journal entry: Sep 25 04:06:28 | variant A (…0080fa95 00 20 … 40)

================================================================================

04:08:44.686810 gx10-3502 kernel: ACPI: BERT 0x0000000087065D18 000030 (v01 MTKID MTKTABLE 00000001 CREA 00000001)

04:08:44.725306 gx10-3502 kernel: BERT: Error records from previous boot:

04:08:44.725316 gx10-3502 kernel: [Hardware Error]: It has been corrected by h/w and requires no further action

04:08:44.725326 gx10-3502 kernel: [Hardware Error]: event severity: corrected

04:08:44.725335 gx10-3502 kernel: [Hardware Error]: Error 0, type: corrected

04:08:44.725345 gx10-3502 kernel: [Hardware Error]: section type: unknown, 3c1e3f4b-1e1a-43df-af28-59820e958e3c

04:08:44.725355 gx10-3502 kernel: [Hardware Error]: section length: 0x3e

04:08:44.725365 gx10-3502 kernel: [Hardware Error]: 00000000: 000d0000 544d0000 0044494b 00000000 …MTKID…

04:08:44.725374 gx10-3502 kernel: [Hardware Error]: 00000010: 00000000 00000010 00000022 56190000 …"…V

04:08:44.725386 gx10-3502 kernel: [Hardware Error]: 00000020: a451e0a4 c2964450 0ae9a1c7 0080fa95 ..Q.PD…

04:08:44.725397 gx10-3502 kernel: [Hardware Error]: 00000030: 00 20 00 00 00 00 00 00 00 00 00 00 00 40 . …@

04:08:44.725407 gx10-3502 kernel: BERT: Total records found: 1

================================================================================

Reported at boot -1 starting Sep 25 14:03:01 | previous boot’s last journal entry: Sep 25 14:00:46 | variant B (…0000fa95 00 a0 … 80)

================================================================================

14:03:01.075382 gx10-3502 kernel: ACPI: BERT 0x0000000087065D18 000030 (v01 MTKID MTKTABLE 00000001 CREA 00000001)

14:03:01.111772 gx10-3502 kernel: BERT: Error records from previous boot:

14:03:01.111782 gx10-3502 kernel: [Hardware Error]: It has been corrected by h/w and requires no further action

14:03:01.111791 gx10-3502 kernel: [Hardware Error]: event severity: corrected

14:03:01.111801 gx10-3502 kernel: [Hardware Error]: Error 0, type: corrected

14:03:01.111811 gx10-3502 kernel: [Hardware Error]: section type: unknown, 3c1e3f4b-1e1a-43df-af28-59820e958e3c

14:03:01.111820 gx10-3502 kernel: [Hardware Error]: section length: 0x3e

14:03:01.111830 gx10-3502 kernel: [Hardware Error]: 00000000: 000d0000 544d0000 0044494b 00000000 …MTKID…

14:03:01.111839 gx10-3502 kernel: [Hardware Error]: 00000010: 00000000 00000010 00000022 56190000 …"…V

14:03:01.111851 gx10-3502 kernel: [Hardware Error]: 00000020: a451e0a4 c2964450 0ae9a1c7 0000fa95 ..Q.PD…

14:03:01.111860 gx10-3502 kernel: [Hardware Error]: 00000030: 00 a0 00 00 00 00 00 00 00 00 00 00 00 80 …

14:03:01.111870 gx10-3502 kernel: BERT: Total records found: 1

2

bert_records.txt (25.2 KB)

aer_storm_logs.txt (122.1 KB)

3

anyone ?

4

Same MTKID BERT record on HP ZGX (GB10) — 2 nodes, SGLang TP=2 over RoCE, one chassis silently hard-resets

Related data point from a different OEM. To be clear about what does and doesn’t match: I see the same firmware error record and the same one-chassis-out-of-identical-units pattern, but not the AER storm itself. My box never logs Surprise Down or mlx5 recovery and never hangs — it just resets with zero kernel trace, which corresponds to your unexplained Sep 25 04:06 event rather than your storm hangs. Posting it here because the record appears to be a generic firmware reset log whose byte variants may encode the cause, and a third variant from a second OEM should help whoever can decode it.

Setup

  • 2× HP ZGX Nano G1n (GB10), direct QSFP between the ConnectX-7 ports
  • SGLang, Qwen3.8-Flash-Next NVFP4, TP=2 over RoCEv2/NCCL, ~102 GB of 128 GB in use per node
  • Both nodes log insufficient power on the PCIe slot (27W) for all four mlx5 functions at boot
Node DGX OS Kernel BIOS Driver CX-7 fw IOMMU
hapy-1 7.2.3 6.17.0-1031-nvidia 5.36_0ACUM026 580.173.02 28.45.4028 iommu.passthrough=0
hapy-2 7.2.3 7.0.0-1019-nvidia 5.36_0ACUM028 580.178.04 28.45.4028 iommu.passthrough=0
hapy-1 (stable) hapy-2 (resets)
HP product no. CZ9K4ET D4CY3ET
HP BIOS 5.36_0ACUM026 5.36_0ACUM028
NVIDIA driver 580.173.02 580.178.04

The two units are different HP SKUs. hapy-2 has been on the newer BIOS/driver since day one and has failed since day one; hapy-1 never has. Same workload, container, cable and NCCL settings.

Symptom

hapy-2 hard-resets under TP=2 load; hapy-1 stays up and only sees the peer vanish. last -x shows reboot with no preceding shutdown every time. Seven events in three days (uptimes 21 min to ~2 h); survives 12 h idle.

How I looked

sudo dmesg -T | grep -iE "xid|oom|nvrm"
journalctl --list-boots
sudo journalctl -b -1 -k | grep -iE "xid|nvrm|panic|oops|thermal|watchdog|mlx5|cx7"
sudo journalctl -b -1 -n 80 --no-pager
last -x reboot shutdown | head
sudo ls -la /sys/fs/pstore/
sudo dmesg | grep -iE "BERT|GHES|APEI|hardware error|reset reason|SError"
sudo nvidia-smi -q -d ECC,ROW_REMAPPER,PAGE_RETIREMENT
sudo dmidecode -s bios-version; cat /proc/driver/nvidia/version
sudo mstflint -d 0000:01:00.0 q | head

GPU telemetry was recorded continuously with nvidia-smi dmon -s pucv -d 2 -o T to /var/log, with vm.dirty_expire_centisecs=100 vm.dirty_writeback_centisecs=100 so the file is current to ~1 s when the box dies.

Evidence

  • Kernel log of the dying boot: nothing. Last line is a routine earlyoom heartbeat (22 GB free, flat for 10+ min), then the new boot. No Xid, no AER, no mlx5 error, no oops, no thermal trip, no OOM. pstore empty. Hardware watchdog is disabled by the kernel.
  • GPU telemetry at the last reset: 32–33 W, 62–63 °C, sm 91–94 %, pclk 2489 MHz, pviol 0, tviol 0. The GPU was mostly waiting on NCCL.
  • BERT on the next boot — same GUID and payload prefix as yours, third byte variant:
ACPI: BERT 0x0000000087065D18 000030 (v01 MTKID  MTKTABLE 00000001 CREA 00000001)
[Hardware Error]: event severity: corrected
[Hardware Error]: section type: unknown, 3c1e3f4b-1e1a-43df-af28-59820e958e3c
[Hardware Error]: 00000000: 000d0000 544d0000 0044494b 00000000  ......MTKID.....
[Hardware Error]: 00000010: 00000000 00000020 00000022 56190000  .... ..."......V
[Hardware Error]: 00000020: a451e0a4 c2964450 0ae9a1c7 0000fa95  ..Q.PD..........
[Hardware Error]: 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00

vs. yours: offset 0x14 = 0x20 (yours 0x10), bytes 0x30/0x3D = 00/00 (yours a0/80 and 20/40). Same BERT table address across two OEMs.

  • ECC / row-remapper counters report N/A on GB10, so no memory comparison possible.

Ruled out

OOM, GPU power/thermal throttling, software (identical stack on the stable peer), driver revision alone (your Unit A and my stable node both run 580.173.02), OEM-specific firmware (ASUS and HP show the same record).

5

Thanks for the detailed report — the evidence points in a clear direction.

The core finding: Both CX7 root ports asserting SDES within ~50 ms of each other is the key. Those are independent x4 links, so a simultaneous drop points to something upstream of both — a momentary disturbance on the PCIe power rail or a PERST# event on the SoC — not a NIC firmware or link-layer issue. The insufficient power on PCIe slot (27W) message is present on all units at every boot and is not what’s differentiating Unit A.

Immediate ask: please run dgx-spark-fieldiag on Unit A, ideally immediately after the next event before rebooting. This captures the SoC/EC power-rail history and BERT dumps in a form the hardware team can analyze.

On your specific questions:

  1. Known issue / firmware fix? Not a known software-fixable defect. The single-chassis isolation — same cable, same peers, same workload, plus an unexplained idle reset Sep 25 — points to a unit-level hardware fault. We need the fieldiag bundle before any hardware disposition can be made.
  2. BERT/MTKID decoding: Linux can’t decode the 3c1e3f4b… vendor section; it requires ASUS/MediaTek engineering. The fieldiag bundle is the path to getting that read. (Worth noting: variant B correlates with your large-storm crashes, variant A with clean reboots — the record does distinguish reset type, even if we can’t read the payload today. @rustend’s third variant from HP ZGX is useful context and we’ll pass that along.)
  3. RMA? Contact your OEM vendor directly — ASUS for the Ascent GX10, HP for the ZGX Nano G1n. NVIDIA can’t issue replacements for OEM hardware. Attach the fieldiag output when you contact them and reference this thread.
  4. AER recovery hang: Expected behavior in the current kernel. When the NIC leaves the bus, the AER recovery thread spins on config-space reads to a device that never responds, producing the 130 s soft lockup you see. There’s no software mitigation once the device is gone — the fix is preventing the hardware event.

Side findings noted for the community: iommu.passthrough=1 throughput improvement is expected (SMMU translation overhead on Arm with large RDMA MRs — this is the recommended setting). The ibv_reg_mr ENOMEM regression on 7.0.0-1019-nvidia is confirmed.

6

Follow-up: hapy-2 silent reset with 1 s telemetry (HP ZGX Nano G1n)

One more reset today, after about 11.5 hours of uptime. Since my last post I added a lot more observability, especially around power, so maybe this helps, @Neill.

I haven’t run dgx-spark-fieldiag yet. It isn’t installed on these HP units, but dgx-spark-fieldiag 2.0.4-1 is in the DGX OS repo they’re configured with. Going by https://docs.nvidia.com/pdf/userguide-dgx-spark-fieldiag.pdf, partnerdiag --field runs the full ~35 min stress suite (Secure Boot off, DOCA-OFED for CX7Stress); I don’t see a mode that only collects logs or the SoC/EC power-rail history. Is the full run what you want right after the next reset, and is it valid on the HP ZGX Nano G1n? Or is there a separate way to pull the power-rail history?

AI-assisted summary of the data

hapy-2 hard-reset again on 2026-09-29 at 12:08:59.9 CEST. It was running SGLang TP=2 decode over RoCEv2 (direct DAC, 200G) with hapy-1. The job had been up since 00:40, with requests on and off all morning; the request in progress started at 12:05 after about 15 minutes idle.

Since the last post I added per-second telemetry on both nodes (written with fsync and shipped off-box through a disk-backed queue) and wall-power metering on a NETIO PowerPDU, polled from a third machine. So this time there is data up to the last second, from inside and outside the box.

Short version: hapy-2 went from normal operation to a dead link in under 0.7 s. Nothing in the OS, the drivers or any sensor moved beforehand, the supply was clean, and the only trace is the same MediaTek BERT record as always. That fits the upstream power-rail / PERST# explanation from Neill’s reply rather than anything software-side.

hapy-2 (resets) hapy-1 (stable)
Model HP ZGX Nano G1n HP ZGX Nano G1n
BIOS 5.36_0ACUM028 (2026-07-16) 5.36_0ACUM026 (2026-03-23)
Kernel / driver 7.0.0-1019-nvidia / 580.178.04 6.17.0-1031-nvidia / 580.173.02
ConnectX-7 fw 28.45.4028 28.45.4028
IOMMU iommu.passthrough=0 iommu.passthrough=0

Both log mlx5_pcie_event: Detected insufficient power on the PCIe slot (27W) at every boot, as in the GX10 reports.

What was measured, and what it showed

  • Moment of the reset: hapy-1’s kernel logged mlx5_core … Link down on both CX-7 ports at 12:08:59.945, within 3 ms of each other. hapy-2’s last sample is from 12:08:59.267, so the gap is at most 0.68 s.
  • GPU (last sample): 35.2 W, 65 °C, 88 % util, SM 2476 MHz. The clocks_event_reasons mask was 0 the whole time, and the driver’s counters show 0 s of power capping (and 0 s HW brake / thermal slowdown) in the preceding 3 minutes. nvidia-smi was still answering (sample age 0.2 s).
  • Thermal: hottest ACPI zone 76.5 °C (TSOC / TS1P), critical trip 104.8 °C. A slow rise since the request started, no step.
  • Memory: 82.9 % of 124.6 GB used, flat since the request started. PSI memory and CPU 0.
  • Supply: wall power steady at 133–140 W until the reset, then 26 W (PF 0.56) at 12:09:10 while the board restarted. Mains 239.5–240.1 V at 50.02 Hz, no sag; the outlet stayed on and the PDU did not reboot.
  • PCIe: AER counters 0 on every device; link speed and width unchanged (32 GT/s x4 on both CX-7 root ports).
  • Kernel: no AER, SDES, mlx5, Xid, soft-lockup or hung-task line in the whole boot. pstore (efi) empty. The last journal entry was at 12:08:09, a routine earlyoom status line.
  • Peer: hapy-1’s IB link_downed counter did not increment, although both ports were down for 15.5 s; only the port state (DOWN / Disabled) shows it. The links came back at 12:09:15.4, before hapy-2’s kernel had booted. hapy-1’s NCCL then spins on the missing peer (96 % util at ~20 W); the job does not recover on its own.

fig1-hapy2-last-6-min
fig1-hapy2-last-6-min1720×1932 210 KB

fig2-last-30s-and-peer
fig2-last-30s-and-peer1720×1348 137 KB

fig3-wall-power-mains
fig3-wall-power-mains1720×1034 68.4 KB

Timeline (CEST)

Time Event Source
00:40:32 flashnext-tp2 (SGLang TP=2) started; head on hapy-2, worker on hapy-1 sparkrun
10:11–11:50 Requests come and go, 1–2 at a time SGLang metrics
11:50–12:05 Idle: GPU 11.5 W, 47 °C hapy-2 1 s log
12:05:00 Next request starts hapy-2 1 s log
12:08:09 Last hapy-2 journal entry hapy-2 journal
12:08:59.267 Last hapy-2 sample, all values normal hapy-2 1 s log
12:08:59.945 Both CX-7 links down on the peer (the reset) hapy-1 kernel
12:09:10 hapy-2 at 26 W, PF 0.56 PDU
12:09:15.4 CX-7 links back up hapy-1 kernel
12:09:21 hapy-2 kernel boots: BERT: Total records found: 1 hapy-2 kernel

BERT record

[Hardware Error]:   section type: unknown, 3c1e3f4b-1e1a-43df-af28-59820e958e3c
[Hardware Error]:   section length: 0x3e
[Hardware Error]:   00000000: 000d0000 544d0000 0044494b 00000000  ......MTKID.....
[Hardware Error]:   00000010: 00000000 00000020 00000022 56190000  .... ..."......V
[Hardware Error]:   00000020: a451e0a4 c2964450 0ae9a1c7 0000fa95  ..Q.PD..........
[Hardware Error]:   00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00        ..............
Offset hapy-2 (every reset) GX10 unit A (earlier in this thread)
0x14 0x20 0x10
0x2C 0x0000fa95 variant A 0x0080fa95 · variant B (storm) 0x0000fa95
0x30–0x3D all zero variant A 20 … 40 · variant B a0 … 80

The payload is byte-identical across all 9 hapy-2 hard resets in my monitoring history (23–29 Sep). Clean shutdowns in the same period left no BERT record.

7

Adding an ASUS GX10 with the same firmware set as your Unit A, because on this chassis the resets carried on after the ConnectX-7 was off the bus entirely. That seems to support @Neill’s reading that the double SDES is a symptom of something upstream rather than the cause.

Unit: ASUS Ascent GX10, BIOS GX10DGX.0105.2026.0505.1153, EC 0x02000006, SoC 0x03000007, USB-C PD 0x516, CX7 fw 28.45.4028, kernel 7.0.0-1019-nvidia, driver 580.178.04 (the same kernel and driver as @rustend’s hapy-2).

Phase 1, Sep 19-20: your signature. The unit was one rank of a 3-node vLLM tensor-parallel group over direct QSFP (RoCEv2). In the middle of a run of hard resets, both CX7 root ports (0000:00:00.0 and 0002:00:00.0) logged Uncorrectable (Fatal) SDES in the same second, preceded by a corrected RxErr on 0002:00:00.0. Another boot that day ended with NETDEV WATCHDOG transmit timeouts on both CX7 second ports. It hard-reset many times in about 12 hours, including runs of boots that lasted only 5 to 7 s. No BERT records. A capsule downgrade (BIOS 0104, EC 0x02000005) changed nothing. After a full ASUS recovery-image reflash and an update back to the firmware above, it was redeployed as below.

Phase 2, Sep 20 to today: same chassis, no QSFP cables at all. It now runs as a single node on LAN only. With no transceivers, cx7-pcie-hotplug powers the CX7 off at every boot: no Mellanox functions on the bus, and no SDES or fatal AER since (only the corrected RxErr from the hotplug power-down at boot). Dates UTC:

  • Sep 20-21: 9 deliberate load-and-reboot cycles, all clean, NVMe unsafe_shutdowns unchanged.
  • Sep 24, after about 3 days up: power loss, then 7 boots of 6 to 7 s each before one stayed up.
  • Sep 24 (after ~20 h) and Sep 27 (after ~2.4 days): single power losses.
  • Sep 29, after ~2.3 days: power loss about 1 min 50 s into a single-node vLLM prefill benchmark, then 2 boots of 6 to 7 s. Telemetry collected off-box every 2 s: GPU 96%, 40 to 51 W, 62 C max, 8.2 GiB host memory free, nothing moving beforehand.
  • 13 unclean power-offs in all, matching NVMe unsafe_shutdowns 192 to 205.
  • The short boots end at the same point each time, about 6 s in, as the NVIDIA X session and the GPU containers start.
  • No kernel trace, no pstore, and no BERT record after any of them (we never get the MTKID record you both see).

So on this unit the double Surprise Down was one symptom while the NIC was on the bus, and the resets outlived the NIC. Happy to share the boot list and telemetry.

8

your terrible trackers and javascript cause my scroll to break. Unbelievable after spending 18k on nvidia hardware

9

Thanks Neill

I do already have a fieldiag run from Unit A (r9.257.8), but it isn’t the post-event capture you asked for. It was taken Sep 26 03:57 UTC, about 9 hours and two reboots after the last event (SDES on both root ports Sep 25 18:57:14 / 18:57:16 UTC, followed by the soft lockups).

Result: PASS, all 9 tests OK (GpuStress, C2CStress, CpuStress1/2, PowerStress, ThermalStress, FioSSD, MemStress, CX7Stress). CX7Stress ran ib_write_bw at ~101.6 Gb/s on both NICs against an expected 85. An earlier run the same night reported FAIL only because CX7Stress couldn’t start MST; the other 8 tests passed. So the unit passes diagnostics in a healthy state, which fits an intermittent fault. It has now been up 3 days 17 hours without a recurrence. I have a capture script staged to run fieldiag immediately after the next event, before any reboot, and I’ll post that bundle as soon as I have it. I can attach it here or send it however you prefer. I’ll also open a case with ASUS referencing this thread.

Understood on the AER recovery hang, the BERT decoding and iommu.passthrough.

10

Unit ASUS GX10 (DGX Spark / GB10), hostname gx10-3502,

         serial W1MSAG003973BLC

Role head node, 192.168.100.111

Peers gx10-efa8 (192.168.100.122), gx10-398b (192.168.100.133)

Fabric 3x direct QSFP DACs in a directed ring, all four CX7

         functions in use per node

Software kernel 6.17.0-1031-nvidia on all nodes,

         CX7 firmware 28.45.4028, iommu.passthrough=1

Times are CDT unless marked UTC (CDT = UTC-5). [inference] marks

interpretation; everything else is read directly from logs.

1. SUMMARY

----------

Unit A keeps resetting itself. The failures always involve its two

ConnectX-7 PCIe links, and they happen whether the box is under load,

idle, or still booting. Three forms have now been observed:

A. Full AER storm (12 recorded, latest Sep 30 06:03)

 - Both GB10 root ports feeding the CX7 (0000:00:00.0 and

   0002:00:00.0, each Gen5 x4) report Surprise Down (SDES) within

   \~50 ms of each other.

 - The mlx5 driver reports fatal health errors and the AER recovery

   path soft-locks the CPUs.

 - The box hangs for minutes, then hard-resets.

 - The next boot carries a BERT record (MediaTek MTKID, variant A or B).

B. Silent hard reset (latest Oct 01 02:42, idle)

 - Nothing is persisted on the head: no AER, no lockups, no shutdown.

 - The next boot carries BERT variant A.

 - Both peers see a \~46 s link-down.

 - Same pattern as the unexplained Sep 25 04:06 reset.

C. Early-boot reset loop (Oct 01 07:57-08:05, after a clean reboot)

 - Five consecutive boots each died \~10-20 s after kernel start,

   seconds after the CX7 fabric links came up. The sixth boot stayed

   up.

 - Warm resets: the CX7 links stayed powered through them.

 - No BERT, nothing persisted.

 - A clean reboot took \~9.5 min instead of \~80 s.

 - One earlier instance: Sep 25 19:03, the first boot after storm #11.

In all three forms:

  • The peers never log a PCIe/AER error; they only see the link go down.

  • Only this one chassis is affected. efa8 and 398b, the same model on the

    same software, have never stormed.

Two resets in ~20 h while idle (A and B), then five resets during one

reboot (C), rule out the workload as the trigger.

[inference] The evidence points to a unit-level hardware fault upstream of

both x4 links:

  • CX7 module, PCIe, or the power/PERST# path.

  • NVIDIA’s forum reply (Sep 28) reached the same reading: "PCIe power

    rail or a PERST# event … unit-level hardware fault".

Operational effects:

  • Every reset drops the GLM-5.3 TP=3 ring.

  • After a reset, each worker’s head-facing RoCE v2 GID moves from index 3

    to index 4, and start-tp3.sh refuses to launch until that is fixed.

2. WHAT HAS BEEN RULED OUT

--------------------------

  • Workload / load

    Storms #11 and #12 and event B happened idle; event C happened

    during boot with no engine running.

  • IOMMU mode

    #11 and #12 and B and C all happened with iommu.passthrough=1.

  • efa8 PD firmware mismatch

    #11 happened after it was updated (0x1 → 0x516).

  • A specific worker

    The faults continued after swapping efa8 → 398b, and happened with

    both attached. Not fully separated: no storm in ~40 h with 398b

    alone.

  • CX7 firmware, BIOS, kernel cmdline

    Identical to the peers.

  • Firmware update during event C

    fwupd history shows only the EC update of 2026-08-23

    (0x00000001 → 0x02000006). No capsule was applied.

  • Cable (weak)

    PHY CRC/symbol/FEC counters were 0; the same cable showed no faults

    on other nodes. Electrical/ground-loop effects remain untested.

  • 27 W slot-power warning

    Logged by every unit; NVIDIA says it is not a differentiator.

3. CHRONOLOGICAL NARRATIVE

--------------------------

Sep 20-25 Early history (DeepSeek TP=2 lane, head ↔ efa8, direct DAC)

Storms:

  • #1-#9: Sep 21 18:25 to Sep 22 15:44. Most began 4 s to 4.6 min after

    a container start. Several hung for 1+ h with 250-345 soft lockups

    before the reset.

  • #10: Sep 22 19:57. TX timeout only, no AER.

  • #11: Sep 25 13:57. Idle, in a restart loop.

Unexplained resets with BERT variant A:

  • Sep 20 ~19:12 and Sep 21 ~16:23.

  • Sep 25 04:06: idle, peers saw a 46 s link-down (normal reboot:

    12-15 s).

Changes made during this period, with storms continuing after each:

  • efa8 PD firmware update

  • iommu.passthrough=1 on all nodes

  • a trial through a MikroTik switch, then back to direct DAC

Fieldiag:

  • Sep 25 22:57: PASS (logs-20260926-035721.tgz). Healthy baseline only;

    no post-event fieldiag has ever been run.

Sep 25 19:03 UTC First early-boot failure (precedent for form C)

  • The boot after storm #11 (e6b6b12b) had CX7 link up at 11.7 s of

    uptime; its journal ends at 14.5 s on “Loading containers: done.”

  • The next boot (19:04:39 UTC) stayed up.

Sep 26-28 Worker swapped to 398b, then the 3-node ring built

  • No storms in ~40 h with 398b as the only worker.

  • Sep 27-28: the cables on 398b were swapped to make a directed ring

    (confirmed with LLDP), fabric re-addressed (tp3-f0/f1, tp3-p2f0/p2f1),

    and RDMA verified at ~111.6 Gb/s in all six directions.

  • Sep 27 22:02: the remote kernel-log stream to

    398b:~/head-kernel-stream.log stopped. It was later found dead and has

    captured nothing since.

Sep 30 06:03:16 STORM #12 - form A

  • GLM TP=3 ring up ~45 h, idle ~7 h.

  • Head kernel (UTC):

    11:03:16.453 pcieport 0000:00:00.0: AER: Uncorrectable (Fatal) …

    11:03:17.325 pcieport 0000:00:00.0: [ 5] SDES (First)

    11:03:17.325 mlx5_core 0002:01:00.0: poll_health: Fatal error 1

    11:03:17.325 mlx5_core 0000:01:00.0: mlx5_pci_err_detected …

               pci channel state = 2

    11:03:17.325 pcieport 0002:00:00.0: AER: Uncorrectable (Non-Fatal)

               ... \[ 5\] SDES

    11:03:18.2-.8 mlx5_core 0000:01:00.1 / 0002:01:00.1: Fatal error 1

  • Then 4 soft lockups, 26 TX timeouts, and "mlx5e_safe_reopen_channels

    failed recovering from a tx_timeout". The journal ends 06:07:22 with

    no shutdown (hard reset).

  • Next boot at 06:11: BERT variant A

    (… 0ae9a1c7 0080fa95 / 00 20 … 40)

  • Workers: link down at 06:03:16 only, no PCIe errors.

Sep 30 ~08:00-08:15 Aftermath

  • start-tp3.sh: “rank 1 GID index 3 is EMPTY on rocep1s0f0”.

  • The RoCE v2 IPv4 GID had moved to index 4 on each worker’s head-facing

    port (efa8 rocep1s0f0, 398b rocep1s0f1); the other ports stayed at

    index 3.

  • Both workers rebooted at 08:15.

Oct 01 02:41-02:44 SILENT RESET - form B

  • The head’s journal ends at 07:41:53 UTC with routine avahi lines.

    Before that: only cron, anacron, sysstat and DHCPv6. No kernel errors,

    no shutdown. No engine running, only the glm53-nfs container.

  • Both workers: head-facing link down 07:42:24 UTC, up 07:43:11 (46 s,

    restored during the head’s firmware POST). No worker PCIe errors.

  • Next head boot (07:44:15 UTC): BERT variant A, byte-identical to

    storm #12’s.

  • [inference] Either an instant reset/power event, or a storm that hung

    the head before journald flushed. Without the off-box stream the two

    cannot be told apart.

  • The GID shift recurred:

    efa8 rocep1s0f0, roceP2p1s0f0 (-> head) gid4

    398b rocep1s0f1, roceP2p1s0f1 (-> head) gid4

    ports facing the other worker gid3

Oct 01 07:55:46 Operator reboot of all three nodes

  • Head: clean `sudo reboot`, “Journal stopped” at 07:55:49.

  • Workers rebooted by the operator; back at 07:58:10 (efa8) and

    07:58:27 (398b), error-free.

Oct 01 07:57-08:05 EARLY-BOOT RESET LOOP - form C

  • Five head boots in a row, ~100 s apart:

    12:57:11, 12:58:49, 13:00:28, 13:02:12, 13:03:46 UTC

  • Each journal ends at 8.9-9.2 s of uptime, right after:

    [~7-8 s] mlx5_core …: Link up (all four CX7 functions)

    [~9.2 s] dockerd: Loading containers: done. (glm53-nfs restart)

  • Each boot: 0 storm lines, 0 shutdown markers, no BERT.

  • Workers saw only the head’s driver-load link reset (~7-8 s after each

    kernel start, 14.5 s down). The link then stayed up through each

    crash, so these were warm resets.

  • POST takes ~80 s on this unit, so each failed boot lived ~10-20 s:

    seconds after its fabric links came up.

  • The sixth boot (13:05:28 UTC = 08:05 CDT) stayed up; 0 errors at 5 min

    of uptime.

11

when the aer storm happens or I have to reboot, it get stuck in a boot loop for a while

12

Thanks for the continued updates — the new data since our last comment substantially advances the diagnosis.

@joeym82956’s hotplug finding settles the upstream question

With the CX7 fully powered off via hotplug and resets still continuing, the ConnectX-7 itself — its firmware, PCIe link, and power draw — is eliminated as the primary fault source. The AER storms while the NIC was on the bus were a secondary manifestation of the underlying fault; the reset mechanism exists independently of the NIC. This is the clearest evidence yet that the fault lives in the power or PERST# path upstream of both x4 segments.

@rustend’s telemetry

GPU at 35.2W, 65°C, 88% utilization, wall power steady 133–140W right up to the reset — then 26W. No precursor of any kind. This is consistent with an instantaneous hardware event, not a thermal trip, OOM, or workload-induced failure.

On the fieldiag PASS (@leafy-inn.0z, Sep 26)

A PASS run 9 hours after an event, including CX7Stress at ~101.6 Gb/s, means the hardware was functional at collection time — not that there’s no fault. For an intermittent failure like this, fieldiag is only meaningful when run immediately post-event in the failed state, before rebooting. The capture script you’ve staged is exactly right.

A clarification on the insufficient power on PCIe slot (27W) message

To add precision to what we said earlier: this message is benign and generated because the PCIe slot does not advertise its power capability to the NIC — not because the NIC is actually underpowered. It appears on every GB10 unit at every boot and is unrelated to the fault being investigated.

On the GID index shift (@leafy-inn.0z, post #10)

GID index migration (index 3 → 4) is expected behavior when a RoCE port resets and the GID table is rebuilt on the next link-up. The immediate workaround is show_gids to verify the current mapping and update your start-tp3.sh config accordingly. A more durable fix is to reference the GID by its associated IP address rather than by hardcoded index. The root cause remains the hardware reset, not a separate RoCE issue.

Next steps

We’re raising this for engineering review internally given the multi-OEM pattern (same BERT GUID across ASUS GX10 and HP ZGX, third ASUS unit confirming resets survive NIC removal). For hardware disposition on Unit A: contact ASUS support directly — for OEM units, ASUS hand

13

I’m facing this exact problem too on a Founders Edition DGX Spark.

I’ve raised an RMA but it would be lovely if there was a firmware fix for it or something.

The RMA process so far has been… tiresome and fruitless.

14

Thanks for the report, @home24. What you’re seeing matches the pattern in this thread: a hardware-level fault upstream of the ConnectX-7’s PCIe links, not a firmware or driver issue. The multi-OEM evidence (ASUS GX10, HP ZGX, and now the Founders Edition) points to a common power/PERST# path on the GB10 platform.

For the Founders Edition, you should work with NVIDIA support directly for the RMA — the process you’ve already started is the right path. If you can, capture the same kind of logs this thread has: dmesg around the event, any AER/SDES entries, BERT records from /sys/firmware/acpi/tables/BERT, and the output of dgx-spark-fieldiag (guide). Even a short kernel log from the failed boot will help us confirm whether you’re seeing the same signature.

Also, to be clear: the ‘insufficient power on PCIe slot (27W)’ message you may see at boot is benign — it’s just the slot not advertising its power capability, and it appears on all GB10 units. It’s not related to this fault.

Let us know if your RMA case stalls or if you’d like us to escalate internally — we’re tracking this as a cross-OEM issue.