pc1_log.txt (807.1 KB)
pc2_log.txt (1.2 MB)
BPMP firmware fatal error in EMC DVFS library (dvfs_training_sequence.c, ERR_UID=9999113) hangs Jetson AGX Orin during a reboot stress test
1. Summary
During a continuous reboot stress test, a Jetson AGX Orin (T234) intermittently fails to
complete the boot: about 19 s into the boot the BPMP firmware aborts inside its EMC DVFS
library while switching the EMC refresh/timing state from the “derated” table at
3199000 kHz:
refresh_change: switching from derated at 3199000 kHz failed
FATAL ERROR [FILE=platform/drivers/emc/lib/t234/dvfs_training_sequence.c, ERR_UID=9999113]: fatal error in emc dvfs library via log_error
After this message the BPMP prints its crash dump and stops responding permanently.
Every BPMP-mediated service on the CCPLEX then fails (thermal, clocks, EMC bandwidth,
nvidia-modeset), all hung tasks are reported after 122 s, and the board never reboots
again. The failure is intermittent (roughly 1 in a few boots) and has been observed on
two identical units with a byte-identical BPMP call stack.
2. Environment
- Module: Jetson AGX Orin 64GB, P3701-0005 (MB1:
FUSE_SKU_INFO_0 = 0xd0,SKU: Prod) - Carrier: P3737-0000 (BSP is built for
tegra234-p3737-0000+p3701-0005-nv.dtb) [confirm] - BSP: L4T 39.2.1 based custom BSP, JetPack [TO FILL]
- Kernel:
Linux 6.8.12-1021-rt-tegra(PREEMPT_RT), custom build - BPMP firmware tag (from kernel log):
684d9490f24f1c33cad8-bfff9e30f72 bpmp_t234-TE990M-A1_prod.bin: 1,027,008 bytes, md53e4b3a1c12e899fb367a0fd6921f940f- BPMP DTB
tegra234-bpmp-3701-0005-3737-0000.dtb: 256,576 bytes, md5cc0507845e310b40e1b071a1fcc62f3b - Boot mode: Coldboot,
rst_source: 0xb, rst_level: 0x1 BPMP_FW Boot Progress code: 0x190,BPMP_FW Error code: 0x0(identical on passing and failing boots)- Ambient temperature: [TO FILL]; board is warm from the continuous reboot loop (no active cooling) [confirm]
- No
nvpmodelis present in this rootfs, so no power mode is being set by userspace
3. Symptom on the serial console
Last kernel messages before the abort (unit A, pc1_log.txt:7721-7734):
[ 18.785047] [TS:946513380] isx031 7-0030: no found serialer dev
[ 19.003552] [TS:953341652] RTL8261B-VB-CG 10Gbps PHY 6800000.ethernet:01: Not rtl8261x and skip init flow...id = c898
refresh_change: switching from derated at 3199000 kHz failed
FATAL ERROR [FILE=platform/drivers/emc/lib/t234/dvfs_training_sequence.c, ERR_UID=9999113]: fatal error in emc dvfs library via log_error
Unit B is identical, aborting at [ 19.276465] (pc2_log.txt:14503).
BPMP crash dump (identical on both units, unit A pc1_log.txt:7736-7802):
r0_usr 0x50096898
r1_usr 0x00989309
r2_usr 0x50041674
r3_usr 0x5018fc08
r4_usr 0x50002148
...
pc 0x50002148
spsr 0x40000010
fpscr 0x20000010
15: base: 5018e000 size: 00002000 XN P_RW U_RW Non-shareable outer: WB, no WA inner: WB, no WA
global worker@0x50248824 sp 0x5018fbf0 stack: 5018e000 - 5018fffc
call stack:
sp 0x5018fbf0 pc 0x50002148
sp 0x5018fbf0 pc 0x50003360
sp 0x5018fc08 pc 0x5004166a
sp 0x5018fc20 pc 0x50045d42
sp 0x5018fc28 pc 0x5006756c
sp 0x5018fc40 pc 0x500211de
sp 0x5018fe68 pc 0x5004155c
sp 0x5018ff10 pc 0x50043b5a
sp 0x5018ff28 pc 0x5005bbf2
sp 0x5018ff80 pc 0x50043198
sp 0x5018ffa0 pc 0x50044384
sp 0x5018ffc8 pc 0x500456cc
sp 0x5018ffe0 pc 0x50069766
sp 0x5018fff8 pc 0xaaaaaaaa
eht_idx_find: 0xaaaaaaaa not a valid code address
no eidx for 0xaaaaaaaa
Both units abort in the same task (global worker@0x50248824) with the same
13-frame call stack and the same PCs, i.e. a deterministic failure point inside the
EMC DVFS library.
Downstream consequences (unit A, after the abort):
tegra-mc 2c00000.memory-controller: BPMP transfer failed
thermal_zone*: ... (-62) / ... (-16)
nvethernet 6800000.ethernet: failed to enable rx_m_clk / rx_pcs_m_clk
tegra-bpmp bpmp: failed to disable clock eqos_*: -16
nvgpu ... power_node_write failed at busy
nvidia-modeset: Failed to allocate 4500000 KBPS Iso and 4294967295 KBPS Dram
INFO: task ... blocked for more than 122 seconds.
The board then hangs permanently and does not execute the scheduled reboot.
4. Reproduction
- Test: a script runs after each boot, checks eMMC / NVMe / LAN / WiFi / Audio / PCIe /
USB / EtherCAT / CAN / Bluetooth, writes the results to /root/log, then sleeps 30 s
and callsreboot. The reboot loop runs unattended for hours. - Failure rate: intermittent. It occurred on the 2nd captured cold boot of unit A and
the 4th captured cold boot of unit B (the serial capture starts in the middle of the
test, so the absolute cycle numbers are higher). Both units completed many other
boots with no error. - All the peripheral checks of the test script run after the login prompt, i.e. after
the point where the BPMP aborts, so they are not the trigger. - Timing: the abort happens at the very end of the boot (18.6-21.4 s is the normal range
for reaching the login prompt on these units).
5. What we have verified so far
- A passing boot and a failing boot of the same unit were diffed line by line: they are
identical apart from therefresh_changeline and everything after it. No kernel-side
software difference, no DT difference, nothing that indicates a configuration change
between the two boots. - The kernel log contains no EMC/bandwidth request immediately before the abort; the last
messages are camera sensor probes, fourIRQ194/IRQ293: set affinity failed(-22)
messages and an ethernet PHY message, i.e. ordinary boot-completion activity. - We decompiled
tegra234-bpmp-3701-0005-3737-0000.dtb: it contains
nvidia,tegra234-emc-table(nominal) andnvidia,tegra234-emc-table-deratedtables at
204000 / 665600 / 2133000 / 3199000 kHz — 3199000 kHz is the top OPP and the exact
frequency named in the failure message. It also contains two strap-selectable DRAM
configurations (external-memory-2/external-memory-3, with DRAM config version
strings such as..._NoCfgVersion_V1.0.1_V1.5and a...V1.0variant). - The BPMP firmware binary is encrypted (
filereports “data”), so we cannot symbolize
the crash PCs, and there is no source forplatform/drivers/emc/lib/t234/in the BSP. - We found public reports of the same
ERR_UID=9999113on Jetson Thor (t264), described
there as a BPMP-FW issue (“DVFS sequence stalled calibration more than threshold”) and
fixed in a later release. This looks like the same defect class on T234.
6. Impact
The failure permanently wedges the unit: no automatic recovery, no watchdog reset, the
scheduled reboot is never executed. This makes the reboot reliability requirement for
our product unachievable, and the failing step is inside the BPMP firmware, which we
cannot patch or work around from the CCPLEX.
7. Questions / requests
- Is
ERR_UID=9999113fromdvfs_training_sequence.con T234 a known BPMP-FW issue?
Is it the same defect as the t264ERR_UID=9999113report? If so, in which release /
BPMP firmware version is the fix available for T234 (P3701-0005)? - Is there a supported way to avoid this code path, e.g. by capping the EMC maximum
frequency to 2133000 kHz in the BPMP DTB, or by any EMC/DVFS table or strap option?
Would that avoid the failing “switching from derated at 3199000 kHz” sequence, or is
3199000 kHz required by the memory controller configuration? - Does the derated/nominal switch depend on DRAM temperature (i.e. can a thermal
condition make this path fail)? What temperature thresholds does the BPMP use, and
is there a supported way to bias/cool to avoid it during our test? - What additional information should we capture to help localize this (BPMP trace
buffer, additional debugfs nodes, which counters, kernel-side MRQ traces)? - Is it possible to obtain a debug-symbolized or non-encrypted BPMP firmware build for
this analysis, or a list of the relevant EMC DVFS error paths and their ERR_UIDs?
8. Attachments
pc1_log.txt— full serial console log of unit A. The failing boot starts at the
cold boot marker at line 4937 and the abort is at line 7732.pc2_log.txt— full serial console log of unit B. The failing boot starts at the
cold boot marker at line 11287 and the abort is at line 14503.