Jp 7.2.1 hangs Jetson AGX Orin during a reboot stress test

pc1_log.txt (807.1 KB)

pc2_log.txt (1.2 MB)

BPMP firmware fatal error in EMC DVFS library (dvfs_training_sequence.c, ERR_UID=9999113) hangs Jetson AGX Orin during a reboot stress test

1. Summary

During a continuous reboot stress test, a Jetson AGX Orin (T234) intermittently fails to
complete the boot: about 19 s into the boot the BPMP firmware aborts inside its EMC DVFS
library while switching the EMC refresh/timing state from the “derated” table at
3199000 kHz:

refresh_change: switching from derated at 3199000 kHz failed

FATAL ERROR [FILE=platform/drivers/emc/lib/t234/dvfs_training_sequence.c, ERR_UID=9999113]: fatal error in emc dvfs library via log_error

After this message the BPMP prints its crash dump and stops responding permanently.
Every BPMP-mediated service on the CCPLEX then fails (thermal, clocks, EMC bandwidth,
nvidia-modeset), all hung tasks are reported after 122 s, and the board never reboots
again. The failure is intermittent (roughly 1 in a few boots) and has been observed on
two identical units with a byte-identical BPMP call stack.

2. Environment

  • Module: Jetson AGX Orin 64GB, P3701-0005 (MB1: FUSE_SKU_INFO_0 = 0xd0, SKU: Prod)
  • Carrier: P3737-0000 (BSP is built for tegra234-p3737-0000+p3701-0005-nv.dtb) [confirm]
  • BSP: L4T 39.2.1 based custom BSP, JetPack [TO FILL]
  • Kernel: Linux 6.8.12-1021-rt-tegra (PREEMPT_RT), custom build
  • BPMP firmware tag (from kernel log): 684d9490f24f1c33cad8-bfff9e30f72
  • bpmp_t234-TE990M-A1_prod.bin: 1,027,008 bytes, md5 3e4b3a1c12e899fb367a0fd6921f940f
  • BPMP DTB tegra234-bpmp-3701-0005-3737-0000.dtb: 256,576 bytes, md5 cc0507845e310b40e1b071a1fcc62f3b
  • Boot mode: Coldboot, rst_source: 0xb, rst_level: 0x1
  • BPMP_FW Boot Progress code: 0x190, BPMP_FW Error code: 0x0 (identical on passing and failing boots)
  • Ambient temperature: [TO FILL]; board is warm from the continuous reboot loop (no active cooling) [confirm]
  • No nvpmodel is present in this rootfs, so no power mode is being set by userspace

3. Symptom on the serial console

Last kernel messages before the abort (unit A, pc1_log.txt:7721-7734):

[   18.785047] [TS:946513380] isx031 7-0030: no found serialer dev
[   19.003552] [TS:953341652] RTL8261B-VB-CG 10Gbps PHY 6800000.ethernet:01: Not rtl8261x and skip init flow...id = c898
refresh_change: switching from derated at 3199000 kHz failed

FATAL ERROR [FILE=platform/drivers/emc/lib/t234/dvfs_training_sequence.c, ERR_UID=9999113]: fatal error in emc dvfs library via log_error

Unit B is identical, aborting at [ 19.276465] (pc2_log.txt:14503).

BPMP crash dump (identical on both units, unit A pc1_log.txt:7736-7802):

r0_usr  0x50096898
r1_usr  0x00989309
r2_usr  0x50041674
r3_usr  0x5018fc08
r4_usr  0x50002148
...
pc 0x50002148
spsr 0x40000010
fpscr 0x20000010
15: base: 5018e000 size: 00002000 XN P_RW U_RW Non-shareable outer: WB, no WA inner: WB, no WA
global worker@0x50248824 sp 0x5018fbf0 stack: 5018e000 - 5018fffc
call stack:
sp 0x5018fbf0 pc 0x50002148
sp 0x5018fbf0 pc 0x50003360
sp 0x5018fc08 pc 0x5004166a
sp 0x5018fc20 pc 0x50045d42
sp 0x5018fc28 pc 0x5006756c
sp 0x5018fc40 pc 0x500211de
sp 0x5018fe68 pc 0x5004155c
sp 0x5018ff10 pc 0x50043b5a
sp 0x5018ff28 pc 0x5005bbf2
sp 0x5018ff80 pc 0x50043198
sp 0x5018ffa0 pc 0x50044384
sp 0x5018ffc8 pc 0x500456cc
sp 0x5018ffe0 pc 0x50069766
sp 0x5018fff8 pc 0xaaaaaaaa
eht_idx_find: 0xaaaaaaaa not a valid code address
no eidx for 0xaaaaaaaa

Both units abort in the same task (global worker@0x50248824) with the same
13-frame call stack and the same PCs, i.e. a deterministic failure point inside the
EMC DVFS library.

Downstream consequences (unit A, after the abort):

tegra-mc 2c00000.memory-controller: BPMP transfer failed
thermal_zone*: ... (-62) / ... (-16)
nvethernet 6800000.ethernet: failed to enable rx_m_clk / rx_pcs_m_clk
tegra-bpmp bpmp: failed to disable clock eqos_*: -16
nvgpu ... power_node_write failed at busy
nvidia-modeset: Failed to allocate 4500000 KBPS Iso and 4294967295 KBPS Dram
INFO: task ... blocked for more than 122 seconds.

The board then hangs permanently and does not execute the scheduled reboot.

4. Reproduction

  • Test: a script runs after each boot, checks eMMC / NVMe / LAN / WiFi / Audio / PCIe /
    USB / EtherCAT / CAN / Bluetooth, writes the results to /root/log, then sleeps 30 s
    and calls reboot. The reboot loop runs unattended for hours.
  • Failure rate: intermittent. It occurred on the 2nd captured cold boot of unit A and
    the 4th captured cold boot of unit B (the serial capture starts in the middle of the
    test, so the absolute cycle numbers are higher). Both units completed many other
    boots with no error.
  • All the peripheral checks of the test script run after the login prompt, i.e. after
    the point where the BPMP aborts, so they are not the trigger.
  • Timing: the abort happens at the very end of the boot (18.6-21.4 s is the normal range
    for reaching the login prompt on these units).

5. What we have verified so far

  • A passing boot and a failing boot of the same unit were diffed line by line: they are
    identical apart from the refresh_change line and everything after it. No kernel-side
    software difference, no DT difference, nothing that indicates a configuration change
    between the two boots.
  • The kernel log contains no EMC/bandwidth request immediately before the abort; the last
    messages are camera sensor probes, four IRQ194/IRQ293: set affinity failed(-22)
    messages and an ethernet PHY message, i.e. ordinary boot-completion activity.
  • We decompiled tegra234-bpmp-3701-0005-3737-0000.dtb: it contains
    nvidia,tegra234-emc-table (nominal) and nvidia,tegra234-emc-table-derated tables at
    204000 / 665600 / 2133000 / 3199000 kHz — 3199000 kHz is the top OPP and the exact
    frequency named in the failure message. It also contains two strap-selectable DRAM
    configurations (external-memory-2 / external-memory-3, with DRAM config version
    strings such as ..._NoCfgVersion_V1.0.1_V1.5 and a ...V1.0 variant).
  • The BPMP firmware binary is encrypted (file reports “data”), so we cannot symbolize
    the crash PCs, and there is no source for platform/drivers/emc/lib/t234/ in the BSP.
  • We found public reports of the same ERR_UID=9999113 on Jetson Thor (t264), described
    there as a BPMP-FW issue (“DVFS sequence stalled calibration more than threshold”) and
    fixed in a later release. This looks like the same defect class on T234.

6. Impact

The failure permanently wedges the unit: no automatic recovery, no watchdog reset, the
scheduled reboot is never executed. This makes the reboot reliability requirement for
our product unachievable, and the failing step is inside the BPMP firmware, which we
cannot patch or work around from the CCPLEX.

7. Questions / requests

  1. Is ERR_UID=9999113 from dvfs_training_sequence.c on T234 a known BPMP-FW issue?
    Is it the same defect as the t264 ERR_UID=9999113 report? If so, in which release /
    BPMP firmware version is the fix available for T234 (P3701-0005)?
  2. Is there a supported way to avoid this code path, e.g. by capping the EMC maximum
    frequency to 2133000 kHz in the BPMP DTB, or by any EMC/DVFS table or strap option?
    Would that avoid the failing “switching from derated at 3199000 kHz” sequence, or is
    3199000 kHz required by the memory controller configuration?
  3. Does the derated/nominal switch depend on DRAM temperature (i.e. can a thermal
    condition make this path fail)? What temperature thresholds does the BPMP use, and
    is there a supported way to bias/cool to avoid it during our test?
  4. What additional information should we capture to help localize this (BPMP trace
    buffer, additional debugfs nodes, which counters, kernel-side MRQ traces)?
  5. Is it possible to obtain a debug-symbolized or non-encrypted BPMP firmware build for
    this analysis, or a list of the relevant EMC DVFS error paths and their ERR_UIDs?

8. Attachments

  • pc1_log.txt — full serial console log of unit A. The failing boot starts at the
    cold boot marker at line 4937 and the abort is at line 7732.
  • pc2_log.txt — full serial console log of unit B. The failing boot starts at the
    cold boot marker at line 11287 and the abort is at line 14503.

Hi,

There is no such known issue on T234.

Could you directly answer below question? Please do not give any further analysis. Some of them are not really important to the issue itself.

  1. Is that NV devkit or custom board? I saw you enabled many items that seems not on NV devkit (cam_fsync、RTL8261B PHY、KSZ9477、EtherCAT).
    But you also claimed " Carrier: P3737-0000 (BSP is built for tegra234-p3737-0000+p3701-0005-nv.dtb) [confirm]". I am not really sure what you are doing here.

  2. Is there any method that could reproduce this issue on NV devkit? Please make sure you really tried that before you answer us.

  3. We don’t answer your question regarding the BPMP as it is not open source. It should not be something for you to check. The only thing you need to do is answer below question in a very short way.

Hi,

Any answer to the question here so that we could do further check?

The issue has been resolved. Thank you.

How did you resolve this issue? Could you share?