[Jetson AGX Orin 64 R35.2.1] System Reset Due to Kernel Panic on Boot

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
1

Hi NVIDIA Team,

Our client are encountering an issue on Jetson AGX Orin (L4T R35.2.1) where the system unexpectedly resets on cold boot, and the customer suspects that the kernel panic was caused by BCCPLEXWDT (watchdog).

⚠️ Key Symptoms:

  • During cold boot, the system resets automatically without completing boot, causing the boot time longer than the normal boot time(more than 2mins).
  • The issue occurs intermittently, approximately once every 80 cold boots.
  • This appears to happen early in the boot process, before user space is fully initialized.
  • Serial console log shows the following errors:
[20250318 16:19:50:542][   16.584284] nvmap_heap_init: nvmap_heap_init: created heap block cache
[20250318 16:19:50:553][   16.584418] tegra-carveouts tegra-carveouts: fsi :dma coherent mem declare 0x0000000833000000,16777216
[20250318 16:19:50:564][   16.584421] tegra-carveouts tegra-carveouts: assigned reserved memory node fsi-carveout
[20250318 16:19:50:577][   16.584427] tegra-carveouts tegra-carveouts: vpr :dma coherent mem declare 0x0000000849800000,914358272
[20250318 16:19:50:577][   16.584428] tegra-carveouts tegra-carveouts: assigned reserved memory node vpr-carveout
[20250318 16:19:50:587][   16.584432] nvmap_page_pool_init: Total RAM pages: 7831023
[20250318 16:19:50:588][   16.584434] nvmap_page_pool_init: nvmap page pool size: 978877 pages (3823 MB)
[20250318 16:19:50:598][   16.584485] nvmap_background_zero_thread: PP zeroing thread starting.
[20250318 16:19:50:610][   16.584583] misc nvmap: created heap vpr base 0x0000000849800000 size (892928KiB)
[20250318 16:19:50:610][   16.587525] misc nvmap: created heap fsi base 0x0000000833000000 size (16384KiB)
[20250318 16:19:50:621][   16.654273] nvgpu: 17000000.ga10b          nvgpu_nvhost_syncpt_init:135  [INFO]  syncpt_unit_base 60000000 syncpt_unit_size 4000000 size 10000
[20250318 16:19:50:632][   16.654273] 
[20250318 16:19:50:633][   16.688396] systemd[1]: Started Journal Service.
[20250318 16:19:50:666][   16.717598] systemd-journald[290]: Received client request to flush runtime journal.
[20250318 16:19:50:688][   16.743421] Unable to handle kernel paging request at virtual address 0000000108d04000
[20250318 16:19:50:688][   16.751606] Mem abort info:
[20250318 16:19:50:699][   16.754482]   ESR = 0x96000004
[20250318 16:19:50:699][   16.757629]   EC = 0x25: DABT (current EL), IL = 32 bits
[20250318 16:19:50:700][   16.757630]   SET = 0, FnV = 0
[20250318 16:19:50:710][   16.757631]   EA = 0, S1PTW = 0
[20250318 16:19:50:710][   16.757631] Data abort info:
[20250318 16:19:50:712][   16.757632]   ISV = 0, ISS = 0x00000004
[20250318 16:19:50:722][   16.757635]   CM = 0, WnR = 0
[20250318 16:19:50:722][   16.779452] user pgtable: 4k pages, 48-bit VAs, pgdp=00000001085cd000
[20250318 16:19:50:734][   16.779453] [0000000108d04000] pgd=0000000000000000, p4d=0000000000000000
[20250318 16:19:50:734][   16.793084] Internal error: Oops: 96000004 [#1] PREEMPT SMP
[20250318 16:19:50:745][   16.793086] Modules linked in: ina3221(E) pwm_fan(E) nvgpu(E) nvmap(E) ip_tables(E) x_tables(E)
[20250318 16:19:50:756][   16.793097] CPU: 4 PID: 7 Comm: kworker/u16:0 Tainted: G            E     5.10.104-tegra #1
[20250318 16:19:50:767][   16.816368] Hardware name: Unknown Jetson AGX Orin/Jetson AGX Orin, BIOS 2.1-32413640 01/24/2023
[20250318 16:19:50:768][   16.816379] Workqueue: efi_rts_wq efi_call_rts
[20250318 16:19:50:780][   16.829968] pstate: 60c00009 (nZCv daif +PAN +UAO -TCO BTYPE=--)
[20250318 16:19:50:780][   16.829971] pc : 0xbd9c70733950
[20250318 16:19:50:780][   16.829973] lr : 0xbd9c70733810
[20250318 16:19:50:780][   16.842613] sp : ffff8000100a3a60
[20250318 16:19:50:791][   16.842614] x29: ffff8000100a3a70 x28: ffff6d7f4241b400 
[20250318 16:19:50:792][   16.851494] x27: 000000000000003a x26: 0000000000000000 
[20250318 16:19:50:802][   16.851495] x25: 0000bd9c70740000 x24: 0000bd9c70740548 
[20250318 16:19:50:803][   16.862407] x23: 0000bd9c707400a8 x22: 0000000000000062 
[20250318 16:19:50:813][   16.862408] x21: 0000bd9c70740470 x20: 0000bd9c70bd0118 
[20250318 16:19:50:815][   16.873333] x19: 0000000108d04000 x18: 0000000000000000 
[20250318 16:19:50:825][   16.873335] x17: 0000000000000000 x16: 0000000000000000 
[20250318 16:19:50:825][   16.873336] x15: 0daa11d293ca8be4 x14: df61000000000000 
[20250318 16:19:50:837][   16.884268] x13: 0065005300000000 x12: 0000000000000016 
[20250318 16:19:50:837][   16.884270] x11: 00000074006f006f x10: 0042006500720075 
[20250318 16:19:50:848][   16.895198] x9 : ffff8000100a3b78 x8 : 0000000000000000 
[20250318 16:19:50:848][   16.895200] x7 : 0000000000000000 x6 : 0000000000000000 
[20250318 16:19:50:860][   16.895201] x5 : 0000000000000000 x4 : 0000000000000000 
[20250318 16:19:50:860][   16.895202] x3 : 00000000ffff0005 x2 : 00000000482c7e00 
[20250318 16:19:50:871][   16.895204] x1 : 00000000482c7e00 x0 : ffff6d78482c7e00 
[20250318 16:19:50:871][   16.895205] Call trace:
[20250318 16:19:50:871][   16.895208]  0xbd9c70733950
[20250318 16:19:50:871][   16.933373]  0xbd9c70735b90
[20250318 16:19:50:883][   16.933374]  0xbd9c70782de0
[20250318 16:19:50:883][   16.933374]  0xbd9c7078568c
[20250318 16:19:50:883][   16.933380]  __efi_rt_asm_wrapper+0x28/0x48
[20250318 16:19:50:894][   16.946288]  efi_call_rts+0x354/0x410
[20250318 16:19:50:894][   16.946291]  process_one_work+0x1c0/0x4a0
[20250318 16:19:50:894][   16.946294]  worker_thread+0x50/0x420
[20250318 16:19:50:895][   16.957946]  kthread+0x148/0x170
[20250318 16:19:50:906][   16.957948]  ret_from_fork+0x10/0x18
[20250318 16:19:50:906][   16.957952] Code: 17ffffd6 a95207e0 aa008020 f9400413 (b9400262) 
[20250318 16:19:50:917][   16.971224] ---[ end trace bbf5969e48996226 ]---
[20250318 16:19:50:917][   16.975970] Kernel panic - not syncing: Oops: Fatal exception
[20250318 16:19:50:928][   16.975973] SMP: stopping secondary CPUs
[20250318 16:19:50:930][   16.985914] Kernel Offset: 0x3e3d5e660000 from 0xffff800010000000
[20250318 16:19:50:940][   16.985915] PHYS_OFFSET: 0xffff9288c0000000
[20250318 16:19:50:940][   16.985916] CPU features: 0x0040006,4a80aa38
[20250318 16:19:50:940][   16.985917] Memory Limit: none
[20250318 16:19:50:949][   17.003982] ---[ end Kernel panic - not syncing: Oops: Fatal exception ]---

❓ Questions:

  1. What could be the root cause of the kernel panic during the booting on Jetson R35.2.1?
  2. Is there a known issue in R35.2.1 related to BCCPLEXWDT or CPU bring-up that could cause this?
  3. Could this be related to CPU core power gating, clock tree issues, or bootloader misconfiguration?
  4. Our customer is highly concerned about this issue. Are there firmware updates (BPMP, MB2, PSC_FW, etc.) or patches available to address this?
  5. Would upgrading to R35.4.x or R36.x help mitigate this behavior?

Log Attachment:

Full boot log (serial UART dump) is attached, showing the watchdog triggering and reset cycle. Please advise on how we can better diagnose or patch this issue.

Thanks for your help!

20250318-161121-filter - copy.txt (131.8 KB)

20250609094900
202506090949001920×1080 205 KB

4

Hi Jack_Chia,

Are you using the devkit or custom board for AGX Orin?

Do you hit this issue in the reboot stress test?
If so, is the failed rate about 1/80?

The log in the attached file and in the screenshot are not matched.
Or the watchdog is triggered due to system hang with kernel panic?
(i.e. kernel panic happens first and the watchdog trigger reset)

From the current log, the kernel panic was caused by an unhandled page fault at the virtual memory address.
The CPU attempted to access this address, but it was not mapped to any physical memory, leading to a fatal, unrecoverable exception.

I would like to check the FULL log including the previous logs with boot successful.

6

The customer is using a custom board developed and manufactured by us, Realtimes. They initially suspected the issue might be related to BCCPLEXWDT, based on a photo taken during the failure. The analysis now indicates that the issue may be related to virtual memory mapping. Could you please help determine whether this is related to the R35.2.1 system/kernel itself, or if it might be caused by the customer’s application?

Due to intermittent boot delay issues observed in the field, the customer sent the device back to us for further analysis. We used a smart power switch to automate power cycling and executed reboot cycles via script, while monitoring the serial console to capture logs and measure boot intervals.

  • In the first round of testing, 395 cycles were executed, with 5 instances of abnormal boot time recorded.
  • In the second round, 179 cycles were completed, and 1 kernel panic was captured.
  • In the third round, 29 cycles were run, during which another kernel panic was successfully reproduced and logged.

Attached are the serial console logs captured before and after the issue occurred. Please review them at your convenience. Thank you.

20250318-161121.log (647.7 KB)
20250319-140059.log (396.7 KB)

7

Could you share the result of cat /etc/nv_boot_control.conf on your board?

They seem 3 different issues on your board.
Which one would you like to discuss in this topic? 2nd one with kernel panic?
I would suggest creating another topic to discuss about boot time issue.

Have you put the module on the devkit to check if there’s the similar issue?

And what’s the reproduce steps?
From the log you shared, is the kernel panic issue happening just after boot up?

[20250319 14:09:23:917][   15.906529] CPU: 3 PID: 7 Comm: kworker/u16:0 Tainted: G            E     5.10.104-tegra #1
[20250319 14:09:23:931][   15.906530] Hardware name: Unknown Jetson AGX Orin/Jetson AGX Orin, BIOS 2.1-32413640 01/24/2023
[20250319 14:09:23:932][   15.906542] Workqueue: efi_rts_wq efi_call_rts

The kernel panic may be relating to kworker calling UEFI runtime service and UEFI FW has the wrong memory access. It may be the bug in UEFI.
Could you also help to verify with the latest Jetpack 5.1.5(L4T R35.6.2)?

8

Sorry, the device with the issue was sent back to the customer three months ago, so we’re currently unable to retrieve that file and reproduce the issue for Jetpack 5.1.5. and we don’t modify this file normally.

Actually, we’ve identified two boot-related issues.

The first one is that the system occasionally gets stuck during the boot stage — we’ve already found the solution from the forums. It’s solved. bellow is the reference links:

The second issue is the boot delay, which involves the kernel panic we reported.

I don’t have an NVIDIA devkit to test this issue. The reproduce steps are as follows:

  • After powering on and booting into the system desktop, a crontab automatically poweroff the device.
  • Once the smart power switch detects the shutdown, it cuts the power.
  • After a 2-minute delay, the power is turned back on, and this cycle is repeated continuously for testing.

Is this the same as Issue 1 — a bug in R35.2.1? If so, is there any patch available, or has it been resolved in later versions? If yes, could you please share the specific cause and provide the relevant release notes or documentation that describe the fix?

9

I just want to check the details info of your setup.

I think we can focus on the kernel panic issue in this topic.

R35.2.1 is an quite old release to us(2yrs ago) so that we need you to clarify if the issue is reproducible on the devkit with the latest R35.6.2.
Or do you have any plan to update to the latest R35.6.2?

So, is the issue happening on the specific device?
(i.e. can you reproduce it on other boards?)

10

We haven’t reproduced the issue on the devkit using R35.6.2. Would it be possible for your team to set up the environment and help verify the issue?

There are a large number of Jetson Orin devices already deployed in the field, and the cost of upgrading all units to R35.6.2 would be too high at this stage.

According to user feedback, a total of five units have experienced this issue so far.

11

Orin 64GB supported since rel-35.3.1 but not rel-35.2.1.

image
image1104×151 21 KB

Is this Orin64GB module?

If you just repeating some comment with dummy information but not replying our questions, then nothing can help here.

12

Sorry, It’s Jetson AGX Orin 32GB.