Orin 32GB Modulewith DevKit Carrier Board on JetPack5.1.1 cause Kernel Panic

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
1

Hi,
As the topic,
I flash the JetPack5.1.1, L4T35.3.1 on the hardware Orin 32GB Module and AGX Orin DevKit Carrier Board.
There is no errror about flash process. But after I finish the flash process, the module reset and boot fail, and the console shows Kernel Panic.

Is there any idea about it？

My Hardware:
Orin 32GB Module(P3701-0004)
AGX Orin DevKit Carrier Board(P3737-0000)

My Software:
JetPack5.1.1, L4T35.3.1 without modification

My Flash command:

sudo ./flash.sh jetson-agx-orin-devkit mmcblk0p1

Here is the full flash log on host, and full console log on the module.
orin_devkit_jp5.1.1_flash.log (189.8 KB)
orin_devkit_jp5.1.1_console_kernel_panic.log (229.3 KB)

The error in console log is as below:

[   11.595453] Run /init as init process
[   11.611062] Root device found: mmcblk0p1
[   11.619651] Found dev node: /dev/mmcblk0p1
[   11.649202] EXT4-fs (mmcblk0p1): mounted filesystem with ordered data mode. Opts: (null)
[   11.659070] Rootfs mounted over mmcblk0p1
[   11.679587] Switching from initrd to actual rootfs
[   11.783987] systemd[1]: System time before build time, advancing clock.
[   11.830176] systemd[1]: systemd 245.4-4ubuntu3.19 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCO)
[   11.855196] systemd[1]: Detected architecture arm64.
[   11.905638] systemd[1]: No hostname configured.
[   11.910952] systemd[1]: Set hostname to <localhost>.
[   12.106831] random: systemd: uninitialized urandom read (16 bytes read)
[   12.114374] systemd[1]: system-modprobe.slice: unit configures an IP firewall, but the local system does not support BPF/cgroup firewalling.
[   12.128120] systemd[1]: (This warning is only shown for the first unit using IP firewalling.)
[   12.138335] systemd[1]: Created slice system-modprobe.slice.
[   12.145716] random: systemd: uninitialized urandom read (16 bytes read)
[   12.153330] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[   12.164465] random: systemd: uninitialized urandom read (16 bytes read)
[   12.171999] systemd[1]: Reached target Slices.
[   12.177999] systemd[1]: Reached target Swap.
[   12.184033] systemd[1]: Listening on RPCbind Server Activation Socket.
[   12.192314] systemd[1]: Listening on fsck to fsckd communication Socket.
[   12.200734] systemd[1]: Listening on initctl Compatibility Named Pipe.
[   12.209137] systemd[1]: Listening on Journal Audit Socket.
[   12.216374] systemd[1]: Listening on Journal Socket (/dev/log).
[   12.224097] systemd[1]: Listening on Journal Socket.
[   12.230853] systemd[1]: Listening on udev Control Socket.
[   12.238005] systemd[1]: Listening on udev Kernel Socket.
[   12.245907] systemd[1]: Mounting Huge Pages File System...
[   12.254004] systemd[1]: Mounting POSIX Message Queue File System...
[   12.263051] systemd[1]: Mounting Kernel Debug File System...
[   12.271359] systemd[1]: Mounting Kernel Trace File System...
[   12.279905] systemd[1]: Starting Journal Service...
[   12.287410] systemd[1]: Starting Set the console keyboard layout...
[   12.296470] systemd[1]: Starting Create list of static device nodes for the current kernel...
[   12.307912] systemd[1]: Starting Load Kernel Module chromeos_pstore...
[   12.317574] systemd[1]: Starting Load Kernel Module efi_pstore...
[   12.326644] systemd[1]: Starting Load Kernel Module pstore_blk...
[   12.335648] systemd[1]: Starting Load Kernel Module pstore_zone...
[   12.344718] systemd[1]: Starting Load Kernel Module ramoops...
[   12.353354] systemd[1]: Starting NVIDIA specific first-boot udev script...
[   12.363109] systemd[1]: Started Nameserver information manager.
[   12.371072] systemd[1]: Reached target Network (Pre).
[   12.378928] systemd[1]: Starting Set console scheme...
[   12.386484] systemd[1]: Condition check resulted in Set Up Additional Binary Formats being skipped.
[   12.396619] systemd[1]: Condition check resulted in File System Check on Root Device being skipped.
[   12.414385] systemd[1]: Starting Load Kernel Modules...
[   12.422255] systemd[1]: Starting Remount Root and Kernel File Systems...
[   12.430204] EXT4-fs (mmcblk0p1): re-mounted. Opts: (null)
[   12.430987] nvmap_heap_init: nvmap_heap_init: created heap block cache
[   12.444884] tegra-carveouts tegra-carveouts: fsi :dma coherent mem declare 0x0000000833000000,16777216
[   12.445710] systemd[1]: Starting udev Coldplug all Devices...
[   12.455272] tegra-carveouts tegra-carveouts: assigned reserved memory node fsi-carveout
[   12.471072] tegra-carveouts tegra-carveouts: vpr :dma coherent mem declare 0x0000000849800000,914358272
[   12.481535] tegra-carveouts tegra-carveouts: assigned reserved memory node vpr-carveout
[   12.481542] nvmap_page_pool_init: Total RAM pages: 7830522
[   12.497054] nvmap_page_pool_init: nvmap page pool size: 978815 pages (3823 MB)
[   12.497096] nvmap_background_zero_thread: PP zeroing thread starting.
[   12.497183] misc nvmap: created heap vpr base 0x0000000849800000 size (892928KiB)
[   12.500379] misc nvmap: created heap fsi base 0x0000000833000000 size (16384KiB)
[   12.532045] systemd[1]: Started Journal Service.
[   12.566860] systemd-journald[277]: Received client request to flush runtime journal.
[   12.586490] nvgpu: 17000000.ga10b          nvgpu_nvhost_syncpt_init:135  [INFO]  syncpt_unit_base 60000000 syncpt_unit_size 4000000 size 10000
[   12.586490] 
[   14.565230] random: crng init done
[   14.569405] random: 7 urandom warning(s) missed due to ratelimiting
[   14.940248] Internal error: SP/PC alignment exception: 8a000000 [#1] PREEMPT SMP
[   14.948415] Modules linked in: ina3221 pwm_fan nvgpu nvmap ip_tables x_tables
[   14.948429] CPU: 1 PID: 19 Comm: ksoftirqd/1 Not tainted 5.10.104-tegra #1
[   14.948430] Hardware name: Unknown Jetson AGX Orin/Jetson AGX Orin, BIOS 3.1-32827747 03/19/2023
[   14.948432] pstate: 60c00009 (nZCv daif +PAN +UAO -TCO BTYPE=--)
[   14.948437] pc : 0x570e6cb99c9325
[   14.948445] lr : rcu_core+0x274/0x980
[   14.948446] sp : ffff8000101bbc90
[   14.948447] x29: ffff8000101bbc90 x28: ffffb877a9607000 
[   14.948448] x27: ffff0b9bc02eba00 x26: ffff0b9bc02eba00 
[   14.948449] x25: ffff8000101bbd20 x24: ffff0ba2eeb45af0 
[   14.948451] x23: ffffb877a998ae40 x22: ffff0b9bc02eba00 
[   14.948452] x21: ffffb877a98f8998 x20: ffffb877a9607000 
[   14.948453] x19: ffff0ba2eeb45a80 x18: 0000000000000000 
[   14.948455] x17: 0000000000000000 x16: 0000000000000000 
[   14.948456] x15: 0000000000000000 x14: 0000000000000000 
[   14.948457] x13: 00000000fffee918 x12: ffffffffffffff51 
[   14.948458] x11: ffffb877a98f6000 x10: 0000000000000a80 
[   14.948460] x9 : ffff8000101bbd40 x8 : ffff0b9bc02eba00 
[   14.948461] x7 : 0000000000000020 x6 : ffff532b45529000 
[   14.948462] x5 : 0000000000000001 x4 : fffffe2e6f0e2ca0 
[   14.948464] x3 : 0000000080200004 x2 : ffff0b9bcb8b3800 
[   14.948465] x1 : ca570e6cb99c9325 x0 : ffff0b9bcb8b3800 
[   14.948467] Call trace:
[   14.948468]  0x570e6cb99c9325
[   14.948470]  rcu_core_si+0x18/0x20
[   14.948473]  __do_softirq+0x140/0x3e8
[   14.948478]  run_ksoftirqd+0x50/0x60
[   14.948481]  smpboot_thread_fn+0x1c4/0x280
[   14.948482]  kthread+0x148/0x170
[   14.948484]  ret_from_fork+0x10/0x24
[   14.948491] Code: bad PC value
[   14.948499] ---[ end trace f27e1fcd77ee3fe3 ]---
[   14.948501] Kernel panic - not syncing: SP/PC alignment exception: Fatal exception in interrupt
[   14.948502] SMP: stopping secondary CPUs
[   14.948510] Kernel Offset: 0x387797be0000 from 0xffff800010000000
[   14.948511] PHYS_OFFSET: 0xfffff46540000000
[   14.948512] CPU features: 0x0040006,4a80aa38
[   14.948513] Memory Limit: none
[   15.156947] ---[ end Kernel panic - not syncing: SP/PC alignment exception: Fatal exception in interrupt ]---
2

Hi,
If the device cannot be flashed/booted, please refer to the page to get uart log from the device:
Jetson/General debug - eLinux.org
And get logs of host PC and Jetson device for reference. If you are using custom board, you can compare uart log of developer kit and custom board to get more information.
Also please check FAQs:
Jetson AGX Orin FAQ
If possible, we would suggest follow quick start in developer guide to re-flash the system:
Quick Start — NVIDIA Jetson Linux Developer Guide 1 documentation
And see if the issue still persists on a clean-flashed system.
Thanks!

3

Hi carolyuu,

Thanks for your reply.

My debug log has attached above.

My JetPack is downloaded from NV websitde without any modification.
I have flash about 5 ~ 6 times. The error is same, so I think there should something wrong.

4

Hi,
The module may be with new PCN. Please try latest Jetpack 5.1.4 or 6.1. Multiple new PCNs are supported in latest versions.

5

Hi DaneLLL,
Maybe you’re right. I have flashed the JetPack5.1.3 and It works.

Do you know how to confirm this problem is due to the PCNs ?

Because we have released product on JetPack5.1.1 and my customers has applied them to the market.
If the new JetPack (newer than 5.1.3) can fix this problem. We also need to give the customers more precise information to persuade them to upgrade the JetPack to a newer version.

Many Thanks.

More information:
I have applied all the JetPack5.1.1 patch as below. They didn’t work and the error still exists.

微信截图_20241029092421
微信截图_20241029092421889×688 101 KB

6

Any update ?

8

I think you might check the module FAB to check if this is really affected by any of the PCN.

But not “apply all the PCN” when you don’t know which PCN really affect your module.

9

I have checked the PCNs by scan the QR code behind the module.

The result is: 1420523041424,48B02DDA6F69,10,699-13701-0004-500.

And I checked this PCN （699-13701-0004-500），Seems no PCN has effect on this module.

10

Then please try with latest BSP first if there is still such issue.

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.