Kernel Panic in kswapd0 (mem_cgroup_from_obj) on JetPack 6.2.2 R36.5.0

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
, ,
1

Kernel Panic in kswapd0 / mem_cgroup_from_obj on JetPack 6.2.2 (R36.5.0)

Environment

  • Board: NVIDIA Jetson AGX Orin Developer Kit

  • JetPack: 6.2.2

  • Jetson Linux: R36.5.0 (GCID: 43688277, DATE: Fri Jan 16 03:50:45 UTC 2026)

  • Kernel: 5.15.185-tegra

  • Ubuntu: 22.04

  • Power mode: nvpmodel -m 3 (50W)

  • apt upgrade: fully up to date

Symptom

System crashes and reboots suddenly during LLM model loading (Docker container, llama.cpp server, Gemma4-26B Q4_K_M model). Also occurs during apt upgrade.

  • No output on serial console during crash

  • journalctl -b -1 ends abruptly with no shutdown sequence

  • pstore/ramoops captures the kernel panic after reboot

  • Crash is intermittent (does not occur every time)

Additional Findings

  • console-ramoops-0 shows journal file corruption on every boot, confirming repeated unclean shutdowns

  • Crash occurs consistently ~519 seconds after boot

  • SMP: failed to stop secondary CPUs 0-1,4 at panic time

  • dmesg-ramoops-0 and dmesg-ramoops-1 contain identical panic (same crash recorded in two ramoops slots)

Reboot History (last reboot / Jun 6)

reboot   system boot  5.15.185-tegra   Sat Jun  6 21:51   still running
reboot   system boot  5.15.185-tegra   Sat Jun  6 18:20
reboot   system boot  5.15.185-tegra   Sat Jun  6 16:15
reboot   system boot  5.15.185-tegra   Sat Jun  6 14:53
reboot   system boot  5.15.185-tegra   Sat Jun  6 10:19
reboot   system boot  5.15.185-tegra   Sat Jun  6 09:07

Kernel Panic Log (/sys/fs/pstore/dmesg-ramoops-0)

<1>[  519.543956] Unable to handle kernel paging request at virtual address ffff004200007c68
<1>[  519.543970] Mem abort info:
<1>[  519.543972]   ESR = 0x0000000096000005
<1>[  519.543973]   EC = 0x25: DABT (current EL), IL = 32 bits
<1>[  519.543976]   SET = 0, FnV = 0
<1>[  519.543977]   EA = 0, S1PTW = 0
<1>[  519.543978]   FSC = 0x05: level 1 translation fault
<1>[  519.543979] Data abort info:
<1>[  519.543980]   ISV = 0, ISS = 0x00000005
<1>[  519.543981]   CM = 0, WnR = 0
<1>[  519.543982] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000752645000
<1>[  519.543985] [ffff004200007c68] pgd=180000082cd68003, p4d=180000082cd68003, pud=0000000000000000
<0>[  519.543993] Internal error: Oops: 0000000096000005 [#1] PREEMPT SMP
<6>[  519.544168] CPU: 4 PID: 163 Comm: kswapd0 Tainted: G           O      5.15.185-tegra #1
<6>[  519.544173] Hardware name: NVIDIA NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS 36.5.0-gcid-43688277 01/16/2026
<6>[  519.544175] pstate: 804000c9 (Nzcv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
<6>[  519.544179] pc : mem_cgroup_from_obj+0xb0/0x110
<6>[  519.544193] lr : mem_cgroup_from_obj+0x20/0x110
<6>[  519.544229] Call trace:
<6>[  519.544231]  mem_cgroup_from_obj+0xb0/0x110
<6>[  519.544234]  list_lru_add+0xa0/0x130
<6>[  519.544243]  workingset_update_node+0x94/0xb0
<6>[  519.544247]  xas_store+0x2b4/0x560
<6>[  519.544252]  __delete_from_page_cache+0x11c/0x210
<6>[  519.544256]  __remove_mapping+0x140/0x270
<6>[  519.544260]  shrink_page_list+0x64c/0xf60
<6>[  519.544263]  shrink_lruvec+0x528/0xaa0
<6>[  519.544265]  shrink_node+0x3d0/0x770
<6>[  519.544268]  balance_pgdat+0x330/0x6f0
<6>[  519.544271]  kswapd+0x234/0x510
<6>[  519.544273]  kthread+0x184/0x1a0
<6>[  519.544278]  ret_from_fork+0x10/0x20
<0>[  519.544284] Code: 0b010000 1ac32400 f8607840 b4000120 (f9400800)
<4>[  519.544290] ---[ end trace f469f984dd0e6bbb ]---
<0>[  520.554783] Kernel panic - not syncing: Oops: Fatal exception
<2>[  521.559689] SMP: stopping secondary CPUs
<4>[  522.618180] SMP: failed to stop secondary CPUs 0-1,4
<0>[  522.618187] Kernel Offset: 0x32f110240000 from 0xffff800008000000
<0>[  522.618189] PHYS_OFFSET: 0x80000000
<0>[  522.618190] CPU features: 0x0,000005c1,a3332e5a
<0>[  522.618193] Memory Limit: none

console-ramoops-0 (boot log at crash time, relevant excerpt)

[    0.000000] Linux version 5.15.185-tegra ... #1 SMP PREEMPT Thu Jan 15 19:24:38 PST 2026
[    8.371266] systemd-journald[278]: File /var/log/journal/.../system.journal
               corrupted or uncleanly shut down, renaming and replacing.

journalctl -b -1 (last line before crash)

Jun 06 23:17:47 jetson-agx-orin NetworkManager[579]: dhcp4 (wlP1p1s0): state changed new lease

-> No shutdown/panic message. Log ends abruptly.

What was ruled out

  • NOT thermal shutdown (no thermal errors in dmesg)

  • NOT OOM killer (no OOM messages)

  • NOT power supply issue (serial output present until panic, no voltage warnings)

  • NOT storage full (/ partition: 25GB used / 57GB total)

  • apt fully up to date (JetPack 6.2.2 / R36.5.0 is latest)

Similar report

Question

Is this a known bug in kernel 5.15.185-tegra (R36.5.0)? Is there a fix or workaround available for the mem_cgroup_from_obj / kswapd0 kernel panic under memory pressure?

4

Hi g6.67300,

Based on the panic trace, the crash appears to happen in the kernel memory reclaim path (kswapd0), specifically around mem_cgroup_from_obj() while the system is under memory pressure.
In other words, the failure does not seem to point directly to the LLM application itself, but to a kernel-side page reclaim / memory-accounting path being exercised during heavy memory activity.

Since the panic is seen during both model loading and apt upgrade, it looks more related to memory pressure triggering this reclaim path than to one specific user-space workload.
As a next step, it would be useful to confirm whether this reproduces on a clean stock JP6.2.2 setup, without any out-of-tree kernel modules, and also whether it reproduces on the bare AGX Orin without Docker memory limits.

5

Hi KevinFFF,

Thank you for the analysis. Here is what I have found so far.

Regarding Docker memory limits:
I have experienced freezes with llama.cpp running outside of Docker as well (multiple occurrences). However, since I did not have ramoops capture set up at the time, I cannot confirm whether those were the same kswapd0 panic. I now have the capture commands ready for the next occurrence.

Regarding out-of-tree kernel modules:
The only out-of-tree modules present are the Realtek Wi-Fi (rtl8822ce) and Bluetooth (rtk_btusb) drivers:

/lib/modules/5.15.185-tegra/updates/drivers/net/wireless/realtek/rtl8822ce/rtl8822ce.ko
/lib/modules/5.15.185-tegra/updates/drivers/bluetooth/realtek/rtk_btusb.ko

Both files have a timestamp of Jan 16 03:42, which matches the JP6.2.2 / R36.5.0 build date (Jan 16 03:50 UTC). They are not managed by dpkg or DKMS (dkms is not installed), so I believe these were placed during the initial JetPack flash rather than built manually.

Regarding intentional reproduction:
Deliberate reproduction is difficult in practice. The crash occurs during model loading (Gemma4-26B Q4_K_M via llama.cpp), which takes considerable time per attempt, and the failure rate is low. I am not actively stress-testing, but I have the ramoops save commands prepared and will capture the logs if it occurs again during normal use.

6

Thanks for the update.
Since you have also seen similar freezes outside Docker, the issue may not be tied only to Docker memory limits.
Without ramoops from those earlier events, we cannot confirm whether they were the same kswapd0 panic.

For now, please capture the next dmesg-ramoops / console-ramoops logs and lsmod output so we can check whether the same kernel path is being hit again.