Hi I am constantly having issues like the orin dev kit shutting down because of this kernel panic error
<1>[108613.089703] Unable to handle kernel paging request at virtual address ffff4c8d52afe000
<1>[108613.089714] Mem abort info:
<1>[108613.089715] ESR = 0x0000000096000004
<1>[108613.089717] EC = 0x25: DABT (current EL), IL = 32 bits
<1>[108613.089720] SET = 0, FnV = 0
<1>[108613.089721] EA = 0, S1PTW = 0
<1>[108613.089721] FSC = 0x04: level 0 translation fault
<1>[108613.089723] Data abort info:
<1>[108613.089723] ISV = 0, ISS = 0x00000004
<1>[108613.089724] CM = 0, WnR = 0
<1>[108613.089726] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000121656000
<1>[108613.089729] [ffff4c8d52afe000] pgd=0000000000000000, p4d=0000000000000000
<0>[108613.089735] Internal error: Oops: 0000000096000004 [#1] PREEMPT_RT SMP
<6>[108613.089739] Modules linked in: xt_tcpudp(E) xt_mark(E) xt_conntrack(E) xt_MASQUERADE(E) nf_conntrack_netlink(E) xt_addrtype(E) br_netfilter(E) nvidia_modeset(OE) can_raw(E) can(E) lzo_rle(E) lzo_compress(E) zram(E) ec_generic(OE) zsmalloc(E) ec_master(OE) ramoops(E) reed_solomon(E) nf_tables(E) nfnetlink(E) ip6table_nat(E) ip6table_filter(E) ip6_tables(E) iptable_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) libcrc32c(E) iptable_filter(E) algif_hash(E) algif_skcipher(E) af_alg(E) snd_soc_tegra210_admaif(OE) snd_soc_tegra186_asrc(OE) snd_soc_tegra186_arad(OE) snd_soc_tegra210_afc(OE) snd_soc_tegra210_mixer(OE) snd_soc_tegra_pcm(E) snd_soc_tegra210_mvc(OE) snd_soc_tegra210_ope(OE) snd_soc_tegra186_dspk(OE) snd_soc_tegra210_dmic(OE) snd_soc_tegra210_amx(OE) snd_soc_tegra210_sfc(OE) snd_soc_tegra210_adx(OE) snd_soc_tegra210_i2s(OE) snd_soc_tegra210_ahub(OE) tegra210_adma(E) spidev(E) nvvrs_pseq_rtc(OE) gs_usb(E) can_dev(E) rtk_btusb(OE) btusb(E) btrtl(E) btintel(E)
<6>[108613.089804] btbcm(E) bluetooth(E) ecdh_generic(E) ecc(E) snd_soc_tegra_machine_driver(OE) snd_soc_tegra_utils(OE) snd_soc_simple_card_utils(E) crct10dif_ce(E) tegra234_oc_event(OE) r8125(OE) fusb301(OE) nvpmodel_clk_cap(OE) thermal_trip_event(OE) tegra_cactmon_mc_all(OE) tegra234_aon(OE) tegra_aconnect(E) r8168(OE) rtl8822ce(OE) snd_hda_codec_hdmi(E) snd_hda_tegra(E) cfg80211(E) snd_hda_codec(E) pwm_tegra_tachometer(OE) nvidia(OE) nv_imx219(OE) snd_hda_core(E) at24(E) spi_tegra114(E) mc_hwpm(OE) tegra_pcie_dma_test(OE) nvidia_vrs_pseq(OE) rfkill(E) host1x_fence(OE) tegra_pcie_edma(OE) tegra_dce(OE) tsecriscv(OE) bridge(E) stp(E) llc(E) usb_f_ncm(E) usb_f_mass_storage(E) usb_f_acm(E) u_serial(E) usb_f_rndis(E) u_ether(E) nvhost_vi5(OE) nvhost_nvcsi_t194(OE) nvhost_isp5(OE) libcomposite(E) tegra_camera(OE) v4l2_dv_timings(E) nvhost_nvcsi(OE) tegra_camera_platform(OE) capture_ivc(OE) governor_userspace(E) tegra_camera_rtcpu(OE) ivc_bus(OE) tegra_drm(OE) hsp_mailbox_client(OE) ivc_ext(OE)
<6>[108613.089870] v4l2_fwnode(E) v4l2_async(E) videobuf2_dma_contig(E) videobuf2_memops(E) videobuf2_v4l2(E) videobuf2_common(E) tegra_wmark(OE) videodev(E) nvhwpm(OE) cec(E) mc(E) nvhost_capture(OE) drm_kms_helper(E) host1x_nvhost(OE) nvidia_p2p(OE) ina3221(E) nvgpu(OE) governor_pod_scaling(OE) host1x(OE) mc_utils(OE) nvmap(OE) nvsciipc(OE) drm(E) fuse(E) ip_tables(E) x_tables(E) ipv6(E) pwm_fan pwm_tegra tegra_bpmp_thermal tegra_xudc ucsi_ccg typec_ucsi typec nvme nvme_core phy_tegra194_p2u pcie_tegra194
<6>[108613.089915] CPU: 5 PID: 162 Comm: kswapd0 Tainted: G OE 5.15.136-rt-tegra #1
<6>[108613.089919] Hardware name: NVIDIA NVIDIA Jetson Orin Nano Developer Kit/Jetson, BIOS 36.3.0-gcid-36191598 05/06/2024
<6>[108613.089921] pstate: 40400009 (nZcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
<6>[108613.089925] pc : percpu_ref_get_many+0x40/0xd0
<6>[108613.089937] lr : percpu_ref_get_many+0x1c/0xd0
<6>[108613.089940] sp : ffff800008fc3510
<6>[108613.089941] x29: ffff800008fc3510 x28: 0000000000000001 x27: ffff00009d8a902f
<6>[108613.089945] x26: ffff0000800d0a00 x25: 0000000000000001 x24: ffff800008fc3530
<6>[108613.089948] x23: ffff00009d8a6200 x22: ffffb3749e7da670 x21: ffff00009d8a902f
<6>[108613.089950] x20: ffff00009d8a902f x19: 0000000000000001 x18: 0000000000000000
<6>[108613.089953] x17: 0000000000000000 x16: ffffb3749dae7ea0 x15: 0000aaaacadd70b0
<6>[108613.089956] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000080000000
<6>[108613.089958] x11: ffff800008fc36c8 x10: 0000000000000002 x9 : 0000000000000001
<6>[108613.089961] x8 : 0000000000000238 x7 : 0000000000000000 x6 : ffff4c8d52afe000
<6>[108613.089963] x5 : fffffffffffff820 x4 : ffff00008a487000 x3 : 000000000000000c
<6>[108613.089966] x2 : ffff4c8d52afe000 x1 : ffff000080e99f00 x0 : ffff4c8d52afe000
<6>[108613.089969] Call trace:
<6>[108613.089971] percpu_ref_get_many+0x40/0xd0
<6>[108613.089974] refill_obj_stock+0x80/0x1c0
<6>[108613.089979] obj_cgroup_uncharge+0x34/0x50
<6>[108613.089983] memcg_slab_free_hook+0x94/0x160
<6>[108613.089989] kmem_cache_free+0x2d8/0x320
<6>[108613.089991] free_buffer_head+0x44/0xc0
<6>[108613.089996] try_to_free_buffers+0xbc/0x140
<6>[108613.089998] jbd2_journal_try_to_free_buffers+0x120/0x150
<6>[108613.090003] ext4_releasepage+0x58/0x110
<6>[108613.090007] try_to_release_page+0x80/0xb0
<6>[108613.090012] invalidate_inode_page+0xb0/0xd0
<6>[108613.090017] __invalidate_mapping_pages+0x188/0x1e0
<6>[108613.090021] invalidate_mapping_pages+0x40/0x60
<6>[108613.090025] inode_lru_isolate+0x1e0/0x240
<6>[108613.090028] __list_lru_walk_one+0x88/0x150
<6>[108613.090032] list_lru_walk_one+0x6c/0xa0
<6>[108613.090034] prune_icache_sb+0x60/0xd0
<6>[108613.090036] super_cache_scan+0x15c/0x1c0
<6>[108613.090038] do_shrink_slab+0x180/0x490
<6>[108613.090041] shrink_slab+0x214/0x350
<6>[108613.090043] shrink_node+0x3c4/0x760
<6>[108613.090045] balance_pgdat+0x2e8/0x6a0
<6>[108613.090047] kswapd+0x1f8/0x480
<6>[108613.090049] kthread+0x198/0x1b0
<6>[108613.090054] ret_from_fork+0x10/0x20
<0>[108613.090060] Code: 11000442 b9001822 d53cd042 8b020000 (f833001f)
<4>[108613.090067] ---[ end trace 0000000000000002 ]---
My kernel version is: 5.15.136-rt-tegra
HW used- Jetson orin nano devkit
OS version- Ubuntu 22.04
Hi 0e355f7dd4ed74e0f4b2704d4,
I’ve moved your topic to the correct category for Orin Nano.
What’s the Jetpack version in use?
Do you have any customization in kernel which may cause this panic?
1 Like
Jetpack version:
sudo apt-cache show nvidia-jetpack
[sudo] password for cobot:
Package: nvidia-jetpack
Source: nvidia-jetpack (6.0)
Version: 6.0+b106
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-jetpack-runtime (= 6.0+b106), nvidia-jetpack-dev (= 6.0+b106)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_6.0+b106_arm64.deb
Size: 29296
SHA256: 561d38f76683ff865e57b2af41e303be7e590926251890550d2652bdc51401f8
SHA1: ef3fca0c1b5c780b2bad1bafae6437753bd0a93f
MD5sum: 95de21b4fce939dee11c6df1f2db0fa5
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8
Package: nvidia-jetpack
Source: nvidia-jetpack (6.0)
Version: 6.0+b87
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-jetpack-runtime (= 6.0+b87), nvidia-jetpack-dev (= 6.0+b87)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_6.0+b87_arm64.deb
Size: 29298
SHA256: 70be95162aad864ee0b0cd24ac8e4fa4f131aa97b32ffa2de551f1f8f56bc14e
SHA1: 36926a991855b9feeb12072694005c3e7e7b3836
MD5sum: 050cb1fd604a16200d26841f8a59a038
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8
About kernel customization, we have following things added:
Install a non-stock EtherCAT stack from source (not just apt default).
Build EtherCAT kernel modules (make modules) and install them (make modules_install).
Run depmod after EtherCAT module install.
Force EtherCAT master binding to base NIC by setting MASTER0_DEVICE to eth0 MAC.
Force EtherCAT to use the generic driver path (DEVICE_MODULES=“generic”).
Add udev rule KERNEL==“EtherCAT[0-9]*”, MODE=“0666” for device permissions.
Enable and auto-start ethercat.service on boot.
Modify ethercat.service ordering to require/start after network-online.target.
Build and install out-of-tree Realtek r8125 NIC kernel module against target kernel headers.
Package that r8125 module into flashed rootfs/initrd flow.
Blacklist rtl8822ce kernel module on base.
Explicitly unload blacklisted modules if already loaded.
I think we have one more usb to can driver which is added while making the flashing image but other then that we dont have any major out of tree changes.
This are the only modifications which I dont even think are kernel level mods
Steps to reproduce-
This isnt something that we can reproduce because this is a long running system which runs for 8 hrs and the issue itself is not something we can reproduce without having deep knowledge of the linux kernel, so this is the best I can give you
The panic is a kernel memory‑corruption crash : kswapd0 is freeing ext4 pages, goes through kmem_cache_free → memcg_slab_free_hook → obj_cgroup_uncharge → percpu_ref_get_many, and there it dereferences a completely invalid pointer (ffff4c8d52afe000, level‑0 translation fault). That means some earlier code has already corrupted or freed the memcg/objcg/percpu_ref data , and the generic memory‑reclaim path is just where it finally explodes.
I have noticed that you are using rt-kernel.
Please also provide the full log as file here for further check.
I will checkout the packaging of ethercat module and rt kernel packaging, give me some time. Meanwhile here are the full logs
2026-02-27T19-19-27__console-ramoops-0.txt (512.0 KB)
2026-02-27T19-16-45__dmesg-ramoops-0.txt (106.6 KB)
2026-02-27T19-16-46__dmesg-ramoops-1.txt (106.6 KB)
<0>[108613.090060] Code: 11000442 b9001822 d53cd042 8b020000 (f833001f)
<4>[108613.090067] ---[ end trace 0000000000000002 ]---
<6>[108613.090595] EtherCAT 0: Domain 0: Working counter changed to 0/33.
<0>[108613.599168] Kernel panic - not syncing:
It seems the kernel panic is caused from your EtherCAT module.
You can also check if there’s the similar issue if you don’t enable rt-kernel.
1 Like
Thats insightful, and I am checking my kernel driver config as we speak, may I know though that how can something in my kernel driver cause a kernel panic?
Especially if you see the calltrace it points to an issue in the linux kernel and not in the kernel driver itself, if it was actually the ethercat driver wouldnt the call trace have function/API/ABI calls related to that? Just a curious question before I start rooting apart completely the ethercat driver
I checked the build workflow I have answers on it with a few caveats-
The ethercat driver is loaded as a kernel module and is not part of the base kernel image.
If you are asking, that was the ethercat module built with sysroot as the rt kernel of jetson, yes that does happen,
If you are asking the ethercat driver itself has rt-enabled flag, I dont think so
Hi @KevinFFF can anyone respond
0e355f7dd4ed74e0f4b2704d4:
If you are asking the ethercat driver itself has rt-enabled flag, I dont think so
As you are using the rt kernel, please build your driver with rt-enabled.
$ modinfo <EtherCAT module> | egrep 'vermagic'
Or, please check if there’s the similar issue if you use normal kernel.
Hi @KevinFFF Here is the grep which I assume says that the module was compiled with rt kernel as sysroot
$ modinfo ec_masterfp@igh-essen.com fp@igh-essen.com
Is ec_master your kernel module for EtherCAT?
Could you also help to check this?
1 Like
Yes ec_master and ec_generic both are compiled with preempt-rt. I am guessing that is nominal in your perspective for the 5.15.136 rt tegra kernel?
About the non-rt run, I can check it out, but a lot of our userspace applications need preempt-rt feats and cannot work without that. Those userspace applications are responsible for querying this ec_master kernel module and have sched_rr and rtprio set, w/o that we loose the rt behavior of those apps.
Yes, preempt_rt flag is expected for the rt-kernel.ec_master and ec_generic are the custom kernel drivers for EtherCAT, please also ask the help from your vendor.
0e355f7dd4ed74e0f4b2704d4:
About the non-rt run, I can check it out, but a lot of our userspace applications need preempt-rt feats and cannot work without that.
Okay, it is just helping to clarify if the current kernel panic is caused from rt-kernel.
Yes but those drivers arent the one causing a kernel crash, something in kswapd is causing one? Can you answer this @KevinFFF
Even though the crash happens in kswapd, the trace shows it’s using a completely invalid internal pointer, which strongly indicates earlier memory corruption. With your out‑of‑tree modules (ec_master, ec_generic) loaded, the most likely root cause is a custom driver corrupting kernel memory, and kswapd just happens to be where it finally panics.
system
April 8, 2026, 2:43am
23
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.