Kernel Panic Related to NVGPU During Application Runtime

We encountered a kernel panic that appears to be related to NVGPU.

While our control application was running, the system suddenly shut down and reported a kernel panic. Before the shutdown, I noticed a large number of NVGPU-related errors in the system logs.

So far, this issue has occurred only once. The relevant logs are attached below.

[  543.098378] nvgpu: 17000000.gpu      gr_init_ctxsw_falcon_support:857  [ERR]  FECS context switch init error
[  543.098380] nvgpu: 17000000.gpu            nvgpu_finalize_poweron:1095 [ERR]  Failed initialization for: g->ops.gr.gr_init_support
[  543.152252] nvgpu: 17000000.gpu                 gk20a_power_write:127  [ERR]  power_node_write failed at busy
[  543.154349] Unable to handle kernel paging request at virtual address ffff58a811390000
[  543.154356] Mem abort info:
[  543.154357]   ESR = 0x0000000096000004
[  543.154358]   EC = 0x25: DABT (current EL), IL = 32 bits
[  543.154360]   SET = 0, FnV = 0
[  543.154361]   EA = 0, S1PTW = 0
[  543.154361]   FSC = 0x04: level 0 translation fault
[  543.154362] Data abort info:
[  543.154363]   ISV = 0, ISS = 0x00000004
[  543.154364]   CM = 0, WnR = 0
[  543.154364] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000508137000
[  543.154366] [ffff58a811390000] pgd=0000000000000000, p4d=0000000000000000
[  543.154371] Internal error: Oops: 0000000096000004 [#1] PREEMPT_RT SMP
[  543.154374] Modules linked in: nvidia_modeset(O) sl_zedx(OE) sl_max9295(OE) nf_tables nfnetlink sl_max96712(OE) nvme_fabrics option usb_wwan usbserial wlan(OE) lzo_rle lzo_compress zram zsmalloc cfg80211(E) can_gw can_bcm can_raw can ramoops reed_solomon bridge stp llc usb_f_ncm usb_f_mass_storage usb_f_acm u_serial usb_f_rndis u_ether libcomposite snd_soc_tegra210_admaif(O) snd_soc_tegra186_asrc(O) snd_soc_tegra210_afc(O) snd_soc_tegra186_arad(O) snd_soc_tegra_pcm snd_soc_tegra210_mixer(O) snd_soc_tegra210_mvc(O) snd_soc_tegra210_ope(O) snd_soc_tegra186_dspk(O) snd_soc_tegra210_dmic(O) snd_soc_tegra210_adx(O) snd_soc_tegra210_amx(O) snd_soc_tegra210_sfc(O) snd_soc_tegra210_i2s(O) wlan_cnss_core_pcie(OE) igb nvme i2c_algo_bit nvme_core tegra_pcie_dma_test(O) tegra_pcie_edma(O) xhci_pci xhci_pci_renesas snd_soc_tegra210_ahub(O) tegra210_adma spidev nvvrs_pseq_rtc(O) ip6table_filter ip6table_nat ip6_tables xt_MASQUERADE xt_nat xt_addrtype iptable_nat nf_nat xt_tcpudp xt_conntrack
[  543.154430]  nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter snd_soc_tegra_machine_driver(O) crct10dif_ce snd_soc_tegra_utils(O) snd_soc_simple_card_utils tegra234_oc_event(O) tegra23x_perf_uncore(O) mttcan(O) nvpmodel_clk_cap(O) tegra_mce(O) thermal_trip_event(O) tegra_cactmon_mc_all(O) can_dev tegra234_aon(O) nvethernet(O) tegra_aon_ivc_gmsl_config(O) tegra_aon_ivc_gmsl_timestamp(O) tegra_aon_ivc_echo(O) tegra_aconnect uvcvideo nvpps(O) videobuf2_vmalloc snd_hda_codec_hdmi snd_hda_tegra phy_tegra194_p2u snd_hda_codec pwm_tegra_tachometer(O) tegra_xudc snd_hda_core snd_soc_rt5640 snd_soc_rl6231 spi_tegra114 bmi088(O) at24 mc_hwpm(O) nvidia(O) pcie_tegra194 lm90 i2c_nvvrs11(O) host1x_fence(O) nvidia_vrs_pseq(O) tegra_dce(O) nvhost_isp5(O) nvhost_nvcsi_t194(O) nvhost_vi5(O) tegra_camera(O) v4l2_dv_timings nvhost_nvcsi(O) tegra_camera_platform(O) capture_ivc(O) tegra_camera_rtcpu(O) ivc_bus(O) hsp_mailbox_client(O) ivc_ext(O) v4l2_fwnode v4l2_async videobuf2_dma_contig
[  543.154476]  videobuf2_memops videobuf2_v4l2 videobuf2_common governor_userspace videodev mc nvhost_pva(O) tegra_drm(O) nvhost_capture(O) nvhost_nvdla(O) host1x_nvhost(O) tegra_wmark(O) tsecriscv(O) nvhwpm(O) cec drm_kms_helper tegra_se(O) crypto_engine nvidia_p2p(O) ina3221 nvgpu(O) governor_pod_scaling(O) host1x(O) mc_utils(O) nvmap(O) drm nvsciipc(O) fuse ip_tables x_tables ipv6 pwm_fan pwm_tegra tegra_bpmp_thermal [last unloaded: sl_max96712]
[  543.154505] CPU: 5 PID: 52 Comm: rcuc/5 Tainted: G        W  OE     5.15.148-rt-tegra #1 66cd28412c4c02a037f0598c853664914a19e276
[  543.154509] Hardware name: NVIDIA NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS 36.4.3-gcid-38968081 01/08/2025
[  543.154511] pstate: 40400009 (nZcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  543.154513] pc : percpu_ref_get_many+0x3c/0xd0
[  543.154520] lr : percpu_ref_get_many+0x18/0xd0
[  543.154521] sp : ffff8000084f3b60
[  543.154522] x29: ffff8000084f3b60 x28: ffff000fa2c3da38 x27: ffff0000c0000000
[  543.154524] x26: ffff00008000d200 x25: 0000000000000001 x24: ffff8000084f3b80
[  543.154526] x23: ffff0000be375980 x22: ffffa767918aa670 x21: ffff0000c0000000
[  543.154528] x20: ffff0000c0000000 x19: 0000000000000001 x18: 0000000000000000
[  543.154530] x17: ffff58a811390000 x16: ffffa76790b880f0 x15: 0000000000000000
[  543.154532] x14: 00000000000000f3 x13: 0000000000000024 x12: 0000000080000000
[  543.154534] x11: 0000000000000040 x10: ffffa76791e62320 x9 : 000000047c9d227e
[  543.154536] x8 : 000000000000105d x7 : 000000000000166b x6 : ffff58a811390000
[  543.154537] x5 : fffffffffffff7d0 x4 : ffff0000bc571000 x3 : ffffa7679189b008
[  543.154539] x2 : ffff58a811390000 x1 : ffff000080521f00 x0 : ffff58a811390000
[  543.154541] Call trace:
[  543.154543]  percpu_ref_get_many+0x3c/0xd0
[  543.154544]  refill_obj_stock+0x7c/0x1b0
[  543.154547]  obj_cgroup_uncharge+0x30/0x40
[  543.154550]  memcg_slab_free_hook+0x90/0x160
[  543.154554]  kmem_cache_free+0x2d0/0x320
[  543.154557]  put_pid.part.0+0x88/0xc0
[  543.154560]  delayed_put_pid+0x28/0x40
[  543.154561]  rcu_core+0x288/0x920
[  543.154564]  rcu_cpu_kthread+0xc0/0x130
[  543.154566]  smpboot_thread_fn+0x2d8/0x320
[  543.154568]  kthread+0x194/0x1b0
[  543.154570]  ret_from_fork+0x10/0x20
[  543.154574] Code: 11000442 b9001822 d53cd042 8b020000 (f833001f) 
[  543.154577] ---[ end trace 0000000000000004 ]---
[  543.181196] tegra-hsierrrptinj: Callback for 0x0001 already registered
[  543.181204] nvgpu: 17000000.gpu                nvgpu_cic_mon_init:75   [ERR]  Err inj callback registration failed: -22
[  543.250331] nvgpu: 17000000.gpu                nvgpu_pmu_cmd_post:600  [ERR]  FBQ cmd setup failed
[  543.250350] nvgpu: 17000000.gpu             nvgpu_pmu_rpc_execute:727  [ERR]  Failed to execute RPC status=0xfffffff4, func=0x3
[  543.250353] nvgpu: 17000000.gpu gv100_pmu_lsfm_bootstrap_ls_falcon:100  [ERR]  Failed to execute RPC, status=0xfffffff4

console-ramoops-0.log (512.0 KB)

journal.log (83.6 MB)

Any assistance, troubleshooting suggestions, or guidance would be greatly appreciated.

— 🤖 This is an automated AI response. I’m here to help, but please verify important details! —

Based on the provided context, I could not find clear support for that in the provided Jetson AGX Orin documentation. However, I can suggest some general troubleshooting steps that may help resolve the issue.

  1. Check the system logs for any error messages related to NVGPU or kernel panic.
  2. Verify that the NVGPU driver is properly installed and configured.
  3. Check the power management settings to ensure that the system is not experiencing any power-related issues.
  4. Try to reproduce the issue by running the control application again and see if the kernel panic occurs consistently.
  5. If the issue persists, try to gather more information about the kernel panic by running the dmesg command and checking the system logs for any error messages.

It’s also worth noting that the kernel panic message mentions a “FECS context switch init error” and a “power_node_write failed” error, which may indicate a hardware or power-related issue. If you’re still experiencing issues after trying the above steps, it may be helpful to consult with NVIDIA support or a qualified engineer for further assistance.

— We support team will also look into your issue to have further suggestions soon. please stay tuned. —

In addition, we observed another abnormal issue on the same machine today. When running the dmesg command, it terminated with a Segmentation fault. Unfortunately, the machine was manually powered off shortly afterward, so we were unable to collect any additional diagnostic information. There are also no corresponding records in journalctl for the time when this occurred.

There is also the following log excerpt:

Jul 11 16:02:57 30C-ubuntu kernel: EXT4-fs (mmcblk0p1): error count since last fsck: 67
Jul 11 16:02:57 30C-ubuntu kernel: EXT4-fs (mmcblk0p1): initial error at time 1783649481: ext4_validate_block_bitmap:420
Jul 11 16:02:57 30C-ubuntu kernel: EXT4-fs (mmcblk0p1): last error at time 1783754296: ext4_validate_block_bitmap:420

Which Jetpack version is in use here? There are critical patch mentioned here that you might need to apply.