We encountered a kernel panic that appears to be related to NVGPU.
While our control application was running, the system suddenly shut down and reported a kernel panic. Before the shutdown, I noticed a large number of NVGPU-related errors in the system logs.
So far, this issue has occurred only once. The relevant logs are attached below.
[ 543.098378] nvgpu: 17000000.gpu gr_init_ctxsw_falcon_support:857 [ERR] FECS context switch init error
[ 543.098380] nvgpu: 17000000.gpu nvgpu_finalize_poweron:1095 [ERR] Failed initialization for: g->ops.gr.gr_init_support
[ 543.152252] nvgpu: 17000000.gpu gk20a_power_write:127 [ERR] power_node_write failed at busy
[ 543.154349] Unable to handle kernel paging request at virtual address ffff58a811390000
[ 543.154356] Mem abort info:
[ 543.154357] ESR = 0x0000000096000004
[ 543.154358] EC = 0x25: DABT (current EL), IL = 32 bits
[ 543.154360] SET = 0, FnV = 0
[ 543.154361] EA = 0, S1PTW = 0
[ 543.154361] FSC = 0x04: level 0 translation fault
[ 543.154362] Data abort info:
[ 543.154363] ISV = 0, ISS = 0x00000004
[ 543.154364] CM = 0, WnR = 0
[ 543.154364] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000508137000
[ 543.154366] [ffff58a811390000] pgd=0000000000000000, p4d=0000000000000000
[ 543.154371] Internal error: Oops: 0000000096000004 [#1] PREEMPT_RT SMP
[ 543.154374] Modules linked in: nvidia_modeset(O) sl_zedx(OE) sl_max9295(OE) nf_tables nfnetlink sl_max96712(OE) nvme_fabrics option usb_wwan usbserial wlan(OE) lzo_rle lzo_compress zram zsmalloc cfg80211(E) can_gw can_bcm can_raw can ramoops reed_solomon bridge stp llc usb_f_ncm usb_f_mass_storage usb_f_acm u_serial usb_f_rndis u_ether libcomposite snd_soc_tegra210_admaif(O) snd_soc_tegra186_asrc(O) snd_soc_tegra210_afc(O) snd_soc_tegra186_arad(O) snd_soc_tegra_pcm snd_soc_tegra210_mixer(O) snd_soc_tegra210_mvc(O) snd_soc_tegra210_ope(O) snd_soc_tegra186_dspk(O) snd_soc_tegra210_dmic(O) snd_soc_tegra210_adx(O) snd_soc_tegra210_amx(O) snd_soc_tegra210_sfc(O) snd_soc_tegra210_i2s(O) wlan_cnss_core_pcie(OE) igb nvme i2c_algo_bit nvme_core tegra_pcie_dma_test(O) tegra_pcie_edma(O) xhci_pci xhci_pci_renesas snd_soc_tegra210_ahub(O) tegra210_adma spidev nvvrs_pseq_rtc(O) ip6table_filter ip6table_nat ip6_tables xt_MASQUERADE xt_nat xt_addrtype iptable_nat nf_nat xt_tcpudp xt_conntrack
[ 543.154430] nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter snd_soc_tegra_machine_driver(O) crct10dif_ce snd_soc_tegra_utils(O) snd_soc_simple_card_utils tegra234_oc_event(O) tegra23x_perf_uncore(O) mttcan(O) nvpmodel_clk_cap(O) tegra_mce(O) thermal_trip_event(O) tegra_cactmon_mc_all(O) can_dev tegra234_aon(O) nvethernet(O) tegra_aon_ivc_gmsl_config(O) tegra_aon_ivc_gmsl_timestamp(O) tegra_aon_ivc_echo(O) tegra_aconnect uvcvideo nvpps(O) videobuf2_vmalloc snd_hda_codec_hdmi snd_hda_tegra phy_tegra194_p2u snd_hda_codec pwm_tegra_tachometer(O) tegra_xudc snd_hda_core snd_soc_rt5640 snd_soc_rl6231 spi_tegra114 bmi088(O) at24 mc_hwpm(O) nvidia(O) pcie_tegra194 lm90 i2c_nvvrs11(O) host1x_fence(O) nvidia_vrs_pseq(O) tegra_dce(O) nvhost_isp5(O) nvhost_nvcsi_t194(O) nvhost_vi5(O) tegra_camera(O) v4l2_dv_timings nvhost_nvcsi(O) tegra_camera_platform(O) capture_ivc(O) tegra_camera_rtcpu(O) ivc_bus(O) hsp_mailbox_client(O) ivc_ext(O) v4l2_fwnode v4l2_async videobuf2_dma_contig
[ 543.154476] videobuf2_memops videobuf2_v4l2 videobuf2_common governor_userspace videodev mc nvhost_pva(O) tegra_drm(O) nvhost_capture(O) nvhost_nvdla(O) host1x_nvhost(O) tegra_wmark(O) tsecriscv(O) nvhwpm(O) cec drm_kms_helper tegra_se(O) crypto_engine nvidia_p2p(O) ina3221 nvgpu(O) governor_pod_scaling(O) host1x(O) mc_utils(O) nvmap(O) drm nvsciipc(O) fuse ip_tables x_tables ipv6 pwm_fan pwm_tegra tegra_bpmp_thermal [last unloaded: sl_max96712]
[ 543.154505] CPU: 5 PID: 52 Comm: rcuc/5 Tainted: G W OE 5.15.148-rt-tegra #1 66cd28412c4c02a037f0598c853664914a19e276
[ 543.154509] Hardware name: NVIDIA NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS 36.4.3-gcid-38968081 01/08/2025
[ 543.154511] pstate: 40400009 (nZcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 543.154513] pc : percpu_ref_get_many+0x3c/0xd0
[ 543.154520] lr : percpu_ref_get_many+0x18/0xd0
[ 543.154521] sp : ffff8000084f3b60
[ 543.154522] x29: ffff8000084f3b60 x28: ffff000fa2c3da38 x27: ffff0000c0000000
[ 543.154524] x26: ffff00008000d200 x25: 0000000000000001 x24: ffff8000084f3b80
[ 543.154526] x23: ffff0000be375980 x22: ffffa767918aa670 x21: ffff0000c0000000
[ 543.154528] x20: ffff0000c0000000 x19: 0000000000000001 x18: 0000000000000000
[ 543.154530] x17: ffff58a811390000 x16: ffffa76790b880f0 x15: 0000000000000000
[ 543.154532] x14: 00000000000000f3 x13: 0000000000000024 x12: 0000000080000000
[ 543.154534] x11: 0000000000000040 x10: ffffa76791e62320 x9 : 000000047c9d227e
[ 543.154536] x8 : 000000000000105d x7 : 000000000000166b x6 : ffff58a811390000
[ 543.154537] x5 : fffffffffffff7d0 x4 : ffff0000bc571000 x3 : ffffa7679189b008
[ 543.154539] x2 : ffff58a811390000 x1 : ffff000080521f00 x0 : ffff58a811390000
[ 543.154541] Call trace:
[ 543.154543] percpu_ref_get_many+0x3c/0xd0
[ 543.154544] refill_obj_stock+0x7c/0x1b0
[ 543.154547] obj_cgroup_uncharge+0x30/0x40
[ 543.154550] memcg_slab_free_hook+0x90/0x160
[ 543.154554] kmem_cache_free+0x2d0/0x320
[ 543.154557] put_pid.part.0+0x88/0xc0
[ 543.154560] delayed_put_pid+0x28/0x40
[ 543.154561] rcu_core+0x288/0x920
[ 543.154564] rcu_cpu_kthread+0xc0/0x130
[ 543.154566] smpboot_thread_fn+0x2d8/0x320
[ 543.154568] kthread+0x194/0x1b0
[ 543.154570] ret_from_fork+0x10/0x20
[ 543.154574] Code: 11000442 b9001822 d53cd042 8b020000 (f833001f)
[ 543.154577] ---[ end trace 0000000000000004 ]---
[ 543.181196] tegra-hsierrrptinj: Callback for 0x0001 already registered
[ 543.181204] nvgpu: 17000000.gpu nvgpu_cic_mon_init:75 [ERR] Err inj callback registration failed: -22
[ 543.250331] nvgpu: 17000000.gpu nvgpu_pmu_cmd_post:600 [ERR] FBQ cmd setup failed
[ 543.250350] nvgpu: 17000000.gpu nvgpu_pmu_rpc_execute:727 [ERR] Failed to execute RPC status=0xfffffff4, func=0x3
[ 543.250353] nvgpu: 17000000.gpu gv100_pmu_lsfm_bootstrap_ls_falcon:100 [ERR] Failed to execute RPC, status=0xfffffff4
console-ramoops-0.log (512.0 KB)
journal.log (83.6 MB)
Any assistance, troubleshooting suggestions, or guidance would be greatly appreciated.