Kernel Panic when hwrng_fillfn called from ret_from_fork

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.8.1
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
1.9.3.10904
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Once in a while we randomly see errors like these in our dmesg logs:

Mar 28 14:26:51 tegra-ubuntu kernel: Unable to handle kernel NULL pointer dereference at virtual address 000000000000008c
Mar 28 14:26:51 tegra-ubuntu kernel: Mem abort info:
Mar 28 14:26:51 tegra-ubuntu kernel:   ESR = 0x0000000096000004
Mar 28 14:26:51 tegra-ubuntu kernel:   EC = 0x25: DABT (current EL), IL = 32 bits
Mar 28 14:26:51 tegra-ubuntu kernel:   SET = 0, FnV = 0
Mar 28 14:26:51 tegra-ubuntu kernel:   EA = 0, S1PTW = 0
Mar 28 14:26:51 tegra-ubuntu kernel:   FSC = 0x04: level 0 translation fault
Mar 28 14:26:51 tegra-ubuntu kernel: Data abort info:
Mar 28 14:26:51 tegra-ubuntu kernel: printk: console [ttyS2]: printing thread stopped
Mar 28 14:26:51 tegra-ubuntu kernel:   ISV = 0, ISS = 0x00000004
Mar 28 14:26:51 tegra-ubuntu kernel:   CM = 0, WnR = 0
Mar 28 14:26:51 tegra-ubuntu kernel: user pgtable: 4k pages, 48-bit VAs, pgdp=00000001145e0000
Mar 28 14:26:51 tegra-ubuntu kernel: [000000000000008c] pgd=0000000000000000, p4d=0000000000000000
Mar 28 14:26:51 tegra-ubuntu kernel: Internal error: Oops: 96000004 [#1] PREEMPT_RT SMP
Mar 28 14:26:51 tegra-ubuntu kernel: Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack libcrc32c nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter fuse 8021q garp mrp nvidia_modeset(O) lan743x(O) tegra_pcie_dma_test(O) tegra_pcie_edma(O) cdi_mgr(O) snd_soc_tegra_virt_t210ref_pcm(O) cdi_pwm(O) isc_mgr(O) snd_soc_tegra210_virt_alt_admaif(O) isc_pwm(O) cdi_dev(O) tegra_hv_vcpu_yield(O) tegra_hv_pm_ctl(O) isc_dev(O) cdi_gpio(O) isc_gpio(O) tegra_fsicom(O) lm90 tegra210_adma phy_tegra194_p2u spidev mttcan(O) tegra_bpmp_thermal cam_fsync(O) can_dev tegra_uss_io_proxy(O) nvidia(O) tegra_aconnect tegra_xudc tegra_dce(O) spi_tegra114 safety_i2s(O) tsecriscv(O) watchdog_tegra_t18x(O) pcie_tegra194 nvhost_vi5(O) nvhost_isp5(O) nvhost_nvcsi_t194(O) tegra_camera(O) v4l2_dv_timings v4l2_fwnode v4l2_async videobuf2_dma_contig nvhost_nvcsi(O) tegra_camera_platform(O) mc_utils(O) tegra_drm_next(O) capture_ivc(O) cec videobuf2_v4l2 videobuf2_memops
Mar 28 14:26:51 tegra-ubuntu kernel:  videobuf2_common bridge nvhost_pva(O) videodev stp drm_kms_helper llc mc nvhost_capture(O) drm cpuidle_tegra_auto(O) nvhost_nvdla(O) nvhwpm(O) host1x_nvhost(O) camchar(O) camera_diagnostics(O) debug(O) tegra_camera_rtcpu(O) device_group(O) firmwares_class(O) reset_group(O) ivc_bus(O) hsp_mailbox_client(O) clk_group(O) nvgpu(O) tegra_gr_comm(O) nvmap(O) hvc_sysfs(O) tegra_nvvse_cryptodev(O) tegra_hv_vse_safety(O) host1x_fence(O) host1x_next(O) nvsciipc(O) userspace_ivc_mempool(O) ivc_cdev(O) ip_tables x_tables ipv6 nvme(E) nvme_core(E) oak_pci(OE) nvethernet(OE) pinctrl_tegra234(OE) nvpps(OE) tegra194_gte(OE) tegra_bpmp(OE) tegra_vblk(OE) tegra_hv_vblk_oops(OE) tegra_hv(OE) ivc_ext(OE)
Mar 28 14:26:51 tegra-ubuntu kernel: CPU: 5 PID: 638 Comm: hwrng Tainted: G           OE     5.15.98-rt-tegra #1
Mar 28 14:26:51 tegra-ubuntu kernel: Hardware name: p3710-0010 (DT)
Mar 28 14:26:51 tegra-ubuntu kernel: pstate: 80400005 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Mar 28 14:26:51 tegra-ubuntu kernel: pc : tegra_hv_vse_get_db+0x11cc/0x70e0 [tegra_hv_vse_safety]
Mar 28 14:26:51 tegra-ubuntu kernel: lr : tegra_hv_vse_get_db+0x1098/0x70e0 [tegra_hv_vse_safety]
Mar 28 14:26:51 tegra-ubuntu kernel: sp : ffff80000f293c80
Mar 28 14:26:51 tegra-ubuntu kernel: x29: ffff80000f293c80 x28: ffff0001ab01f680 x27: 0000000000000000
Mar 28 14:26:51 tegra-ubuntu kernel: x26: 0000000000000000 x25: ffff80000155ef08 x24: ffff80000f293d47
Mar 28 14:26:51 tegra-ubuntu kernel: x23: 0000000000000000 x22: 0000000000000000 x21: ffff0000849507f0
Mar 28 14:26:51 tegra-ubuntu kernel: x20: ffff00008062b380 x19: ffff0001ab01fe80 x18: 0000000000000000
Mar 28 14:26:51 tegra-ubuntu kernel: x17: 0000000000000002 x16: ffff8000012ca0e0 x15: 0000000000000000
Mar 28 14:26:51 tegra-ubuntu kernel: x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
Mar 28 14:26:51 tegra-ubuntu kernel: x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
Mar 28 14:26:51 tegra-ubuntu kernel: x8 : 0000000000000000 x7 : 0000000000000000 x6 : 0000000000000000
Mar 28 14:26:51 tegra-ubuntu kernel: x5 : ffff0001ab01ffa8 x4 : 0000000000000004 x3 : 0000000000000009
Mar 28 14:26:51 tegra-ubuntu kernel: x2 : 0000000000000000 x1 : ffff000083b51080 x0 : 0000000000000000
Mar 28 14:26:51 tegra-ubuntu kernel: Call trace:
Mar 28 14:26:51 tegra-ubuntu kernel:  tegra_hv_vse_get_db+0x11cc/0x70e0 [tegra_hv_vse_safety]
Mar 28 14:26:51 tegra-ubuntu kernel:  tegra_hv_vse_get_db+0x2828/0x70e0 [tegra_hv_vse_safety]
Mar 28 14:26:51 tegra-ubuntu kernel:  tegra_hv_vse_get_db+0x5590/0x70e0 [tegra_hv_vse_safety]
Mar 28 14:26:51 tegra-ubuntu kernel:  tegra_hv_vse_get_db+0x56f8/0x70e0 [tegra_hv_vse_safety]
Mar 28 14:26:51 tegra-ubuntu kernel:  hwrng_fillfn+0x8c/0x170
Mar 28 14:26:51 tegra-ubuntu kernel:  kthread+0x198/0x1c0
Mar 28 14:26:51 tegra-ubuntu kernel:  ret_from_fork+0x10/0x20
Mar 28 14:26:51 tegra-ubuntu kernel: Code: b9402a60 b9006820 35fffe20 f9400020 (39423002)

We’ve tried to force a crash using:

$ sudo dd if=/dev/hwrng of=/dev/zero bs=1M

But haven’t been able to reliably reproduce it.

We’re reporting it to nvidia in case they’ve also seen this issue.

We haven’t come across this specific issue so far. If you manage to gather more information or reproduce the problem consistently, please share the details.