Kernel error when loading and unloading camera kernel driver module several times

[user27558]

Running R32.5.1 (Jetpack 4.5.1).
Kernel version: 4.9.201-tegra

Hello,
I am developing a custom imx390 camera driver and the driver is an out of tree kernel module (.ko).
Loading and unloading the kernel module with insmod and rmmod works fine and the cameras are working.

But, when unloading and loading the kernel module repeatedly the following error appears in dmesg and the cameras stop working:

[  458.039642] Unable to handle kernel paging request at virtual address 4300303d736595
[  458.039816] Mem abort info:
[  458.039886]   ESR = 0x96000004
[  458.039953]   Exception class = DABT (current EL), IL = 32 bits
[  458.040062]   SET = 0, FnV = 0
[  458.040141]   EA = 0, S1PTW = 0
[  458.040221] Data abort info:
[  458.040274]   ISV = 0, ISS = 0x00000004
[  458.040354]   CM = 0, WnR = 0
[  458.040414] [004300303d736595] address between user and kernel address ranges
[  458.040551] Internal error: Oops: 96000004 [#1] PREEMPT SMP
[  458.040652] Modules linked in: imx390(O+) bnep fuse xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter zram overlay spidev userspace_alert nvgpu bluedroid_pm ip_tables x_tables [last unloaded: imx390]
[  458.041873] CPU: 0 PID: 8620 Comm: v4l_id Tainted: G           O    4.9.201-tegra #2
[  458.042456] Hardware name: NVIDIA Jetson Xavier NX Developer Kit (DT)
[  458.043974] task: ffffffc1ec1b2a00 task.stack: ffffffc1b2fd8000
[  458.049678] PC is at read_phy_mode_from_dt+0x4c/0xb8
[  458.054487] LR is at csi5_mipi_cal+0x28/0x260
[  458.059026] pc : [<ffffff8008b48284>] lr : [<ffffff8008b56548>] pstate: 20400045
[  458.066463] sp : ffffffc1b2fdb6b0
[  458.069179] x29: ffffffc1b2fdb6b0 x28: ffffffc1ec1b2a00 
[  458.074949] x27: ffffffc1b2fdbd48 x26: ffffffc1f55d4810 
[  458.080473] x25: 0000000000000080 x24: ffffffc1f4da4818 
[  458.085885] x23: ffffffc1f4e86018 x22: ffffff80094bbdb8 
[  458.090795] x21: ffffffc1f4cbb020 x20: ffffffc1f4cbb020 
[  458.096558] x19: 4f4300303d73656d x18: 0000000000000000 
[  458.102246] x17: 0000007f902fd070 x16: 0000000000000000 
[  458.108019] x15: 0000000000000000 x14: 000000000052e4b0 
[  458.113708] x13: 0000000000000026 x12: 071c71c71c71c71c 
[  458.119395] x11: 000000000000000b x10: 0000000000000a10 
[  458.125084] x9 : ffffffc1b2fdb450 x8 : ffffffc1ec1b3470 
[  458.130609] x7 : fefefeff646c606d x6 : 000000001340538a 
[  458.136371] x5 : 0000000000000000 x4 : 0000000000000000 
[  458.141708] x3 : 0000000044524f4c x2 : 0000000000000000 
[  458.146794] x1 : ffffffc1f5158810 x0 : 0000000000000158 

[  458.153785] Process v4l_id (pid: 8620, stack limit = 0xffffffc1b2fd8000)
[  458.159914] Call trace:
[  458.162283] [<ffffff8008b48284>] read_phy_mode_from_dt+0x4c/0xb8
[  458.167880] [<ffffff8008b56548>] csi5_mipi_cal+0x28/0x260
[  458.172956] [<ffffff8008b48430>] tegra_csi_mipi_calibrate+0x80/0xd0
[  458.179081] [<ffffff800855aaf8>] tegra194_nvcsi_finalize_poweron+0x88/0xb0
[  458.185471] [<ffffff800852bef4>] nvhost_module_runtime_resume+0xbc/0x280
[  458.191687] [<ffffff800878becc>] pm_generic_runtime_resume+0x3c/0x58
[  458.198065] [<ffffff8008799d30>] __genpd_runtime_resume+0x38/0xa0
[  458.203493] [<ffffff800879c4a4>] genpd_runtime_resume+0xa4/0x210
[  458.209007] [<ffffff800878e214>] __rpm_callback+0x74/0xa0
[  458.214169] [<ffffff800878e274>] rpm_callback+0x34/0x98
[  458.219501] [<ffffff800878f710>] rpm_resume+0x470/0x710
[  458.224666] [<ffffff800878f9fc>] __pm_runtime_resume+0x4c/0x70
[  458.230527] [<ffffff800852ae2c>] nvhost_module_busy+0x5c/0x168
[  458.236304] [<ffffff8008b56f2c>] csi5_power_on+0x24/0x88
[  458.241987] [<ffffff8008b469e8>] tegra_csi_power+0x38/0x158
[  458.247504] [<ffffff8008b46b34>] tegra_csi_s_power+0x2c/0x38
[  458.253107] [<ffffff8008b3b914>] tegra_channel_set_power+0x84/0x198
[  458.259577] [<ffffff8008b46608>] vi5_power_on+0x80/0xa0
[  458.264653] [<ffffff8008b39928>] tegra_channel_open+0x80/0x180
[  458.270515] [<ffffff8008b0d1b8>] v4l2_open+0x80/0x118
[  458.275765] [<ffffff8008261f6c>] chrdev_open+0x94/0x198
[  458.280926] [<ffffff8008258918>] do_dentry_open+0x1d8/0x340
[  458.286701] [<ffffff8008259ed0>] vfs_open+0x58/0x88
[  458.291691] [<ffffff800826d3b0>] do_last+0x530/0xfd0
[  458.296418] [<ffffff800826dee0>] path_openat+0x90/0x378
[  458.301752] [<ffffff800826f450>] do_filp_open+0x70/0xe8
[  458.306568] [<ffffff800825a394>] do_sys_open+0x174/0x258
[  458.312162] [<ffffff800825a4fc>] SyS_openat+0x3c/0x50
[  458.317240] [<ffffff800808395c>] __sys_trace_return+0x0/0x4
[  458.322931] ---[ end trace 8997eaaa2650b04c ]---

According to the stack trace, the error happens in the read_phy_mode_from_dt function located in the csi kernel driver.
The error seems to always happen after the imx390_power_on function is called.

Do you have any idea what could be the issue and how to solve it ?

Thanks !

[linuxdev]

I can’t answer, but a starting place is to understand that a paging request means likely memory is reserved during module insert, and should be released on module exit. I can think of a few things to check:

• Is that address something specific? Meaning does that address get reserved each time, versus the question of whether the memory is ok reserving any address? Let’s say this is not shared memory, and not pinned; in that case your driver may not care what address that memory is at. Perhaps that’s a physical address reserved for hardware, in which case it’ll always be the same address. The goal is to answer whether something might be hanging on to this memory other than your driver.
• Perhaps there is a memory leak such that reserving memory occurs upon insert, but the memory is never released upon removal of the module. Eventually it would run out of memory.
• I suggest that any time memory is reserved that your log print the address. Then when removal of module followed by new insert fails, compare the address reserved to the address blocking and see if it matches a previously reserved memory address.
• If you have a method of logging when memory is released, I suggest you also log release. It is possible that release through other mechanisms would be logged anyway, so after the failure it might be worth a grep of that address via dmesg to see if something else showed up earlier with that address.
• Just observe if that address is the same failing address each time, even if you’ve rebooted.