Kernel Panic - not syncing error on Orin custom Board for IMX296 driver

Autonomous Machines Jetson & Embedded Systems Jetson Orin NX
, , ,
1

Hi,

We have a custom board for Orin. We have flashed it and were able to load OV5693 sensor module and get the video streaming.
However while I try to load VC - MIPI IMX296 module the board boots up but shuts down after 5 seconds.
The video node is present and the driver is loaded as well.
custom_board_imx296.txt (150.3 KB)

Log:

[2023-12-21 14:41:41] [   19.828747] nvidia: loading out-of-tree module taints kernel.
[2023-12-21 14:41:42] [   20.308026] using random self ethernet address
[2023-12-21 14:41:42] [   20.314487] using random host ethernet address
[2023-12-21 14:41:43] [   21.906129] using random self ethernet address
[2023-12-21 14:41:43] [   21.910732] using random host ethernet address
[2023-12-21 14:41:47] [   25.381916] CPU4: shutdown
[2023-12-21 14:41:47] [   25.481746] CPU5: shutdown
[2023-12-21 14:41:47] [   25.638039] CPU6: shutdown
[2023-12-21 14:41:47] [   25.759466] IRQ 116: no longer affine to CPU7
[2023-12-21 14:41:47] [   25.764743] CPU7: shutdown
[2023-12-21 14:41:48] [   26.020593] hot-surface-alert cooling state: 1 -> 0
[2023-12-21 14:41:48] [   26.033359] hot-surface-alert cooling state: 0 -> 1
[2023-12-21 14:42:03] [   40.970627] Unable to handle kernel NULL pointer dereference at virtual address 000000000000022c
[2023-12-21 14:42:03] [   40.979730] Mem abort info:
[2023-12-21 14:42:03] [   40.983038]   ESR = 0x96000045
[2023-12-21 14:42:03] [   40.986300]   EC = 0x25: DABT (current EL), IL = 32 bits
[2023-12-21 14:42:03] [   40.991893]   SET = 0, FnV = 0
[2023-12-21 14:42:03] [   40.995194]   EA = 0, S1PTW = 0
[2023-12-21 14:42:03] [   40.998560] Data abort info:
[2023-12-21 14:42:03] [   41.001656]   ISV = 0, ISS = 0x00000045
[2023-12-21 14:42:03] [   41.005716]   CM = 0, WnR = 1
[2023-12-21 14:42:03] [   41.008786] user pgtable: 4k pages, 48-bit VAs, pgdp=00000001332e0000
[2023-12-21 14:42:03] [   41.015722] [000000000000022c] pgd=0000000113624003, p4d=0000000113624003, pud=0000000000000000
[2023-12-21 14:42:03] [   41.024885] Internal error: Oops: 96000045 [#1] PREEMPT SMP
[2023-12-21 14:42:03] [   41.030627] Modules linked in: nvidia_modeset(OE) fuse(E) lzo_rle(E) lzo_compress(E) zram(E) ramoops(E) reed_solomon(E) loop(E) snd_soc_tegra186_asrc(E) snd_soc_tegra210_ope(E) s)
[2023-12-21 14:42:03] [   41.030762]  nvgpu(E) nvmap(E) ip_tables(E) x_tables(E) [last unloaded: mtd]
[2023-12-21 14:42:03] [   41.129852] CPU: 1 PID: 2250 Comm: pool-gnome-init Tainted: G           OE     5.10.120-tegra #2
[2023-12-21 14:42:03] [   41.138869] Hardware name: Unknown NVIDIA Orin NX Developer Kit/NVIDIA Orin NX Developer Kit, BIOS 4.1-33958178 08/01/2023
[2023-12-21 14:42:03] [   41.150207] pstate: 60400009 (nZCv daif +PAN -UAO -TCO BTYPE=--)
[2023-12-21 14:42:03] [   41.156387] pc : tegra_channel_set_format+0x60/0xf0
[2023-12-21 14:42:03] [   41.161386] lr : tegra_channel_set_format+0x44/0xf0
[2023-12-21 14:42:03] [   41.166392] sp : ffff800014f6bb50
[2023-12-21 14:42:03] [   41.169783] x29: ffff800014f6bb50 x28: ffff6dfad3eed100 
[2023-12-21 14:42:03] [   41.175235] x27: 00000000c0d05605 x26: 0000000000000000 
[2023-12-21 14:42:03] [   41.180688] x25: ffff6dfad3eed400 x24: ffff6dfa8367d1f8 
[2023-12-21 14:42:03] [   41.186135] x23: ffff6dfad3eed100 x22: ffff6dfa80df9410 
[2023-12-21 14:42:03] [   41.191584] x21: ffff6dfa8367d080 x20: ffff6dfad3eed408 
[2023-12-21 14:42:03] [   41.197039] x19: ffff6dfad3eed400 x18: 0000000000000001 
[2023-12-21 14:42:03] [   41.202488] x17: 0000000000000000 x16: 0000000000000000 
[2023-12-21 14:42:03] [   41.207932] x15: ffff6dfa920f0570 x14: ffffffffffffffff 
[2023-12-21 14:42:03] [   41.213386] x13: ffff800014f6bb48 x12: 0000000000000028 
[2023-12-21 14:42:03] [   41.218846] x11: 0101010101010101 x10: ffff7f7f7f7f7f7f 
[2023-12-21 14:42:03] [   41.224293] x9 : fefdff2f64636e6c x8 : ffffffffffffffff 
[2023-12-21 14:42:03] [   41.229746] x7 : fefefefefefefefe x6 : 0000000000800000 
[2023-12-21 14:42:03] [   41.235204] x5 : 0000000000000000 x4 : ffffffffffffffff 
[2023-12-21 14:42:03] [   41.240652] x3 : 0000000000000000 x2 : ffffbc3c6c7cad58 
[2023-12-21 14:42:03] [   41.246104] x1 : 0000000000000000 x0 : 0000000000000000 
[2023-12-21 14:42:03] [   41.251565] Call trace:
[2023-12-21 14:42:03] [   41.254085]  tegra_channel_set_format+0x60/0xf0
[2023-12-21 14:42:03] [   41.258736]  v4l_s_fmt+0x308/0x568
[2023-12-21 14:42:03] [   41.262221]  __video_do_ioctl+0x188/0x3f8
[2023-12-21 14:42:03] [   41.266333]  video_usercopy+0x280/0x790
[2023-12-21 14:42:03] [   41.270269]  video_ioctl2+0x40/0x178
[2023-12-21 14:42:03] [   41.273933]  v4l2_ioctl+0x68/0x90
[2023-12-21 14:42:03] [   41.277331]  __arm64_sys_ioctl+0xac/0xf0
[2023-12-21 14:42:03] [   41.281355]  el0_svc_common.constprop.0+0x80/0x1c8
[2023-12-21 14:42:03] [   41.286275]  do_el0_svc+0x38/0xa8
[2023-12-21 14:42:03] [   41.289676]  el0_svc+0x1c/0x28
[2023-12-21 14:42:03] [   41.292803]  el0_sync_handler+0xa8/0xb0
[2023-12-21 14:42:03] [   41.296737]  el0_sync+0x16c/0x180
[2023-12-21 14:42:03] [   41.300142] Code: d100e063 b9400a60 121e7400 b9000a60 (b9022c60) 
[2023-12-21 14:42:03] [   41.306409] ---[ end trace c6847b5f00b90ec6 ]---
[2023-12-21 14:42:03] [   41.316716] Kernel panic - not syncing: Oops: Fatal exception
[2023-12-21 14:42:03] [   41.322618] SMP: stopping secondary CPUs
[2023-12-21 14:42:03] [   41.326650] Kernel Offset: 0x3c3c5c610000 from 0xffff800010000000
[2023-12-21 14:42:03] [   41.332899] PHYS_OFFSET: 0xffff920680000000
[2023-12-21 14:42:03] [   41.337196] CPU features: 0x08040006,4a80aa38
[2023-12-21 14:42:03] [   41.341667] Memory Limit: none
[2023-12-21 14:42:03] [   41.350152] ---[ end Kernel panic - not syncing: Oops: Fatal exception ]---
[2023-12-21 14:42:04] ��WARNING @ [platform/drivers/mailbox/ivc_link_provider/mail_imo.c]: mail imo TX timeout
[2023-12-21 14:42:04] failed to send thermal trip message to cpu!
[2023-12-21 14:42:05] WARNING @ [platform/drivers/mailbox/ivc_link_provider/mail_imo.c]: mail imo TX timeout
[2023-12-21 14:42:05] failed to send thermal trip message to cpu!
[2023-12-21 14:42:07] WARNING @ [platform/drivers/mailbox/ivc_link_provider/mail_imo.c]: mail imo TX timeout
[2023-12-21 14:42:07] failed to send thermal trip message to cpu!
[2023-12-21 14:42:10] WARNING @ [platform/drivers/mailbox/ivc_link_provider/mail_imo.c]: mail imo TX timeout
[2023-12-21 14:42:10] failed to send thermal trip message to cpu!

We were able to get IMX-296 working on the Orin development board.
On OV5693 sensor integration on custom board I do not encounter this issue.

JetPack: 5.1.2
BSP - 35.4.1

Please advice.

3

Could be the pixel format report incorrect cause the problem.

Thanks

4

@ShaneCCC

Sorry I did not get you.

5

Could be your sensor driver report incorrect format cause the problem.

6

@ShaneCCC

How do i debug this issue further?

This the log obtained after the orin shuts down:

[   40.644272] Unable to handle kernel NULL pointer dereference at virtual address 000000000000022c
[   40.653335] Mem abort info:
[   40.656225]   ESR = 0x96000045
[   40.659363]   EC = 0x25: DABT (current EL), IL = 32 bits
[   40.664844]   SET = 0, FnV = 0
[   40.668027]   EA = 0, S1PTW = 0
[   40.671249] Data abort info:
[   40.674215]   ISV = 0, ISS = 0x00000045
[   40.678163]   CM = 0, WnR = 1
[   40.681223] user pgtable: 4k pages, 48-bit VAs, pgdp=00000001426da000
[   40.687856] [000000000000022c] pgd=000000010d47e003, p4d=000000010d47e003, pud=0000000000000000
[   40.696817] Internal error: Oops: 96000045 [#1] PREEMPT SMP
[   40.702542] Modules linked in: nvidia_modeset(OE) fuse(E) lzo_rle(E) lzo_compress(E) zram(E) ramoops(E) reed_solomon(E) loop(E) snd_soc_tegra186_asrc(E) snd_soc_tegra210_ope(E) snd_soc_tegra210_iqc(E))
[   40.702618]  nvgpu(E) nvmap(E) ip_tables(E) x_tables(E) [last unloaded: mtd]
[   40.801673] CPU: 3 PID: 2354 Comm: pool-gnome-init Tainted: G           OE     5.10.120-tegra #2
[   40.810681] Hardware name: Unknown NVIDIA Orin NX Developer Kit/NVIDIA Orin NX Developer Kit, BIOS 4.1-33958178 08/01/2023
[   40.822007] pstate: 60400009 (nZCv daif +PAN -UAO -TCO BTYPE=--)
[   40.828177] pc : tegra_channel_set_format+0x60/0xf0
[   40.833179] lr : tegra_channel_set_format+0x44/0xf0
[   40.838169] sp : ffff80001e54bb50
[   40.841561] x29: ffff80001e54bb50 x28: ffff2cc500313d00 
[   40.847012] x27: 00000000c0d05605 x26: 0000000000000000 
[   40.852455] x25: ffff2cc553875500 x24: ffff2cc845c5d1f8 
[   40.857897] x23: ffff2cc500313d00 x22: ffff2cc500dfd810 
[   40.863343] x21: ffff2cc845c5d080 x20: ffff2cc553875508 
[   40.868791] x19: ffff2cc553875500 x18: 0000000000000001 
[   40.874233] x17: 0000000000000000 x16: 0000000000000000 
[   40.879677] x15: ffff2cc53fae8570 x14: ffffffffffffffff 
[   40.885116] x13: ffff80001e54bb48 x12: 0000000000000028 
[   40.890563] x11: 0101010101010101 x10: ffff7f7f7f7f7f7f 
[   40.896017] x9 : fefdff2f64636e6c x8 : ffffffffffffffff 
[   40.901459] x7 : fefefefefefefefe x6 : 0000000000800000 
[   40.906898] x5 : 0000000000000000 x4 : ffffffffffffffff 
[   40.912347] x3 : 0000000000000000 x2 : ffffcbcafa25ad58 
[   40.917798] x1 : 0000000000000000 x0 : 0000000000000000 
[   40.923239] Call trace:
[   40.925747]  tegra_channel_set_format+0x60/0xf0
[   40.930393]  v4l_s_fmt+0x308/0x568
[   40.933874]  __video_do_ioctl+0x188/0x3f8
[   40.937990]  video_usercopy+0x280/0x790
[   40.941924]  video_ioctl2+0x40/0x178
[   40.945581]  v4l2_ioctl+0x68/0x90
[   40.948983]  __arm64_sys_ioctl+0xac/0xf0
[   40.953002]  el0_svc_common.constprop.0+0x80/0x1c8
[   40.957908]  do_el0_svc+0x38/0xa8
[   40.961304]  el0_svc+0x1c/0x28
[   40.964436]  el0_sync_handler+0xa8/0xb0
[   40.968363]  el0_sync+0x16c/0x180
[   40.971758] Code: d100e063 b9400a60 121e7400 b9000a60 (b9022c60) 
[   40.978018] ---[ end trace bb317f81424924a5 ]---
[   40.988290] Kernel panic - not syncing: Oops: Fatal exception
[   40.994179] SMP: stopping secondary CPUs
[   40.998200] Kernel Offset: 0x4bcaea0a0000 from 0xffff800010000000
[   41.004449] PHYS_OFFSET: 0xffffd33c00000000
[   41.008731] CPU features: 0x08040006,4a80aa38
[   41.013190] Memory Limit: none
[   41.021885] ---[ end Kernel panic - not syncing: Oops: Fatal exception ]---
7

Reference to ov5693 to implement the IMX-296 from crash again.

Thanks

9

@ShaneCCC

the clock was not synchronized before loading the new module, also updated initramfs for the new driver. hence the issue is resolved.

Thanks.

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.