Using a script to open multiple cameras at the same time causes crash

I worked on orin # R35 (release), REVISION: 1.0, GCID: 31250864, BOARD: t186ref, EABI: aarch64.
When multiple cameras are turned on at the same time, the following errors may occur and cause crash.

[ 2343.408608] ------------[ cut here ]------------
[ 2343.408612] channel context at 1 is busy
[ 2343.408635] WARNING: CPU: 0 PID: 4282 at /home/zh/work/titan5/jet-pack-5.0.2/src/kernel/nvidia/drivers/platform/tegra/rtcpu/capture-ivc.c:176 tegra_capture_ivc_notify_chan_id+0x188/0x1b0
[ 2343.408636] Modules linked in: nvidia_modeset(OE) fuse(E) option(E) usb_wwan(E) usbserial(E) cdc_acm(E) xt_conntrack(E) switch(E) xt_MASQUERADE(E) can_raw(E) nf_conntrack_netlink(E) can(E) nfnetlink(E) mttcan(E) xt_addrtype(E) iptable_filter(E) iptable_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) libcrc32c(E) br_netfilter(E) lzo_rle(E) lzo_compress(E) zram(E) overlay(E) mv88e6xxx(E) ramoops(E) reed_solomon(E) dsa_core(E) loop(E) camera(E) ox03c10(E) imx390(E) nvgpu(E) hid_logitech_hidpp(E) snd_soc_tegra186_asrc(E) aes_ce_blk(E) snd_soc_tegra210_ope(E) snd_soc_tegra186_dspk(E) snd_soc_tegra210_iqc(E) snd_soc_tegra210_mvc(E) crypto_simd(E) snd_soc_tegra186_arad(E) cryptd(E) snd_soc_tegra210_afc(E) mcp251xfd(E) snd_soc_tegra210_dmic(E) aes_ce_cipher(E) snd_soc_tegra210_adx(E) snd_soc_tegra210_amx(E) ghash_ce(E) snd_soc_tegra210_admaif(E) can_dev(E) snd_soc_tegra210_i2s(E) sha2_ce(E) snd_soc_tegra210_mixer(E) snd_soc_tegra210_sfc(E) snd_soc_tegra_pcm(E) sha256_arm64(E)
[ 2343.408688]  sha1_ce(E) pwm_fan(E) snd_soc_tegra_machine_driver(E) pwm_gpio(E) snd_soc_spdif_tx(E) input_leds(E) userspace_alert(E) snd_soc_tegra210_adsp(E) snd_soc_tegra_utils(E) gw5200(E) snd_soc_simple_card_utils(E) max96712(E) max9295(E) max9296(E) snd_hda_codec_hdmi(E) hid_logitech_dj(E) snd_hda_tegra(E) ina3221(E) nct1008(E) tegra_bpmp_thermal(E) ofpart(E) nvadsp(E) snd_hda_codec(E) nvethernet(E) cmdlinepart(E) snd_hda_core(E) snd_soc_tegra210_ahub(E) nvidia(OE) nvmap(E) snd_soc_sgtl5000(E) qspi_mtd(E) tegra210_adma(E) mtd(E) spi_tegra114(E) binfmt_misc(E) max2008X(E) spidev(E) ip_tables(E) x_tables(E)
[ 2343.408723] CPU: 0 PID: 4282 Comm: vi-output, gw5X Tainted: G           OE     5.10.104 #11
[ 2343.408724] Hardware name: Jetson AGX Orin (DT)
[ 2343.408727] pstate: 40c00009 (nZcv daif +PAN +UAO -TCO BTYPE=--)
[ 2343.408729] pc : tegra_capture_ivc_notify_chan_id+0x188/0x1b0
[ 2343.408731] lr : tegra_capture_ivc_notify_chan_id+0x188/0x1b0
[ 2343.408731] sp : ffff80002435bad0
[ 2343.408732] x29: ffff80002435bad0 x28: 0000000000000000 
[ 2343.408734] x27: 0000000000001043 x26: ffff3045739a3000 
[ 2343.408735] x25: ffff3045da8fe6b8 x24: ffff3045043dc088 
[ 2343.408737] x23: 0000000000000057 x22: ffff3045043dc080 
[ 2343.408738] x21: ffff3045043dcb60 x20: 0000000000000001 
[ 2343.408739] x19: 0000000000000057 x18: 0000000000000000 
[ 2343.408741] x17: 0000000000000000 x16: ffffcfd1001ac5f0 
[ 2343.408742] x15: ffff3045808b2270 x14: ffffffffffffffff 
[ 2343.408743] x13: ffffcfd100fb6de8 x12: ffffcfd100fb6a1b 
[ 2343.408745] x11: 0000000000000000 x10: 0000000000000a80 
[ 2343.408746] x9 : ffff80002435bad0 x8 : 2073692031207461 
[ 2343.408748] x7 : 20747865746e6f63 x6 : 00000000358682ed 
[ 2343.408749] x5 : ffff304c2c527958 x4 : 0000000100000220 
[ 2343.408751] x3 : 0000000000000001 x2 : ffff304c2c527960 
[ 2343.408752] x1 : 47a3908bac4de200 x0 : 0000000000000000 
[ 2343.408754] Call trace:
[ 2343.408757]  tegra_capture_ivc_notify_chan_id+0x188/0x1b0
[ 2343.408760]  vi_capture_setup+0x35c/0x670
[ 2343.408765]  tegra_channel_capture_setup+0x24c/0x2b8
[ 2343.408767]  vi5_channel_error_recover+0x1dc/0x210
[ 2343.408769]  tegra_channel_error_recover+0x74/0x90
[ 2343.408771]  tegra_channel_kthread_capture_dequeue+0x124/0x1e0
[ 2343.408774]  kthread+0x148/0x170
[ 2343.408777]  ret_from_fork+0x10/0x18
[ 2343.408779] ---[ end trace c0c2181249ce35da ]---	

[  111.960063] flags: 0x8000000000000000()
[  111.964010] raw: 8000000000000000 dead000000000100 dead000000000122 ffff1c33c0003900
[  111.971984] raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
[  111.979960] page dumped because: non-NULL mapping
[  111.984951] Disabling lock debugging due to kernel taint
[  111.990704] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 131072, datatype 1e
[  112.002202] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
[  112.714643] bwmgr API not supported
[  112.719254] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000520
[  112.728456] Mem abort info:
[  112.731382]   ESR = 0x96000045
[  112.734556]   EC = 0x25: DABT (current EL), IL = 32 bits
[  112.740134]   SET = 0, FnV = 0
[  112.743336]   EA = 0, S1PTW = 0
[  112.746635] Data abort info:
[  112.749662]   ISV = 0, ISS = 0x00000045
[  112.753687]   CM = 0, WnR = 1
[  112.756915] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000195e4a000
[  112.763609] [0000000000000520] pgd=00000001a6db5003, p4d=00000001a6db5003, pud=0000000000000000
[  112.772666] Internal error: Oops: 96000045 [#1] PREEMPT SMP
[  112.778404] Modules linked in: nvidia_modeset(OE) fuse(E) option(E) usb_wwan(E) usbserial(E) cdc_acm(E) switch(E) xt_conntrack(E) can_raw(E) can(E))
[  112.778469]  sha1_ce(E) pwm_gpio(E) snd_soc_tegra_machine_driver(E) pwm_fan(E) snd_soc_spdif_tx(E) userspace_alert(E) snd_soc_tegra210_adsp(E) inpu)
[  112.926305] CPU: 0 PID: 3356 Comm: gst-launch-1.0 Tainted: G    B   W  OE     5.10.104 #11
[  112.934809] Hardware name: Jetson AGX Orin (DT)
[  112.939472] pstate: a0400009 (NzCv daif +PAN -UAO -TCO BTYPE=--)
[  112.945655] pc : tegra_channel_ec_close+0x34/0x60
[  112.950495] lr : tegra_channel_ec_close+0x18/0x60
[  112.955331] sp : ffff800022723a90
[  112.958744] x29: ffff800022723a90 x28: ffff1c3442e1a800 
[  112.964210] x27: 0000000040045613 x26: 0000000000000000 
[  112.969674] x25: ffff800022723d08 x24: ffff1c3ac08ed1f8 
[  112.975140] x23: ffff1c34393b3a00 x22: ffff800022723d08 
[  112.980606] x21: ffff1c3ac08f60a0 x20: ffff1c3ac08ed080 
[  112.986071] x19: ffff1c3ac08f60a0 x18: 0000000000000010 
[  112.991535] x17: 0000000000000000 x16: ffffaaa9db02ddb0 
[  112.996999] x15: ffff1c34393b3f70 x14: 0000000000000000 
[  113.002463] x13: 0000000000000020 x12: 0101010101010101 
[  113.007930] x11: 0000000000000003 x10: 0101010101010101 
[  113.013393] x9 : 00000000fffffffe x8 : 5f35697363203a30 
[  113.018858] x7 : 3030303061353140 x6 : 0000000107613477 
[  113.024323] x5 : 00ffffffffffffff x4 : 001be795d0000000 
[  113.029787] x3 : 0000000000000017 x2 : ffff1c3ac08f63f8 
[  113.035253] x1 : 0000000000000520 x0 : fffffffffffffff8 
[  113.040720] Call trace:
[  113.043232]  tegra_channel_ec_close+0x34/0x60
[  113.047712]  vi5_power_off+0x34/0x60
[  113.051392]  tegra_channel_stop_streaming+0x48/0x60
[  113.056411]  __vb2_queue_cancel+0x3c/0x220
[  113.060628]  vb2_core_streamoff+0x30/0xd0
[  113.064757]  vb2_streamoff+0x30/0x80
[  113.068436]  vb2_ioctl_streamoff+0x54/0x60
[  113.072653]  v4l_streamoff+0x3c/0x50
[  113.076333]  __video_do_ioctl+0x180/0x3f0
[  113.080463]  video_usercopy+0x27c/0x790
[  113.084409]  video_ioctl2+0x3c/0x180
[  113.088088]  v4l2_ioctl+0x64/0x90
[  113.091503]  __arm64_sys_ioctl+0xa8/0xf0
[  113.095538]  el0_svc_common.constprop.0+0x7c/0x1c0
[  113.100468]  do_el0_svc+0x34/0xa0
[  113.103882]  el0_svc+0x1c/0x30
[  113.107027]  el0_sync_handler+0xa8/0xb0
[  113.110973]  el0_sync+0x16c/0x180
[  113.114386] Code: d1002000 54000140 d503201f 9114a001 (a9007c3f) 
[  113.120659] ---[ end trace 29eaeba40b6ceae5 ]---
[  113.130244] Kernel panic - not syncing: Oops: Fatal exception
[  113.136149] SMP: stopping secondary CPUs
[  113.140296] Kernel Offset: 0x2aa9caf40000 from 0xffff800010000000
[  113.146564] PHYS_OFFSET: 0xffffe3cd40000000
[  113.150858] CPU features: 0x0040006,4a80aa38
[  113.155255] Memory Limit: none
[  113.163217] ---[ end Kernel panic - not syncing: Oops: Fatal exception ]---

Could you please help solve it? Thank you.

hello skr_giao,

could you please check you’re able to fetch the camera stream individually.
please check you’ll see NULL pointer dereference also.

hello JerryChang
Thank you for your reply.
When I turned on the camera alone, it didn’t crash when I turned on the other cameras more than 0.4s apart.
I think the cause of the crash may be related to the phenomenon described in this post.

hello skr_giao,

let’s keep using single thread for tracking the same failure.
so…
may I know how many camera stream you had.
is it accepted to have workarounds with some delay before enable camera stream?

we’ve worked on the camera init sequence refinements, there’re bug fixes to address the similar issue.
please expect the next Jetpack release will also include this bug fixes.

hello JerryChang,
I have 12 cameras, three 8M, nine 2M, hooked up to an orin,I’ll try to solve the problem by delay first.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.