运行系统卡顿1秒钟

Robotics & Edge Computing Jetson & Embedded Systems Jetson Xavier NX
,
1

我们使用的jetpack5.1的sdk，打入了实时内核patch，在读取传感器数据中发现在特定时刻系统的通信会停止约1秒钟时间，读取数据和向外发送数据看起来都停止了。如下是故障日志：

Feb  3 23:20:51 nvidia kernel: [10609.882138] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 131072
Feb  3 23:20:59 nvidia kernel: [10618.103687] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 256
Feb  3 23:20:59 nvidia kernel: [10618.148757] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
Feb  3 23:20:59 nvidia kernel: [10618.181951] [RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] "General error queue is out of sync with frame queue. ts=10639112363424 sof_ts=10639113717824 gerror_code=2 gerror_data=400060 notify_bits=20000"
Feb  3 23:21:02 nvidia kernel: [10620.665930] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Feb  3 23:21:02 nvidia kernel: [10620.665947] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Feb  3 23:21:02 nvidia kernel: [10620.677304] (NULL device *): vi_capture_control_message: NULL VI channel received
Feb  3 23:21:02 nvidia kernel: [10620.677312] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=0, csi_port=0
Feb  3 23:21:02 nvidia kernel: [10620.677317] (NULL device *): vi_capture_control_message: NULL VI channel received
Feb  3 23:21:02 nvidia kernel: [10620.677318] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 0 vc- 0
Feb  3 23:21:02 nvidia kernel: [10620.688043] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
Feb  3 23:21:02 nvidia kernel: [10620.690862] ar0233yuv 2-001b: ar0233yuv_stop_streaming: stop streaming
Feb  3 23:21:03 nvidia kernel: [10620.693007] bwmgr API not supported
Feb  3 23:21:03 nvidia kernel: [10620.707815] ------------[ cut here ]------------
Feb  3 23:21:03 nvidia kernel: [10620.707823] WARNING: CPU: 0 PID: 4766 at drivers/media/common/videobuf2/videobuf2-core.c:1956 __vb2_queue_cancel+0x1b4/0x220
Feb  3 23:21:03 nvidia kernel: [10620.707838] Modules linked in: nvidia_modeset(OE) fuse(E) xt_conntrack(E) xt_MASQUERADE(E) nf_conntrack_netlink(E) nfnetlink(E) xt_addrtype(E) iptable_filter(E) iptable_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) libcrc32c(E) br_netfilter(E) mttcan(E) can_dev(E) lzo_rle(E) lzo_compress(E) can_raw(E) zram(E) can(E) overlay(E) ramoops(E) reed_solomon(E) snd_soc_tegra210_ope(E) snd_soc_tegra186_dspk(E) snd_soc_tegra186_asrc(E) snd_soc_tegra210_iqc(E) snd_soc_tegra186_arad(E) snd_soc_tegra210_mvc(E) snd_soc_tegra210_adx(E) snd_soc_tegra210_amx(E) snd_soc_tegra210_afc(E) snd_soc_tegra210_dmic(E) snd_soc_tegra210_i2s(E) snd_soc_tegra210_mixer(E) snd_soc_tegra210_admaif(E) snd_soc_tegra210_sfc(E) snd_soc_tegra_pcm(E) loop(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) ghash_ce(E) sha2_ce(E) sha256_arm64(E) sha1_ce(E) snd_soc_spdif_tx(E) snd_soc_tegra_machine_driver(E) pps_gen_gpio(E) heart_ctl(E) nct1008(E) snd_soc_tegra210_adsp(E) snd_soc_tegra_utils(E)
Feb  3 23:21:03 nvidia kernel: [10620.707891]  snd_soc_simple_card_utils(E) userspace_alert(E) tegra_bpmp_thermal(E) nvadsp(E) snd_soc_tegra210_ahub(E) tegra210_adma(E) snd_soc_rt5640(E) snd_hda_codec_hdmi(E) snd_soc_rl6231(E) ar0233yuv(E) snd_hda_tegra(E) snd_hda_codec(E) snd_hda_core(E) spi_tegra114(E) nvidia(OE) ina3221(E) pwm_fan(E) nvgpu(E) binfmt_misc(E) nvmap(E) ip_tables(E) x_tables(E) [last unloaded: mtd]
Feb  3 23:21:03 nvidia kernel: [10620.707913] CPU: 0 PID: 4766 Comm: syzk-video Tainted: G        W  OE     5.10.104-rt63-tegra #92
Feb  3 23:21:03 nvidia kernel: [10620.707916] Hardware name: Unknown Jetson AGX Orin/Jetson AGX Orin, BIOS 202210.2-26fc186-dirty 04/01/2023
Feb  3 23:21:03 nvidia kernel: [10620.707919] pstate: 60400009 (nZCv daif +PAN -UAO -TCO BTYPE=--)
Feb  3 23:21:03 nvidia kernel: [10620.707921] pc : __vb2_queue_cancel+0x1b4/0x220
Feb  3 23:21:03 nvidia kernel: [10620.707924] lr : __vb2_queue_cancel+0x40/0x220
Feb  3 23:21:03 nvidia kernel: [10620.707926] sp : ffff8000241e3b00
Feb  3 23:21:03 nvidia kernel: [10620.707927] x29: ffff8000241e3b00 x28: ffff041786b56e00 
Feb  3 23:21:03 nvidia kernel: [10620.707929] x27: 0000000040045613 x26: 0000000000000000 
Feb  3 23:21:03 nvidia kernel: [10620.707931] x25: ffff8000241e3d08 x24: ffff0415c11771f8 
Feb  3 23:21:03 nvidia kernel: [10620.707933] x23: ffff0416bd519e00 x22: ffff8000241e3d08 
Feb  3 23:21:03 nvidia kernel: [10620.707935] x21: ffff0415c1177740 x20: 0000000000000001 
Feb  3 23:21:03 nvidia kernel: [10620.707936] x19: ffff0415c1177740 x18: 0000000000000001 
Feb  3 23:21:03 nvidia kernel: [10620.707938] x17: 0000000000000000 x16: ffffab6b9cb6122c 
Feb  3 23:21:03 nvidia kernel: [10620.707940] x15: ffffffffffffffff x14: ffffff0000000000 
Feb  3 23:21:03 nvidia kernel: [10620.707941] x13: ffffffffffffffff x12: 0000000000000028 
Feb  3 23:21:03 nvidia kernel: [10620.707943] x11: 0101010101010101 x10: ffff7f7f7f7f7f7f 
Feb  3 23:21:03 nvidia kernel: [10620.707944] x9 : fefdff3064636e6c x8 : ffffffffffffffff 
Feb  3 23:21:03 nvidia kernel: [10620.707946] x7 : ffffab6b9de75df8 x6 : 0000004d6db73751 
Feb  3 23:21:03 nvidia kernel: [10620.707948] x5 : 00ffffffffffffff x4 : 000f69a9efa93a3e 
Feb  3 23:21:03 nvidia kernel: [10620.707949] x3 : 0000000000000017 x2 : ffffab6b9ccf4970 
Feb  3 23:21:03 nvidia kernel: [10620.707951] x1 : ffff041cc31f05c0 x0 : 0000000000000002 
Feb  3 23:21:03 nvidia kernel: [10620.707953] Call trace:
Feb  3 23:21:03 nvidia kernel: [10620.707955]  __vb2_queue_cancel+0x1b4/0x220
Feb  3 23:21:03 nvidia kernel: [10620.707957]  vb2_core_streamoff+0x34/0xd0
Feb  3 23:21:03 nvidia kernel: [10620.707960]  vb2_streamoff+0x34/0x80
Feb  3 23:21:03 nvidia kernel: [10620.707963]  vb2_ioctl_streamoff+0x58/0x70
Feb  3 23:21:03 nvidia kernel: [10620.707966]  v4l_streamoff+0x40/0x50
Feb  3 23:21:03 nvidia kernel: [10620.707969]  __video_do_ioctl+0x188/0x400
Feb  3 23:21:03 nvidia kernel: [10620.707971]  video_usercopy+0x280/0x7b0
Feb  3 23:21:03 nvidia kernel: [10620.707973]  video_ioctl2+0x40/0x180
Feb  3 23:21:03 nvidia kernel: [10620.707975]  v4l2_ioctl+0x68/0x90
Feb  3 23:21:03 nvidia kernel: [10620.707979]  __arm64_sys_ioctl+0xac/0xf0
Feb  3 23:21:03 nvidia kernel: [10620.707985]  el0_svc_common.constprop.0+0x80/0x1d0
Feb  3 23:21:03 nvidia kernel: [10620.707990]  do_el0_svc+0x38/0xb0
Feb  3 23:21:03 nvidia kernel: [10620.707993]  el0_svc+0x1c/0x30
Feb  3 23:21:03 nvidia kernel: [10620.707998]  el0_sync_handler+0xa8/0xb0
Feb  3 23:21:03 nvidia kernel: [10620.708000]  el0_sync+0x16c/0x180
Feb  3 23:21:03 nvidia kernel: [10620.708004] ---[ end trace 000000000000041a ]---
Feb  3 23:21:03 nvidia kernel: [10621.713111] videobuf2_common: driver bug: stop_streaming operation is leaving buf 0000000049eb9711 in active state
Feb  3 23:21:03 nvidia kernel: [10621.713122] videobuf2_common: driver bug: stop_streaming operation is leaving buf 00000000d0f90dbc in active state
Feb  3 23:21:03 nvidia kernel: [10621.713748] mttcan c320000.mttcan can1: mttcan_poll_ir: some msgs lost on in Q0
Feb  3 23:21:03 nvidia kernel: [10621.714074] mttcan c310000.mttcan can0: mttcan_poll_ir: some msgs lost on in Q0
Feb  3 23:21:03 nvidia kernel: [10621.714468] ar0233yuv 2-0022: ar0233yuv_stop_streaming: stop streaming
Feb  3 23:21:03 nvidia kernel: [10621.714516] ar0233yuv 2-001f: ar0233yuv_stop_streaming: stop streaming
Feb  3 23:21:03 nvidia kernel: [10621.714712] mttcan c310000.mttcan can0: mttcan_poll_ir: some msgs lost on in Q0
Feb  3 23:21:03 nvidia kernel: [10621.715253] ar0233yuv 2-0021: ar0233yuv_stop_streaming: stop streaming
Feb  3 23:21:03 nvidia kernel: [10621.729431] bwmgr API not supported
Feb  3 23:21:03 nvidia kernel: [10621.731284] bwmgr API not supported
Feb  3 23:21:03 nvidia kernel: [10621.734230] bwmgr API not supported
Feb  3 23:21:03 nvidia kernel: [10621.741333] ------------[ cut here ]------------
Feb  3 23:21:03 nvidia kernel: [10621.741340] WARNING: CPU: 1 PID: 3808 at drivers/media/common/videobuf2/videobuf2-core.c:1956 __vb2_queue_cancel+0x1b4/0x220
Feb  3 23:21:03 nvidia kernel: [10621.741353] Modules linked in: nvidia_modeset(OE) fuse(E) xt_conntrack(E) xt_MASQUERADE(E) nf_conntrack_netlink(E) nfnetlink(E) xt_addrtype(E) iptable_filter(E) iptable_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) libcrc32c(E) br_netfilter(E) mttcan(E) can_dev(E) lzo_rle(E) lzo_compress(E) can_raw(E) zram(E) can(E) overlay(E) ramoops(E) reed_solomon(E) snd_soc_tegra210_ope(E) snd_soc_tegra186_dspk(E) snd_soc_tegra186_asrc(E) snd_soc_tegra210_iqc(E) snd_soc_tegra186_arad(E) snd_soc_tegra210_mvc(E) snd_soc_tegra210_adx(E) snd_soc_tegra210_amx(E) snd_soc_tegra210_afc(E) snd_soc_tegra210_dmic(E) snd_soc_tegra210_i2s(E) snd_soc_tegra210_mixer(E) snd_soc_tegra210_admaif(E) snd_soc_tegra210_sfc(E) snd_soc_tegra_pcm(E) loop(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) ghash_ce(E) sha2_ce(E) sha256_arm64(E) sha1_ce(E) snd_soc_spdif_tx(E) snd_soc_tegra_machine_driver(E) pps_gen_gpio(E) heart_ctl(E) nct1008(E) snd_soc_tegra210_adsp(E) snd_soc_tegra_utils(E)
Feb  3 23:21:03 nvidia kernel: [10621.741402]  snd_soc_simple_card_utils(E) userspace_alert(E) tegra_bpmp_thermal(E) nvadsp(E) snd_soc_tegra210_ahub(E) tegra210_adma(E) snd_soc_rt5640(E) snd_hda_codec_hdmi(E) snd_soc_rl6231(E) ar0233yuv(E) snd_hda_tegra(E) snd_hda_codec(E) snd_hda_core(E) spi_tegra114(E) nvidia(OE) ina3221(E) pwm_fan(E) nvgpu(E) binfmt_misc(E) nvmap(E) ip_tables(E) x_tables(E) [last unloaded: mtd]
Feb  3 23:21:03 nvidia kernel: [10621.741422] CPU: 1 PID: 3808 Comm: syzk-pdriver Tainted: G        W  OE     5.10.104-rt63-tegra #92
Feb  3 23:21:03 nvidia kernel: [10621.741426] Hardware name: Unknown Jetson AGX Orin/Jetson AGX Orin, BIOS 202210.2-26fc186-dirty 04/01/2023
Feb  3 23:21:03 nvidia kernel: [10621.741427] pstate: 60400009 (nZCv daif +PAN -UAO -TCO BTYPE=--)
Feb  3 23:21:03 nvidia kernel: [10621.741429] pc : __vb2_queue_cancel+0x1b4/0x220
Feb  3 23:21:03 nvidia kernel: [10621.741432] lr : __vb2_queue_cancel+0x40/0x220
Feb  3 23:21:03 nvidia kernel: [10621.741434] sp : ffff800020eebb00
Feb  3 23:21:03 nvidia kernel: [10621.741435] x29: ffff800020eebb00 x28: ffff0415c13b3000 
Feb  3 23:21:03 nvidia kernel: [10621.741437] x27: 0000000040045613 x26: 0000000000000000 
Feb  3 23:21:03 nvidia kernel: [10621.741438] x25: ffff800020eebd08 x24: ffff041cc31f01f8 
Feb  3 23:21:03 nvidia kernel: [10621.741440] x23: ffff041697610000 x22: ffff800020eebd08 
Feb  3 23:21:03 nvidia kernel: [10621.741441] x21: ffff041cc31f0740 x20: 0000000000000001 
Feb  3 23:21:03 nvidia kernel: [10621.741443] x19: ffff041cc31f0740 x18: 0000000000000001 
Feb  3 23:21:03 nvidia kernel: [10621.741444] x17: 0000000000000000 x16: ffffab6b9cb6122c 
Feb  3 23:21:03 nvidia kernel: [10621.741445] x15: ffffffffffffffff x14: ffffff0000000000 
Feb  3 23:21:03 nvidia kernel: [10621.741446] x13: ffffffffffffffff x12: 0000000000000028 
Feb  3 23:21:03 nvidia kernel: [10621.741448] x11: 0101010101010101 x10: ffff7f7f7f7f7f7f 
Feb  3 23:21:03 nvidia kernel: [10621.741449] x9 : fefdff3064636e6c x8 : ffffffffffffffff 
Feb  3 23:21:03 nvidia kernel: [10621.741451] x7 : fefefefefefefefe x6 : 0000004d6fa34fe1 
Feb  3 23:21:03 nvidia kernel: [10621.741452] x5 : 00ffffffffffffff x4 : 00105dee3658f363 
Feb  3 23:21:03 nvidia kernel: [10621.741453] x3 : 0000000000000017 x2 : ffffab6b9ccf4970 
Feb  3 23:21:03 nvidia kernel: [10621.741454] x1 : ffff041cc31f05c0 x0 : 0000000000000001 
Feb  3 23:21:03 nvidia kernel: [10621.741457] Call trace:
Feb  3 23:21:03 nvidia kernel: [10621.741458]  __vb2_queue_cancel+0x1b4/0x220
Feb  3 23:21:03 nvidia kernel: [10621.741460]  vb2_core_streamoff+0x34/0xd0
Feb  3 23:21:03 nvidia kernel: [10621.741462]  vb2_streamoff+0x34/0x80
Feb  3 23:21:03 nvidia kernel: [10621.741465]  vb2_ioctl_streamoff+0x58/0x70
Feb  3 23:21:03 nvidia kernel: [10621.741468]  v4l_streamoff+0x40/0x50
Feb  3 23:21:03 nvidia kernel: [10621.741471]  __video_do_ioctl+0x188/0x400
Feb  3 23:21:03 nvidia kernel: [10621.741473]  video_usercopy+0x280/0x7b0
Feb  3 23:21:03 nvidia kernel: [10621.741475]  video_ioctl2+0x40/0x180
Feb  3 23:21:03 nvidia kernel: [10621.741476]  v4l2_ioctl+0x68/0x90
Feb  3 23:21:03 nvidia kernel: [10621.741480]  __arm64_sys_ioctl+0xac/0xf0
Feb  3 23:21:03 nvidia kernel: [10621.741484]  el0_svc_common.constprop.0+0x80/0x1d0
Feb  3 23:21:03 nvidia kernel: [10621.741489]  do_el0_svc+0x38/0xb0
Feb  3 23:21:03 nvidia kernel: [10621.741491]  el0_svc+0x1c/0x30
Feb  3 23:21:03 nvidia kernel: [10621.741495]  el0_sync_handler+0xa8/0xb0
Feb  3 23:21:03 nvidia kernel: [10621.741497]  el0_sync+0x16c/0x180
Feb  3 23:21:03 nvidia kernel: [10621.741499] ---[ end trace 000000000000041b ]---
Feb  3 23:21:03 nvidia kernel: [10621.741987] ------------[ cut here ]------------
Feb  3 23:21:03 nvidia kernel: [10621.741991] WARNING: CPU: 7 PID: 3651 at drivers/media/common/videobuf2/videobuf2-core.c:1956 __vb2_queue_cancel+0x1b4/0x220
Feb  3 23:21:03 nvidia kernel: [10621.742001] Modules linked in: nvidia_modeset(OE) fuse(E) xt_conntrack(E) xt_MASQUERADE(E) nf_conntrack_netlink(E) nfnetlink(E) xt_addrtype(E) iptable_filter(E) iptable_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) libcrc32c(E) br_netfilter(E) mttcan(E) can_dev(E) lzo_rle(E) lzo_compress(E) can_raw(E) zram(E) can(E) overlay(E) ramoops(E) reed_solomon(E) snd_soc_tegra210_ope(E) snd_soc_tegra186_dspk(E) snd_soc_tegra186_asrc(E) snd_soc_tegra210_iqc(E) snd_soc_tegra186_arad(E) snd_soc_tegra210_mvc(E) snd_soc_tegra210_adx(E) snd_soc_tegra210_amx(E) snd_soc_tegra210_afc(E) snd_soc_tegra210_dmic(E) snd_soc_tegra210_i2s(E) snd_soc_tegra210_mixer(E) snd_soc_tegra210_admaif(E) snd_soc_tegra210_sfc(E) snd_soc_tegra_pcm(E) loop(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) ghash_ce(E) sha2_ce(E) sha256_arm64(E) sha1_ce(E) snd_soc_spdif_tx(E) snd_soc_tegra_machine_driver(E) pps_gen_gpio(E) heart_ctl(E) nct1008(E) snd_soc_tegra210_adsp(E) snd_soc_tegra_utils(E)
Feb  3 23:21:03 nvidia kernel: [10621.742039]  snd_soc_simple_card_utils(E) userspace_alert(E) tegra_bpmp_thermal(E) nvadsp(E) snd_soc_tegra210_ahub(E) tegra210_adma(E) snd_soc_rt5640(E) snd_hda_codec_hdmi(E) snd_soc_rl6231(E) ar0233yuv(E) snd_hda_tegra(E) snd_hda_codec(E) snd_hda_core(E) spi_tegra114(E) nvidia(OE) ina3221(E) pwm_fan(E) nvgpu(E) binfmt_misc(E) nvmap(E) ip_tables(E) x_tables(E) [last unloaded: mtd]
Feb  3 23:21:03 nvidia kernel: [10621.742055] CPU: 7 PID: 3651 Comm: syzk-pdriver Tainted: G        W  OE     5.10.104-rt63-tegra #92
Feb  3 23:21:03 nvidia kernel: [10621.742058] Hardware name: Unknown Jetson AGX Orin/Jetson AGX Orin, BIOS 202210.2-26fc186-dirty 04/01/2023
Feb  3 23:21:03 nvidia kernel: [10621.742059] pstate: 60400009 (nZCv daif +PAN -UAO -TCO BTYPE=--)
Feb  3 23:21:03 nvidia kernel: [10621.742062] pc : __vb2_queue_cancel+0x1b4/0x220
Feb  3 23:21:03 nvidia kernel: [10621.742064] lr : __vb2_queue_cancel+0x40/0x220
Feb  3 23:21:03 nvidia kernel: [10621.742066] sp : ffff80001cffbb00
Feb  3 23:21:03 nvidia kernel: [10621.742067] x29: ffff80001cffbb00 x28: ffff0419815baa00 
Feb  3 23:21:03 nvidia kernel: [10621.742069] x27: 0000000040045613 x26: 0000000000000000 
Feb  3 23:21:03 nvidia kernel: [10621.742070] x25: ffff80001cffbd08 x24: ffff041cc31f21f8 
Feb  3 23:21:03 nvidia kernel: [10621.742072] x23: ffff04167c52bc00 x22: ffff80001cffbd08 
Feb  3 23:21:03 nvidia kernel: [10621.742073] x21: ffff041cc31f2740 x20: 0000000000000001 
Feb  3 23:21:03 nvidia kernel: [10621.742075] x19: ffff041cc31f2740 x18: 0000000000000001 
Feb  3 23:21:03 nvidia kernel: [10621.742076] x17: 0000000000000000 x16: ffffab6b9cb6122c 
Feb  3 23:21:03 nvidia kernel: [10621.742078] x15: ffffffffffffffff x14: ffffff0000000000 
Feb  3 23:21:03 nvidia kernel: [10621.742079] x13: ffffffffffffffff x12: 0000000000000028 
Feb  3 23:21:03 nvidia kernel: [10621.742081] x11: 0101010101010101 x10: ffff7f7f7f7f7f7f 
Feb  3 23:21:03 nvidia kernel: [10621.742082] x9 : fefdff3064636e6c x8 : ffffffffffffffff 
Feb  3 23:21:03 nvidia kernel: [10621.742084] x7 : ffffab6b9de75df8 x6 : 0000004d6fa34fe1 
Feb  3 23:21:03 nvidia kernel: [10621.742085] x5 : 00ffffffffffffff x4 : 00105dee3658f363 
Feb  3 23:21:03 nvidia kernel: [10621.742086] x3 : 0000000000000017 x2 : ffffab6b9ccf4970 
Feb  3 23:21:03 nvidia kernel: [10621.742088] x1 : ffff041cc31f05c0 x0 : 0000000000000001 
Feb  3 23:21:03 nvidia kernel: [10621.742090] Call trace:
Feb  3 23:21:03 nvidia kernel: [10621.742091]  __vb2_queue_cancel+0x1b4/0x220
Feb  3 23:21:03 nvidia kernel: [10621.742093]  vb2_core_streamoff+0x34/0xd0
Feb  3 23:21:03 nvidia kernel: [10621.742095]  vb2_streamoff+0x34/0x80
Feb  3 23:21:03 nvidia kernel: [10621.742099]  vb2_ioctl_streamoff+0x58/0x70
Feb  3 23:21:03 nvidia kernel: [10621.742101]  v4l_streamoff+0x40/0x50
Feb  3 23:21:03 nvidia kernel: [10621.742103]  __video_do_ioctl+0x188/0x400
Feb  3 23:21:03 nvidia kernel: [10621.742105]  video_usercopy+0x280/0x7b0
Feb  3 23:21:03 nvidia kernel: [10621.742107]  video_ioctl2+0x40/0x180
Feb  3 23:21:03 nvidia kernel: [10621.742109]  v4l2_ioctl+0x68/0x90
Feb  3 23:21:03 nvidia kernel: [10621.742112]  __arm64_sys_ioctl+0xac/0xf0
Feb  3 23:21:03 nvidia kernel: [10621.742116]  el0_svc_common.constprop.0+0x80/0x1d0
Feb  3 23:21:03 nvidia kernel: [10621.742120]  do_el0_svc+0x38/0xb0
Feb  3 23:21:03 nvidia kernel: [10621.742123]  el0_svc+0x1c/0x30
Feb  3 23:21:03 nvidia kernel: [10621.742127]  el0_sync_handler+0xa8/0xb0
Feb  3 23:21:03 nvidia kernel: [10621.742129]  el0_sync+0x16c/0x180
Feb  3 23:21:03 nvidia kernel: [10621.742131] ---[ end trace 000000000000041c ]---
Feb  3 23:21:03 nvidia kernel: [10621.745395] ------------[ cut here ]------------
Feb  3 23:21:03 nvidia kernel: [10621.745398] WARNING: CPU: 0 PID: 4829 at drivers/media/common/videobuf2/videobuf2-core.c:1956 __vb2_queue_cancel+0x1b4/0x220
Feb  3 23:21:03 nvidia kernel: [10621.745407] Modules linked in: nvidia_modeset(OE) fuse(E) xt_conntrack(E) xt_MASQUERADE(E) nf_conntrack_netlink(E) nfnetlink(E) xt_addrtype(E) iptable_filter(E) iptable_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) libcrc32c(E) br_netfilter(E) mttcan(E) can_dev(E) lzo_rle(E) lzo_compress(E) can_raw(E) zram(E) can(E) overlay(E) ramoops(E) reed_solomon(E) snd_soc_tegra210_ope(E) snd_soc_tegra186_dspk(E) snd_soc_tegra186_asrc(E) snd_soc_tegra210_iqc(E) snd_soc_tegra186_arad(E) snd_soc_tegra210_mvc(E) snd_soc_tegra210_adx(E) snd_soc_tegra210_amx(E) snd_soc_tegra210_afc(E) snd_soc_tegra210_dmic(E) snd_soc_tegra210_i2s(E) snd_soc_tegra210_mixer(E) snd_soc_tegra210_admaif(E) snd_soc_tegra210_sfc(E) snd_soc_tegra_pcm(E) loop(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) ghash_ce(E) sha2_ce(E) sha256_arm64(E) sha1_ce(E) snd_soc_spdif_tx(E) snd_soc_tegra_machine_driver(E) pps_gen_gpio(E) heart_ctl(E) nct1008(E) snd_soc_tegra210_adsp(E) snd_soc_tegra_utils(E)
Feb  3 23:21:03 nvidia kernel: [10621.745443]  snd_soc_simple_card_utils(E) userspace_alert(E) tegra_bpmp_thermal(E) nvadsp(E) snd_soc_tegra210_ahub(E) tegra210_adma(E) snd_soc_rt5640(E) snd_hda_codec_hdmi(E) snd_soc_rl6231(E) ar0233yuv(E) snd_hda_tegra(E) snd_hda_codec(E) snd_hda_core(E) spi_tegra114(E) nvidia(OE) ina3221(E) pwm_fan(E) nvgpu(E) binfmt_misc(E) nvmap(E) ip_tables(E) x_tables(E) [last unloaded: mtd]
Feb  3 23:21:03 nvidia kernel: [10621.745457] CPU: 0 PID: 4829 Comm: syzk-video Tainted: G        W  OE     5.10.104-rt63-tegra #92
Feb  3 23:21:03 nvidia kernel: [10621.745460] Hardware name: Unknown Jetson AGX Orin/Jetson AGX Orin, BIOS 202210.2-26fc186-dirty 04/01/2023
Feb  3 23:21:03 nvidia kernel: [10621.745461] pstate: 60400009 (nZCv daif +PAN -UAO -TCO BTYPE=--)
Feb  3 23:21:03 nvidia kernel: [10621.745463] pc : __vb2_queue_cancel+0x1b4/0x220
Feb  3 23:21:03 nvidia kernel: [10621.745465] lr : __vb2_queue_cancel+0x40/0x220
Feb  3 23:21:03 nvidia kernel: [10621.745467] sp : ffff800024cfbb00
Feb  3 23:21:03 nvidia kernel: [10621.745468] x29: ffff800024cfbb00 x28: ffff0419816e6600 
Feb  3 23:21:03 nvidia kernel: [10621.745470] x27: 0000000040045613 x26: 0000000000000000 
Feb  3 23:21:03 nvidia kernel: [10621.745472] x25: ffff800024cfbd08 x24: ffff041cc31f41f8 
Feb  3 23:21:03 nvidia kernel: [10621.745473] x23: ffff0416ff163c00 x22: ffff800024cfbd08 
Feb  3 23:21:03 nvidia kernel: [10621.745475] x21: ffff041cc31f4740 x20: 0000000000000001 
Feb  3 23:21:03 nvidia kernel: [10621.745476] x19: ffff041cc31f4740 x18: 0000000000000001 
Feb  3 23:21:03 nvidia kernel: [10621.745478] x17: 0000000000000000 x16: ffffab6b9cb6122c 
Feb  3 23:21:03 nvidia kernel: [10621.745479] x15: ffffffffffffffff x14: ffffff0000000000 
Feb  3 23:21:03 nvidia kernel: [10621.745481] x13: ffffffffffffffff x12: 0000000000000028 
Feb  3 23:21:03 nvidia kernel: [10621.745482] x11: 0101010101010101 x10: ffff7f7f7f7f7f7f 
Feb  3 23:21:03 nvidia kernel: [10621.745483] x9 : fefdff3064636e6c x8 : ffffffffffffffff 
Feb  3 23:21:03 nvidia kernel: [10621.745485] x7 : ffffab6b9de75df8 x6 : 0000004d6fa34fe1 
Feb  3 23:21:03 nvidia kernel: [10621.745486] x5 : 00ffffffffffffff x4 : 00105dee3658f363 
Feb  3 23:21:03 nvidia kernel: [10621.745488] x3 : 0000000000000017 x2 : ffffab6b9ccf4970 
Feb  3 23:21:03 nvidia kernel: [10621.745489] x1 : ffff041cc31f05c0 x0 : 0000000000000001 
Feb  3 23:21:03 nvidia kernel: [10621.745491] Call trace:
Feb  3 23:21:03 nvidia kernel: [10621.745492]  __vb2_queue_cancel+0x1b4/0x220
Feb  3 23:21:03 nvidia kernel: [10621.745494]  vb2_core_streamoff+0x34/0xd0
Feb  3 23:21:03 nvidia kernel: [10621.745497]  vb2_streamoff+0x34/0x80
Feb  3 23:21:03 nvidia kernel: [10621.745499]  vb2_ioctl_streamoff+0x58/0x70
Feb  3 23:21:03 nvidia kernel: [10621.745502]  v4l_streamoff+0x40/0x50
Feb  3 23:21:03 nvidia kernel: [10621.745504]  __video_do_ioctl+0x188/0x400
Feb  3 23:21:03 nvidia kernel: [10621.745506]  video_usercopy+0x280/0x7b0
Feb  3 23:21:03 nvidia kernel: [10621.745508]  video_ioctl2+0x40/0x180
Feb  3 23:21:03 nvidia kernel: [10621.745510]  v4l2_ioctl+0x68/0x90
Feb  3 23:21:03 nvidia kernel: [10621.745513]  __arm64_sys_ioctl+0xac/0xf0
Feb  3 23:21:03 nvidia kernel: [10621.745517]  el0_svc_common.constprop.0+0x80/0x1d0
Feb  3 23:21:03 nvidia kernel: [10621.745521]  do_el0_svc+0x38/0xb0
Feb  3 23:21:03 nvidia kernel: [10621.745524]  el0_svc+0x1c/0x30
Feb  3 23:21:03 nvidia kernel: [10621.745527]  el0_sync_handler+0xa8/0xb0
Feb  3 23:21:03 nvidia kernel: [10621.745529]  el0_sync+0x16c/0x180
Feb  3 23:21:03 nvidia kernel: [10621.745531] ---[ end trace 000000000000041d ]---
Feb  3 23:21:03 nvidia kernel: [10621.942617] videobuf2_common: driver bug: stop_streaming operation is leaving buf 000000007607cbcd in active state
Feb  3 23:21:03 nvidia kernel: [10621.943264] videobuf2_common: driver bug: stop_streaming operation is leaving buf 00000000c889d5a3 in active state
Feb  3 23:21:03 nvidia kernel: [10621.947081] mttcan c320000.mttcan can1: mttcan_poll_ir: some msgs lost on in Q0
Feb  3 23:21:03 nvidia kernel: [10621.947397] mttcan c310000.mttcan can0: mttcan_poll_ir: some msgs lost on in Q0
Feb  3 23:21:03 nvidia kernel: [10621.951649] videobuf2_common: driver bug: stop_streaming operation is leaving buf 00000000a7298fb0 in active state
Feb  3 23:21:03 nvidia kernel: [10621.972945] bwmgr API not supported
Feb  3 23:21:03 nvidia kernel: [10621.994492] ar0233yuv 2-001b: ar0233yuv_set_mode: mode index:0
Feb  3 23:21:03 nvidia kernel: [10621.994507] ar0233yuv 2-001b: ar0233yuv_start_streaming: start streaming
Feb  3 23:21:03 nvidia kernel: [10622.005954] [RCE] ERROR: camera-ip/nvcsi/nvcsi.c:1925 [nvcsi_stream_set_config] "MIPI clock rate not known. Using 250000 kHz
Feb  3 23:21:03 nvidia kernel: [10622.005961] [RCE] "
Feb  3 23:21:03 nvidia kernel: [10622.008591] bwmgr API not supported
Feb  3 23:21:03 nvidia kernel: [10622.010222] bwmgr API not supported
Feb  3 23:21:03 nvidia kernel: [10622.017921] bwmgr API not supported
Feb  3 23:21:03 nvidia kernel: [10622.029890] ar0233yuv 2-0022: ar0233yuv_set_mode: mode index:0
Feb  3 23:21:03 nvidia kernel: [10622.029898] ar0233yuv 2-0022: ar0233yuv_start_streaming: start streaming
Feb  3 23:21:03 nvidia kernel: [10622.031498] ar0233yuv 2-0021: ar0233yuv_set_mode: mode index:0
Feb  3 23:21:03 nvidia kernel: [10622.031505] ar0233yuv 2-0021: ar0233yuv_start_streaming: start streaming
Feb  3 23:21:03 nvidia kernel: [10622.039292] ar0233yuv 2-001f: ar0233yuv_set_mode: mode index:0
Feb  3 23:21:03 nvidia kernel: [10622.039300] ar0233yuv 2-001f: ar0233yuv_start_streaming: start streaming

故障出现在10620时刻。日志中关闭摄像头是应用层逻辑，当检测到摄像头没有数据后会执行关闭摄像头操作，检测超时时间约500ms。故障在持续大约1秒后会自动恢复。这会是哪里的问题？

5

hello 525554536,

it looks VI did not receiving frames correctly. there’re discarding frame reported for dropping capture buffers.
as you can see, it has error recover mechanism attempts to restore the capture channel, even though it keeps failed due to NULL VI channel.
hence,
when you try to terminate the capture application, it shows the following kernel panic.
it’s internally kernel operation to handle that panic, that’s why you’ll have ~1sec system stall.

let’s verify your sensor functionality,
is your sensor driver works normally before applying RT patches?

6

我们测试过丢帧后关闭摄像头的操作，并不会引起系统卡顿。异常摄像头的数量从1个到7个都做过对应测试。针对现场环境，丢帧问题是有雨设备线缆过长导致的且无法针对这个问题做硬件更改，所以只能只兼容这个错误，引入了断流重新打开摄像头的操作。所以应该并不单纯是摄像头本身驱动的问题。还有日志中也有提到了CAN的报错信息“some msgs lost on in Q0”，这是表明系统处于完全卡死状态，无法响应接收中断吗？

7

hello 525554536,

you may dig into kernel layer for understanding why there’s a kernel panic,
for example, is it related to multi-cam scenario? is this happened for single camera use-case?

8

摄像头的报错在打了vi-output占用率100%的优化补丁后就会一直存在。并不是系统卡顿时所独有的或者新增的。

9

please point-out the patch files you’ve applied.

10

image
image967×372 30.5 KB

11

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks