All cameras stop work suddenly on orin&jetpack 5.1.3

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
1

Hi, nvidia teams:
our board use" jetson orin & jetpack 5.1.3", we use orin csi 4 lane mode, and connect 7 cameras to orin csi ports, one 8M cameras, and six 2M cameras. there is a problem sometimes: all cameras stop works suddenly and can not recover. the log is as follow, I upload logs in the attachment. so what is the reason and how to solve it?
kern.log_2025-03-04_14_10_50.zip (4.8 KB)

Mar 4 14:04:14 mos kernel: [ 3348.427427] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 131072
Mar 4 14:04:14 mos kernel: [ 3348.427430] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 131072
Mar 4 14:04:14 mos kernel: [ 3348.454525] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 131072
Mar 4 14:04:14 mos kernel: [ 3348.454562] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 131072
Mar 4 14:04:14 mos kernel: [ 3348.464794] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
Mar 4 14:04:14 mos kernel: [ 3348.466717] [RCE] VM0 deactivating.VM0 activating.ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] “General error queue is out of sync with frame queue. ts=3361214364320 sof_ts=3361214864704 gerror_code=2 gerror_data=a4 notify_bits=0”
Mar 4 14:04:14 mos kernel: [ 3348.466721] [RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] “General error queue is out of sync with frame queue. ts=3361214372928 sof_ts=3361214871328 gerror_code=2 gerror_data=a5 notify_bits=0”
Mar 4 14:04:14 mos kernel: [ 3348.466724] [RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] “General error queue is out of sync with frame queue. ts=3361214360160 sof_ts=3361214859008 gerror_code=2 gerror_data=a4 notify_bits=0”
Mar 4 14:04:14 mos kernel: [ 3348.466728] [RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] “General error queue is out of sync with frame queue. ts=3361214366144 sof_ts=3361214864992 gerror_code=2 gerror_data=a5 notify_bits=0”
Mar 4 14:04:14 mos kernel: [ 3348.470690] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
Mar 4 14:04:14 mos kernel: [ 3348.470964] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
Mar 4 14:04:14 mos kernel: [ 3348.478043] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
Mar 4 14:04:17 mos kernel: [ 3351.226712] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Mar 4 14:04:17 mos kernel: [ 3351.230650] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Mar 4 14:04:17 mos kernel: [ 3351.235880] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Mar 4 14:04:17 mos kernel: [ 3351.246575] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Mar 4 14:04:17 mos kernel: [ 3351.271226] (NULL device *): vi_capture_control_message: NULL VI channel received
Mar 4 14:04:17 mos kernel: [ 3351.271323] (NULL device *): vi_capture_control_message: NULL VI channel received
Mar 4 14:04:17 mos kernel: [ 3351.274963] (NULL device *): vi_capture_control_message: NULL VI channel received
Mar 4 14:04:17 mos kernel: [ 3351.274967] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=4, csi_port=4
Mar 4 14:04:17 mos kernel: [ 3351.274970] (NULL device *): vi_capture_control_message: NULL VI channel received
Mar 4 14:04:17 mos kernel: [ 3351.274971] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 4 vc- 1
Mar 4 14:04:17 mos kernel: [ 3351.275501] ------------[ cut here ]------------
Mar 4 14:04:17 mos kernel: [ 3351.275502] channel context at 1 is busy
Mar 4 14:04:17 mos kernel: [ 3351.275530] WARNING: CPU: 0 PID: 94316 at /home/zhidao/jenkins-workspace/workspace/MOGOOS-OS-A741/A744/Kernel_src/kernel/nvidia/drivers/platform/tegra/rtcpu/capture-ivc.c:176 tegra_capture_ivc_notify_chan_id+0x188/0x1b0
Mar 4 14:04:17 mos kernel: [ 3351.275531] Modules linked in: mogo_max20087 mogo_imx390 fuse nvidia_modeset(O) mttcan can_dev can_raw can xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c xt_addrtype iptable_filter br_netfilter lzo_rle lzo_compress zram overlay ramoops reed_solomon nvgpu snd_soc_tegra186_asrc snd_soc_tegra210_ope snd_soc_tegra186_dspk snd_soc_tegra210_iqc snd_soc_tegra210_afc snd_soc_tegra210_mvc snd_soc_tegra186_arad snd_soc_tegra210_dmic aes_ce_blk snd_soc_tegra210_amx snd_soc_tegra210_adx crypto_simd cryptd snd_soc_tegra210_i2s snd_soc_tegra210_admaif aes_ce_cipher ghash_ce snd_soc_tegra210_mixer snd_soc_tegra_pcm snd_soc_tegra210_sfc sha2_ce sha256_arm64 sha1_ce snd_soc_tegra210_adsp snd_hda_codec_hdmi snd_soc_spdif_tx snd_soc_tegra_machine_driver pwm_fan userspace_alert snd_soc_tegra_utils nct1008 ina3221 snd_soc_simple_card_utils nvidia(O) tegra_bpmp_thermal snd_hda_tegra i2c_nvvrs11 snd_hda_codec nvadsp snd_soc_rt5640
Mar 4 14:04:17 mos kernel: [ 3351.275588] snd_hda_core snd_soc_tegra210_ahub tegra210_adma mogo_max9295a snd_soc_rl6231 spi_tegra114 binfmt_misc nvmap trace_noschedule trace_irqoff ip_tables x_tables [last unloaded: mtd]
Mar 4 14:04:17 mos kernel: [ 3351.275601] CPU: 0 PID: 94316 Comm: vi-output, mogo Tainted: G O 5.10.192-tegra #1
Mar 4 14:04:17 mos kernel: [ 3351.275602] Hardware name: Jetson AGX Orin Developer Kit (DT)
Mar 4 14:04:17 mos kernel: [ 3351.275604] pstate: 40c00009 (nZcv daif +PAN +UAO -TCO BTYPE=–)
Mar 4 14:04:17 mos kernel: [ 3351.275606] pc : tegra_capture_ivc_notify_chan_id+0x188/0x1b0
Mar 4 14:04:17 mos kernel: [ 3351.275607] lr : tegra_capture_ivc_notify_chan_id+0x188/0x1b0
Mar 4 14:04:17 mos kernel: [ 3351.275608] sp : ffff800026583ad0
Mar 4 14:04:17 mos kernel: [ 3351.275609] x29: ffff800026583ad0 x28: 0000000000000000
Mar 4 14:04:17 mos kernel: [ 3351.275610] x27: 0000000000001043 x26: ffff3b4604341c00
Mar 4 14:04:17 mos kernel: [ 3351.275612] x25: ffff3b44fce3e838 x24: ffff3b4404dee088
Mar 4 14:04:17 mos kernel: [ 3351.275613] x23: 0000000000000047 x22: ffff3b4404dee080
Mar 4 14:04:17 mos kernel: [ 3351.275615] x21: ffff3b4404dee960 x20: 0000000000000001
Mar 4 14:04:17 mos kernel: [ 3351.275616] x19: 0000000000000047 x18: 0000000000000000
Mar 4 14:04:17 mos kernel: [ 3351.275617] x17: 0000000000000000 x16: 0000000000000000
Mar 4 14:04:17 mos kernel: [ 3351.275619] x15: ffff3b4547c922f0 x14: ffffffffffffffff
Mar 4 14:04:17 mos kernel: [ 3351.275620] x13: ffffd6d708420e28 x12: ffffd6d708420a5b
Mar 4 14:04:17 mos kernel: [ 3351.275621] x11: 0000000000000000 x10: 0000000000000ab0
Mar 4 14:04:17 mos kernel: [ 3351.275623] x9 : ffff800026583ad0 x8 : 2073692031207461
Mar 4 14:04:17 mos kernel: [ 3351.275624] x7 : 20747865746e6f63 x6 : 0000000017d1e356
Mar 4 14:04:17 mos kernel: [ 3351.275626] x5 : ffff3b532cfb09b8 x4 : 00000000fffff433
Mar 4 14:04:17 mos kernel: [ 3351.275627] x3 : 0000000000000001 x2 : ffff3b532cfb09c0
Mar 4 14:04:17 mos kernel: [ 3351.275628] x1 : 6fa42fa04af43200 x0 : 0000000000000000
Mar 4 14:04:17 mos kernel: [ 3351.275630] Call trace:
Mar 4 14:04:17 mos kernel: [ 3351.275632] tegra_capture_ivc_notify_chan_id+0x188/0x1b0
Mar 4 14:04:17 mos kernel: [ 3351.275636] vi_capture_setup+0x35c/0x660
Mar 4 14:04:17 mos kernel: [ 3351.275637] tegra_channel_capture_setup+0xe4/0x2c0
Mar 4 14:04:17 mos kernel: [ 3351.275638] vi5_channel_error_recover+0x124/0x250
Mar 4 14:04:17 mos kernel: [ 3351.275641] tegra_channel_error_recover+0x40/0x80
Mar 4 14:04:17 mos kernel: [ 3351.275642] tegra_channel_kthread_capture_dequeue+0x124/0x1e0
Mar 4 14:04:17 mos kernel: [ 3351.275645] kthread+0x148/0x170
Mar 4 14:04:17 mos kernel: [ 3351.275648] ret_from_fork+0x10/0x24
Mar 4 14:04:17 mos kernel: [ 3351.275649] —[ end trace fe5df8a01257c6b0 ]—
Mar 4 14:04:17 mos kernel: [ 3351.275762] tegra194-vi5 13e40000.host1x:vi0@15c00000: failed to update control callback
Mar 4 14:04:17 mos kernel: [ 3351.275806] tegra-camrtc-capture-vi tegra-capture-vi: vi capture setup failed
Mar 4 14:04:17 mos kernel: [ 3351.278950] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=4, csi_port=4
Mar 4 14:04:17 mos kernel: [ 3351.287075] (NULL device *): vi_capture_control_message: NULL VI channel received
Mar 4 14:04:17 mos kernel: [ 3351.291891] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=5, csi_port=6
Mar 4 14:04:17 mos kernel: [ 3351.291894] (NULL device *): vi_capture_control_message: NULL VI channel received
Mar 4 14:04:17 mos kernel: [ 3351.291897] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 5 vc- 2
Mar 4 14:04:17 mos kernel: [ 3351.296614] (NULL device *): vi_capture_control_message: NULL VI channel received
Mar 4 14:04:17 mos kernel: [ 3351.302781] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
Mar 4 14:04:17 mos kernel: [ 3351.306533] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=5, csi_port=6
Mar 4 14:04:17 mos kernel: [ 3351.314353] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 4 vc- 0
Mar 4 14:04:17 mos kernel: [ 3351.323397] (NULL device *): vi_capture_control_message: NULL VI channel received
Mar 4 14:04:17 mos kernel: [ 3351.323401] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 5 vc- 3
Mar 4 14:04:17 mos kernel: [ 3351.324014] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
Mar 4 14:04:17 mos kernel: [ 3351.330010] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
Mar 4 14:04:18 mos kernel: [ 3352.314781] tegra194-vi5 13e40000.host1x:vi0@15c00000: capture control message timed out
Mar 4 14:04:18 mos kernel: [ 3352.326911] tegra194-vi5 13e40000.host1x:vi0@15c00000: vi_capture_release: release channel IVC failed

2

Hi,

For the camera basic functionality first needs to check the device and driver configuration.
You can reference to below program guide for the detailed information of device tree and driver implementation.

Please refer to Applications Using V4L2 IOCTL Directly by using V4L2 IOCTL to verify basic camera functionality.

Once confirm the configure and still failed below link help to get log and some information and some tips for debug.

Thanks!

5

Hi，carolyuu:
Thanks for your reply, This is just an occasional bug. Most of the time, the camera works fine. So there should be no problem with the camera configuration, this is a stability issue.

6

Please apply the changes from below link to verify.

7

Hi.ShanCCC:
Thanks for your reply, we will try it, is there a method to flash “camera-rtcpu-t234-rce.img” only?

8

Check below.

9

OK, thanks！ we will try it.

10

Hi,ShanCCC:
I try flash “camera-rtcpu-t234-rce. img” and upload the flash log. so
1、how to check the update is success?
2、Is there more simple method to update “camera-rtcpu-t234-rce.img”? for example using “dd” command in shell.
flash_rce.log.txt (71.9 KB)

11

Check the sha1 number by below command.

sudo dmesg | grep -i rtcpu