What is the actural spec for Orin MIPI CPHY？

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
,
1

Dear support team：
It looks like the VI block is hitting a performance bottleneck, I can’t open more than 13 Video devices at the same time. the kernel log:

[  184.094341] [RCE] BUG: camera-ip/vi5/vi5.c:415 [vi5_check_falcon_failure] "VI FALCON FAILURE: 0x40000000"
[  184.159960] tegra186-cam-rtcpu bc00000.rtcpu: Alert: Camera RTCPU gone bad! restoring it immediately!!
[  185.102355] tegra194-vi5 13e40000.host1x:vi1@14c00000: capture control message timed out
[  185.110734] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_control_send_message: failed to send IVC control message
[  186.638320] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 40000 ms
[  186.638328] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 40000 ms
[  186.638336] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 40000 ms
[  186.638347] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 40000 ms
[  186.638353] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 40000 ms
[  186.638359] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 40000 ms
[  186.638395] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 40000 ms
[  186.638560] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 40000 ms
[  186.638638] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 40000 ms
[  186.638894] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 40000 ms
[  186.639009] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  186.639022] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 40000 ms
[  186.639039] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 40000 ms
[  186.639637] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  186.639800] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  186.640219] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  186.640310] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  186.640444] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  186.640525] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  186.640544] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 40000 ms
[  186.640769] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  186.640820] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_release: control failed, errno 1
[  186.640824] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_release: control failed, errno 1
[  186.640903] video4linux video13: vi capture release failed
[  186.640906] video4linux video15: vi capture release failed
[  186.640909] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
[  186.640911] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
[  186.640964] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_release: control failed, errno 1
[  186.640976] tegra194-vi5 13e40000.host1x:vi0@15c00000: vi_capture_release: control failed, errno 1
[  186.641008] video4linux video14: vi capture release failed
[  186.641012] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
[  186.641019] video4linux video2: vi capture release failed
[  186.641022] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
[  186.641060] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_release: control failed, errno 1
[  186.641098] video4linux video12: vi capture release failed
[  186.641103] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
[  186.641773] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  186.641786] tegra194-vi5 13e40000.host1x:vi0@15c00000: vi_capture_release: control failed, errno 1
[  186.641832] video4linux video8: vi capture release failed
[  186.641835] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
[  186.641888] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  186.641900] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_release: control failed, errno 1
[  186.641946] video4linux video6: vi capture release failed
[  186.641949] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
[  186.642011] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  186.658007] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_release: control failed, errno 1
[  186.658011] tegra194-vi5 13e40000.host1x:vi0@15c00000: vi_capture_release: control failed, errno 1
[  186.658054] video4linux video10: vi capture release failed
[  186.658058] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
[  186.658800] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  186.659345] tegra194-vi5 13e40000.host1x:vi0@15c00000: vi_capture_release: control failed, errno 1
[  186.659386] video4linux video0: vi capture release failed
[  186.659388] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
[  186.662340] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_release: control failed, errno 1
[  186.662372] video4linux video4: vi capture release failed
[  186.662374] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
[  186.666087] tegra194-vi5 13e40000.host1x:vi0@15c00000: vi_capture_release: control failed, errno 1
[  186.667192] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  186.669584] tegra194-vi5 13e40000.host1x:vi0@15c00000: vi_capture_release: control failed, errno 1
[  186.669620] video4linux video9: vi capture release failed
[  186.669623] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
[  186.675441] video4linux video5: vi capture release failed
[  186.684643] video4linux video1: vi capture release failed
[  186.694441] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
[  186.712312] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
[  413.223054] ------------[ cut here ]------------
[  413.227822] refcount_t: addition on 0; use-after-free.
[  413.227841] WARNING: CPU: 11 PID: 7525 at lib/refcount.c:25 refcount_warn_saturate+0x98/0x140
[  413.236615] Modules linked in: scsensor_site3(E) scsensor_site2(E) scsensor_site1(E) scsensor_site0(E) nvidia_modeset(OE) fuse(E) lzo_rle(E) lzo_compress(E) zram(E) ramoops(E) reed_solomon(E) bnep(E) loop(E) snd_soc_tegra186_asrc(E) snd_soc_tegra210_iqc(E) snd_soc_tegra210_ope(E) snd_soc_tegra210_mvc(E) snd_soc_tegra186_dspk(E) snd_soc_tegra186_arad(E) snd_soc_tegra210_afc(E) snd_soc_tegra210_admaif(E) snd_soc_tegra210_dmic(E) aes_ce_blk(E) snd_soc_tegra210_adx(E) crypto_simd(E) snd_soc_tegra210_amx(E) snd_soc_tegra_pcm(E) snd_soc_tegra210_mixer(E) snd_soc_tegra210_i2s(E) snd_soc_tegra210_sfc(E) cryptd(E) aes_ce_cipher(E) nvgpu(E) ghash_ce(E) sha2_ce(E) rtk_btusb(E) sha256_arm64(E) sha1_ce(E) snd_hda_codec_hdmi(E) snd_soc_tegra210_adsp(E) btusb(E) ucsi_ccg(E) snd_soc_tegra_machine_driver(E) input_leds(E) rtl8822ce(E) btrtl(E) typec_ucsi(E) snd_soc_tegra_utils(E) btbcm(E) snd_hda_tegra(E) btintel(E) snd_soc_simple_card_utils(E) pwm_fan(E) snd_soc_spdif_tx(E) snd_hda_codec(E)
[  413.236670]  snd_soc_rt5640(E) i40e(E) typec(E) nct1008(E) snd_soc_tegra210_ahub(E) ina3221(E) nvadsp(E) snd_soc_rl6231(E) snd_hda_core(E) tegra_bpmp_thermal(E) userspace_alert(E) cfg80211(E) tegra210_adma(E) spi_tegra114(E) nvidia(OE) binfmt_misc(E) nvmap(E) ip_tables(E) x_tables(E) [last unloaded: mtd]
[  413.236693] CPU: 11 PID: 7525 Comm: v4l2-ctl-t186 Tainted: G           OE     5.10.104-tegra #50
[  413.236694] Hardware name: Unknown Jetson AGX Orin/Jetson AGX Orin, BIOS 3.1-32827747 03/19/2023
[  413.236697] pstate: 60400009 (nZCv daif +PAN -UAO -TCO BTYPE=--)
[  413.236699] pc : refcount_warn_saturate+0x98/0x140
[  413.236700] lr : refcount_warn_saturate+0x98/0x140
[  413.236701] sp : ffff8000323a3a70
[  413.236702] x29: ffff8000323a3a70 x28: ffff0f1ec042ba00 
[  413.236704] x27: ffffbf7eaf4e7000 x26: 0000ffff8c0be93c 
[  413.236706] x25: fffffffffffffe00 x24: ffff0f1ec042c000 
[  413.236708] x23: ffff0f258195c1f8 x22: ffff0f25819620a0 
[  413.236710] x21: ffff0f1e8a653a30 x20: ffff0f258195c550 
[  413.236712] x19: ffff0f1e8a653a00 x18: 0000000000000000 
[  413.236714] x17: 0000000000000000 x16: 0000000000000000 
[  413.236716] x15: 0000000000000000 x14: 0000000000000000 
[  413.236717] x13: 0000000000000000 x12: 0000000000000000 
[  413.236719] x11: 0000000000000000 x10: 0000000000000a80 
[  413.236721] x9 : ffff8000323a3a70 x8 : 6572662d72657466 
[  413.236723] x7 : 612d657375203b30 x6 : c0000000ffffefff 
[  413.236725] x5 : ffff0f25aec0e958 x4 : ffffbf7eaf7f7968 
[  413.236727] x3 : 0000000000000001 x2 : ffff0f25aec0e960 
[  413.236729] x1 : 0000000000000000 x0 : 0000000000000000 
[  413.236731] Call trace:
[  413.236733]  refcount_warn_saturate+0x98/0x140
[  413.236736]  kthread_stop+0x4c/0x260
[  413.236740]  vi5_channel_stop_kthreads+0x44/0x60
[  413.236741]  vi5_channel_stop_streaming+0xc0/0xe0
[  413.236743]  tegra_channel_stop_streaming+0x3c/0x70
[  413.236745]  __vb2_queue_cancel+0x40/0x220
[  413.236746]  vb2_core_queue_release+0x30/0x60
[  413.236747]  _vb2_fop_release+0x88/0xb0
[  413.236749]  tegra_channel_close+0x68/0x190
[  413.236751]  v4l2_release+0xc0/0x100
[  413.236754]  __fput+0x80/0x240
[  413.236756]  ____fput+0x24/0x30
[  413.236758]  task_work_run+0x88/0x140
[  413.236760]  do_exit+0x354/0xab0
[  413.236762]  do_group_exit+0x4c/0xb0
[  413.236764]  get_signal+0x104/0x830
[  413.236767]  do_notify_resume+0x184/0xa00
[  413.236768]  work_pending+0xc/0x384
[  413.236769] ---[ end trace 3dafc625905151bc ]---
[  413.242292] ------------[ cut here ]------------
[  413.247050] refcount_t: underflow; use-after-free.
[  413.247062] WARNING: CPU: 11 PID: 7525 at lib/refcount.c:28 refcount_warn_saturate+0xec/0x140
[  413.255837] Modules linked in: scsensor_site3(E) scsensor_site2(E) scsensor_site1(E) scsensor_site0(E) nvidia_modeset(OE) fuse(E) lzo_rle(E) lzo_compress(E) zram(E) ramoops(E) reed_solomon(E) bnep(E) loop(E) snd_soc_tegra186_asrc(E) snd_soc_tegra210_iqc(E) snd_soc_tegra210_ope(E) snd_soc_tegra210_mvc(E) snd_soc_tegra186_dspk(E) snd_soc_tegra186_arad(E) snd_soc_tegra210_afc(E) snd_soc_tegra210_admaif(E) snd_soc_tegra210_dmic(E) aes_ce_blk(E) snd_soc_tegra210_adx(E) crypto_simd(E) snd_soc_tegra210_amx(E) snd_soc_tegra_pcm(E) snd_soc_tegra210_mixer(E) snd_soc_tegra210_i2s(E) snd_soc_tegra210_sfc(E) cryptd(E) aes_ce_cipher(E) nvgpu(E) ghash_ce(E) sha2_ce(E) rtk_btusb(E) sha256_arm64(E) sha1_ce(E) snd_hda_codec_hdmi(E) snd_soc_tegra210_adsp(E) btusb(E) ucsi_ccg(E) snd_soc_tegra_machine_driver(E) input_leds(E) rtl8822ce(E) btrtl(E) typec_ucsi(E) snd_soc_tegra_utils(E) btbcm(E) snd_hda_tegra(E) btintel(E) snd_soc_simple_card_utils(E) pwm_fan(E) snd_soc_spdif_tx(E) snd_hda_codec(E)
[  413.255877]  snd_soc_rt5640(E) i40e(E) typec(E) nct1008(E) snd_soc_tegra210_ahub(E) ina3221(E) nvadsp(E) snd_soc_rl6231(E) snd_hda_core(E) tegra_bpmp_thermal(E) userspace_alert(E) cfg80211(E) tegra210_adma(E) spi_tegra114(E) nvidia(OE) binfmt_misc(E) nvmap(E) ip_tables(E) x_tables(E) [last unloaded: mtd]
[  413.255894] CPU: 11 PID: 7525 Comm: v4l2-ctl-t186 Tainted: G        W  OE     5.10.104-tegra #50
[  413.255895] Hardware name: Unknown Jetson AGX Orin/Jetson AGX Orin, BIOS 3.1-32827747 03/19/2023
[  413.255896] pstate: 60400009 (nZCv daif +PAN -UAO -TCO BTYPE=--)
[  413.255898] pc : refcount_warn_saturate+0xec/0x140
[  413.255900] lr : refcount_warn_saturate+0xec/0x140
[  413.255900] sp : ffff8000323a3a70
[  413.255901] x29: ffff8000323a3a70 x28: ffff0f1ec042ba00 
[  413.255903] x27: ffffbf7eaf4e7000 x26: 0000ffff8c0be93c 
[  413.255906] x25: fffffffffffffe00 x24: ffff0f1ec042c000 
[  413.255907] x23: ffff0f258195c1f8 x22: ffff0f25819620a0 
[  413.255909] x21: ffff0f1e8a653a30 x20: 0000000000000000 
[  413.255911] x19: ffff0f1e8a653a00 x18: 0000000000000010 
[  413.255913] x17: 0000000000000000 x16: ffffbf7eadb03210 
[  413.255915] x15: ffff0f1ec042bf70 x14: ffffffffffffffff 
[  413.255917] x13: ffff8000b23a36a7 x12: ffff8000323a36af 
[  413.255919] x11: 0000000000000000 x10: 0000000000000a80 
[  413.255921] x9 : ffff8000323a3a70 x8 : 72657466612d6573 
[  413.255923] x7 : 75203b776f6c6672 x6 : c0000000ffffefff 
[  413.255925] x5 : ffff0f25aec0e958 x4 : ffffbf7eaf7f7968 
[  413.255927] x3 : 0000000000000001 x2 : ffff0f25aec0e960 
[  413.255928] x1 : 0000000000000000 x0 : 0000000000000000 
[  413.255931] Call trace:
[  413.255933]  refcount_warn_saturate+0xec/0x140
[  413.255934]  kthread_stop+0x24c/0x260
[  413.255937]  vi5_channel_stop_kthreads+0x44/0x60
[  413.255938]  vi5_channel_stop_streaming+0xc0/0xe0
[  413.255940]  tegra_channel_stop_streaming+0x3c/0x70
[  413.255942]  __vb2_queue_cancel+0x40/0x220
[  413.255943]  vb2_core_queue_release+0x30/0x60
[  413.255944]  _vb2_fop_release+0x88/0xb0
[  413.255946]  tegra_channel_close+0x68/0x190
[  413.255948]  v4l2_release+0xc0/0x100
[  413.255950]  __fput+0x80/0x240
[  413.255951]  ____fput+0x24/0x30
[  413.255953]  task_work_run+0x88/0x140
[  413.255955]  do_exit+0x354/0xab0
[  413.255956]  do_group_exit+0x4c/0xb0
[  413.255958]  get_signal+0x104/0x830
[  413.255960]  do_notify_resume+0x184/0xa00
[  413.255962]  work_pending+0xc/0x384
[  413.255963] ---[ end trace 3dafc625905151bd ]---
[  413.317706] [RCE] WARNING: core/hsp/hsp-vm.c:283 [camrtc_hsp_full_ss_isr] "VM0 issued IVC in inactive state."
[  414.285696] tegra194-vi5 13e40000.host1x:vi0@15c00000: capture control message timed out
[  414.294073] tegra194-vi5 13e40000.host1x:vi0@15c00000: vi_capture_control_send_message: failed to send IVC control message
[  414.315538] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=0, csi_port=0
[  414.347737] bwmgr API not supported
[  414.347877] bwmgr req failed for 8
[  414.355082] misc tegra_camera_ctrl: tegra_camera_isomgr_request: ICC failed to reserve 13500000 KBps
[  414.374539] misc tegra_camera_ctrl: tegra_camera_update_isobw: failed to reserve 13500000 KBps with isomgr
[  414.384491] tegra194-vi5 13e40000.host1x:vi0@15c00000: vi_capture_release: setup channel first
[  414.413477] video4linux video0: vi capture release failed

It looks like there is a problem with the camera-ip module, this part of the code is not open source

primo@VM:~/project/nvidia/Linux_for_Tegra$ grep -nr "VI FALCON FAILURE"

Binary file rootfs/opt/ota_package/t19x/bl_only_payload matches
Binary file rootfs/opt/ota_package/t19x/TEGRA_BL.Cap matches
Binary file rootfs/opt/ota_package/t23x/bl_only_payload matches
Binary file rootfs/opt/ota_package/t23x/bl_only_payload_3701_000 matches
Binary file rootfs/opt/ota_package/t23x/TEGRA_BL_3701_000.Cap matches
Binary file rootfs/opt/ota_package/t23x/TEGRA_BL.Cap matches
Binary file bootloader/camera-rtcpu-t194-rce.img matches
Binary file bootloader/system.img matches
Binary file bootloader/system.img.raw matches
Binary file bootloader/camera-rtcpu-t234-rce.img matches

Could you give an accurate spec of the MIPI CPHY? Or what module is limiting the ability to turn on 16 cameras at the same time？

3

hello 541449841,

could you please also share the data-rate of your total 16-camera streams.
as you can see here… Topic 258111, CPHY has validate up-to 3.0 Gsps.
that’s also mentioned in the [Release Notes (r35.4.1) ]

2.3. Camera

  • Issue 4175410
    • There is a limitation on the C-PHY-supported data rate. C-PHY has been tested to work up to 3.0Gsps per trio, so there might be stream failures if data rate goes higher than 3.0Gsps.

BTW,
may I know which Jetpack release you’re using?
could you please moving to the latest release, i.e. JetPack 5.1.2 for verification.
thanks

4

the data-rate is 2.6Gsps

5.1.1 ,we will moving to the latest release(5.1.2) moving to the latest release a little bit later

5

Hi,Jerry
We got the same verification as before,this is the kernel log when Orin crashed:


[   62.678278] site3 30-001f: Detected SCSENSOR sensor
[   68.013711] bwmgr API not supported
[   74.625457] bwmgr API not supported
[   78.772035] bwmgr API not supported
[   82.793627] bwmgr API not supported
[   90.010787] bwmgr API not supported
[   99.014419] bwmgr API not supported
[  103.210202] bwmgr API not supported
[  107.248363] bwmgr API not supported
[  111.743426] bwmgr API not supported
[  119.520061] bwmgr API not supported
[  119.523786] bwmgr req failed for 8
[  119.531020] misc tegra_camera_ctrl: tegra_camera_isomgr_request: ICC failed to reserve 11250000 KBps
[  119.540436] misc tegra_camera_ctrl: tegra_camera_update_isobw: failed to reserve 11250000 KBps with isomgr
[  124.178021] bwmgr API not supported
[  124.178147] bwmgr req failed for 8
[  124.185385] misc tegra_camera_ctrl: tegra_camera_isomgr_request: ICC failed to reserve 12375000 KBps
[  124.194806] misc tegra_camera_ctrl: tegra_camera_update_isobw: failed to reserve 12375000 KBps with isomgr
[  128.364450] bwmgr API not supported
[  128.364573] bwmgr req failed for 8
[  128.371832] misc tegra_camera_ctrl: tegra_camera_isomgr_request: ICC failed to reserve 13500000 KBps
[  128.381258] misc tegra_camera_ctrl: tegra_camera_update_isobw: failed to reserve 13500000 KBps with isomgr
[  133.502373] bwmgr API not supported
[  133.502481] bwmgr req failed for 8
[  133.509728] misc tegra_camera_ctrl: tegra_camera_isomgr_request: ICC failed to reserve 14625000 KBps
[  133.519126] misc tegra_camera_ctrl: tegra_camera_update_isobw: failed to reserve 14625000 KBps with isomgr
[  133.556547] [RCE] BUG: camera-ip/vi5/vi5.c:415 [vi5_check_falcon_failure] "VI FALCON FAILURE: 0x40000000"
[  133.597504] tegra186-cam-rtcpu bc00000.rtcpu: Alert: Camera RTCPU gone bad! restoring it immediately!!
[  133.631956] ------------[ cut here ]------------
[  133.631981] WARNING: CPU: 7 PID: 2732 at /home/hisi/Downloads/Orin35.4.1/Linux_for_Tegra/source/public/kernel/nvidia/drivers/platform/tegra/rtcpu/capture-ivc.c:42 tegra_capture_ivc_tx+0x120/0x150
[  133.649916] Modules linked in: scsensor_site3(E) scsensor_site2(E) scsensor_site1(E) scsensor_site0(E) fuse(E) nvidia_modeset(OE) lzo_rle(E) lzo_compress(E) zram(E) ramoops(E) reed_solomon(E) bnep(E) loop(E) snd_soc_tegra186_dspk(E) snd_soc_tegra210_mvc(E) aes_ce_blk(E) crypto_simd(E) snd_soc_tegra210_ope(E) snd_soc_tegra186_asrc(E) snd_soc_tegra186_arad(E) snd_soc_tegra210_iqc(E) snd_soc_tegra210_afc(E) cryptd(E) snd_soc_tegra210_admaif(E) snd_soc_tegra210_dmic(E) snd_soc_tegra210_adx(E) snd_soc_tegra210_amx(E) aes_ce_cipher(E) snd_soc_tegra210_mixer(E) snd_soc_tegra210_sfc(E) snd_soc_tegra210_i2s(E) snd_soc_tegra_pcm(E) ghash_ce(E) sha2_ce(E) rtk_btusb(E) sha256_arm64(E) nvgpu(E) rtl8822ce(E) sha1_ce(E) btusb(E) snd_hda_codec_hdmi(E) snd_soc_tegra210_adsp(E) ucsi_ccg(E) btrtl(E) snd_soc_tegra_machine_driver(E) input_leds(E) snd_hda_tegra(E) btbcm(E) snd_soc_tegra_utils(E) typec_ucsi(E) btintel(E) pwm_fan(E) max96712(E) snd_soc_spdif_tx(E) snd_soc_simple_card_utils(E) cfg80211(E) typec(E)
[  133.650045]  nvadsp(E) userspace_alert(E) i2c_nvvrs11(E) snd_hda_codec(E) nct1008(E) ina3221(E) snd_soc_tegra210_ahub(E) tegra_bpmp_thermal(E) snd_soc_rt5640(E) snd_hda_core(E) tegra210_adma(E) snd_soc_rl6231(E) spi_tegra114(E) nvidia(OE) binfmt_misc(E) nvmap(E) ip_tables(E) x_tables(E) [last unloaded: mtd]
[  133.650093] CPU: 7 PID: 2732 Comm: vi-output, site Tainted: G           OE     5.10.120-tegra #1
[  133.650097] Hardware name: Unknown Jetson AGX Orin Developer Kit/Jetson AGX Orin Developer Kit, BIOS 4.1-33958178 08/01/2023
[  133.650102] pstate: 60c00009 (nZCv daif +PAN +UAO -TCO BTYPE=--)
[  133.650108] pc : tegra_capture_ivc_tx+0x120/0x150
[  133.650113] lr : tegra_capture_ivc_tx+0x30/0x150
[  133.650116] sp : ffff80002831bc30
[  133.650118] x29: ffff80002831bc30 x28: ffff800013607000 
[  133.650125] x27: 0000000000000001 x26: ffff1a32c20b6080 
[  133.650131] x25: 0000000000000010 x24: 0000000000002000 
[  133.650137] x23: ffff80002831bd18 x22: ffff1a2bfde9bb00 
[  133.650142] x21: ffff1a2bc3b3f400 x20: 0000000000000010 
[  133.650148] x19: ffff1a2bc4818080 x18: 0000007ff3c00000 
[  133.650153] x17: 0000000000000000 x16: ffffda9bcce693a0 
[  133.650159] x15: 000000000000002b x14: 00000000000000c4 
[  133.650164] x13: 0000000000000000 x12: ffff8000132e3000 
[  133.650170] x11: 0000000000000000 x10: 0000000000001000 
[  133.650176] x9 : ffff800013607000 x8 : 00000aac00000000 
[  133.650181] x7 : 0000004200000000 x6 : ffff1a2bc6931830 
[  133.650187] x5 : 000000000000006e x4 : 0000000000000000 
[  133.650192] x3 : 0000000b00000aac x2 : ffffda9bcbe20d70 
[  133.650198] x1 : 0000000000000000 x0 : 0000000000000000 
[  133.650203] Call trace:
[  133.650210]  tegra_capture_ivc_tx+0x120/0x150
[  133.650216]  tegra_capture_ivc_capture_submit+0x38/0x60
[  133.650223]  vi_capture_request+0xc8/0x1a0
[  133.650229]  tegra_channel_kthread_capture_enqueue+0x2dc/0x4c0
[  133.650235]  kthread+0x148/0x170
[  133.650240]  ret_from_fork+0x10/0x24
[  133.650244] ---[ end trace 914c465eefbe0e2a ]---
[  133.655049] tegra194-vi5 13e40000.host1x:vi0@15c00000: IVC capture submit failed
[  133.662686] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request dispatch err -5
[  134.548545] tegra194-vi5 13e40000.host1x:vi1@14c00000: capture control message timed out
[  134.556894] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_control_send_message: failed to send IVC control message
[  136.116572] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  136.125729] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  136.135362] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_release: control failed, errno 1
[  136.144631] video4linux video12: vi capture release failed
[  136.150283] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
[  136.376543] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  136.385684] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  136.395347] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_release: control failed, errno 1
[  136.404616] video4linux video7: vi capture release failed
[  136.410450] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
[  136.884561] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  136.893749] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  136.903463] tegra194-vi5 13e40000.host1x:vi0@15c00000: vi_capture_release: control failed, errno 1
[  136.912792] video4linux video3: vi capture release failed
[  136.918413] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
[  138.932955] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  138.944548] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  138.954307] tegra194-vi5 13e40000.host1x:vi0@15c00000: vi_capture_release: control failed, errno 1
[  138.963589] video4linux video2: vi capture release failed
[  138.969154] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
[  139.960545] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  139.969709] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  139.979430] tegra194-vi5 13e40000.host1x:vi0@15c00000: vi_capture_release: control failed, errno 1
[  139.988911] video4linux video9: vi capture release failed
[  139.994490] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
6

I just used the cat command to turn on the camera one by one

cat /dev/video0
cat /dev/video1
cat /dev/video2
cat /dev/video3
cat /dev/video4
cat /dev/video5
cat /dev/video6
...
cat /dev/video12
Orin Crashed

Using the reverse order or any order, Orin always crashes when it opens the 13th device

7

hello 541449841,

FYI, we’ve internally test 12-cam use-case. let me check whether this is limitation.

8

BTW, please also refer to Topic 258971 for kernel patch to address race condition of multi-cam use-case.

9

I got it

10

hello 541449841,

these errors also catch my eye.
it looks like a failure to set bandwidth, and it’s returning ENOMEM error.

[  124.178147] bwmgr req failed for 8
[  124.185385] misc tegra_camera_ctrl: tegra_camera_isomgr_request: ICC failed to reserve 12375000 KBps
[  124.194806] misc tegra_camera_ctrl: tegra_camera_update_isobw: failed to reserve 12375000 KBps with isomgr

it’s tegra_camera_platform.c driver to send the bandwidth request.
for example, $public_sources/kernel_src/kernel/nvidia/drivers/video/tegra/camera/tegra_camera_platform.c

static int tegra_camera_isomgr_request(
                struct tegra_camera_info *info, uint iso_bw, uint lt)
{

        dev_dbg(info->dev,
                "%s++ bw=%u, lt=%u\n", __func__, iso_bw, lt);
...

                ret = icc_set_bw(info->icc_iso_path_handle,
                        iso_bw, (u32)info->max_bw);
                if (ret) {
                        dev_err(info->dev,
                        "%s: ICC failed to reserve %u KBps\n",
                        __func__, iso_bw);

could you please enable the dev_dbg() debug prints,
or, replace it with pr_info() to compare the iso_bw between works/failure use-case.
thanks

11

ok , I 'll try it later

12

It still crashed when stopping the video channel.

13

Hi,Jerry
the kernel log shows:

   85.980338] site0 30-0013: Detected SCSENSOR sensor
[  107.468039] bwmgr API not supported
[  107.468047] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=1125000, lt=0
[  120.609850] bwmgr API not supported
[  120.609858] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=2250000, lt=0
[  150.896356] bwmgr API not supported
[  150.896365] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=3375000, lt=0
[  159.642940] bwmgr API not supported
[  159.642951] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=4500000, lt=0
[  168.568657] bwmgr API not supported
[  168.568664] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=5625000, lt=0
[  176.713823] bwmgr API not supported
[  176.713832] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=6750000, lt=0
[  188.024111] bwmgr API not supported
[  188.024119] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=7875000, lt=0
[  194.146054] bwmgr API not supported
[  194.146062] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=9000000, lt=0
[  217.120649] bwmgr API not supported
[  217.120658] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=10125000, lt=0
[  227.678749] bwmgr API not supported
[  227.678757] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=11250000, lt=0
[  227.691813] bwmgr req failed for 8
[  227.695425] misc tegra_camera_ctrl: tegra_camera_isomgr_request: ICC failed to reserve 11250000 KBps
[  227.704818] misc tegra_camera_ctrl: tegra_camera_update_isobw: failed to reserve 11250000 KBps with isomgr
[  236.113429] bwmgr API not supported
[  236.113437] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=12375000, lt=0
[  236.126490] bwmgr req failed for 8
[  236.130095] misc tegra_camera_ctrl: tegra_camera_isomgr_request: ICC failed to reserve 12375000 KBps
[  236.139514] misc tegra_camera_ctrl: tegra_camera_update_isobw: failed to reserve 12375000 KBps with isomgr
[  243.540928] bwmgr API not supported
[  243.540935] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=13500000, lt=0
[  243.554056] bwmgr req failed for 8
[  243.557697] misc tegra_camera_ctrl: tegra_camera_isomgr_request: ICC failed to reserve 13500000 KBps
[  243.567159] misc tegra_camera_ctrl: tegra_camera_update_isobw: failed to reserve 13500000 KBps with isomgr
[  253.194091] bwmgr API not supported
[  253.194099] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=14625000, lt=0
[  253.207021] [RCE] BUG: camera-ip/vi5/vi5.c:415 [vi5_check_falcon_failure] "VI FALCON FAILURE: 0x40000000"
[  253.207267] bwmgr req failed for 8
[  253.210904] misc tegra_camera_ctrl: tegra_camera_isomgr_request: ICC failed to reserve 14625000 KBps
[  253.220308] misc tegra_camera_ctrl: tegra_camera_update_isobw: failed to reserve 14625000 KBps with isomgr
[  253.291955] tegra186-cam-rtcpu bc00000.rtcpu: Alert: Camera RTCPU gone bad! restoring it immediately!!
[  254.260190] tegra194-vi5 13e40000.host1x:vi0@15c00000: capture control message timed out
[  254.268549] tegra194-vi5 13e40000.host1x:vi0@15c00000: vi_capture_control_send_message: failed to send IVC control message
[  255.823918] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  255.833063] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  255.842770] tegra194-vi5 13e40000.host1x:vi0@15c00000: vi_capture_release: control failed, errno 1
[  255.852046] video4linux video12: vi capture release failed
[  255.857715] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
[  256.079870] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  256.089027] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  256.098695] tegra194-vi5 13e40000.host1x:vi0@15c00000: vi_capture_release: control failed, errno 1
14

kernel1.log (100 KB)

16

hello 541449841,

did you enable camera one-by-one? according to the logs, the request bandwidth keep increasing.

17

yes,I opened camera one-by-one.

1、v4l2-ctl -d /dev/video0 --stream-mmap
2、v4l2-ctl -d /dev/video1 --stream-mmap
3、v4l2-ctl -d /dev/video2 --stream-mmap
4、v4l2-ctl -d /dev/video3 --stream-mmap
.....
13、v4l2-ctl -d /dev/video12 --stream-mmap
...crashed

the request bandwidth shouldn’t keep increasing,right?

18

hello 541449841,

those commands keep camera streaming continuously.
please try ctrl^c to shutdown the stream, let’s check the approach to enable cameras individually, assume the bandwidth should be the same for each video node.

19

yes,the bandwidth is the same for each one.

  156.329210] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=1125000, lt=0
[  158.792904] bwmgr API not supported
[  158.792918] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=0, lt=0
[  162.215847] bwmgr API not supported
[  162.215856] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=1125000, lt=0
[  162.262019] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
[  162.297491] [RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] "General error queue is out of sync with frame queue. ts=182818425792 sof_ts=182819183584 gerror_code=2 gerror_data=a0 notify_bits=200004"
[  164.793239] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  164.802409] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  164.813081] (NULL device *): vi_capture_control_message: NULL VI channel received
[  164.820799] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=0, csi_port=0
[  164.831467] (NULL device *): vi_capture_control_message: NULL VI channel received
[  164.839186] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 0 vc- 2
[  164.849940] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  168.257373] bwmgr API not supported
[  168.257386] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=0, lt=0
[  171.456007] bwmgr API not supported
[  171.456017] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=1125000, lt=0
[  171.515995] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
[  171.520786] [RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] "General error queue is out of sync with frame queue. ts=192042613632 sof_ts=192073763520 gerror_code=2 gerror_data=400 notify_bits=40030"
[  174.264423] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  174.273583] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  174.284100] (NULL device *): vi_capture_control_message: NULL VI channel received
[  174.291818] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=0, csi_port=0
[  174.302465] (NULL device *): vi_capture_control_message: NULL VI channel received
[  174.310177] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 0 vc- 1
[  174.320933] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  176.213878] bwmgr API not supported
[  176.213889] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=0, lt=0
[  178.613473] bwmgr API not supported
[  178.613481] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=1125000, lt=0
[  178.670004] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
[  178.712098] [RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] "General error queue is out of sync with frame queue. ts=199223478848 sof_ts=199227987072 gerror_code=2 gerror_data=800660 notify_bits=40030"
[  181.431919] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  181.441101] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  181.451079] (NULL device *): vi_capture_control_message: NULL VI channel received
[  181.458792] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=0, csi_port=0
[  181.469450] (NULL device *): vi_capture_control_message: NULL VI channel received
[  181.477159] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 0 vc- 0
[  181.487917] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  183.567339] bwmgr API not supported
[  183.567351] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=0, lt=0
[  186.166751] bwmgr API not supported
[  186.166758] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=1125000, lt=0
[  188.733372] bwmgr API not supported
[  188.733381] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=0, lt=0
[  191.750500] bwmgr API not supported
[  191.750508] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=1125000, lt=0
[  191.801001] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
[  191.855294] [RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] "General error queue is out of sync with frame queue. ts=212354462016 sof_ts=212358964672 gerror_code=2 gerror_data=222 notify_bits=200010"
[  194.487193] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  194.496378] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  194.507063] (NULL device *): vi_capture_control_message: NULL VI channel received
[  194.514795] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=2, csi_port=2
[  194.525454] (NULL device *): vi_capture_control_message: NULL VI channel received
[  194.533209] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 2 vc- 2
[  194.544004] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  196.000302] bwmgr API not supported
[  196.000313] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=0, lt=0
[  199.502270] bwmgr API not supported
[  199.502280] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=1125000, lt=0
[  199.558157] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
[  199.602939] [RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] "General error queue is out of sync with frame queue. ts=220110533344 sof_ts=220115041568 gerror_code=2 gerror_data=400262 notify_bits=220010"
[  202.167181] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  202.176346] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  202.187508] (NULL device *): vi_capture_control_message: NULL VI channel received
[  202.195227] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=2, csi_port=2
[  202.205904] (NULL device *): vi_capture_control_message: NULL VI channel received
[  202.213630] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 2 vc- 1
[  202.224396] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  204.289386] bwmgr API not supported
[  204.289400] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=0, lt=0
[  207.725314] bwmgr API not supported
[  207.725323] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=1125000, lt=0
[  207.776999] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
[  207.798590] [RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] "General error queue is out of sync with frame queue. ts=228330279840 sof_ts=228334788064 gerror_code=2 gerror_data=800662 notify_bits=240030"
[  210.358505] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  210.367677] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  210.378047] (NULL device *): vi_capture_control_message: NULL VI channel received
[  210.385762] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=2, csi_port=2
[  210.396414] (NULL device *): vi_capture_control_message: NULL VI channel received
[  210.404127] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 2 vc- 0
[  210.414914] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  213.173571] bwmgr API not supported
[  213.173586] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=0, lt=0
[  216.160343] bwmgr API not supported
[  216.160353] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=1125000, lt=0
[  217.012447] bwmgr API not supported
[  217.012459] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=0, lt=0
[  220.197914] bwmgr API not supported
[  220.197922] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=1125000, lt=0
[  220.248003] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
[  220.250128] [RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] "General error queue is out of sync with frame queue. ts=240800959776 sof_ts=240805462496 gerror_code=2 gerror_data=224 notify_bits=200010"
[  222.902266] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  222.911461] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  222.922410] (NULL device *): vi_capture_control_message: NULL VI channel received
[  222.930133] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=4, csi_port=4
[  222.940810] (NULL device *): vi_capture_control_message: NULL VI channel received
[  222.948546] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 4 vc- 2
[  222.959375] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  225.411475] bwmgr API not supported
[  225.411489] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=0, lt=0
[  228.348873] bwmgr API not supported
[  228.348882] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=1125000, lt=0
[  228.404006] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
[  228.417882] [RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] "General error queue is out of sync with frame queue. ts=248956712384 sof_ts=248961220672 gerror_code=2 gerror_data=400264 notify_bits=220010"
[  231.094016] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  231.103193] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  231.113068] (NULL device *): vi_capture_control_message: NULL VI channel received
[  231.120908] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=4, csi_port=4
[  231.131615] (NULL device *): vi_capture_control_message: NULL VI channel received
[  231.139364] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 4 vc- 1
[  231.150197] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  233.167778] bwmgr API not supported
[  233.167791] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=0, lt=0
[  237.141564] bwmgr API not supported
[  237.141573] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=1125000, lt=0
[  239.797738] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  239.806920] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  239.817228] (NULL device *): vi_capture_control_message: NULL VI channel received
[  239.824940] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=4, csi_port=4
[  239.835607] (NULL device *): vi_capture_control_message: NULL VI channel received
[  239.843326] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 4 vc- 0
[  239.854071] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  242.613497] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  242.622654] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  242.633065] (NULL device *): vi_capture_control_message: NULL VI channel received
[  242.640789] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=4, csi_port=4
[  242.651452] (NULL device *): vi_capture_control_message: NULL VI channel received
[  242.659169] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 4 vc- 0
[  242.669945] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  245.205639] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  245.214784] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  245.225101] (NULL device *): vi_capture_control_message: NULL VI channel received
[  245.232833] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=4, csi_port=4
[  245.243493] (NULL device *): vi_capture_control_message: NULL VI channel received
[  245.251206] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 4 vc- 0
[  245.261959] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  245.274738] bwmgr API not supported
[  245.274749] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=0, lt=0
[  246.492842] bwmgr API not supported
[  246.492851] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=1125000, lt=0
[  249.013374] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  249.022536] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  249.033135] (NULL device *): vi_capture_control_message: NULL VI channel received
[  249.040866] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=4, csi_port=4
[  249.051531] (NULL device *): vi_capture_control_message: NULL VI channel received
[  249.059278] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 4 vc- 0
[  249.070062] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  249.083522] bwmgr API not supported
[  249.083530] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=0, lt=0
[  252.937831] bwmgr API not supported
[  252.937840] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=1125000, lt=0
[  254.479822] bwmgr API not supported
[  254.479836] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=0, lt=0
[  256.965110] bwmgr API not supported
[  256.965118] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=1125000, lt=0
[  257.016005] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
[  257.021124] [RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] "General error queue is out of sync with frame queue. ts=277569247104 sof_ts=277573749824 gerror_code=2 gerror_data=225 notify_bits=200010"
[  259.765274] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  259.774453] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  259.785070] (NULL device *): vi_capture_control_message: NULL VI channel received
[  259.792879] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=5, csi_port=6
[  259.803590] (NULL device *): vi_capture_control_message: NULL VI channel received
[  259.811325] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 5 vc- 2
[  259.822149] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  261.581410] bwmgr API not supported
[  261.581417] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=0, lt=0
[  264.102975] bwmgr API not supported
[  264.102983] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=1125000, lt=0
[  264.139996] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
[  264.160968] [RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] "General error queue is out of sync with frame queue. ts=284692701184 sof_ts=284697209440 gerror_code=2 gerror_data=225 notify_bits=200010"
[  266.676931] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  266.686100] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  266.696077] (NULL device *): vi_capture_control_message: NULL VI channel received
[  266.703797] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=5, csi_port=6
[  266.714465] (NULL device *): vi_capture_control_message: NULL VI channel received
[  266.722182] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 5 vc- 1
[  266.732990] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  267.739031] bwmgr API not supported
[  267.739044] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=0, lt=0
[  270.485403] bwmgr API not supported
[  270.485412] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=1125000, lt=0
[  270.528013] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
[  270.560815] [RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] "General error queue is out of sync with frame queue. ts=291081065152 sof_ts=291085573408 gerror_code=2 gerror_data=225 notify_bits=200010"
[  273.076949] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  273.086110] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  273.096128] (NULL device *): vi_capture_control_message: NULL VI channel received
[  273.103837] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=5, csi_port=6
[  273.114510] (NULL device *): vi_capture_control_message: NULL VI channel received
[  273.122226] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 5 vc- 0
[  273.133037] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  274.893324] bwmgr API not supported
[  274.893331] misc tegra_camera_ctrl: tegra_camera_isomgr_request++michael:20230808 bw=0, lt=0
20

One more Bug is :
when opened and closed /dev/video0, /dev/video1,2,3 will hard to stream out in the first second.

21

What’s the camera connect with virtual channel?
How about DPHY configure?

22 
camera 0 |--vc0 4096x3072   --- port0 (serial_a vc_0)
        |--vc1 4096x3072  --- port0 (serial_a vc_1)
        |--vc2 4096x3072  --- port0 (serial_a vc_2)
        |--vc3 4096x384    --- port0 (serial_a vc_3)

camera 1|--vc0 4096x3072   --- port2 (serial_c vc_0)
        |--vc1 4096x3072  --- port2(serial_c vc_1)
        |--vc2 4096x3072  --- port2 (serial_c vc_2)
        |--vc3 4096x384    --- port2 (serial_c vc_3)

camera 2|--vc0 4096x3072   --- port4 (serial_e vc_0)
        |--vc1 4096x3072  --- port4 (serial_e vc_1)
        |--vc2 4096x3072  --- port4 (serial_e vc_2)
        |--vc3 4096x384    --- port4 (serial_e vc_3)

camera 3 --vc0 4096x3072   --- port6 (serial_g vc_0)
        |--vc1 4096x3072  --- port6 (serial_g vc_1)
        |--vc2 4096x3072  --- port6 (serial_g vc_2)
        |--vc3 4096x384    --- port6 (serial_g vc_3)

We don’t have the equipment to verify the DPHY configuration.