Problem when trying to use at least 3 virtual channels on the same CSI interface with GMSL

Hardware Platform: Orin

===== L4T Version =====

R36 (release), REVISION: 4.7, GCID: 42132812

===== DeepStream Version =====
deepstream-app version 7.1.0
DeepStreamSDK 7.1.0
CUDA Driver Version: 12.6
CUDA Runtime Version: 12.6
TensorRT Version: 10.3
cuDNN Version not available
libNVWarp360 Version: 2.0.1d3

Hello, I am having an issue when using a GMSL serializer/de-serializer to connect 4 cameras to the same physical CSI_E port and then feed those 4 cameras to the VI brick 4 through the NVCSI port 4. The signal flow is indicated by the red line on this image:

To achieve that, I’m using 4 virtual channels 0 thrugh 3. This is a snippet of my current device tree which defines the 4 VI inputs:

					// VI-4 - virtual channel 0
					port@8 {
						reg = <8>;
						status = "okay";

						ar0234_vi_in8: endpoint {
							vc-id = <0>;
							port-index = <4>;
							bus-width = <2>;
							remote-endpoint = <&ar0234_csi_out8>;
							status = "okay";
						};
					};

					// VI-4 - virtual channel 1
					port@9 {
						reg = <9>;
						status = "okay";

						ar0234_vi_in9: endpoint {
							vc-id = <1>;
							port-index = <4>;
							bus-width = <2>;
							remote-endpoint = <&ar0234_csi_out9>;
							status = "okay";
						};
					};

					// VI stream 4 - virtual channel 2
					port@13 {
						reg = <13>;
						status = "okay";

						ar0234_vi_in13: endpoint {
							vc-id = <2>;
							port-index = <4>;
							bus-width = <2>;
							remote-endpoint = <&ar0234_csi_out13>;
							status = "okay";
						};
					};

					// VI stream 4 - virtual channel 3
					port@14 {
						reg = <14>;
						status = "okay";

						ar0234_vi_in14: endpoint {
							vc-id = <3>;
							port-index = <4>;
							bus-width = <2>;
							remote-endpoint = <&ar0234_csi_out14>;
							status = "okay";
						};
					};

This all works fine when using up to two virtual channels streaming data (for two cameras) at the same time. As soon as I add a third virtual channel (the third camera), I start seeing those errors in dmesg:

[ 3556.758144] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 131072
[ 3556.758151] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 131072
[ 3556.758155] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 131072

And when this happens, the image I see on all streams (including the third one) seems to be shuffled with itself.

Basically adding the third stream breaks all the streams, including the two previous ones that were working fine.

The resolutions being streamed for each sensor is 1920x1200. This happens even at only 2 fps, so doesn’t look like it is a bandwidth issue. I also have made many different combinations with two streams working at the same time, and they are all fine, so each individual virtual channel is properly set up.

Any ideas on what could be happening here?

How about run single camera individually?

Do you boost the clocks to try.

sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee  /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate

Hi, thanks for the reply!

Running a single stereo camera (two streams) works fine

Are those commands the equivalent of running jetson_clocks? Because I already had ran jetson_clocks on all of my tests.

I kind of found out the problem: there was some overflow happening on my GMSL deserializer, I checked thist by reading a register in the deserializer. The respective deserializer PHYs were running at 1500 Mbps. I had to bump them to 2500 Mbps for this scrambling of the image to stop happening.

But I also had to run skew calibration after starting the streams (I start the skew calibration by writing a certain value to a register of the de-seralizer). According to this Nvidia doc:

This is expected if the PHY data rate is > 1.50 Gpbs, which is indeed the case.

Since I increased the PHY frequency, I also had to update this device tree property for each of my image sensors:

serdes_pix_clk_hz = "500000000";

My sensor uses 10 bits per pixel and 2 CSI lanes, so this value makes sense considering I’m using 2.5Gbps on the PHYs (previously it had the value 299000000 and the PHYs were running at 1.5Gbps ).

This seemed to work, but now I have an intermitent issue where sometimes when I start the streams, it seems that the deskew process doesn’t work and see kernel messages like this which eventually lead to a kernel panic:

[ 1625.990318] channel context at 0 is busy
[ 1625.990382] WARNING: CPU: 0 PID: 8990 at /dvs/git/dirty/git-master_linux/out/nvidia/kernel-gvs_kernel_variant/kernel-nvidia-oot/nvidia-oot/drivers/platform/tegra/rtcpu/capture-ivc.c:195 tegra_capture_ivc_n]
[ 1625.990394] Modules linked in: imx728(OE) cam_cdi_tsc(OE) ar0234_4hawk(OE) xt_conntrack(E) iptable_mangle(E) xt_MASQUERADE(E) xt_tcpudp(E) xt_mark(E) nvidia_drm(OE) nvidia_modeset(OE) lzo_rle(E) lzo_compre)
[ 1625.990451]  snd_soc_tegra210_i2s(OE) snd_soc_tegra210_ahub(OE) tegra210_adma(E) spidev(E) nvvrs_pseq_rtc(OE) snd_soc_tegra_machine_driver(OE) snd_soc_tegra_utils(OE) snd_soc_simple_card_utils(E) crct10dif)
[ 1625.990503]  videobuf2_memops(E) tegra_drm(OE) videobuf2_v4l2(E) nvhost_pva(OE) videobuf2_common(E) nvhost_nvdla(OE) tegra_wmark(OE) videodev(E) cec(E) mc(E) nvhwpm(OE) nvhost_capture(OE) tsecriscv(OE) teg)
[ 1625.990537] CPU: 0 PID: 8990 Comm: vi-output, ar02 Tainted: G           OE     5.15.148-tegra #1
[ 1625.990541] Hardware name: NVIDIA NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS 36.4.3-gcid-38968081 01/08/2025
[ 1625.990542] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 1625.990545] pc : tegra_capture_ivc_notify_chan_id+0x1a0/0x1c0 [capture_ivc]
[ 1625.990549] lr : tegra_capture_ivc_notify_chan_id+0x1a0/0x1c0 [capture_ivc]
[ 1625.990551] sp : ffff80000cb2bb10
[ 1625.990552] x29: ffff80000cb2bb10 x28: 0000000000000000 x27: 0000000000000000
[ 1625.990557] x26: 0000000000001043 x25: ffffb6f5f8b54b98 x24: ffff00008a512088
[ 1625.990561] x23: 0000000000000064 x22: ffff00008a512d00 x21: ffff00008a512080
[ 1625.990564] x20: 0000000000000000 x19: 0000000000000064 x18: ffffffffffffffff
[ 1625.990568] x17: 0000000000000000 x16: ffffb6f61f615c40 x15: ffffb6f620bdcaad
[ 1625.990571] x14: ffffffffffffffff x13: ffffb6f620bdcaa3 x12: 6e6f63206c656e6e
[ 1625.990575] x11: ffffffffffffffff x10: ffffb6f620823dc0 x9 : ffff80000cb2bb10
[ 1625.990578] x8 : ffff80000cb2bb10 x7 : 7973756220736920 x6 : 7973756220736920
[ 1625.990582] x5 : ffff000fa2b9c9f0 x4 : 00000000fffffa2d x3 : ffffb6f620853280
[ 1625.990585] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff000155f22e80
[ 1625.990588] Call trace:
[ 1625.990590]  tegra_capture_ivc_notify_chan_id+0x1a0/0x1c0 [capture_ivc]
[ 1625.990593]  vi_capture_setup+0x664/0x700 [tegra_camera]
[ 1625.990611]  tegra_channel_capture_setup+0x10c/0x2c0 [tegra_camera]
[ 1625.990623]  vi5_channel_error_recover+0x128/0x1f0 [tegra_camera]
[ 1625.990635]  tegra_channel_error_recover+0x80/0xa0 [tegra_camera]
[ 1625.990647]  tegra_channel_kthread_capture_dequeue+0x148/0x200 [tegra_camera]
[ 1625.990658]  kthread+0x184/0x190
[ 1625.990664]  ret_from_fork+0x10/0x20
[ 1625.990668] ---[ end trace 7c680b35e297824a ]---
[ 1626.694316] tegra194-vi5 13e00000.host1x:vi1@14c00000: failed to update control callback
[ 1626.694356] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_shutdown--
[ 1626.694368] tegra-camrtc-capture-vi tegra-capture-vi: vi capture setup failed
[ 1626.694375] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
[ 1626.694393] tegra-camrtc-capture-vi tegra-capture-vi: vi_capture_init++
[ 1626.694459] (NULL device *): vi_capture_shutdown--
[ 1626.694467] tegra-camrtc-capture-vi tegra-capture-vi: vi_capture_init++
[ 1626.694504] tegra194-vi5 13e00000.host1x:vi1@14c00000: chan flags 4163
[ 1626.694506] tegra194-vi5 13e00000.host1x:vi1@14c00000: chan mask ffffffffffffffff
[ 1626.694507] tegra194-vi5 13e00000.host1x:vi1@14c00000: queue depth 4
[ 1626.694508] tegra194-vi5 13e00000.host1x:vi1@14c00000: request size 384
[ 1626.694509] tegra194-vi5 13e00000.host1x:vi1@14c00000: csi_stream_id 5
[ 1626.694510] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi unit id 1
[ 1626.694511] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi2 chan mask ffffffffffffffff
[ 1626.694533] tegra194-vi5 13e00000.host1x:vi1@14c00000: 0 GoS tables configured.
[ 1626.694534] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_ivc_send_control: sending chan_id 101 msg_id 30
[ 1626.694609] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_ivc_send_control: response chan_id 101 msg_id 17
[ 1626.694615] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[ 1626.697359] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_request: sending chan_id 1 msg_id 1 buf:0
[ 1626.697532] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_status: waiting for status, timeout:2500 ms
[ 1626.712822] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_request: sending chan_id 1 msg_id 1 buf:1
[ 1626.727517] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_request: sending chan_id 1 msg_id 1 buf:2
[ 1626.738732] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_request: sending chan_id 1 msg_id 1 buf:3
[ 1629.316826] tegra194-vi5 13e00000.host1x:vi1@14c00000: capture status timed out
[ 1629.316837] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[ 1629.316848] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[ 1629.316852] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_ivc_send_control: sending chan_id 1 msg_id 20
[ 1629.317625] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_ivc_send_control: response chan_id 1 msg_id 21
[ 1629.317674] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_shutdown--
[ 1629.317712] tegra-camrtc-capture-vi tegra-capture-vi: vi_capture_init++
[ 1629.317773] (NULL device *): vi_capture_shutdown--
[ 1629.317785] tegra-camrtc-capture-vi tegra-capture-vi: vi_capture_init++
[ 1629.317833] tegra194-vi5 13e00000.host1x:vi1@14c00000: chan flags 4163
[ 1629.317836] tegra194-vi5 13e00000.host1x:vi1@14c00000: chan mask ffffffffffffffff
[ 1629.317837] tegra194-vi5 13e00000.host1x:vi1@14c00000: queue depth 4
[ 1629.317838] tegra194-vi5 13e00000.host1x:vi1@14c00000: request size 384
[ 1629.317840] tegra194-vi5 13e00000.host1x:vi1@14c00000: csi_stream_id 5
[ 1629.317841] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi unit id 1
[ 1629.317841] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi2 chan mask ffffffffffffffff
[ 1629.317864] tegra194-vi5 13e00000.host1x:vi1@14c00000: 0 GoS tables configured.
[ 1629.317866] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_ivc_send_control: sending chan_id 102 msg_id 30
[ 1629.317924] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_ivc_send_control: response chan_id 102 msg_id 17
[ 1629.317928] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[ 1629.319773] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_request: sending chan_id 1 msg_id 1 buf:0
[ 1629.319798] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_status: waiting for status, timeout:2500 ms
[ 1629.330298] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_request: sending chan_id 1 msg_id 1 buf:1
[ 1629.341028] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_request: sending chan_id 1 msg_id 1 buf:2
[ 1629.351681] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_request: sending chan_id 1 msg_id 1 buf:3
[ 1630.607911] ------------[ cut here ]------------
[ 1630.607918] refcount_t: addition on 0; use-after-free.
[ 1630.607928] WARNING: CPU: 3 PID: 8864 at lib/refcount.c:25 refcount_warn_saturate+0xa0/0x150
[ 1630.607942] Modules linked in: imx728(OE) cam_cdi_tsc(OE) ar0234_4hawk(OE) xt_conntrack(E) iptable_mangle(E) xt_MASQUERADE(E) xt_tcpudp(E) xt_mark(E) nvidia_drm(OE) nvidia_modeset(OE) lzo_rle(E) lzo_compre)
[ 1630.607991]  snd_soc_tegra210_i2s(OE) snd_soc_tegra210_ahub(OE) tegra210_adma(E) spidev(E) nvvrs_pseq_rtc(OE) snd_soc_tegra_machine_driver(OE) snd_soc_tegra_utils(OE) snd_soc_simple_card_utils(E) crct10dif)
[ 1630.608034]  videobuf2_memops(E) tegra_drm(OE) videobuf2_v4l2(E) nvhost_pva(OE) videobuf2_common(E) nvhost_nvdla(OE) tegra_wmark(OE) videodev(E) cec(E) mc(E) nvhwpm(OE) nvhost_capture(OE) tsecriscv(OE) teg)
[ 1630.608062] CPU: 3 PID: 8864 Comm: image-server-cl Tainted: G        W  OE     5.15.148-tegra #1
[ 1630.608065] Hardware name: NVIDIA NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS 36.4.3-gcid-38968081 01/08/2025
[ 1630.608066] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 1630.608069] pc : refcount_warn_saturate+0xa0/0x150
[ 1630.608071] lr : refcount_warn_saturate+0xa0/0x150
[ 1630.608072] sp : ffff8000267aba20
[ 1630.608073] x29: ffff8000267aba20 x28: ffff00010a61df00 x27: 0000000040045613
[ 1630.608075] x26: ffffb6f5f8b4ef48 x25: 0000000000000000 x24: ffff8000267abcd8
[ 1630.608077] x23: ffff00008a5311f8 x22: ffff8000267abcd8 x21: ffff000155f22eb0
[ 1630.608079] x20: ffff00008a531550 x19: ffff000155f22e80 x18: 0000000000000000
[ 1630.608081] x17: 0000000000000000 x16: ffffb6f61e4066a0 x15: 0000aaaabed52720
[ 1630.608083] x14: 000000000000025b x13: 0a2e656572662d72 x12: 657466612d657375
[ 1630.608085] x11: 203b30206e6f206e x10: 6f69746964646120 x9 : 612d657375203b30
[ 1630.608087] x8 : 206e6f206e6f6974 x7 : 69646461203a745f x6 : 746e756f63666572
[ 1630.608089] x5 : ffff000fa2bf39f0 x4 : 00000000fffffa7d x3 : ffffb6f620853a00
[ 1630.608091] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff0000dc32ec80
[ 1630.608093] Call trace:
[ 1630.608094]  refcount_warn_saturate+0xa0/0x150
[ 1630.608096]  kthread_stop+0x138/0x250
[ 1630.608100]  vi5_channel_stop_kthreads+0x58/0x80 [tegra_camera]
[ 1630.608119]  vi5_channel_stop_streaming+0x128/0x140 [tegra_camera]
[ 1630.608130]  tegra_channel_stop_streaming+0x40/0x70 [tegra_camera]
[ 1630.608141]  vb2_buffer_done+0x2e4/0x4e0 [videobuf2_common]
[ 1630.608146]  vb2_core_streamoff+0x38/0x180 [videobuf2_common]
[ 1630.608149]  vb2_ioctl_streamoff+0x74/0x1c0 [videobuf2_v4l2]
[ 1630.608154]  v4l2_video_std_frame_period+0x2b4/0x1170 [videodev]
[ 1630.608163]  v4l2_norm_to_name+0x530/0x15d0 [videodev]
[ 1630.608170]  video_usercopy+0x280/0x680 [videodev]
[ 1630.608177]  video_ioctl2+0x44/0x90 [videodev]
[ 1630.608183]  v4l2_prio_check+0x10c/0x160 [videodev]
[ 1630.608190]  __arm64_sys_ioctl+0xb4/0x100
[ 1630.608193]  invoke_syscall+0x5c/0x130
[ 1630.608197]  el0_svc_common.constprop.0+0x64/0x110
[ 1630.608200]  do_el0_svc+0x74/0xa0
[ 1630.608202]  el0_svc+0x28/0x80
[ 1630.608207]  el0t_64_sync_handler+0xa4/0x130
[ 1630.608209]  el0t_64_sync+0x1a4/0x1a8
[ 1630.608211] ---[ end trace 7c680b35e297824b ]---
[ 1631.613234] ------------[ cut here ]------------
[ 1631.613238] refcount_t: underflow; use-after-free.
[ 1631.613249] WARNING: CPU: 6 PID: 8864 at lib/refcount.c:28 refcount_warn_saturate+0xf4/0x150
[ 1631.613260] Modules linked in: imx728(OE) cam_cdi_tsc(OE) ar0234_4hawk(OE) xt_conntrack(E) iptable_mangle(E) xt_MASQUERADE(E) xt_tcpudp(E) xt_mark(E) nvidia_drm(OE) nvidia_modeset(OE) lzo_rle(E) lzo_compre)
[ 1631.613305]  snd_soc_tegra210_i2s(OE) snd_soc_tegra210_ahub(OE) tegra210_adma(E) spidev(E) nvvrs_pseq_rtc(OE) snd_soc_tegra_machine_driver(OE) snd_soc_tegra_utils(OE) snd_soc_simple_card_utils(E) crct10dif)
[ 1631.613351]  videobuf2_memops(E) tegra_drm(OE) videobuf2_v4l2(E) nvhost_pva(OE) videobuf2_common(E) nvhost_nvdla(OE) tegra_wmark(OE) videodev(E) cec(E) mc(E) nvhwpm(OE) nvhost_capture(OE) tsecriscv(OE) teg)
[ 1631.613376] CPU: 6 PID: 8864 Comm: image-server-cl Tainted: G        W  OE     5.15.148-tegra #1
[ 1631.613379] Hardware name: NVIDIA NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS 36.4.3-gcid-38968081 01/08/2025
[ 1631.613380] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 1631.613383] pc : refcount_warn_saturate+0xf4/0x150
[ 1631.613384] lr : refcount_warn_saturate+0xf4/0x150
[ 1631.613386] sp : ffff8000267aba20
[ 1631.613387] x29: ffff8000267aba20 x28: ffff00010a61df00 x27: 0000000040045613
[ 1631.613389] x26: ffffb6f5f8b4ef48 x25: 0000000000000000 x24: ffff8000267abcd8
[ 1631.613391] x23: ffff00008a5311f8 x22: ffff8000267abcd8 x21: ffff000155f22eb0
[ 1631.613393] x20: 0000000000000000 x19: ffff000155f22e80 x18: 0000000000000000
[ 1631.613395] x17: 0000000000000000 x16: ffffb6f61e4066a0 x15: 0000aaaabed52720
[ 1631.613397] x14: 000000000000025b x13: 0a2e656572662d72 x12: 657466612d657375
[ 1631.613399] x11: 203b776f6c667265 x10: 646e75203a745f74 x9 : 72657466612d6573
[ 1631.613401] x8 : 75203b776f6c6672 x7 : 65646e75203a745f x6 : 746e756f63666572
[ 1631.613402] x5 : ffff000fa2c4a9f0 x4 : 00000000fffffaa9 x3 : ffffb6f620853e20
[ 1631.613404] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff0000dc32ec80
[ 1631.613407] Call trace:
[ 1631.613408]  refcount_warn_saturate+0xf4/0x150
[ 1631.613410]  kthread_stop+0x230/0x250
[ 1631.613413]  vi5_channel_stop_kthreads+0x58/0x80 [tegra_camera]
[ 1631.613430]  vi5_channel_stop_streaming+0x128/0x140 [tegra_camera]
[ 1631.613441]  tegra_channel_stop_streaming+0x40/0x70 [tegra_camera]
[ 1631.613452]  vb2_buffer_done+0x2e4/0x4e0 [videobuf2_common]
[ 1631.613456]  vb2_core_streamoff+0x38/0x180 [videobuf2_common]
[ 1631.613459]  vb2_ioctl_streamoff+0x74/0x1c0 [videobuf2_v4l2]
[ 1631.613464]  v4l2_video_std_frame_period+0x2b4/0x1170 [videodev]
[ 1631.613473]  v4l2_norm_to_name+0x530/0x15d0 [videodev]
[ 1631.613480]  video_usercopy+0x280/0x680 [videodev]
[ 1631.613486]  video_ioctl2+0x44/0x90 [videodev]
[ 1631.613493]  v4l2_prio_check+0x10c/0x160 [videodev]
[ 1631.613500]  __arm64_sys_ioctl+0xb4/0x100
[ 1631.613503]  invoke_syscall+0x5c/0x130
[ 1631.613507]  el0_svc_common.constprop.0+0x64/0x110
[ 1631.613509]  do_el0_svc+0x74/0xa0
[ 1631.613512]  el0_svc+0x28/0x80
[ 1631.613516]  el0t_64_sync_handler+0xa4/0x130
[ 1631.613518]  el0t_64_sync+0x1a4/0x1a8
[ 1631.613520] ---[ end trace 7c680b35e297824c ]---
[ 1631.880812] tegra194-vi5 13e00000.host1x:vi1@14c00000: capture status timed out
[ 1631.880820] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[ 1631.880828] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[ 1631.880830] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_ivc_send_control: sending chan_id 1 msg_id 20
[ 1631.881286] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_ivc_send_control: response chan_id 1 msg_id 21
[ 1631.881331] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_shutdown--
[ 1631.881363] tegra-camrtc-capture-vi tegra-capture-vi: vi_capture_init++
[ 1631.881425] (NULL device *): vi_capture_shutdown--
[ 1631.881434] tegra-camrtc-capture-vi tegra-capture-vi: vi_capture_init++
[ 1631.881470] tegra194-vi5 13e00000.host1x:vi1@14c00000: chan flags 4163
[ 1631.881472] tegra194-vi5 13e00000.host1x:vi1@14c00000: chan mask ffffffffffffffff
[ 1631.881474] tegra194-vi5 13e00000.host1x:vi1@14c00000: queue depth 4
[ 1631.881475] tegra194-vi5 13e00000.host1x:vi1@14c00000: request size 384
[ 1631.881476] tegra194-vi5 13e00000.host1x:vi1@14c00000: csi_stream_id 5
[ 1631.881476] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi unit id 1
[ 1631.881477] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi2 chan mask ffffffffffffffff
[ 1631.881500] tegra194-vi5 13e00000.host1x:vi1@14c00000: 0 GoS tables configured.
[ 1631.881501] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_ivc_send_control: sending chan_id 103 msg_id 30
[ 1631.881553] tegra194-vi5 13e00000.host1x:vi1@14c00000: vi_capture_ivc_send_control: response chan_id 103 msg_id 17
[ 1631.881557] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[ 1632.375378] Unable to handle kernel paging request at virtual address 5fa315f9499e07bf
[ 1632.375383] Mem abort info:
[ 1632.375383]   ESR = 0x0000000096000004
[ 1632.375384]   EC = 0x25: DABT (current EL), IL = 32 bits
[ 1632.375386]   SET = 0, FnV = 0
[ 1632.375387]   EA = 0, S1PTW = 0
[ 1632.375387]   FSC = 0x04: level 0 translation fault
[ 1632.375388] Data abort info:
[ 1632.375389]   ISV = 0, ISS = 0x00000004
[ 1632.375389]   CM = 0, WnR = 0
[ 1632.375390] [5fa315f9499e07bf] address between user and kernel address ranges
[ 1632.375392] Internal error: Oops: 0000000096000004 [#1] PREEMPT SMP
[ 1632.375395] Modules linked in: imx728(OE) cam_cdi_tsc(OE) ar0234_4hawk(OE) xt_conntrack(E) iptable_mangle(E) xt_MASQUERADE(E) xt_tcpudp(E) xt_mark(E) nvidia_drm(OE) nvidia_modeset(OE) lzo_rle(E) lzo_compre)
[ 1632.375435]  snd_soc_tegra210_i2s(OE) snd_soc_tegra210_ahub(OE) tegra210_adma(E) spidev(E) nvvrs_pseq_rtc(OE) snd_soc_tegra_machine_driver(OE) snd_soc_tegra_utils(OE) snd_soc_simple_card_utils(E) crct10dif)
[ 1632.375472]  videobuf2_memops(E) tegra_drm(OE) videobuf2_v4l2(E) nvhost_pva(OE) videobuf2_common(E) nvhost_nvdla(OE) tegra_wmark(OE) videodev(E) cec(E) mc(E) nvhwpm(OE) nvhost_capture(OE) tsecriscv(OE) teg)
[ 1632.375495] CPU: 0 PID: 9083 Comm: dpkg Tainted: G        W  OE     5.15.148-tegra #1
[ 1632.375497] Hardware name: NVIDIA NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS 36.4.3-gcid-38968081 01/08/2025
[ 1632.375499] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 1632.375501] pc : __kmalloc+0x124/0x3c0
[ 1632.375508] lr : __kmalloc+0xec/0x3c0
[ 1632.375509] sp : ffff800023b33a30
[ 1632.375510] x29: ffff800023b33a30 x28: ffff000089c06040 x27: ffffb6f6207c9000
[ 1632.375512] x26: ffff000104ae0000 x25: ffffb6f620bb9000 x24: 9697b6f61e742e00
[ 1632.375514] x23: 0000000000000033 x22: 0000000000000dc0 x21: 0000000000000000
[ 1632.375516] x20: ffffb6f6201d0a60 x19: ffff000080002200 x18: 0000000000000000
[ 1632.375518] x17: 0000000037ce1fc3 x16: 000000005c847b9b x15: 000000005c847b9b
[ 1632.375520] x14: 000000006aaf4349 x13: 00000000087c15d0 x12: 0000000070dbeda3
[ 1632.375522] x11: 000000002153003c x10: 0000000070dbeda3 x9 : 00000000803e96e6
[ 1632.375524] x8 : 00000000c23233d0 x7 : 000000006cf8e067 x6 : e087f3ef489ef87f
[ 1632.375526] x5 : 0000000070dbeda3 x4 : 0000000000000040 x3 : 0000000000000000
[ 1632.375527] x2 : bf079e49f915a35f x1 : 0000000018093400 x0 : 5fa315f9499e077f
[ 1632.375530] Call trace:
[ 1632.375531]  __kmalloc+0x124/0x3c0
[ 1632.375532]  ext4_htree_store_dirent+0x50/0x150
[ 1632.375536]  htree_dirblock_to_tree+0x198/0x340
[ 1632.375540]  ext4_htree_fill_tree+0xd8/0x370
[ 1632.375541]  ext4_readdir+0x6d4/0x8e0
[ 1632.375543]  iterate_dir+0x17c/0x1d0
[ 1632.375545]  __arm64_sys_getdents64+0x74/0x220
[ 1632.375546]  invoke_syscall+0x5c/0x130
[ 1632.375550]  el0_svc_common.constprop.0+0x64/0x110
[ 1632.375553]  do_el0_svc+0x74/0xa0
[ 1632.375555]  el0_svc+0x28/0x80

Are there any drawbacks or limitations from the Orin side on the max frequency that the PHY’s of the deserializer can go? I’m suspecting this is the cause of this new issue

Apply below changes to verify again.

[Camera][VI] Add semaphore to avoid multi-cam race condition
https://forums.developer.nvidia.com/t/337341/12/
(Fixed in Jetpack 6.2.2 r36.5)

[Camera][VI] Memory leakage fix with error recovery
https://forums.developer.nvidia.com/t/337609/11/
(Fixed in Jetpack 6.2.2 r36.5)
[Camera][VI] fixes to enable two separate cameras on the same NVCSI brick
https://forums.developer.nvidia.com/t/358947/19
[Camera][VI] kernel panic when accessing non-existent camera on developer kit
https://forums.developer.nvidia.com/t/344342/8
(Fixed in Jetpack 6.2.2 r36.5)

I was able to fix the problem by using 4 CSI lanes instead of just 2. I think the problem was in the GMSL deserializer side, not in anything from the Nvidia Stack.

Using 4 lanes allows the de-serializer PHY to sample the incoming data at half the necessary clock speed, fixing the problem.