Issues when capturing two camera virtual channel inputs via CSI-A/CSI-B ports

Hi all,

I’m using Jetson TX2 module and L4T r32.3.1 kernel and have an issue when capturing two camera
streams from single source which is connected to CSI-A/CSI-B ports, using 4 data lanes and one clock
lane. The camera source is connected using CSI_A_CLK lane and two data lanes of the CSI-A port and
two data lanes of the CSI-B port (CSI_A_D0/CSI_A_D1 and CSI_B_D0/CSI_B_D1). Since only one clock lane is connected, I have to use virtual channels to capture two different camera streams.

The first stream is using CSI2 virtual channel 0 (VC0), the second stream uses CSI2 virtual
channel 1 (VC1). If only one camera (VC0 stream) is connected and the media pipeline links are
initialized properly, I can capture the VC0 camera data using v4l2-ctl.
But if both cameras are connected and media pipeline is initialized, I can only successfully capture
VC1 stream. When VC0 capturing is started, I get “PXL_SOF syncpt timeout” errors.
Any idea what might be wrong here? Or has anyone a suggestion how to debug this issue?
Below is the dmesg output and trace buffer dump taken after interruption of the
hanging command “v4l2-ctl -d /dev/video0 --stream-mmap --stream-count=3” for capturing VC0 stream:

[ 3225.197599] nvcsi 150c0000.nvcsi: settle time reading from props
[ 3225.197622] nvcsi 150c0000.nvcsi: csi_port: 0
[ 3225.197863] nvcsi 150c0000.nvcsi: settle time reading from props
[ 3225.197880] nvcsi 150c0000.nvcsi: csi_port: 0
[ 3225.198220] tegra_channel_g_selection: irs10x0c 8-003d
[ 3225.205285] tegra-vi4 15700000.vi: Create Surface with imgW=224, imgH=172, memFmt=32
[ 3225.207806] nvcsi 150c0000.nvcsi: csi4_start_streaming port_idx=0, lanes=4
[ 3225.207826] nvcsi 150c0000.nvcsi: csi4_stream_init
[ 3225.207854] nvcsi 150c0000.nvcsi: csi4_stream_config
[ 3225.207879] nvcsi 150c0000.nvcsi: csi4_stream_config (0) read VC0_DPCM_CTRL = 00000000
[ 3225.207895] nvcsi 150c0000.nvcsi: settle time reading from props
[ 3225.207913] nvcsi 150c0000.nvcsi: discontinuous_clk = 0 reading from props
[ 3225.207927] nvcsi 150c0000.nvcsi: settle time reading from props
[ 3225.207940] nvcsi 150c0000.nvcsi: csi4_phy_config
[ 3225.207958] nvcsi 150c0000.nvcsi: NVCSI_CIL_CONFIG = 00000000
[ 3225.207983] nvcsi 150c0000.nvcsi: cil core clock: 204, csi clock: 400
[ 3225.207997] nvcsi 150c0000.nvcsi: cil_settingtime was autocalculated
[ 3225.208013] nvcsi 150c0000.nvcsi: csi settle time: 33, cil settle time: 20
[ 3225.409455] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[ 3225.415900] tegra-vi4 15700000.vi: tegra_channel_error_recovery: attempting to reset the capture channel
[ 3225.416449] nvcsi 150c0000.nvcsi: csi4_stop_streaming port_idx=0, lanes=4
[ 3225.416469] nvcsi 150c0000.nvcsi: settle time reading from props
[ 3225.416486] nvcsi 150c0000.nvcsi: discontinuous_clk = 0 reading from props
[ 3225.416502] nvcsi 150c0000.nvcsi: settle time reading from props
[ 3225.416515] nvcsi 150c0000.nvcsi: csi4_phy_config
[ 3225.416534] nvcsi 150c0000.nvcsi: NVCSI_CIL_CONFIG = 00000004
[ 3225.416557] nvcsi 150c0000.nvcsi: csi4_stream_check_status
[ 3225.416583] nvcsi 150c0000.nvcsi: csi4_cil_check_status 397
[ 3225.416604] nvcsi 150c0000.nvcsi: csi4_start_streaming port_idx=0, lanes=4
[ 3225.416617] nvcsi 150c0000.nvcsi: csi4_stream_init
[ 3225.416643] nvcsi 150c0000.nvcsi: csi4_stream_config
[ 3225.416665] nvcsi 150c0000.nvcsi: csi4_stream_config (0) read VC0_DPCM_CTRL = 00000000
[ 3225.416678] nvcsi 150c0000.nvcsi: settle time reading from props
[ 3225.416693] nvcsi 150c0000.nvcsi: discontinuous_clk = 0 reading from props
[ 3225.416705] nvcsi 150c0000.nvcsi: settle time reading from props
[ 3225.416718] nvcsi 150c0000.nvcsi: csi4_phy_config
[ 3225.416734] nvcsi 150c0000.nvcsi: NVCSI_CIL_CONFIG = 00000004
[ 3225.416757] nvcsi 150c0000.nvcsi: cil core clock: 204, csi clock: 400
[ 3225.416770] nvcsi 150c0000.nvcsi: cil_settingtime was autocalculated
[ 3225.416786] nvcsi 150c0000.nvcsi: csi settle time: 33, cil settle time: 20
[ 3225.417310] tegra-vi4 15700000.vi: Create Surface with imgW=224, imgH=172, memFmt=32
[ 3225.417430] video4linux video0: free_ring_buffers: capture init latency is 215 ms
[ 3225.621463] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11

Trace dump:

# tracer: nop
#
# entries-in-buffer/entries-written: 556/556   #P:4
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
        v4l2-ctl-3044  [000] ....  3225.191290: tegra_channel_open: vi-output, irs10x0c 8-003d
        v4l2-ctl-3044  [000] ....  3225.192574: tegra_channel_set_power: irs10x0c 8-003d : 0x1
        v4l2-ctl-3044  [000] ....  3225.196234: tegra_channel_set_power: 150c0000.nvcsi--6 : 0x1
        v4l2-ctl-3044  [000] ....  3225.196242: csi_s_power: enable : 0x1
        v4l2-ctl-3044  [005] ....  3225.205229: tegra_channel_capture_setup: vnc_id 0 W 224 H 172 fmt 20
 vi-output, irs1-3045  [000] ....  3225.205572: tegra_channel_set_stream: enable : 0x1
 vi-output, irs1-3045  [000] ....  3225.207775: tegra_channel_set_stream: 150c0000.nvcsi--6 : 0x1
 vi-output, irs1-3045  [000] ....  3225.207782: csi_s_stream: enable : 0x1
 vi-output, irs1-3045  [000] ....  3225.208029: tegra_channel_set_stream: irs10x0c 8-003d : 0x1
     kworker/3:0-2705  [003] ....  3225.213445: rtos_queue_peek_from_isr_failed: tstamp:101490445711 queue:0x0b4b4500
     kworker/3:0-2705  [003] ....  3225.213453: rtcpu_start: tstamp:101490449028
     kworker/3:0-2705  [003] ....  3225.213458: rtos_queue_send_from_isr_failed: tstamp:101490463075 queue:0x0b4a7258
     kworker/3:0-2705  [003] ....  3225.213461: rtos_queue_send_from_isr_failed: tstamp:101490463183 queue:0x0b4aad68
     kworker/3:0-2705  [003] ....  3225.213463: rtos_queue_send_from_isr_failed: tstamp:101490463289 queue:0x0b4ac998
     kworker/3:0-2705  [003] ....  3225.213466: rtos_queue_send_from_isr_failed: tstamp:101490463397 queue:0x0b4ae518
     kworker/3:0-2705  [003] ....  3225.213468: rtos_queue_send_from_isr_failed: tstamp:101490463502 queue:0x0b4af2d8
     kworker/3:0-2705  [003] ....  3225.213470: rtos_queue_send_from_isr_failed: tstamp:101490463607 queue:0x0b4b0098
     kworker/3:0-2705  [003] ....  3225.213473: rtos_queue_send_from_isr_failed: tstamp:101490463712 queue:0x0b4b0e58
     kworker/3:0-2705  [003] ....  3225.213475: rtos_queue_send_from_isr_failed: tstamp:101490463817 queue:0x0b4b1c18
     kworker/3:0-2705  [003] ....  3225.213479: rtos_queue_send_failed: tstamp:101490464251 queue:0x0b4a7258
     kworker/3:0-2705  [003] ....  3225.213481: rtos_queue_send_from_isr_failed: tstamp:101490469752 queue:0x0b4a7258
     kworker/3:0-2705  [003] ....  3225.213484: rtos_queue_send_from_isr_failed: tstamp:101490469858 queue:0x0b4aad68
     kworker/3:0-2705  [003] ....  3225.213486: rtos_queue_send_from_isr_failed: tstamp:101490469965 queue:0x0b4ac998
     kworker/3:0-2705  [003] ....  3225.213488: rtos_queue_send_from_isr_failed: tstamp:101490470071 queue:0x0b4ae518
     kworker/3:0-2705  [003] ....  3225.213491: rtos_queue_send_from_isr_failed: tstamp:101490470176 queue:0x0b4af2d8
     kworker/3:0-2705  [003] ....  3225.213493: rtos_queue_send_from_isr_failed: tstamp:101490470281 queue:0x0b4b0098
     kworker/3:0-2705  [003] ....  3225.213495: rtos_queue_send_from_isr_failed: tstamp:101490470386 queue:0x0b4b0e58
     kworker/3:0-2705  [003] ....  3225.213498: rtos_queue_send_from_isr_failed: tstamp:101490470494 queue:0x0b4b1c18
     kworker/3:0-2705  [003] ....  3225.213500: rtos_queue_send_failed: tstamp:101490471434 queue:0x0b4a7258
     kworker/3:0-2705  [003] ....  3225.381428: rtos_queue_peek_from_isr_failed: tstamp:101495446044 queue:0x0b4b4500
 vi-output, irs1-3045  [005] ....  3225.417255: tegra_channel_capture_setup: vnc_id 0 W 224 H 172 fmt 20
 vi-output, irs1-3045  [005] ....  3225.417451: tegra_channel_capture_frame: sof:-549620691636.-271027366144
     kworker/3:0-2705  [003] ....  3225.437411: rtos_queue_send_from_isr_failed: tstamp:101497061196 queue:0x0b4a7258
     kworker/3:0-2705  [003] ....  3225.437418: rtos_queue_send_from_isr_failed: tstamp:101497061304 queue:0x0b4aad68
     kworker/3:0-2705  [003] ....  3225.437421: rtos_queue_send_from_isr_failed: tstamp:101497061412 queue:0x0b4ac998
     kworker/3:0-2705  [003] ....  3225.437423: rtos_queue_send_from_isr_failed: tstamp:101497061518 queue:0x0b4ae518
...
     kworker/3:0-2705  [003] ....  3227.733552: rtos_queue_send_from_isr_failed: tstamp:101568734468 queue:0x0b4b0098
     kworker/3:0-2705  [003] ....  3227.733554: rtos_queue_send_from_isr_failed: tstamp:101568734573 queue:0x0b4b0e58
     kworker/3:0-2705  [003] ....  3227.733557: rtos_queue_send_from_isr_failed: tstamp:101568734677 queue:0x0b4b1c18
     kworker/3:0-2705  [003] ....  3227.733559: rtos_queue_send_failed: tstamp:101568735101 queue:0x0b4a7258
     kworker/3:0-2705  [003] ....  3227.789428: rtos_queue_peek_from_isr_failed: tstamp:101570445942 queue:0x0b4b4500
        v4l2-ctl-3044  [003] ....  3227.817430: tegra_channel_set_stream: 150c0000.nvcsi--6 : 0x0
        v4l2-ctl-3044  [003] ....  3227.817436: csi_s_stream: enable : 0x0
        v4l2-ctl-3044  [000] ....  3227.820402: tegra_channel_set_power: irs10x0c 8-003d : 0x0
        v4l2-ctl-3044  [000] ....  3227.820418: tegra_channel_set_power: 150c0000.nvcsi--6 : 0x0
        v4l2-ctl-3044  [000] ....  3227.820422: csi_s_power: enable : 0x0

Media entity links seem to be correct, here is what I see with media-ctl:

# media-ctl -p
Media controller API version 0.1.0

Media device information
------------------------
driver          tegra-vi4
model           NVIDIA Tegra Video Input Device
serial
bus info
hw revision     0x3
driver version  0.0.0

Device topology
- entity 1: 150c0000.nvcsi--6 (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
                <- "irs10x0c 8-003d":0 [ENABLED]
        pad1: Source
                -> "vi-output, irs10x0c 8-003d":0 [ENABLED]

- entity 18: 150c0000.nvcsi--5 (2 pads, 2 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev1
        pad0: Sink
                <- "ov5695d3 8-0049":0 [ENABLED]
        pad1: Source
                -> "vi-output, ov5695d3 8-0049":0 [ENABLED]

- entity 121: irs10x0c 8-003d (1 pad, 1 link)
              type V4L2 subdev subtype Sensor flags 0
              device node name /dev/v4l-subdev2
        pad0: Source
                [fmt:Y12_1X12/224x172 field:none colorspace:unknown
                 crop:(0,0)/224x172]
                -> "150c0000.nvcsi--6":0 [ENABLED]

- entity 123: vi-output, irs10x0c 8-003d (1 pad, 1 link)
              type Node subtype V4L flags 0
              device node name /dev/video0
        pad0: Sink
                <- "150c0000.nvcsi--6":1 [ENABLED]

- entity 135: ov5695d3 8-0049 (1 pad, 1 link)
              type V4L2 subdev subtype Sensor flags 0
              device node name /dev/v4l-subdev3
        pad0: Source
                [fmt:SBGGR10_1X10/2592x1944 field:none colorspace:srgb]
                -> "150c0000.nvcsi--5":0 [ENABLED]

- entity 137: vi-output, ov5695d3 8-0049 (1 pad, 1 link)
              type Node subtype V4L flags 0
              device node name /dev/video1
        pad0: Sink
                <- "150c0000.nvcsi--5":1 [ENABLED]

My cameras and NVCSI/VI port description in device-tree looks like:

&vi_base {
        ports {
                #address-cells = <1>;
                #size-cells = <0>;
                vi_port_0: port@0 {
                        status = "okay";
                        reg = <0>;
                        vi_in_0: endpoint {
                                status = "okay";
                                vc-id = <0>;
                                port-index = <0>;
                                bus-width = <4>;
                                remote-endpoint = <&csi_out_0>;
                        };
                };

                vi_port_1: port@1 {
                        status = "okay";
                        reg = <1>;
                        vi_in_1: endpoint {
                                status = "okay";
                                vc-id = <1>;
                                port-index = <0>;
                                bus-width = <4>;
                                remote-endpoint = <&csi_out_1>;
                        };
                };
        };
};
&csi_base {
        chan_0: channel@0 {
                status = "okay";
                reg = <0>;
                ports {
                        #address-cells = <1>;
                        #size-cells = <0>;
                        port@0 {
                                status = "okay";
                                reg = <0>;
                                csi_in_0: endpoint@0 {
                                        status = "okay";
                                        port-index = <0>;
                                        bus-width = <4>;
                                        remote-endpoint = <&sensor_out_0>;
                                };
                        };
                        port@1 {
                                status = "okay";
                                reg = <1>;
                                csi_out_0: endpoint@1 {
                                        status = "okay";
                                        remote-endpoint = <&vi_in_0>;
                                };
                        };
                };
        };
        chan_1: channel@1 {
                status = "okay";
                reg = <1>;
                ports {
                        #address-cells = <1>;
                        #size-cells = <0>;
                        port@0 {
                                status = "okay";
                                reg = <0>;
                                csi_in_1: endpoint@2 {
                                        status = "okay";
                                        port-index = <0>;
                                        bus-width = <4>;
                                        remote-endpoint = <&sensor_out_1>;
                                };
                        };
                        port@1 {
                                status = "okay";
                                reg = <1>;
                                csi_out_1: endpoint@3 {
                                        status = "okay";
                                        remote-endpoint = <&vi_in_1>;
                                };
                        };
                };
        };
};
q0_link0: irs238xc@3d {
        status = "okay";
        compatible = "irs238xc";
        #address-cells = <1>;
        #size-cells = <0>;
        reg = <0x3d>;
        physical-addr = <0x3d>;         /* 7-bit address, which is what the UB953 wants */
        devnode = "video0";
        mclk = "extperiph1";            /* Note: extperiph1 <= 24MHz */
        frame-sync-mode = <1>;
        physical_w = "224";
        physical_h = "172";

        deserializer = <&ub954q0>;

        set_mode_delay_ms = "100000";   /* wait up to 100s for first frame */

        /* 5 Mpixel, 30fps linear mode */
        mode0 {
                mclk_khz = "24000";
                num_lanes = "2";
                tegra_sinterface = "serial_a";
                phy_mode = "DPHY";
                discontinuous_clk = "no";
                dpcm_enable = "false";
                cil_settletime = "0";

                active_w = "224";
                active_h = "172";
                pixel_t = "raw_y12";
                csi_pixel_bit_depth = "12";
                readout_orientation = "0";
                line_length = "224";
                inherent_gain = "1";
                mclk_multiplier = "170.0";
                pix_clk_hz = "400000000";

                gain_factor = "10";
                min_gain_val = "10";/* 1DB*/
                max_gain_val = "160";/* 16DB*/
                step_gain_val = "1";
                default_gain = "10";
                min_hdr_ratio = "1";
                max_hdr_ratio = "1";
                framerate_factor = "1000000";
                min_framerate = "1816577";/*1.816577 */
                max_framerate = "30000000";/*30*/
                step_framerate = "1";
                default_framerate = "30000000";
                exposure_factor = "1000000";
                min_exp_time = "34";/* us */
                max_exp_time = "550385";/* us */
                step_exp_time = "1";
                default_exp_time = "33334";/* us */
                embedded_metadata_height = "0";
        };

        ports {
                #address-cells = <1>;
                #size-cells = <0>;

                status = "okay";
                port@0 {
                        reg = <0>;
                        sensor_out_0: endpoint {
                                vc-id = <0>;
                                status = "okay";
                                port-index = <0>;
                                bus-width = <4>;
                                remote-endpoint = <&csi_in_0>;
                        };
                };
        };
};
q0_link1: ov5695d3@49 {
        status = "okay";
        compatible = "d3,ov5695d3";
        #address-cells = <1>;
        #size-cells = <0>;
        reg = <0x49>;
        physical-addr = <0x36>;         /* 7-bit address */
        devnode = "video1";
        mclk = "extperiph1";            /* extperiph1 <= 24MHz */
        frame-sync-mode = <1>;
        physical_w = "3.684";           /* mm */
        physical_h = "2.763";           /* mm */
        sensor_model = "ov5695d3";

        use_decibel_gain = "false";
        use_sensor_mode_id = "true";

        deserializer = <&ub954q0>;

        set_mode_delay_ms = "100000";

        /* 5 Mpixel, 30fps linear mode */
        mode0 {
                mclk_khz = "24000";
                mclk_multiplier = "170.0";
                num_lanes = <2>;
                tegra_sinterface = "serial_a";
                phy_mode = "DPHY";
                discontinuous_clk = "no";
                dpcm_enable = "false";
                cil_settletime = "0";
                csi_pixel_bit_depth = "10";
                mode_type = "bayer";
                pixel_phase = "bggr";
                readout_orientation = "0";
                inherent_gain = "1";
                serdes_pix_clk_hz = "4000000000";
                pix_clk_hz = "176160768";
                vc_id = "0";
                min_gain_val = "0.0625";
                max_gain_val = "255.9375";
                active_w = "2592";
                active_h = "1944";
                line_length = "2623";
                min_framerate = "30";
                max_framerate = "30";
                min_exp_time = "68";
                max_exp_time = "32996";
                min_hdr_ratio = "1";
                max_hdr_ratio = "1";
                embedded_metadata_height = "0";
        };
        ports {
                #address-cells = <1>;
                #size-cells = <0>;

                status = "okay";
                port@0 {
                        reg = <0>;
                        sensor_out_1: endpoint {
                                vc-id = <1>;
                                status = "okay";
                                port-index = <0>;
                                bus-width = <4>;
                                remote-endpoint = <&csi_in_1>;
                        };
                };
        };
};

Another observation is that if I first start capturing VC1 stream using “v4l2-ctl -d /dev/video1 --stream-mmap --stream-count=10000” command, then capturing VC0 stream can be started by “v4l2-ctl -d /dev/video0 --stream-mmap --stream-count=10000” and I can get correct data of the two streams.
But after /dev/video1 capturing ist stopped, the capturing from /dev/video0 hangs again
(with “PXL_SOF syncpt timeout” errors). Any idea how to fix this? Thanks!

1 Like

Did you connect two sensor to TX2 directly?
For the virtual channel may need connect the sensor to a bridge like gmsl chip then connect to TX2.

The two sensors are connected via bridge DS90UB954-Q1 & 2x DS90UB953-Q1:

1st sensor (RAW12)    2nd sensor(RAW10)
  244 X 172 pix        2592 X 1944 pix
   2xCSI2 lanes         2xCSI2 lanes
       ||                   ||
       \/                   \/
      UB953                UB953
    serializer           serializer
        |                    |
        '--coax---..---coax--'
                  ||
                 UB954
              4xCSI2 lanes
                 ||||
                 ||||
           Tegra TX2 CSI2 input

The bridge is already initialized to forward the packets and maps CSI2 RAW12 stream to VC0
and RAW10 stream to VC1.

Have you try both of them are the same output raw10 or raw12?

I’ve tried to connect two cameras, and these combinations work:
2x RAW12
2x RAW10

Looks like irs238xc lost below property?

                                            mode_type = "bayer";
                                            pixel_phase = "rggb";

I have inserted mode_type and pixel_phase properties into the mode node and did a quick test.
The behavior did not change, still seeing same issue.

@agustg650e
I review the symptom again. I think it could be the problem from the UB953 chip setting.

Thanks four your suggestion! We have double-checked our UB953 and UB954 configuration and also checked the UB954 and UB935 status registers after activating the VC0 stream (1st sensor sending RAW12 data). This was okay, no issues were reported via status registers. UB954 could detect expected frame resolution values in LINE_LEN_0/1 and LINE_COUNT_HI/LINE_COUNT_LO registers. So, FPD-Link III was not a problem here.
Further debugging revealed that the driver for RAW12 sensor must call camera_common_s_power() function in its .s_power() callback of the v4l2_subdev_core_ops struct, then this problem does not appear any more.