Start/stop streaming with Virtual channels issue

[Carbure]

We are using 2 cameras connected to a single CSI port through GMSL and using virtual channels (Orin nano platform + JPack 6.1):
Camera 1 is an imx715 using vc-id=<0>, the second is a adsd3500 sensor using vc-id=<1>

Our DT : (it’s possible to have config with 2 x 2 cams)

tegra-capture-vi { 
    num-channels = <4>;
	status="okay";
	ports {
		#address-cells = <1>;
		#size-cells = <0>;
		status="okay";
		port@0 {
			reg = <0>;
			Sensor_vi_in0: endpoint {
				status="okay";
				vc-id = <0> ;
				port-index = <0>;
				bus-width = <2>;
				remote-endpoint = <&imx715_csi_out0>;
			};
		};
		port@1 {
			reg = <1>;
			Sensor_vi_in1: endpoint {
				status="okay";
				vc-id = <1>;
				port-index = <0>;
				bus-width = <2>;
				remote-endpoint = <&adsd3500_csi_out1>;
			};
		};
		port@2 {
			reg = <2>;
			Sensor_vi_in2: endpoint {
				status="okay";
				vc-id = <0> ;
				port-index = <1>;
				bus-width = <2>;
				remote-endpoint = <&imx715_csi_out2>;
			};
		};
		port@3 {
			reg = <3>;
			Sensor_vi_in3: endpoint {
				status="okay";
				vc-id = <1> ;
				port-index = <1>;
				bus-width = <2>;
				remote-endpoint = <&adsd3500_csi_out3>;
			};
		};
	};
};

We are using 2 applications running in parallel using these sensors separately: the application using the imx715 is based on ARGUS and the adsd3500 application uses v4l2.
So we can start and then stop the 2 cameras in any order…. worked perfectly with JetPack 5.1.2

With Jetpack6.1 if we stop the first camera (imx715) by stopping the application using it (VC=0), the application using the second camera does not receive data any more and we have a timeout in the kernel …
dmesg:

[ 2275.432709] imx715 9-001a: [T300] imx715_start_streaming entered.

[ 2275.472054] adsd3500 9-0040: [T300] adsd3500_start_streaming entered.

[ 2282.808832] imx715 9-001a: [T300] imx715_stop_streaming entered

[ 2285.989635] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 3000 ms

[ 2285.989664] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel

[ 2285.991165] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel

[ 2289.061533] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 3000 ms

[ 2289.061560] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel

[ 2289.062964] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel

[ 2292.069436] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 3000 ms

[ 2292.069465] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel

[ 2292.071143] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel

[ 2292.071197] adsd3500 9-0040: [T300] adsd3500_stop_streaming entered.

[ 2293.388227] tegra-camrtc-capture-vi tegra-capture-vi: subdev imx715 9-001a unbind

[ 2293.389269] adsd3500 9-0040: adsd3500_remove

[ 2293.391624] tegra-camrtc-capture-vi tegra-capture-vi: subdev adsd3500 9-0040 unbind

[ 2293.391777] adsd3500 9-0040: [T300] adsd3500_remove: ADSD3500 driver removed.

But the measurement with an oscilloscope shows that the mipi data on VC=1 still arrive without any problem until we stop the adsd3500 camera

This does not happen if we stop the camera using *adsd3500 first (*VC=1…)… Data are still received on VC=0 ???

Any idea what could produce this behavior?

[DaneLLL]

Hi,
For information, do you observe the issue when launching two cameras through v4l2? Or it is specific to launching one through v4ls and the other through Argus?

[Carbure]

Hi DaneLLL

Thanks for the fast reply and suggestion…
With v4l2-ctl on both sensors, I have exactly the same behaviour …

[JerryChang]

hello Carbure,

please gather the logs for checking.
could you follow the steps to enable VI tracing logs.
for instance,

modprobe rtcpu_debug    #for JP-6.1 or later.
echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo > /sys/kernel/debug/tracing/trace
cat /sys/kernel/debug/tracing/trace

[Carbure]

Hi Jerry

I searched in the drivers of the sensor vendors to find differences when stopping streaming by activating kernel dynamic debugging.

Only the RGB sensor is calling camera_common_s_power (struct v4l2_subdev *sd, int on) with on=0
(file camera_common.c)

Commenting out the “else” part like this, solves the problem:

int camera_common_s_power(struct v4l2_subdev *sd, int on)
{
	int err = 0;
	struct camera_common_data *s_data = to_camera_common_data(sd->dev);

	if (!s_data)
		return -EINVAL;
	trace_camera_common_s_power("status", on);
	if (on) {
		err = call_s_op(s_data, power_on);
		………………….etc............
	}
#if 0	 // Power off leads to timeout errors with Sick T-300 camera 
        //if stopping RGB sensor , TOF cannot receive data anymore ....
	else {
        camera_common_dpd_enable(s_data);  
		camera_common_mclk_disable(s_data);
		call_s_op(s_data, power_off);
	}
#endif
	return err;
}

It works fine but this solution does not seem clean to me…what would be the right method?

[JerryChang]

hello Carbure,

it’s the kernel init stage to power-up for device registration.
is your camera stream also sending out while GMSL chip power-on?

[Carbure]

Hi Jerry
GMSL chips (ser-des) are powered on … they are managed independently of the drivers because of conflicts between the 2 sensor drivers. ( theses drivers never act on GMSL components)
GMSL once powered on, the video pipes are disabled → Then one or both cameras are started … then the GMSL video pipes are activated … the video pipes are disabled after the 2 cameras has been stopped

[JerryChang]

hello Carbure,

please also collect VI tracing logs for more details.

[Carbure]

traces.txt (575.9 KB)

It’s a big file … ;-)

[JerryChang]

hello Carbure,

did you gather the logs with the issue reproduced?
according to the tracing logs, it seems there’re validate frames on the CSI channel.

[Carbure]

Yes :

tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 3000 ms

and no data forTOF sensor

[JerryChang]

hello Carbure,

@Carbure

commenting out the “else” part like this, solves the problem:

that should not root cause the issue as those code also present on JP-5.1.2

could you please check whether your TOF sensor driver have below operations.
for instance,

static struct v4l2_subdev_core_ops cam_subdev_core_ops = {
  	.s_power = camera_common_s_power,
 };

[Carbure]

Hi Jerry
In the TOF driver, I have

static const struct v4l2_subdev_core_ops adsd3500_core_ops = {
	.s_power = camera_common_s_power,
#ifdef CONFIG_VIDEO_ADV_DEBUG
	.g_register = adsd3500_g_register,
	.s_register = adsd3500_s_register,
#endif
};

but I have also :

static const struct v4l2_subdev_ops adsd3500_subdev_ops = {
#ifdef CONFIG_VIDEO_ADV_DEBUG
	.core = &adsd3500_core_ops,
#endif
	.video = &adsd3500_video_ops,
	.pad = &adsd3500_subdev_pad_ops,
};

with CONFIG_VIDEO_ADV_DEBUG is not set

[JerryChang]

hello Carbure,

you may try making .core = &adsd3500_core_ops available.
since it’s adsd3500_core_ops to call s_power for sensor power-on.
for instance,

static const struct v4l2_subdev_ops adsd3500_subdev_ops = {
	.core = &adsd3500_core_ops,
	.video = &adsd3500_video_ops,
	.pad = &adsd3500_subdev_pad_ops,
};

[Carbure]

Hi Jerry
I had the same Idea …
But doing that, leads to a kernel Panic… I think , at this point , I’ve to contact the manufacturer of the driver …

[JerryChang]

hello Carbure,

you may see-also r36.4.3 release note of [4. Implementation Details] for some camera integration changes.