TX2 problems with 2 MIPI CSI-2 cameras; drivers as LKM; L4T 32.3.1

I’m using Jetson TX2, new Jetpack 4.3 with L4T 32.3.1.
Image sensors orders:
1.) IMX290 (I2C MUX address 30 and first MIPI CSI port)
2.) IMX296 (I2C MUX address 32 and second MIPI CSI-2 port).

I have problem with multistream(probing and stream with 2 image sensors). When i use image sensor drivers as LKM - loadable kernel modules: in sources/kernel/kernel-4.9/arch/arm64/ in configs tegra_defconfig file i set”CONFIG_VIDEO_IMX290=m” and ”CONFIG_VIDEO_IMX296=m” and then only streaming from the IMX296 works, but does not work stream from IMX290.
In dmesg I see:

[ 5.380161] imx296 32-001a: probing v4l2 sensor
[ 5.381489] imx296 32-001a: tegracam sensor driver:imx296_v2.0.6
[ 5.384889] imx290 30-001a: probing v4l2 sensor
[ 5.390223] imx290 30-001a: tegracam sensor driver:imx290_v2.0.6
[ 5.489447] imx290 30-001a: imx290_write_reg: i2c write failed, 0x3001 = 1
[ 5.496938] imx290 30-001a: imx290_read_buffered_reg: error setting register hold
[ 5.505539] imx290 30-001a: imx290_calculate_line_time: unable to read hmax
[ 5.513829] tegra-vi4 15700000.vi: subdev imx296 32-001a bound
[ 5.521218] imx290 30-001a: imx290_write_reg: i2c write failed, 0x304b = 0
[ 5.522190] imx296 32-001a: Detected imx296 sensor
[ 5.529642] imx290 30-001a: imx290_power_off: error setting XVS XHS to Hi-Z
[ 5.586493] tegra-vi4 15700000.vi: subdev imx290 30-001a bound
[ 5.587040] imx290 30-001a: Detected imx290 sensor

I2C communication with both sensors works! Checked with qv4l2!

It is not clear to me:
1.) Why the: “…vi: subdev imx296 32-001a bound” is before the: “…vi: subdev imx290 30-001a bound”??? (VI @32 is bound before VI @30 why?)
When i replace positions of sensors: IMX296(I2C MUX address 30 and first MIPI CSI port) and IMX290(I2C MUX address 32 and second MIPI CSI-2 port) everything works fine! Maybe because of the alphabet or?

2.) If i use drivers included in kernel image -> in sources/kernel/kernel-4.9/arch/arm64/ in configs tegra_defconfig file i set”CONFIG_VIDEO_IMX290=y” and ”CONFIG_VIDEO_IMX296=y” everything works fine - irrelevant to sensor positions!
Right detection sequence: 1.) Probing I2C MUX @30: IMX290; 2.) Detected IMX290 ; 3.)Probing I2C MUX @32: IMX296; 4.) Detected IMX296; 5.) VI 290 bound 6.) VI 296 bound

3.) loadable kernel modules works parallel? I ask because in dmesg (see print above) the sensor detection sequence is intertwined – not in right order as expected and described in the previous step !?

4.) have you tried streaming with two different sensors with drivers installed as LKM?

There’s no problem for the multiple sensors for the LKM like E3333 6 ov5693 worked without problem.
What do you means multistreaming not working? What’s you command for multistreaming test?

Everything works if both two are equal sensors (for example two imx290 sensors), but with different sensors at the same time (for example one imx290 and one imx296) is a problem! Have You tried with different sensors?

I mean two different sensors are connected and an error occurs in dmesg when starting the stream:

[ 59.931209] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[ 59.937748] tegra-vi4 15700000.vi: tegra_channel_error_recovery: attempting to reset the capture channel
[ 59.948043] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) ERROR_STATUS2VI_VC0 = 0x00000004
[ 59.956890] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) INTR_STATUS 0x0000000c
[ 59.964969] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) ERR_INTR_STATUS 0x0000000c
[ 60.179126] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11

This error only occurs when i use the drivers as LKM, if they are used directly in the kernel Image then everything is fine without error !
I suspect that the problem is with VI bound because then probing sequence and VI bound is not in the expected order.
It is a different order (probing sequence and VI bound) if I use LKM or put drivers in the Image kernel! Why ?

argus_camera or qv4l2. I don’t think it matters because everything works fine if the drivers are at the kernel Image.

Does it work by manual insmod with the right sequency?



TEST1:
- Hardware configuration
first CSI port --> IMX290
second CSI port --> IMX296

- insmod sequence:
	TEST a:
		1.) insmod imx290.ko
		2.) insmod imx296.ko
		RESULTS: IMX290 work, IMX296 work!

	TEST b: 
		1.) insmod imx296.ko
		2.) insmod imx290.ko
		RESULTS: IMX296 work, IMX290 does NOT work!!!


TEST2:
- Hardware configuration:
first CSI port --> IMX296
second CSI port --> IMX290

- insmod sequence:
	TEST a:
		1.) insmod imx296.ko
		2.) insmod imx290.ko
		RESULTS: IMX296 work, IMX290 does NOT work!!!

	TEST b: 
		1.) insmod imx290.ko
		2.) insmod imx296.ko
		RESULTS: IMX290 work, IMX296 work!


Could you check any different by media-ctl -p for the working and nonworking case?

TEST1:

  • Hardware configuration
    first CSI port --> IMX290
    second CSI port --> IMX296

  • insmod sequence:

TEST 1a:
1.) insmod imx290.ko
2.) insmod imx296.ko
RESULTS: IMX290 work, IMX296 work!

Media controller API version 0.1.0

Media device information
------------------------
driver          tegra-vi4
model           NVIDIA Tegra Video Input Device
serial          
bus info        
hw revision     0x3
driver version  0.0.0

Device topology
- entity 1: 150c0000.nvcsi--6 (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
	pad0: Sink
		<- "imx290 30-001a":0 [ENABLED]
	pad1: Source
		-> "vi-output, imx290 30-001a":0 [ENABLED]

- entity 4: 150c0000.nvcsi--4 (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev1
	pad0: Sink
		<- "imx296 32-001a":0 [ENABLED]
	pad1: Source
		-> "vi-output, imx296 32-001a":0 [ENABLED]

- entity 7: imx290 30-001a (1 pad, 1 link)
            type V4L2 subdev subtype Sensor flags 0
            device node name /dev/v4l-subdev2
	pad0: Source
		[fmt:SRGGB10_1X10/1920x1080 field:none colorspace:srgb]
		-> "150c0000.nvcsi--6":0 [ENABLED]

- entity 9: vi-output, imx290 30-001a (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video0
	pad0: Sink
		<- "150c0000.nvcsi--6":1 [ENABLED]

- entity 23: imx296 32-001a (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev3
	pad0: Source
		[fmt:SRGGB10_1X10/1456x1088 field:none colorspace:srgb]
		-> "150c0000.nvcsi--4":0 [ENABLED]

- entity 25: vi-output, imx296 32-001a (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video1
	pad0: Sink
		<- "150c0000.nvcsi--4":1 [ENABLED]

TEST 1b:
1.) insmod imx296.ko
2.) insmod imx290.ko
RESULTS: IMX296 work, IMX290 does NOT work!!!

Media controller API version 0.1.0

Media device information
------------------------
driver          tegra-vi4
model           NVIDIA Tegra Video Input Device
serial          
bus info        
hw revision     0x3
driver version  0.0.0

Device topology
- entity 1: 150c0000.nvcsi--6 (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
	pad0: Sink
		<- "imx290 30-001a":0 [ENABLED]
	pad1: Source
		-> "vi-output, imx290 30-001a":0 [ENABLED]

- entity 4: 150c0000.nvcsi--4 (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev1
	pad0: Sink
		<- "imx296 32-001a":0 [ENABLED]
	pad1: Source
		-> "vi-output, imx296 32-001a":0 [ENABLED]

- entity 7: imx296 32-001a (1 pad, 1 link)
            type V4L2 subdev subtype Sensor flags 0
            device node name /dev/v4l-subdev2
	pad0: Source
		[fmt:SRGGB10_1X10/1456x1088 field:none colorspace:srgb]
		-> "150c0000.nvcsi--4":0 [ENABLED]

- entity 9: vi-output, imx296 32-001a (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video0
	pad0: Sink
		<- "150c0000.nvcsi--4":1 [ENABLED]

- entity 23: imx290 30-001a (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev3
	pad0: Source
		[fmt:SRGGB10_1X10/1920x1080 field:none colorspace:srgb]
		-> "150c0000.nvcsi--6":0 [ENABLED]

- entity 25: vi-output, imx290 30-001a (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video1
	pad0: Sink
		<- "150c0000.nvcsi--6":1 [ENABLED]



TEST2:

  • Hardware configuration:
    first CSI port --> IMX296
    second CSI port --> IMX290

  • insmod sequence:

TEST 2a:
1.) insmod imx296.ko
2.) insmod imx290.ko
RESULTS: IMX296 work, IMX290 does NOT work!!!

Media controller API version 0.1.0

Media device information
------------------------
driver          tegra-vi4
model           NVIDIA Tegra Video Input Device
serial          
bus info        
hw revision     0x3
driver version  0.0.0

Device topology
- entity 1: 150c0000.nvcsi--6 (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
	pad0: Sink
		<- "imx296 30-001a":0 [ENABLED]
	pad1: Source
		-> "vi-output, imx296 30-001a":0 [ENABLED]

- entity 4: 150c0000.nvcsi--4 (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev1
	pad0: Sink
		<- "imx290 32-001a":0 [ENABLED]
	pad1: Source
		-> "vi-output, imx290 32-001a":0 [ENABLED]

- entity 7: imx296 30-001a (1 pad, 1 link)
            type V4L2 subdev subtype Sensor flags 0
            device node name /dev/v4l-subdev2
	pad0: Source
		[fmt:SRGGB10_1X10/1456x1088 field:none colorspace:srgb]
		-> "150c0000.nvcsi--6":0 [ENABLED]

- entity 9: vi-output, imx296 30-001a (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video0
	pad0: Sink
		<- "150c0000.nvcsi--6":1 [ENABLED]

- entity 23: imx290 32-001a (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev3
	pad0: Source
		[fmt:SRGGB10_1X10/1920x1080 field:none colorspace:srgb]
		-> "150c0000.nvcsi--4":0 [ENABLED]

- entity 25: vi-output, imx290 32-001a (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video1
	pad0: Sink
		<- "150c0000.nvcsi--4":1 [ENABLED]

TEST 2b:
1.) insmod imx290.ko
2.) insmod imx296.ko
RESULTS: IMX290 work, IMX296 work!

Media controller API version 0.1.0

Media device information
------------------------
driver          tegra-vi4
model           NVIDIA Tegra Video Input Device
serial          
bus info        
hw revision     0x3
driver version  0.0.0

Device topology
- entity 1: 150c0000.nvcsi--6 (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
	pad0: Sink
		<- "imx296 30-001a":0 [ENABLED]
	pad1: Source
		-> "vi-output, imx296 30-001a":0 [ENABLED]

- entity 4: 150c0000.nvcsi--4 (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev1
	pad0: Sink
		<- "imx290 32-001a":0 [ENABLED]
	pad1: Source
		-> "vi-output, imx290 32-001a":0 [ENABLED]

- entity 7: imx290 32-001a (1 pad, 1 link)
            type V4L2 subdev subtype Sensor flags 0
            device node name /dev/v4l-subdev2
	pad0: Source
		[fmt:SRGGB10_1X10/1920x1080 field:none colorspace:srgb]
		-> "150c0000.nvcsi--4":0 [ENABLED]

- entity 9: vi-output, imx290 32-001a (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video0
	pad0: Sink
		<- "150c0000.nvcsi--4":1 [ENABLED]

- entity 23: imx296 30-001a (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev3
	pad0: Source
		[fmt:SRGGB10_1X10/1456x1088 field:none colorspace:srgb]
		-> "150c0000.nvcsi--6":0 [ENABLED]

- entity 25: vi-output, imx296 30-001a (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video1
	pad0: Sink
		<- "150c0000.nvcsi--6":1 [ENABLED]