/dev/video0 node not appearing when loading custom driver to capture pre-configured streaming camera data without i2c

bala.krishnan.gk · September 2, 2022, 10:53am

Hi,
we have designed an interface board to capture CSI data from an already configured and streaming camera and direct it to the Jetson Orin camera connector. I’ve written the driver and the .dtsi files (based on the imx185 and ov5693 cameras) to only do the port binding between the internal nvcsi bus and video input, and the driver attempts to register /dev/video0 if it gets data on the CSI connector. The driver also assumes the camera is already streaming, so I have no mode table for it and every part regarding i2c communication has either been avoided or simply returns 0 in the driver code.

The kernel and the driver compiled successfully, I’ve replaced /boot/Image, /boot/dtb/kernel_tegra234-p3701-0000-p3737-0000.dtb and nvgpu.ko. Decompiling the .dtb file shows that the parameters in the sensor .dtsi file are available correctly.

I’ve attached the sensor driver code, sensor and platform .dtsi files for the camera, .dts main platform file and dmesg log for your reference.

Here are my problems:

I load the driver module in the orin, start camera streaming from another system, but I observe no /dev/video0.
Under the path /proc/device-tree/i2c@3180000/tca9546@70/i2c@0, I see none of the device tree parameters of my camera like I see for all the other camera .dtsi files, for example imx390, imx185, etc.
The command sudo media-ctl -p -d /dev/media0 shows an empty device topology.

Media controller API version 5.10.65

Media device information
------------------------
driver          tegra-camrtc-ca
model           NVIDIA Tegra Video Input Device
serial          
bus info        
hw revision     0x3
driver version  5.10.65

Device topology

I’ve also carefully looked through the .dtsi port bindings, but everything looks similar to the ov5693 and imx185 .dtsi files and seem correct.

I would appreciate it a lot if someone could let me know if I’ve made any mistake in the .dtsi files, the driver code or if I’m doing something wrong with the way the driver module is being loaded as a kernel module. I’ll be glad to hear further debugging tips as well.
nv_ovx1f.c (23.4 KB)
tegra234-camera-rvs215.dtsi (8.4 KB)
tegra234-p3737-0000-camera-rvs215.dtsi (1.6 KB)
tegra234-p3701-0000-p3737-0000.dts (4.0 KB)
dmesg_kernel_log.txt (75.6 KB)

ShaneCCC · September 5, 2022, 5:42am

Remove the tca9546 have sensor under i2c@3180000 directly to try.

bala.krishnan.gk · September 5, 2022, 12:08pm

Hi Shane, thank you for the reply!

I tried having the sensor under i2c@3180000, compiled the driver and reloaded the kernel module. There is unfortunately still no video0 node under /dev. Also the camera .dtsi parameters are again missing when checked under /proc/device-tree/i2c@3180000/.
I changed the following lines in the .dtsi file:

Under tegra-camera-platform, proc-device-tree = “/proc/device-tree/i2c@3180000/ovx1f@36”;
Commented out the lines related to tca9546@70/i2c@0 in both tegra234-camera-rvs215.dtsi and tegra234-p3737-0000-camera-rvs215.dtsi

I also tried changing the deserialiser configuration on the main board where the camera is connected, the deserialiser is now configured to receive YUV422 8 bit data and it will redirect the data to a CSI connector on the board which is in turn connected to our interface board with the Orin. So the lines in the .dtsi files related to resolution and pix_clk_hz are also changed. However, this also did not help in obtaining /dev/video0.

I’ve checked the deserialiser conifgurations with another frame grabber for the same camera and it receives images properly.

Please let me know whether there is something else I can check or something else which I could try.

Here are the updated files for your reference:
tegra234-p3737-0000-camera-rvs215.dtsi (1.6 KB)
tegra234-camera-rvs215.dtsi (8.6 KB)

ShaneCCC · September 5, 2022, 1:05pm

Confirm if the system apply your dtb by below command.

sudo cat /proc/device-tree/nvidia,dtsfilename

bala.krishnan.gk · September 5, 2022, 1:15pm

Running this command gives me:
/dvs/git/dirty/git-master_linux/kernel/kernel-5.10/arch/arm64/boot/dts/../../../../../../hardware/nvidia/platform/t23x/concord/kernel-dts/tegra234-p3701-0000-p3737-0000.dts

I’m not sure how to interpret this, this is the correct path of the .dts file from the downloaded L4T BSP sources (hardware/nvidia/platform/t23x/concord/kernel-dts/tegra234-p3701-0000-p3737-0000.dts), so I assume the system applied it?

I also manually copy the .dts file after rebuilding kernel from the path kernel_out/arch/arm64/boot/dts/nvidia/tegra234-p3701-0000-p3737-0000.dtb to /boot/dtb and rename it to kernel_tegra234-p3701-0000-p3737-0000.dtb
I’m not sure if this step affects anything, I just noticed this in the Jetson Linux r35.1 page under kernel customization section and it was missing in the r34.1 section (r34.1 is the jetson linux version on my Orin), so I did this step just in case.

ShaneCCC · September 5, 2022, 2:22pm

If applied your dtb the file patch should be the source path of your host.
Did you modify the /boot/extlinux/extlinux.conf to add FDT?

bala.krishnan.gk · September 6, 2022, 8:23am

Thank you for pointing out the FDT entry!

I’ve added the FDT entry in extlinux.conf and I can see the ovx1f .dtsi properties under /proc/device-tree/i2c@3180000.
Here is the output of sudo media-ctl -p -d /dev/media0

Media controller API version 5.10.65

Media device information
------------------------
driver          tegra-camrtc-ca
model           NVIDIA Tegra Video Input Device
serial          
bus info        
hw revision     0x3
driver version  5.10.65

Device topology
- entity 1: 13e40000.host1x:nvcsi@15a00000- (2 pads, 0 link)
            type V4L2 subdev subtype Unknown flags 0
	pad0: Sink
	pad1: Source

I hope the port bindings are correct. But /dev/video0 is still not appearing, the dmesg logs show errors in the sensor driver and there is a warning about extending pixel format for yuv_uyvy8.

Here is the dmesg log for your reference:
kernel_log.txt (449.6 KB)

The sensor_common.c and camera_common.c already have the yuv formats but in 1x16 mode. Is the default bits per pixel 16 bits for YUV formats?
Here are my sensor_common.c and camera_common.c files for your reference:
camera_common.c (28.6 KB)
sensor_common.c (21.4 KB)

Does this mean I must change the minimum bits per pixel in the device tree to 16 instead of 8?

I tried changing the csi_pixel_bit_depth under sensor properties to 16 and min_bits_per_pixel under tegra_camera_platform to 16. I’ve compiled again, replaced the .dts file in /boot/dtb. Loading the driver now crashes the Orin.

ShaneCCC · September 6, 2022, 11:43am

Yes, the csi_pixel_bit_depth is 16 for YUV sensor.
Don’t modify the min_bits_per_pixel to try.

bala.krishnan.gk · September 6, 2022, 11:56am

Hi Shane,

I modified the csi_pixel_bit_depth to 16 and kept the min_bits_per_pixel to 8, but the Orin still crashes. Could it be an issue with the driver code?
And is the port binding correct? I’ve written the device tree to link VI port 0 to NvCSI out port 0, NvCSI input is connected to sensor output, and the camera is a 1x4 lane camera. CSI 0 and CSI 1 data pins and CSI 0 clock pins of the jetson camera connector are mapped to our interface board as it is mentioned in the product design guide.

bala.krishnan.gk · September 7, 2022, 4:11pm

Hello again Shane, I’ve now rewritten my driver based on the ov5693.c sensor driver and just removed the i2c part and tegra ctrl ops parts, added a dummy mode table and some dummy gain, exposure values in the device tree and I’m now able to see /dev/video0, but there is a problem.

When I use v4l2src or Gstreamer to view the video stream, I get a completely green image.
Here is my dmesg in case it’s useful.
kernel_log.txt (633.8 KB)

Here is the output of v4l2-ctl -d /dev/video0 --all

Driver Info:
	Driver name      : tegra-video
	Card type        : vi-output, ovx1f 2-0036
	Bus info         : platform:tegra-capture-vi:0
	Driver version   : 5.10.65
	Capabilities     : 0x84200001
		Video Capture
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04200001
		Video Capture
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : tegra-camrtc-ca
	Model            : NVIDIA Tegra Video Input Device
	Serial           : 
	Bus info         : 
	Media version    : 5.10.65
	Hardware revision: 0x00000003 (3)
	Driver version   : 5.10.65
Interface Info:
	ID               : 0x03000008
	Type             : V4L Video
Entity Info:
	ID               : 0x00000006 (6)
	Name             : vi-output, ovx1f 2-0036
	Function         : V4L2 I/O
	Pad 0x01000007   : 0: Sink
	  Link 0x0200000c: from remote pad 0x1000003 of entity '13e40000.host1x:nvcsi@15a00000-': Data, Enabled
Priority: 2
Video input : 0 (Camera 0: ok)
Format Video Capture:
	Width/Height      : 1100/884
	Pixel Format      : 'UYVY' (UYVY 4:2:2)
	Field             : None
	Bytes per Line    : 2200
	Size Image        : 1944800
	Colorspace        : sRGB
	Transfer Function : Default (maps to sRGB)
	YCbCr/HSV Encoding: Default (maps to ITU-R 601)
	Quantization      : Default (maps to Limited Range)
	Flags             : 

Camera Controls

           sensor_configuration 0x009a2032 (u32)    : min=0 max=4294967295 step=1 default=0 [22] flags=read-only, volatile, has-payload
         sensor_mode_i2c_packet 0x009a2033 (u32)    : min=0 max=4294967295 step=1 default=0 [1026] flags=read-only, volatile, has-payload
      sensor_control_i2c_packet 0x009a2034 (u32)    : min=0 max=4294967295 step=1 default=0 [1026] flags=read-only, volatile, has-payload
                    bypass_mode 0x009a2064 (intmenu): min=0 max=1 default=0 value=0
				0: 0 (0x0)
				1: 1 (0x1)
                override_enable 0x009a2065 (intmenu): min=0 max=1 default=0 value=0
				0: 0 (0x0)
				1: 1 (0x1)
                   height_align 0x009a2066 (int)    : min=1 max=16 step=1 default=1 value=1
                     size_align 0x009a2067 (intmenu): min=0 max=2 default=0 value=0
				0: 1 (0x1)
				1: 65536 (0x10000)
				2: 131072 (0x20000)
               write_isp_format 0x009a2068 (int)    : min=1 max=1 step=1 default=1 value=1
       sensor_signal_properties 0x009a2069 (u32)    : min=0 max=4294967295 step=1 default=0 [30][18] flags=read-only, has-payload
        sensor_image_properties 0x009a206a (u32)    : min=0 max=4294967295 step=1 default=0 [30][16] flags=read-only, has-payload
      sensor_control_properties 0x009a206b (u32)    : min=0 max=4294967295 step=1 default=0 [30][36] flags=read-only, has-payload
              sensor_dv_timings 0x009a206c (u32)    : min=0 max=4294967295 step=1 default=0 [30][16] flags=read-only, has-payload
               low_latency_mode 0x009a206d (bool)   : default=0 value=0
               preferred_stride 0x009a206e (int)    : min=0 max=65535 step=1 default=0 value=0
                   sensor_modes 0x009a2082 (int)    : min=0 max=30 step=1 default=30 value=1 flags=read-only

Here is the gstreamer pipeline I tried(unsure if it is correct, I would like to have your suggestions here):
gst-launch-1.0 v4l2src device=/dev/video0 ! ‘video/x-raw, format=(string)UYVY, width=(int)1100, height=(int)884, framerate=(fraction)30/1’ ! xvimagesink -ev

Here is the output of sudo media-ctl -p -d /dev/media0:

Media controller API version 5.10.65

Media device information
------------------------
driver          tegra-camrtc-ca
model           NVIDIA Tegra Video Input Device
serial          
bus info        
hw revision     0x3
driver version  5.10.65

Device topology
- entity 1: 13e40000.host1x:nvcsi@15a00000- (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
	pad0: Sink
		<- "ovx1f 2-0036":0 [ENABLED]
	pad1: Source
		-> "vi-output, ovx1f 2-0036":0 [ENABLED]

- entity 4: ovx1f 2-0036 (1 pad, 1 link)
            type V4L2 subdev subtype Sensor flags 0
            device node name /dev/v4l-subdev1
	pad0: Source
		[fmt:UYVY8_1X16/1100x884 field:none colorspace:srgb]
		-> "13e40000.host1x:nvcsi@15a00000-":0 [ENABLED]

- entity 6: vi-output, ovx1f 2-0036 (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video0
	pad0: Sink
		<- "13e40000.host1x:nvcsi@15a00000-":1 [ENABLED]

And here is the output of the V4l2 compliance test

v4l2-compliance SHA: not available, 64 bits

Compliance test for tegra-video device /dev/video0:

Driver Info:
	Driver name      : tegra-video
	Card type        : vi-output, ovx1f 2-0036
	Bus info         : platform:tegra-capture-vi:0
	Driver version   : 5.10.65
	Capabilities     : 0x84200001
		Video Capture
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04200001
		Video Capture
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : tegra-camrtc-ca
	Model            : NVIDIA Tegra Video Input Device
	Serial           : 
	Bus info         : 
	Media version    : 5.10.65
	Hardware revision: 0x00000003 (3)
	Driver version   : 5.10.65
Interface Info:
	ID               : 0x03000008
	Type             : V4L Video
Entity Info:
	ID               : 0x00000006 (6)
	Name             : vi-output, ovx1f 2-0036
	Function         : V4L2 I/O
	Pad 0x01000007   : 0: Sink
	  Link 0x0200000c: from remote pad 0x1000003 of entity '13e40000.host1x:nvcsi@15a00000-': Data, Enabled

Required ioctls:
	test MC information (see 'Media Driver Info' above): OK
	test VIDIOC_QUERYCAP: OK

Allow for multiple opens:
	test second /dev/video0 open: OK
	test VIDIOC_QUERYCAP: OK
	test VIDIOC_G/S_PRIORITY: OK
	test for unlimited opens: OK

Debug ioctls:
	test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
	test VIDIOC_LOG_STATUS: OK

Input ioctls:
	test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
	test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
	test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
	test VIDIOC_ENUMAUDIO: OK (Not Supported)
	test VIDIOC_G/S/ENUMINPUT: OK
	test VIDIOC_G/S_AUDIO: OK (Not Supported)
	Inputs: 1 Audio Inputs: 0 Tuners: 0

Output ioctls:
	test VIDIOC_G/S_MODULATOR: OK (Not Supported)
	test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
	test VIDIOC_ENUMAUDOUT: OK (Not Supported)
	test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
	test VIDIOC_G/S_AUDOUT: OK (Not Supported)
	Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
	test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
	test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
	test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
	test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls (Input 0):
	test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
	test VIDIOC_QUERYCTRL: OK
	test VIDIOC_G/S_CTRL: OK
	test VIDIOC_G/S/TRY_EXT_CTRLS: OK
	test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
	test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
	Standard Controls: 1 Private Controls: 15

Format ioctls (Input 0):
		fail: v4l2-test-formats.cpp(280): duplicate format 59565955 (UYVY)
	test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: FAIL
		fail: v4l2-test-formats.cpp(1280): ret && node->has_frmintervals
	test VIDIOC_G/S_PARM: FAIL
	test VIDIOC_G_FBUF: OK (Not Supported)
	test VIDIOC_G_FMT: OK
	test VIDIOC_TRY_FMT: OK
	test VIDIOC_S_FMT: OK
	test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
	test Cropping: OK (Not Supported)
	test Composing: OK (Not Supported)
	test Scaling: OK (Not Supported)

Codec ioctls (Input 0):
	test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
	test VIDIOC_G_ENC_INDEX: OK (Not Supported)
	test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls (Input 0):
		fail: v4l2-test-buffers.cpp(715): q.create_bufs(node, 1, &fmt) != EINVAL
	test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: FAIL
	test VIDIOC_EXPBUF: OK
	test Requests: OK (Not Supported)

Total for tegra-video device /dev/video0: 45, Succeeded: 42, Failed: 3, Warnings: 0

I would love to have some further debugging tips, I’m still unsure if I’m receiving proper data.

ShaneCCC · September 8, 2022, 5:13am

Get the trace log to check.

https://elinux.org/Jetson/l4t/Camera_BringUp

bala.krishnan.gk · September 8, 2022, 10:10am

Hi Shane, here is the trace log:
trace.txt (15.2 KB)

Here are my updated driver and .dtsi files for your reference:
nv_ovx1f.c (14.4 KB)
tegra234-camera-rvs215.dtsi (9.6 KB)
ovx1f_mode_tbls.h (1.2 KB)

I also notice that in dmesg, the device name is different when compared with the device tree. It appears as ovx1f 2-0036 during the probe, when it actually should be ovx1f 30-0036 according to my device tree. I am not sure if affects anything, but I did some modifications to the driver code and the mode table by referring to the below forum post:

And strangely, /dev/video0 appears even if I connect nothing to the CSI camera connector on the Orin. And the same green images are seen with gstreamer command.

ShaneCCC · September 8, 2022, 1:22pm

Could you update the debug firmware to collect the log.

bala.krishnan.gk · September 8, 2022, 1:25pm

Hi Shane, I’ll try that out now.
But this needs an ubuntu host PC to flash the Orin right? Or can I run that command from within the Orin itself?
And should I update to r35.1 before I do this?

ShaneCCC · September 8, 2022, 1:48pm

Need PC and install ubuntu 18.04 for install sdkmanager.

bala.krishnan.gk · September 9, 2022, 2:27pm

Hi Shane, I updated to r35.1, updated the debug firmware.

Here are the trace logs and dmesg logs for the same issue:
dmesg.txt (136.4 KB)
trace.txt (503.0 KB)

ShaneCCC · September 12, 2022, 4:19am

Looks like NVCSI/VI didn’t receive any validate package form MIPI bus.

bala.krishnan.gk · September 12, 2022, 12:17pm

Hi Shane, you were right, the cable we were using inverted the odd pin signals and even pin signals when going from the CSI connector on the deserialiser board to our interface board on the Orin.

Now we’ve patched the cable and we’ve probed and checked that there is CSI signal and clock appearing on the interface board on the right pins.

Loading the driver again and running the following gstreamer command again crashes the Orin.
gst-launch-1.0 v4l2src device=/dev/video0 ! “video/x-raw, format=(string)UYVY, width=(int)1100, height=(int)884, framerate=(fraction)30/1” ! xvimagesink -ev

Logs:
dmesg.txt (271.2 KB)
trace.txt (7.7 KB)

ShaneCCC · September 12, 2022, 1:16pm

The trace log tell the SOT error(Start Of Transmission)
That could be the ths_settle problem.
Make sure the timing like below.

bala.krishnan.gk · September 15, 2022, 8:49am

Hello again Shane,

we checked the CSI timing and the amplitudes of the signal on the 8 pins of the 4 CSI lanes we are using on our interface board and the board carrying the deser.

When the interface board is connected to the Orin, we notice on the pin on the interface board corresponding to pin 13 of the jetson camera (CSI0_D1_P), that there is a voltage drop. Instead of the usual 1.2V, the observed amplitude is 0.9 V.

On disconnecting the adaptor board from the Orin, and measuring the signal only on the matching adaptor board pin, we observe the full 1.2 V on the same pin. The voltage drop for some reason occurs only when connected to the Orin.

On every other pin, the timing and the amplitudes of the CSI signal are all fine with and without the Orin connected. The clock signals also seems fine.

Do you know if there is any configuration parameter in the driver development which could change voltage only for that particular pin? I don’t recall ever configuring pins directly, I would be happy to hear your suggestions.