Custom camera driver and device tree

euskadi · November 19, 2019, 3:19pm

I have a working custom camera (not a MIPI sensor, but the output of a MIPI IP Core) connected to the Jetson TX2 camera expansion header. I use 4 lanes of data. The pixel format is RAW10, the framerate is 50 fps, and the resolution is 1920x1080.

I’ve built a device tree and a driver for the camera based on the Sensor Driver Programming Guide, and I achieved it to be loaded and recognized (appears a /dev/video0 entry). Anyway, when trying to stream video (nvgstcapture-1.0), I get the following error:

[  +1,504481] fence timeout on [ffffffc18e485900] after 1500ms
[  +0,000006] fence timeout on [ffffffc18e485180] after 1500ms
[  +0,000002] name=[nvhost_sync:44], current value=0 waiting value=1
[  +0,000004] name=[nvhost_sync:31], current value=0 waiting value=1
[  +0,000001] ---- mlocks ----
[  +0,000004] fence timeout on [ffffffc18e485000] after 1500ms
[  +0,000001] ---- mlocks ----
[  +0,000004] name=[nvhost_sync:34], current value=0 waiting value=1
[  +0,000002] 8: locked by channel 7
[  +0,000001] ---- mlocks ----
[  +0,000003] 8: locked by channel 7
[  +0,000007] 8: locked by channel 7

[  +0,000007] ---- syncpts ----

[  +0,000004] ---- syncpts ----

The pipeline I use is the following one:

gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)50/1' ! nvoverlaysink -e

Anyway, many of the properties that must be defined in the DT and driver are not relevant for me (e.g. power functions, sensor dimensions, mclk …), as the camera is already boot up and transmitting data through the CSI lanes. Is there any simpler way to build a driver that captures data in the specified format?

JerryChang · November 20, 2019, 3:09am

hello euskadi,

that fence timeout failure means low-level driver cannot receive the sensor frame successfully.
had you connect the oscilloscope to probe the sensor signaling and confirm the it’s following MIPI specification?

according to Camera Architecture Stack, could you please narrow down the issue by skipping [Camera Core].
please have an alternative way to access camera sensor via v4l2 standard controls.
for example,

$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=100

you might also review [Sensor Software Driver Programming Guide], and check Sensor Pixel Clock session.
please ensure you’re configure sensor pixel clock correctly.

BTW,
if you still encounter the same failures with v4l2 standard control,
please enable VI tracing logs via debugfs and sharing the details for investigation.
thanks

echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo > /sys/kernel/debug/tracing/trace
cat /sys/kernel/debug/tracing/trace

euskadi · November 20, 2019, 9:39am

Hello JerryChang,

Here are the results. Before running them, I’ve updated the pixel_clk to 480000000, considering 4 data lanes, 10 bits per pixel and 1.2Gbps per lane. Regarding the debugging with the oscilloscope, I connected it and can see there is data transmitting, centered around ~200mV and 400mVpp.

When using the pipeline to skip the Camera Core, I get the following kernel messages:

...
[  265.218087] tegra-vi4 15700000.vi: tegra_channel_error_recovery: attempting to reset the capture channel
[  265.431631] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[  265.438118] tegra-vi4 15700000.vi: tegra_channel_error_recovery: attempting to reset the capture channel
[  265.651565] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[  265.658167] tegra-vi4 15700000.vi: tegra_channel_error_recovery: attempting to reset the capture channel
[  265.871460] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
...

Regarding the trace log, I get the following:

root@nvidia-desktop:/home/nvidia# cat /sys/kernel/debug/tracing/trace
# tracer: nop
#
# entries-in-buffer/entries-written: 387/387   #P:4
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
     kworker/0:0-4     [000] ....   135.110966: rtos_queue_peek_from_isr_failed: tstamp:4553073768 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   135.222944: rtos_queue_peek_from_isr_failed: tstamp:4558073727 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   135.390949: rtos_queue_peek_from_isr_failed: tstamp:4563073733 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   135.558937: rtos_queue_peek_from_isr_failed: tstamp:4568073748 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   135.726919: rtos_queue_peek_from_isr_failed: tstamp:4573073757 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   135.894928: rtos_queue_peek_from_isr_failed: tstamp:4578073763 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   136.062958: rtos_queue_peek_from_isr_failed: tstamp:4583073758 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   136.230991: rtos_queue_peek_from_isr_failed: tstamp:4588073779 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   136.342860: rtos_queue_peek_from_isr_failed: tstamp:4593073759 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   136.510920: rtos_queue_peek_from_isr_failed: tstamp:4598073838 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   136.678869: rtos_queue_peek_from_isr_failed: tstamp:4603073764 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   136.850853: rtos_queue_peek_from_isr_failed: tstamp:4608073776 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   137.018892: rtos_queue_peek_from_isr_failed: tstamp:4613073785 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   137.186927: rtos_queue_peek_from_isr_failed: tstamp:4618073821 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   137.298895: rtos_queue_peek_from_isr_failed: tstamp:4623073819 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   137.466916: rtos_queue_peek_from_isr_failed: tstamp:4628073845 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   137.634938: rtos_queue_peek_from_isr_failed: tstamp:4633073831 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   137.802948: rtos_queue_peek_from_isr_failed: tstamp:4638073845 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   137.970918: rtos_queue_peek_from_isr_failed: tstamp:4643073851 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   138.138839: rtos_queue_peek_from_isr_failed: tstamp:4648073832 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   138.307034: rtos_queue_peek_from_isr_failed: tstamp:4653073840 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   138.418904: rtos_queue_peek_from_isr_failed: tstamp:4658073897 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   138.586913: rtos_queue_peek_from_isr_failed: tstamp:4663073906 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   138.754858: rtos_queue_peek_from_isr_failed: tstamp:4668073886 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   138.922893: rtos_queue_peek_from_isr_failed: tstamp:4673073893 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   139.091021: rtos_queue_peek_from_isr_failed: tstamp:4678073902 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   139.258871: rtos_queue_peek_from_isr_failed: tstamp:4682670935 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   154.046814: rtos_queue_peek_from_isr_failed: tstamp:5146378888 queue:0x0b4b4500
     kworker/0:0-4     [000] ....   154.046818: rtcpu_start: tstamp:5146380436
     kworker/0:0-4     [000] ....   154.046820: rtos_queue_send_from_isr_failed: tstamp:5146389264 queue:0x0b4a7258
     kworker/0:0-4     [000] ....   154.046820: rtos_queue_send_from_isr_failed: tstamp:5146389372 queue:0x0b4aad68
     kworker/0:0-4     [000] ....   154.046821: rtos_queue_send_from_isr_failed: tstamp:5146389480 queue:0x0b4ac998
     kworker/0:0-4     [000] ....   154.046822: rtos_queue_send_from_isr_failed: tstamp:5146389587 queue:0x0b4ae518
     kworker/0:0-4     [000] ....   154.046830: rtos_queue_send_from_isr_failed: tstamp:5146389696 queue:0x0b4af2d8
     kworker/0:0-4     [000] ....   154.046831: rtos_queue_send_from_isr_failed: tstamp:5146389799 queue:0x0b4b0098
     kworker/0:0-4     [000] ....   154.046832: rtos_queue_send_from_isr_failed: tstamp:5146389903 queue:0x0b4b0e58
     kworker/0:0-4     [000] ....   154.046832: rtos_queue_send_from_isr_failed: tstamp:5146390008 queue:0x0b4b1c18
     kworker/0:0-4     [000] ....   154.046834: rtos_queue_send_failed: tstamp:5146390492 queue:0x0b4a7258
...
...

Nevertheless, would it be possible to set up a much simpler device tree? Things I would remove that are externally controlled are:

GPIO camera control

Physical_w & Physical_h

clocks (clocks, clock-names, mclk, clock-frequency and mclk_khz)

gpios

regulators

Then, in the driver, I’d just match the device for probing the sensor, and then start streaming.

JerryChang · November 21, 2019, 2:39am

hello euskadi,

could you please attach the oscilloscope results for checking.
due to PXL_SOF syncpt timeout failure means VI driver waiting for start-of-frame signaling timeout.
your VI tracing logs also not shown any messages about sensor registers.
you may check Sensor Software Driver Programming Guide for the device tree properties descriptions.
you might also check reference device tree for reference,
for example,
$l4t-r32.2/public_sources/kernel_src/hardware/nvidia/platform/t18x/common/kernel-dts/t18x-common-modules/tegra186-camera-e3326-a00.dtsi
please refer to Using the Main Platform Device Tree File chapter if you’re going to setup device tree simpler.
suggest you could also disassembler your own built dtb file into txt file for reviewing
for example,

$ dtc -I dtb -O dts -o output.txt $OUT/Linux_for_Tegra/kernel/dtb/tegra186-quill-p3310-1000-c03-00-base.dtb

euskadi · November 21, 2019, 8:35am

Hi JerryChang,

Thanks for the reply. Here are the results. I also attach the device tree at the end. It is pretty simple, as all the parameters are externally controlled, and only i2c commands are needed to modify them. Moreover, there is no need for a clock neither, as it is also externally included. Besides, is the clock used for frame sync? My MIPI clock lane goes at 120MHz and the data lane at 1200Mhz. I start the data transmission when the START_STREAMING function is called by sending a command through i2c, and it works fine, as the oscilloscope shows a plane signal before the start_streaming function is called, and shows a signal (the one showed in (1)) after being called.

I’m reading a differential data lane (D0-,D0+). The oscilloscope has a limit of 1Gsps, and the data lane is at 1.2Ghz, so I’m not looking deeply into the data, but only if something is being sent or not.

I’ve looked at it, and I’ve not implemented many properties (gain, mclk, sensor dimensions…) as they are externally controlled. I’ve set the mclk_khz to 120000 (120MHz) and the pix_clk_hz to 480000000 (1.2Ghz*4/10)
The way I include the device tree is as you suggested. I comment the following includes in the main device tree, and include mine:

#include <t18x-common-platforms/tegra186-quill-common-p3310-1000-a00.dtsi>
#include <t18x-common-platforms/tegra186-quill-power-tree-p3310-1000-a00-00.dtsi>
//#include <t18x-common-platforms/tegra186-quill-camera-modules.dtsi>
#include <t18x-common-modules/tegra186-display-e3320-1000-a00.dtsi>

/* comms dtsi file should be included after gpio dtsi file */
#include <t18x-common-platforms/tegra186-quill-comms.dtsi>
#include <t18x-common-plugin-manager/tegra186-quill-p3310-1000-a00-plugin-manager.dtsi>
#include <t18x-common-modules/tegra186-super-module-e2614-p2597-1000-a00.dtsi>
#include <t18x-common-plugin-manager/tegra186-quill-display-plugin-manager.dtsi>
#include <t18x-common-prod/tegra186-priv-quill-p3310-1000-a00-prod.dtsi>
//#include <t18x-common-plugin-manager/tegra186-quill-camera-plugin-manager.dtsi>
#include <t18x-common-modules/tegra186-camera-cam2000-a00.dtsi>

Device Tree

/ {
	host1x {
		vi@15700000 {
			num-channels = <1>;
			ports {
				#address-cells = <1>;
				#size-cells = <0>;
				status = "okay";
				port@0 {
					reg = <0>;
					status = "okay";
					/* VI block input */
					cam2000_vi_in0: endpoint {
						status = "okay";
						port-index = <0>;
						bus-width = <4>;
						remote-endpoint = <&cam2000_csi_out0>;
					};
				};
			};
		};
		nvcsi@150c0000 {
			num-channels = <1>;
			#address-cells = <1>;
			#size-cells = <0>;
			channel@0 {
				reg = <0>;
				ports {
					#address-cells = <1>;
					#size-cells = <0>;
					/* NVCSI block input */
					port@0 {
						reg = <0>;
						cam2000_csi_in0: endpoint@0 {
							status = "okay";
							port-index = <0>;
							bus-width = <4>;
							remote-endpoint = <&cam2000_out0>;
						};
					};
					/* NVCSI block output */
					port@1 {
						reg = <1>;
						cam2000_csi_out0: endpoint@1 {
							remote-endpoint = <&cam2000_vi_in0>;
						};
					};
				};
			};
		};
	};
};
/ {
	i2c@3180000 {
		status = "okay";
		cam2000_a@24 {
			compatible = "THH,cam2000";
			reg = <0x24>;
			devnode = "video0";
			sensor_model ="cam2000";
			mode0 {
				num_lanes = "4";
				tegra_sinterface = "serial_a";
				phy_mode = "DPHY";
				discontinuous_clk = "yes";
				dpcm_enable = "false";
				cil_settletime = "0";

				csi_pixel_bit_depth = "10";
				mode_type = "bayer";
				pixel_phase = "rggb";
				active_w = "1920";
				active_h = "1080";
				readout_orientation = "0";
				line_length = "1920";
				inherent_gain = "1";
				pix_clk_hz = "480000000";
			};
			ports {
				#address-cells = <1>;
				#size-cells = <0>;
				port@0 {
					reg = <0>;
					cam2000_out0: endpoint {
						port-index = <0>;
						bus-width = <4>;
						remote-endpoint = <&cam2000_csi_in0>;
					};
				};
			};
		};
	};
};
/ {
	tegra-camera-platform {
		compatible = "nvidia, tegra-camera-platform";
		num_csi_lanes = <4>;
		max_lane_speed = <1500000>;
		min_bits_per_pixel = <10>;
		vi_peak_byte_per_pixel = <2>;
		vi_bw_margin_pct = <25>;
		isp_peak_byte_per_pixel = <5>;
		isp_bw_margin_pct = <25>;
		modules {
			module0 {
				badge = "cam2000_bottom_THH";
				position = "bottom";
				orientation = "0";
				status = "okay";
				drivernode0 {
					/* PCL support driver (classically known as guid) */
					pcl_id = "v4l2_sensor";
					/* Driver v4l2 device name */
					devname = "cam2000 2-0024";
					/* Device-tree hierarchy to driver instance */
					proc-device-tree = "/proc/device-tree/i2c@3180000/cam2000_a@24";
				};
			};
		};
	};
};

JerryChang · November 21, 2019, 9:14am

hello euskadi,

you might still have details in the device tree,
please refer to below and also have modification for verification,
thanks

please keep mclk_khz property in the device tree, since it’ll used by sensor_signal_properties struct.
could you please review your line_length settings, this value usually larger then active_w. you might check the sensor init register table for the settings.
please also add embedded_metadata_height = “0”; property in the end of mode table. since VI will parse this property for cropping.

euskadi · November 22, 2019, 8:15am

Hi JerryChang,
The result is the same. I’ll try to look at the clock lane to see if there is something wrong. Else, the only thing that comes to my mind is that the synchronization is not correctly produced. How is the data synchronized? With the MIPI_CLK lanes? And how is that lane sampled in the TX2?
Do you know any other source of error for this to happen?
Regarding the line_length/active_w, which should be the difference? active_w is the actual resolution, but line_length? I see that its definition is “Pixel line width horizontal timing size for the sensor mode”, but how is it calculated? Does it consider the line blanking or so?

euskadi · November 22, 2019, 1:10pm

I’ve analysed the Clock (+) and one data line (+), and it shows like this:

Is this normal? Shouldn’t there be a LP state at around 1.2V as MIPI specifies in the D-PHY physical layer specification sheet between packets? Or the signal looks good?

Trumany · November 25, 2019, 2:23am

Hi, seems no 1.2V part in blue waveform. Please refer to MIPI D-PHY spec to check the signals, to capture good quality signals, the bandwidth of oscilloscope should be >3.75GHz.