Failure in image capture and streaming using v4l2

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin NX
1

Hello, our team has developed a camera driver and device tree file for the IMX327 sensor on an Orin NX (with a custom carrier board). Currently, the camera can successfully capture video using commands such as video-viewer and GStreamer. However, we’ve encountered an issue: whenever we use the v4l2-ctl command to stream or capture RAW format images, the system always hangs without any response. I’m not sure what’s causing this issue.

Could anyone please help clarify the reason behind this behavior? If additional debugging information is needed, please let me know. Thanks, everyone!

nx4@nx4-desktop:~$ v4l2-ctl -d /dev/video0 --stream-count=1 --stream-mmap --stream-to=frame.raw --verbose
VIDIOC_QUERYCAP: ok
		VIDIOC_REQBUFS returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_STREAMON returned 0 (Success)
cap dqbuf: 0 seq:      0 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq:      0 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 1 seq:      1 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 2 seq:      2 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 3 seq:      3 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq:      0 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq:      0 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 1 seq:      1 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 2 seq:      2 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 3 seq:      3 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq:      0 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq:      0 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 1 seq:      1 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 2 seq:      2 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 3 seq:      3 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
^C
nx4@nx4-desktop:~$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=100
^C

Tracing logs captured while streaming:

root@nx4-desktop:/home/nx4# cat /sys/kernel/debug/tracing/trace
# tracer: nop
#
# entries-in-buffer/entries-written: 21/21   #P:6
#
#                                _-----=> irqs-off
#                               / _----=> need-resched
#                              | / _---=> hardirq/softirq
#                              || / _--=> preempt-depth
#                              ||| /     delay
#           TASK-PID     CPU#  ||||   TIMESTAMP  FUNCTION
#              | |         |   ||||      |         |
     kworker/3:2-13525   [003] ....  3318.999692: rtcpu_string: tstamp:104390796804 id:0x04010000 str:"VM0 deactivating."
        v4l2-ctl-16143   [000] ....  3321.546677: tegra_channel_open: vi-output, imx327 30-001a
        v4l2-ctl-16143   [000] ....  3321.552527: tegra_channel_set_power: imx327 30-001a : 0x1
        v4l2-ctl-16143   [000] ....  3321.552536: camera_common_s_power: status : 0x1
        v4l2-ctl-16143   [000] ....  3321.567772: tegra_channel_set_power: 13e40000.host1x:nvcsi@15a00000- : 0x1
        v4l2-ctl-16143   [000] ....  3321.567776: csi_s_power: enable : 0x1
        v4l2-ctl-16143   [002] ....  3321.568326: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1080 fmt c4
        v4l2-ctl-16143   [002] ....  3321.574022: tegra_channel_set_stream: enable : 0x1
        v4l2-ctl-16143   [004] ....  3321.577226: tegra_channel_set_stream: 13e40000.host1x:nvcsi@15a00000- : 0x1
        v4l2-ctl-16143   [004] ....  3321.577229: csi_s_stream: enable : 0x1
        v4l2-ctl-16143   [004] ....  3321.577537: tegra_channel_set_stream: imx327 30-001a : 0x1
     kworker/3:2-13525   [003] ....  3321.599626: rtcpu_string: tstamp:104471177158 id:0x04010000 str:"VM0 activating."
     kworker/3:2-13525   [003] ....  3321.599629: rtcpu_vinotify_event: tstamp:104471818950 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:3343081320416 data:0x759d580010000000
     kworker/3:2-13525   [003] ....  3321.599630: rtcpu_vinotify_event: tstamp:104471819103 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:3343081326880 data:0x0000000031000001
     kworker/3:2-13525   [003] ....  3321.599630: rtcpu_vinotify_event: tstamp:104471819256 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:3343081367648 data:0x759d550010000000
     kworker/3:2-13525   [003] ....  3321.599630: rtcpu_vinotify_event: tstamp:104471819388 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:3343081374176 data:0x0000000031000002
 vi-output, imx3-16145   [001] ....  3324.359929: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1080 fmt c4
     kworker/3:2-13525   [003] ....  3324.387587: rtcpu_vinotify_event: tstamp:104558625445 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:3345867535680 data:0x759d580010000000
     kworker/3:2-13525   [003] ....  3324.387589: rtcpu_vinotify_event: tstamp:104558625580 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:3345867578336 data:0x0000000031000001
     kworker/3:2-13525   [003] ....  3324.387590: rtcpu_vinotify_event: tstamp:104558625733 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:3345867595968 data:0x759d550010000000
     kworker/3:2-13525   [003] ....  3324.387590: rtcpu_vinotify_event: tstamp:104558625864 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:3345867656384 data:0x0000000031000002


nx4@nx4-desktop:~$ v4l2-compliance -d /dev/video0
v4l2-compliance SHA: not available, 64 bits

Compliance test for tegra-video device /dev/video0:

Driver Info:
	Driver name      : tegra-video
	Card type        : vi-output, imx327 30-001a
	Bus info         : platform:tegra-capture-vi:1
	Driver version   : 5.10.120
	Capabilities     : 0x84200001
		Video Capture
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04200001
		Video Capture
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : tegra-camrtc-ca
	Model            : NVIDIA Tegra Video Input Device
	Serial           : 
	Bus info         : 
	Media version    : 5.10.120
	Hardware revision: 0x00000003 (3)
	Driver version   : 5.10.120
Interface Info:
	ID               : 0x0300000e
	Type             : V4L Video
Entity Info:
	ID               : 0x0000000c (12)
	Name             : vi-output, imx327 30-001a
	Function         : V4L2 I/O
	Pad 0x0100000d   : 0: Sink
	  Link 0x02000012: from remote pad 0x1000003 of entity '13e40000.host1x:nvcsi@15a00000-': Data, Enabled

Required ioctls:
	test MC information (see 'Media Driver Info' above): OK
	test VIDIOC_QUERYCAP: OK

Allow for multiple opens:
	test second /dev/video0 open: OK
	test VIDIOC_QUERYCAP: OK
	test VIDIOC_G/S_PRIORITY: OK
	test for unlimited opens: OK

Debug ioctls:
	test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
	test VIDIOC_LOG_STATUS: OK

Input ioctls:
	test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
	test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
	test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
	test VIDIOC_ENUMAUDIO: OK (Not Supported)
	test VIDIOC_G/S/ENUMINPUT: OK
	test VIDIOC_G/S_AUDIO: OK (Not Supported)
	Inputs: 1 Audio Inputs: 0 Tuners: 0

Output ioctls:
	test VIDIOC_G/S_MODULATOR: OK (Not Supported)
	test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
	test VIDIOC_ENUMAUDOUT: OK (Not Supported)
	test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
	test VIDIOC_G/S_AUDOUT: OK (Not Supported)
	Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
	test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
	test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
	test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
	test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls (Input 0):
		warn: v4l2-test-controls.cpp(92): Gain: (max - min) % step != 0
		warn: v4l2-test-controls.cpp(92): Exposure: (max - min) % step != 0
	test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
	test VIDIOC_QUERYCTRL: OK
	test VIDIOC_G/S_CTRL: OK
	test VIDIOC_G/S/TRY_EXT_CTRLS: OK
	test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
	test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
	Standard Controls: 1 Private Controls: 20

Format ioctls (Input 0):
	test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
		fail: v4l2-test-formats.cpp(1280): ret && node->has_frmintervals
	test VIDIOC_G/S_PARM: FAIL
	test VIDIOC_G_FBUF: OK (Not Supported)
	test VIDIOC_G_FMT: OK
	test VIDIOC_TRY_FMT: OK
	test VIDIOC_S_FMT: OK
	test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
	test Cropping: OK (Not Supported)
	test Composing: OK (Not Supported)
	test Scaling: OK (Not Supported)

Codec ioctls (Input 0):
	test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
	test VIDIOC_G_ENC_INDEX: OK (Not Supported)
	test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls (Input 0):
		fail: v4l2-test-buffers.cpp(715): q.create_bufs(node, 1, &fmt) != EINVAL
	test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: FAIL
	test VIDIOC_EXPBUF: OK
	test Requests: OK (Not Supported)

Total for tegra-video device /dev/video0: 45, Succeeded: 43, Failed: 2, Warnings: 2
3

Here is the kernel log information when running the v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=100 command.

[  172.635576] bwmgr API not supported
[  172.636573] imx327_set_mode
[  172.640194] imx327_write_table
[  172.858786] imx327_start_streaming
[  172.858792] imx327_write_table
[  172.889739] +++++++++++++++++++++start_streaming+++++++++++++++++++++++
[  175.335762] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  175.345031] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  175.355937] (NULL device *): vi_capture_control_message: NULL VI channel received
[  175.363716] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=0, csi_port=0
[  175.374515] (NULL device *): vi_capture_control_message: NULL VI channel received
[  175.382335] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 0 vc- 0
[  175.393303] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  178.151690] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  178.160855] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  178.170907] (NULL device *): vi_capture_control_message: NULL VI channel received
[  178.178634] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=0, csi_port=0
[  178.189293] (NULL device *): vi_capture_control_message: NULL VI channel received
[  178.197016] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 0 vc- 0
[  178.207815] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  180.971607] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  180.980810] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  180.990931] (NULL device *): vi_capture_control_message: NULL VI channel received
[  180.998651] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=0, csi_port=0
[  181.009320] (NULL device *): vi_capture_control_message: NULL VI channel received
[  181.017037] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 0 vc- 0
[  181.027841] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
4

Look like didn’t receive any data from the trace log.
Suppose it could be incomplete sensor REG configuration cause the problem.

5

I’m not very familiar with this, but is it possible that an incomplete register configuration is causing v4l2-ctl to fail when capturing RAW images, while still allowing GStreamer to capture video successfully? Currently, the camera functions normally when using video-viewer to acquire video images.

Additionally, can GStreamer pipeline commands capture RAW format image output?

6

It’s working by gst-launch-1.0 nvarguscamerasrc … command?
You can use nvargus_nvraw for raw capture alternative.

Thanks

7

1)Thank you for your response. Yes, the following commands can all successfully capture video images. Additionally, we have previously confirmed that the nvargus_nvraw command can indeed capture RAW format images for imx327.

nvgstcapture-1.0 --sensor-id=0
gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! nvegltransform ! nveglglessink
gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=(int)1920,height=(int)1080,framerate=(fraction)30/1,format=(string)NV12' ! nvvidconv ! 'video/x-raw,width=(int)1920,height=(int)1080,format=(string)I420' ! nv3dsink sync=false

2)Our team is currently developing a new ToF camera with an MIPI CSI-2 interface on the Orin NX. We have completed the initial setup of the device tree and part of the driver. At this stage, we have confirmed that the camera registers as a V4L2 sub-device and that I2C communication is functioning correctly. We are attempting to modify the registers to enable the camera to enter streaming mode.

Now, we want to verify whether there is an actual data stream flowing through the MIPI channel. Initially, I planned to use the v4l2-ctl command for this verification, but it also hangs and blocks execution. I recall facing a similar issue when developing the IMX327 driver, which is why I created this post.

Currently, I am unsure how to check if data is being transmitted through the MIPI channel. Do you have any suggestions or insights regarding this?

8

Review the port-index/tegra_sinterface/num_lanes/bus-width then,
Maybe make the control function(CID) in sensor driver to dummy function to make sure the REG configuration the same both for v4l2-ctl with argus.

9

Here is the relevant portion of the device tree for the ToF camera. These node properties should be correct, right? I am using a 4-lane ToF camera, which is connected to the third interface (J21) on the custom carrier board.

图片
图片980×522 31.9 KB

		i2c_cam2:i2c@2 {
			reg = <2>;
			i2c-mux,deselect-on-exit; 
			#address-cells = <1>;
			#size-cells = <0>;
			status = "okay";
			mlx75027_cam: mlx75027@57 {
				compatible = "melexis,mlx75027";  
				/* I2C device address */
				reg = <0x57>;  

				/* V4L2 device node location */
				devnode = "video2";  

				/* Physical dimensions of sensor */
				physical_w = "5.6";
				physical_h = "4.8";

				/* Define any required hw resources needed by driver */
				/* ie. clocks, io pins, power sources */
				//avdd-reg = "vana";
				// iovdd-reg = "vif";
				/*dvdd-reg = "vdig";*/
				sensor_model ="mlx75027";  
				clocks = <0x02 0x24>;  
				clock-names = "extperiph1";
				mclk = "extperiph1";
			
				/* Defines number of frames to be dropped by driver internally after applying */
				/* sensor crop settings. Some sensors send corrupt frames after applying */
				/* crop co-ordinates */
				// post_crop_frame_drop = "10";  //useless
				// set_mode_delay_ms ="500";  //useless

				/* Convert Gain to unit of dB (decibel) befor passing to kernel driver */
				// use_decibel_gain = "true";

				/* if true, delay gain setting by one frame to be in sync with exposure */
				// delayed_gain = "true";

				/* enable CID_SENSOR_MODE_ID for sensor modes selection */
				use_sensor_mode_id = "true";  

				/* WAR to prevent banding by reducing analog gain. Bug 2229902 */
				// limit_analog_gain = "true";

				reset-gpios = <&tca9534_38 2 0>;  
				status = "okay";

				mode0 {
					mclk_khz = "8000"; 
					num_lanes = "4";
					tegra_sinterface = "serial_c";
					phy_mode = "DPHY";
					discontinuous_clk = "no";  //must be "no"
					dpcm_enable = "false";
					cil_settletime = "0";
					lane_polarity = "0";   /*Orin NX Only*/

					active_w = "640";
					active_h = "480";
					mode_type = "bayer";
					pixel_phase = "rggb";
					csi_pixel_bit_depth = "12";  //bit per pixel
					dynamic_pixel_bit_depth = "12";

					readout_orientation = "0";
					line_length = "640";     //Total width
					inherent_gain = "1";
					// mclk_multiplier = "4";  //pix_clk_hz/mclk_khz
					pix_clk_hz = "80000000";  

					gain_factor = "1";
					min_gain_val = "3"; /* 0.3dB */
					max_gain_val = "180"; /* 20dB */
					step_gain_val = "3"; /* 0.3dB */
					default_gain = "16";
					min_hdr_ratio = "1";
					max_hdr_ratio = "1";
					
					framerate_factor = "1000000";
					min_framerate = "1500000"; /* 1.5 */
					max_framerate = "60000000"; /* 50 */
					step_framerate = "1";
					default_framerate= "60000000";

					exposure_factor = "1000000";
					min_exp_time = "18"; /* /us */
					max_exp_time = "20000"; /* /us */
					step_exp_time = "18";
					default_exp_time = "2000";/* /us */
					
					embedded_metadata_height = "0";
				};
				
				ports {
					#address-cells = <1>;
					#size-cells = <0>;

					port@0 {
						reg = <0>;
						mlx75027_out2: endpoint {
							port-index = <2>;
							bus-width = <4>;
							remote-endpoint = <&mlx75027_csi_in2>;
						};
					};
				};
			};
		};
	/* nvcsi@15a00000 setting */
	fragment@15 {
		target = <&nvcsi>;
		__overlay__ {
			num-channels = <0x03>;
			channel@2{
				reg = <2>;
				status = "okay";
				ports {
					#address-cells = <1>;
					#size-cells = <0>;
					port@0 {
						reg = <0>;
						status = "okay";
						mlx75027_csi_in2: endpoint@4 {
							port-index = <2>;
							bus-width = <4>;
							status = "okay";
							remote-endpoint = <&mlx75027_out2>;
						};
					};

					port@1 {
						reg = <1>;
						mlx75027_csi_out2: endpoint@5 {
							status = "okay";
							remote-endpoint = <&mlx75027_vi_in2>;
						};
					};
				};
			};
		};
	};

	/* tegra-capture-vi setting */
	fragment@16 {
		target-path = "/";
		__overlay__ {
			tegra-capture-vi {
				status = "okay";
				num-channels = <3>;
				ports{
					port@2{
						#address-cells = <0x01>;
						#size-cells = <0x00>;
						reg = <2>;
						status = "okay";
						mlx75027_vi_in2:endpoint{
							port-index = <2>;
							bus-width = <4>;
							status = "okay";
							remote-endpoint = <&mlx75027_csi_out2>;
						};
					};
				};
			};
		};
	};

	/* tegra-camera-platform setting */
	fragment@17 {
		target = <&tcp>;
		__overlay__ {
			num_csi_lanes = <8>;
			modules {
				cam_module2: module2 {
					badge = "mlx75027_topright_liimx185";
					position = "topright";
					orientation = "1";
					status = "okay";
					cam_module2_drivernode0: drivernode0 {
						status = "okay";
						pcl_id = "v4l2_sensor";
						devname = "mlx75027 32-0057";
						proc-device-tree = "/proc/device-tree/i2c@3180000/tca9548@77/i2c@2/mlx75027@57";
					};
				};
			};
		};
	};

We have written a fairly complete register table and successfully written it into the sensor via the driver. I also used the driver’s read function to verify that the register values were indeed written correctly. By modifying the mode registers, we have set the sensor to video streaming mode. Since the driver is not yet fully implemented, it may not be outputting valid data, but at this stage, I am more concerned about whether there is any MIPI data stream being transmitted.

Do you know how to check if the MIPI channel is outputting data on the Orin NX?

图片
图片689×122 14.6 KB

10

You may need to probe by scope.

Thanks