GStreamer Works, v4l2 Returns Empty File

goodarzm · January 28, 2021, 6:11am

We have the following setup:

2x Leopard Imaging LI-IMX390-GMSL2 cameras
ConnectTech Xavier GMSL Camera Platform (8xGMSL2 to CSI)
Jetson AGX Xavier DevKit running JetPack 4.4.1 (L4T r32.4.4 along with the relevant ConnectTech BSP)

We have been able to set up a GStreamer pipeline to view the camera feed using the following command:

gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1' ! nvoverlaysink -e

However, when we have tried to do the same thing with v4l2 we have been less lucky. More specifically, when running

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat='RG12' --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 --stream-to=test.raw --verbose

the test.raw file remains empty and we get the following output

VIDIOC_QUERYCAP: ok
VIDIOC_S_EXT_CTRLS: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: ok
Format Video Capture:
Width/Height      : 1920/1080
Pixel Format      : 'RG12'
Field             : None
Bytes per Line    : 3840
Size Image        : 4147200
Colorspace        : sRGB
Transfer Function : Default (maps to sRGB)
YCbCr/HSV Encoding: Default (maps to ITU-R 601)
Quantization      : Default (maps to Full Range)
Flags             : 
VIDIOC_REQBUFS: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_STREAMON: ok
Index    : 0
Type     : Video Capture
Flags    : mapped, error
Field    : None
Sequence : 0
Length   : 4147200
Bytesused: 4147200
Timestamp: 0.000000s (Monotonic, End-of-Frame)

Index    : 0
Type     : Video Capture
Flags    : mapped, error
Field    : None
Sequence : 0
Length   : 4147200
Bytesused: 4147200
Timestamp: 0.000000s (Monotonic, End-of-Frame)

Index    : 1
Type     : Video Capture
Flags    : mapped, error
Field    : None
Sequence : 1
Length   : 4147200
Bytesused: 4147200
Timestamp: 0.000000s (Monotonic, End-of-Frame)

Index    : 2
Type     : Video Capture
Flags    : mapped, error
Field    : None
Sequence : 2
Length   : 4147200
Bytesused: 4147200
Timestamp: 0.000000s (Monotonic, End-of-Frame)

Index    : 3
Type     : Video Capture
Flags    : mapped, error
Field    : None
Sequence : 3
Length   : 4147200
Bytesused: 4147200
Timestamp: 0.000000s (Monotonic, End-of-Frame)

Index    : 0
Type     : Video Capture
Flags    : mapped, error
Field    : None
Sequence : 0
Length   : 4147200
Bytesused: 4147200
Timestamp: 0.000000s (Monotonic, End-of-Frame)

Index    : 0
Type     : Video Capture
Flags    : mapped, error
Field    : None
Sequence : 0
Length   : 4147200
Bytesused: 4147200
Timestamp: 0.000000s (Monotonic, End-of-Frame)

Index    : 1
Type     : Video Capture
Flags    : mapped, error
Field    : None
Sequence : 1
Length   : 4147200
Bytesused: 4147200
Timestamp: 0.000000s (Monotonic, End-of-Frame)

Index    : 2
Type     : Video Capture
Flags    : mapped, error
Field    : None
Sequence : 2
Length   : 4147200
Bytesused: 4147200
Timestamp: 0.000000s (Monotonic, End-of-Frame)

Index    : 3
Type     : Video Capture
Flags    : mapped, error
Field    : None
Sequence : 3
Length   : 4147200
Bytesused: 4147200
Timestamp: 0.000000s (Monotonic, End-of-Frame)

The camera and driver information are as follows:

Driver Info (not using libv4l2):
Driver name   : tegra-video
Card type     : vi-output, imx390 30-001b
Bus info      : platform:15c10000.vi:6
Driver version: 4.9.140
Capabilities  : 0x84200001
	Video Capture
	Streaming
	Extended Pix Format
	Device Capabilities
Device Caps   : 0x04200001
	Video Capture
	Streaming
	Extended Pix Format
Priority: 2
Video input : 0 (Camera 6: ok)
Format Video Capture:
Width/Height      : 1920/1080
Pixel Format      : 'RG12'
Field             : None
Bytes per Line    : 3840
Size Image        : 4147200
Colorspace        : sRGB
Transfer Function : Default (maps to sRGB)
YCbCr/HSV Encoding: Default (maps to ITU-R 601)
Quantization      : Default (maps to Full Range)
Flags             : 

Camera Controls

                 group_hold 0x009a2003 (bool)   : default=0 value=0 flags=execute-on-write
                 hdr_enable 0x009a2004 (intmenu): min=0 max=1 default=0 value=0
                       gain 0x009a2009 (int64)  : min=0 max=300 step=3 default=0 value=0 flags=slider
                   exposure 0x009a200a (int64)  : min=59 max=33333 step=1 default=33333 value=59 flags=slider
                 frame_rate 0x009a200b (int64)  : min=30000000 max=30000000 step=1 default=30000000 value=30000000 flags=slider
             exposure_short 0x009a200c (int64)  : min=59 max=33333 step=1 default=33333 value=59 flags=slider
                bypass_mode 0x009a2064 (intmenu): min=0 max=1 default=0 value=0
            override_enable 0x009a2065 (intmenu): min=0 max=1 default=0 value=0
               height_align 0x009a2066 (int)    : min=1 max=16 step=1 default=1 value=1
                 size_align 0x009a2067 (intmenu): min=0 max=2 default=0 value=0
           write_isp_format 0x009a2068 (int)    : min=1 max=1 step=1 default=1 value=1
   sensor_signal_properties 0x009a2069 (u32)    : min=0 max=4294967295 step=1 default=0 [30][18] flags=read-only, has-payload
    sensor_image_properties 0x009a206a (u32)    : min=0 max=4294967295 step=1 default=0 [30][16] flags=read-only, has-payload
  sensor_control_properties 0x009a206b (u32)    : min=0 max=4294967295 step=1 default=0 [30][36] flags=read-only, has-payload
          sensor_dv_timings 0x009a206c (u32)    : min=0 max=4294967295 step=1 default=0 [30][16] flags=read-only, has-payload
           low_latency_mode 0x009a206d (bool)   : default=0 value=0
           preferred_stride 0x009a206e (int)    : min=0 max=65535 step=1 default=0 value=0
               sensor_modes 0x009a2082 (int)    : min=0 max=30 step=1 default=30 value=1 flags=read-only

I’d appreciate any help letting us know why v4l2 is not working and what should be done to fix it.

On a side note, v4l2 shows that the camera has an “hdr_enable” control. Is it possible to somehow enable that through the GStreamer pipeline? I’m really keen on testing out the camera’s HDR capabilities.

ShaneCCC · January 28, 2021, 7:54am

Try if --stream-count=10 may be the start of few frame not ready.

goodarzm · January 28, 2021, 4:03pm

I’ve tried that, but it makes no difference. The test.raw file is still empty.

ShaneCCC · January 28, 2021, 6:19pm

@SimonZhu
Could you help to check if need driver update for this device.

Thanks

goodarzm · January 28, 2021, 11:14pm

Update: when I run v4l2-compliance I get:

v4l2-compliance SHA   : not available

Driver Info:
	Driver name   : tegra-video
	Card type     : vi-output, imx390 30-001b
	Bus info      : platform:15c10000.vi:6
	Driver version: 4.9.140
	Capabilities  : 0x84200001
		Video Capture
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps   : 0x04200001
		Video Capture
		Streaming
		Extended Pix Format

Compliance test for device /dev/video0 (not using libv4l2):

Required ioctls:
	test VIDIOC_QUERYCAP: OK

Allow for multiple opens:
	test second video open: OK
	test VIDIOC_QUERYCAP: OK
	test VIDIOC_G/S_PRIORITY: OK
	test for unlimited opens: OK

Debug ioctls:
	test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
	test VIDIOC_LOG_STATUS: OK

Input ioctls:
	test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
	test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
	test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
	test VIDIOC_ENUMAUDIO: OK (Not Supported)
	test VIDIOC_G/S/ENUMINPUT: OK
	test VIDIOC_G/S_AUDIO: OK (Not Supported)
	Inputs: 1 Audio Inputs: 0 Tuners: 0

Output ioctls:
	test VIDIOC_G/S_MODULATOR: OK (Not Supported)
	test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
	test VIDIOC_ENUMAUDOUT: OK (Not Supported)
	test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
	test VIDIOC_G/S_AUDOUT: OK (Not Supported)
	Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
	test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
	test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
	test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
	test VIDIOC_G/S_EDID: OK (Not Supported)

Test input 0:

	Control ioctls:
		test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
		test VIDIOC_QUERYCTRL: OK
		test VIDIOC_G/S_CTRL: OK
		test VIDIOC_G/S/TRY_EXT_CTRLS: OK
		test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
		test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
		Standard Controls: 1 Private Controls: 18

	Format ioctls:
		test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
		fail: v4l2-test-formats.cpp(1184): ret && node->has_frmintervals
		test VIDIOC_G/S_PARM: FAIL
		test VIDIOC_G_FBUF: OK (Not Supported)
		test VIDIOC_G_FMT: OK
		test VIDIOC_TRY_FMT: OK
		test VIDIOC_S_FMT: OK
		test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
		test Cropping: OK (Not Supported)
		test Composing: OK (Not Supported)
		test Scaling: OK (Not Supported)

	Codec ioctls:
		test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
		test VIDIOC_G_ENC_INDEX: OK (Not Supported)
		test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

	Buffer ioctls:
		test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
		test VIDIOC_EXPBUF: OK

Test input 0:


Total: 43, Succeeded: 42, Failed: 1, Warnings: 0

Given that VIDIOC_G/S_PARM test fails and from what I can gather from here, the problem seems to be with the frame rate defined in the driver. However, when I run v4l2-ctl -d /dev/video1 --list-formats-ext I get

ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'RG12'
	Name        : 12-bit Bayer RGRG/GBGB
		Size: Discrete 1920x1080
			Interval: Discrete 0.033s (30.000 fps)

which strangely is the correct frame rate.

Finally, running the v4l2 pipeline above (writing to test.raw) returns the following on dmesg:

[Jan28 18:18] imx390 30-001c: imx390_power_on: power on
[  +0.003847] imx390 30-001c: imx390_set_gain: db: 30
[  +0.016014] tegra194-vi5 15c10000.vi: Attempting retry 4
[  +0.055654] [RCE] vi5_hwinit: firmware CL2018101701 protocol version 2.2
[  +0.221771] imx390 30-001c: imx390_set_gain: db: 30
[  +2.454085] tegra194-vi5 15c10000.vi: no reply from camera processor
[  +0.000186] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms
[  +0.000195] tegra194-vi5 15c10000.vi: err_rec: attempting to reset the capture channel
[  +0.002536] tegra194-vi5 15c10000.vi: Attempting retry 4
[  +0.000016] tegra194-vi5 15c10000.vi: err_rec: successfully reset the capture channel
[  +2.556985] tegra194-vi5 15c10000.vi: no reply from camera processor
[  +0.000194] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms
[  +0.000181] tegra194-vi5 15c10000.vi: err_rec: attempting to reset the capture channel
[  +0.002396] tegra194-vi5 15c10000.vi: Attempting retry 4
[  +0.000016] tegra194-vi5 15c10000.vi: err_rec: successfully reset the capture channel
[  +2.557154] tegra194-vi5 15c10000.vi: no reply from camera processor
[  +0.000154] tegra194-vi5 15c10000.vi: uncorr_err: request timed out after 2500 ms
[  +0.000156] tegra194-vi5 15c10000.vi: err_rec: attempting to reset the capture channel

Any help figuring this out would be appreciated.

SimonZhu · January 29, 2021, 7:16am

Our current IMX390-GMSL2 driver for Nvidia Jetson AGX Xavier Devkit is based on R32.4.3 (Jetpack 4.4).

goodarzm · January 29, 2021, 7:58am

So, should I downgrade from r32.4.4 to 32.4.3?

SimonZhu · January 29, 2021, 12:19pm

Hi goodarzm,
Yes, our driver can work on R32.4.3 directly, or you can apply our R32.4.3 driver patches to R32.4.4 kernel code and do minor modifications. It may work too.

goodarzm · January 30, 2021, 2:59am

Thanks for helping me with this. Could you please also let me know if there is any documentation available for the driver? I am asking this because the driver info above shows that the camera has a lot of controls (e.g. hdr_enable) that I want to understand more in order to use them.

SimonZhu · January 30, 2021, 3:47am

Hi goodarzm,
There are many Jetson documents on Jetson Download Center. Please check it.

goodarzm · February 2, 2021, 1:58am

Hi @SimonZhu ,

I downgraded to L4T r32.4.3 but the problem still persists. Essentially nothing seems to have changed. I still get an empty test.raw file with v4l2, and v4l2-compliance still shows the same error. Any suggestions on how I should proceed?

SimonZhu · February 3, 2021, 3:44am

Hi goodarzm,
Could you take a picture about the whole camera kit? Are you using our deserializer board and driver?
Based on our record, we can capture raw data with below command.
v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=RG12 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=3 --stream-to=imx390.raw -d /dev/video0

goodarzm · February 3, 2021, 4:23am

I am using the setup mentioned in the original post (with L4T downgraded to r32.4.3), i.e.

2x Leopard Imaging LI-IMX390-GMSL2 cameras
ConnectTech Xavier GMSL Camera Platform (8xGMSL2 to CSI)
Jetson AGX Xavier DevKit running JetPack 4.4 (L4T r32.4.3 along with the relevant ConnectTech BSP)

As mentioned there the GStreamer pipeline using libargus works, but when I try to get the raw frames using v4l2 I get an empty file.

SimonZhu · February 3, 2021, 4:44am

Hi goodarzm,
Thanks. Our driver was built for Nvidia AGX Devkit with our MIPI and Deserializer board, and it may not work with ConnectTech carrier board. If your driver was from ConnectTech, please check with them about this issue.

goodarzm · February 3, 2021, 5:00am

Thanks for letting me know. I will reach out to them.