We are testing with two cameras, both do not support USB3 and negotiate speed to 480Mbit/s.
Here’re the capabilities as reported by v4l2-ctl:
albertr@tx2:~$ sudo v4l2-ctl --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
Index : 0
Type : Video Capture
Pixel Format: 'H264' (compressed)
Name : H.264
Size: Discrete 1920x1440
Interval: Discrete 0.033s (30.000 fps)
Size: Discrete 1280x720
Interval: Discrete 0.033s (30.000 fps)
Size: Discrete 640x480
Interval: Discrete 0.033s (30.000 fps)
Size: Discrete 320x240
Interval: Discrete 0.033s (30.000 fps)
Index : 1
Type : Video Capture
Pixel Format: 'MJPG' (compressed)
Name : Motion-JPEG
Size: Discrete 1920x1440
Interval: Discrete 0.033s (30.000 fps)
Size: Discrete 1280x720
Interval: Discrete 0.033s (30.000 fps)
Size: Discrete 640x480
Interval: Discrete 0.033s (30.000 fps)
Size: Discrete 320x240
Interval: Discrete 0.033s (30.000 fps)
albertr@tx2:~$
Quite interesting is that if I compile gst-plugins-good-1.8.0 without linking to libv4l library, then it negotiate format to I420!
I’m trying to figure out how I can force the same when compiling without linking to libv4l library. Below are some snippets from negotiation sequence when it’s using libv4l library.
0:00:04.028593041 3548 0x7fec50 DEBUG v4l2 gstv4l2object.c:1106:gst_v4l2_ob
ject_fill_format_list:<v4l2src0> getting src format enumerations
0:00:04.028659025 3548 0x7fec50 INFO v4l2 gstv4l2object.c:1143:gst_v4l2_ob
ject_fill_format_list:<v4l2src0> got 6 format(s):
0:00:04.028674097 3548 0x7fec50 INFO v4l2 gstv4l2object.c:1149:gst_v4l2_ob
ject_fill_format_list:<v4l2src0> MJPG
0:00:04.028699185 3548 0x7fec50 INFO v4l2 gstv4l2object.c:1149:gst_v4l2_ob
ject_fill_format_list:<v4l2src0> YU12 (emulated)
0:00:04.028721649 3548 0x7fec50 INFO v4l2 gstv4l2object.c:1149:gst_v4l2_ob
ject_fill_format_list:<v4l2src0> YV12 (emulated)
0:00:04.028737393 3548 0x7fec50 INFO v4l2 gstv4l2object.c:1149:gst_v4l2_ob
ject_fill_format_list:<v4l2src0> BGR3 (emulated)
0:00:04.028750705 3548 0x7fec50 INFO v4l2 gstv4l2object.c:1149:gst_v4l2_ob
ject_fill_format_list:<v4l2src0> RGB3 (emulated)
0:00:04.028781329 3548 0x7fec50 INFO v4l2 gstv4l2object.c:1149:gst_v4l2_ob
ject_fill_format_list:<v4l2src0> H264
0:00:04.122161706 3548 0x7fec50 DEBUG v4l2 gstv4l2object.c:3812:gst_v4l2_object_get_caps:<v4l2src0> ret: image/jpeg, width=(int)1920, height=(int)1440, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1; image/jpeg, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1; image/jpeg, width=(int)640, height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1; image/jpeg, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1; video/x-raw, format=(string)I420, width=(int)1920, height=(int)1440, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)30/1; video/x-raw, format=(string)I420, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)30/1; video/x-raw, format=(string)I420, width=(int)640, height=(int)480, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)30/1; video/x-raw, format=(string)I420, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)30/1; video/x-raw, format=(string)YV12, width=(int)1920, height=(int)1440, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)30/1; video/x-raw, format=(string)YV12, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)30/1; video/x-raw, format=(string)YV12, width=(int)640, height=(int)480, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)30/1; video/x-raw, format=(string)YV12, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)30/1; video/x-raw, format=(string)BGR, width=(int)1920, height=(int)1440, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)30/1; video/x-raw, format=(string)BGR, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)30/1; video/x-raw, format=(string)BGR, width=(int)640, height=(int)480, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)30/1; video/x-raw, format=(string)BGR, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)30/1; video/x-raw, format=(string)RGB, width=(int)1920, height=(int)1440, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)30/1; video/x-raw, format=(string)RGB, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)30/1; video/x-raw, format=(string)RGB, width=(int)640, height=(int)480, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)30/1; video/x-raw, format=(string)RGB, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)30/1; video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, width=(int)1920, height=(int)1440, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1; video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1; video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, width=(int)640, height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1; video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1
0:00:04.131981456 3548 0x7fec50 DEBUG v4l2src gstv4l2src.c:385:gst_v4l2src_negotiate:<v4l2src0> caps: video/x-raw, format=(string)I420, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)30/1
0:00:04.132222896 3548 0x7fec50 DEBUG video-info video-info.c:300:gst_video_info_from_caps: parsing caps video/x-raw, format=(string)I420, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)30/1
0:00:04.132384304 3548 0x7fec50 DEBUG v4l2 gstv4l2object.c:3078:gst_v4l2_object_set_format_full:<v4l2src0> progressive video
0:00:04.132414320 3548 0x7fec50 DEBUG v4l2 gstv4l2object.c:3212:gst_v4l2_object_set_format_full:<v4l2src0> Desired format 1280x720, format YU12 stride: 1280
0:00:04.132476656 3548 0x7fec50 DEBUG v4l2 gstv4l2object.c:3263:gst_v4l2_object_set_format_full:<v4l2src0> Desired format is 1280x720, format YU12, nb planes 1
0:00:04.132531344 3548 0x7fec50 DEBUG v4l2 gstv4l2object.c:3272:gst_v4l2_object_set_format_full:<v4l2src0> stride 1280
0:00:04.139354708 3548 0x7fec50 DEBUG v4l2 gstv4l2object.c:3306:gst_v4l2_object_set_format_full:<v4l2src0> Got format of 1280x720, format YU12, nb planes 1, colorspace 0
0:00:04.139435220 3548 0x7fec50 DEBUG v4l2 gstv4l2object.c:3316:gst_v4l2_object_set_format_full:<v4l2src0> stride 1280, sizeimage 1382400
0:00:04.139472532 3548 0x7fec50 DEBUG v4l2 gstv4l2object.c:3357:gst_v4l2_object_set_format_full:<v4l2src0> Desired framerate: 30/1
0:00:04.139511924 3548 0x7fec50 DEBUG v4l2 gstv4l2object.c:3373:gst_v4l2_object_set_format_full:<v4l2src0> Got framerate: 30/1
0:00:04.152201724 3548 0x7fec50 INFO v4l2 gstv4l2object.c:3405:gst_v4l2_object_set_format_full:<v4l2src0> Set framerate to 30/1
0:00:04.152316028 3548 0x7fec50 DEBUG v4l2 gstv4l2object.c:2974:gst_v4l2_object_save_format:<v4l2src0> Got sizeimage 1382400
0:00:04.153337885 3548 0x7fec50 DEBUG v4l2bufferpool gstv4l2bufferpool.c:503:gst_v4l2_buffer_pool_set_config:<v4l2src0:pool:src> config GstBufferPoolConfig, caps=(GstCaps)"video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)1280\,\ height\=\(int\)720\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)30/1", size=(uint)1382400, min-buffers=(uint)0, max-buffers=(uint)0, allocator=(GstAllocator)"NULL", params=(GstAllocationParams)NULL;
0:00:04.153705757 3548 0x7fec50 DEBUG video-info video-info.c:300:gst_video_info_from_caps: parsing caps video/x-raw, format=(string)I420, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)30/1
It works very well, but CPU utilization of the core where gstreamer run on is fluctuating around 55% when using the following pipeline:
$PREFIX/bin/gst-launch-1.0 -vvv v4l2src device=/dev/video0 ! 'video/x-raw, width=1280, height=720, framerate=30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=1280, height=720, format=I420, framerate=30/1' ! nvoverlaysink -e -vvv --gst-debug-no-color=1
I’m trying to see if using userptr buffers allocation can help to reduce CPU load and latency.
Let me know if you have any ideas.
BTW, I’ve also tried to use the following pipeline and while technically it “works”, the CPU load shoot over the roof and I’m getting missed frames and choppy video:
$PREFIX/bin/gst-launch-1.0 -vvv v4l2src device=/dev/video0 ! 'image/jpeg,width=1280,height=720' ! nvjpegdec ! nvvidconv ! 'video/x-raw(memory:NVMM), width=1280, height=720, format=I420, framerate=30/1' ! nvoverlaysink -e -vvv --gst-debug-no-color=1
From what I understand I420 is the same as YU12, but with reversed order of Y and U planes. Both are 12bit per pixel RAW formats and 30 fps @ 720p would be about 320Mbit/s. So how it can be negotiated by libv4l library if v4l2-ctl reports no RAW formats supported by this camera?
-albertr