gstreamer nvvidconv not working with video/x-raw caps on both sides

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

I have an USB camera that I access using v4l2src. The format of the raw camera pixels is I420 and I want to convert it with nvvidconv to BGRx to further process it in opencv.

Nvvidconv seems to work fine as long as the sink or source side uses the ‘video/x-raw(memory:NVMM)’ capabilitiy. If both the sink and source sides use the ‘video/x-raw’ capability, I get an internal data flow error. To illustrate:

The following works where I put a video encoder on the source side of nvvidconv to be able to use ‘video/x-raw(memory:NVMM)’ on the source side and ‘video/x-raw’ on the sink side:

gst-launch-1.0 v4l2src device="/dev/video1" ! 'video/x-raw, format=(string)I420' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)NV12' ! omxh264enc ! video/x-h264, stream-format=byte-stream ! rtph264pay ! udpsink host=10.66.24.128

The following also works where I replace the v4l2src with nvcamerasrc (CSI camera i.s.o USB camera) so the sink side is ‘video/x-raw(memory:NVMM)’ and the source side is ‘video/x-raw’:

gst-launch-1.0 -v nvcamerasrc ! video/x-raw(memory:NVMM), format=I420 ! nvvidconv ! 'video/x-raw, format=(string)BGRx' ! ximagesink

However if I want to use the v4l2src on the input and get a BGRx video stream on the output:

gst-launch-1.0 -v v4l2src device="/dev/video1" ! 'video/x-raw, format=(string)I420' ! nvvidconv ! 'video/x-raw, format=(string)BGRx' ! ximagesink

I get the following error:

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)1280\,\ height\=\(int\)1024\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)15/2"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)1280\,\ height\=\(int\)1024\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)15/2"
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = "video/x-raw\,\ width\=\(int\)1280\,\ height\=\(int\)1024\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)15/2\,\ format\=\(string\)BGRx"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = "video/x-raw\,\ width\=\(int\)1280\,\ height\=\(int\)1024\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)15/2\,\ format\=\(string\)BGRx"
/GstPipeline:pipeline0/GstXImageSink:ximagesink0.GstPad:sink: caps = "video/x-raw\,\ width\=\(int\)1280\,\ height\=\(int\)1024\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)15/2\,\ format\=\(string\)BGRx"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = "video/x-raw\,\ width\=\(int\)1280\,\ height\=\(int\)1024\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)15/2\,\ format\=\(string\)BGRx"
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)1280\,\ height\=\(int\)1024\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)15/2"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)1280\,\ height\=\(int\)1024\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive\,\ framerate\=\(fraction\)15/2"
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data flow error.
Additional debug info:
gstbasesrc.c(2948): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming task paused, reason error (-5)
Execution ended after 0:00:01.495050436
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

I was able to work around it using 2 hardware accelerated conversions but that’s just a waste of resources:

gst-launch-1.0 -v v4l2src device="/dev/video1" ! 'video/x-raw, format=(string)I420' ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvvidconv ! 'video/x-raw, format=(string)BGRx' ! ximagesink

Is it possible to fix this or is there a better workaround than using two catenated hardware conversions?

Thanks in advance

2

nvvidconv requiring one of its input or output to be in NVMM memory is usual.
If you want to do the conversion with hw acceleration, you need to get contiguous memory (this is mainly what NVMM memory is, AFAIK) and copy your frames there. This is what does the first nvvidconv in your pipeline. The second one can then do the conversion and write its output into standard memory. Not sure there is such a waste of resources. This datapath looks correct to me for HW conversion. You will just have one more gstreamer thread, but a thread having nothing to do would just cost a few bytes of RAM.

Alternatives are mostly on CPU side and depend on final format expected in your opencv application.
If it ends up you need BGR, you may use videoconvert plugin or make the conversion with opencv (recent versions of opencv support capturing some video formats more than usual BGR). This will result in CPU load, but may be ok depending on what your app requires, if you can enable all cores and boost clocks, and if you keep low resolution and framerate.

There is also a subset of opencv conversions available with GPU, but here again you would have to allocate some GPU memory and copy. Not sure opencv capture is able to read frame directly into a pre-allocated buffer in Unified or pinned memory (I haven’t be able to do this, if someone knows how to do this, I’d be happy to learn). In such case you will have to copy your frames into such buffer before calling the GPU convert functions, so at the end it may be less efficient than your gstreamer pipeline.

3

Please also refer to [url]https://devtalk.nvidia.com/default/topic/1012417/jetson-tx1/tx1-gstreamer-nvvidconv-will-not-pass-out-of-nvmm-memory/post/5162187/#5162187[/url]

Thank HoneyP.