Gstreamer pipeline using Opencv

Autonomous Machines Jetson & Embedded Systems Jetson TX1
#1

Hi

On a project, I need to get through a gstreamer pipeline the feed from a 4k camera and send it to the appsink for further processing. We get the feed in I420 format then we need to convert it to BGR in order to get it to the appsink, but I ran into several issues. My first command was the following :

cv::VideoCapture("v4l2src device=/dev/video1 ! video/x-raw, width=(int)3840, height=(int)2160,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw(memory:NVMM), format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink")

debug says videoconvert0 cannot link to appsink0, but I don’t understand why the BGR is not supported as it should be. I tried several other combinations to no avail.

Maybe videoconvert is failing somewhere, in addition I would rather have everything handled by the GPU (I think videoconvert is CPU processing) so I heard about nvivafilter but it only supports nv12 and RGBA. Is it possible to add the BGR format to the cuda process (using nvsample_cudaprocess)?

#2

You may try to feed videoconvert from cpu memory, I don’t think it supports NVMM memory.

Anyway, videoconvert is slow and for 4K it will probably be very slow.

Using nvivafilter would give more chances to get it. You could use it with such a pipeline:

gst-launch-1.0 v4l2src device=/dev/video1 ! 'video/x-raw, width=(int)3840, height=(int)2160,format=(string)I420, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)RGBA' ! nvivafilter customer-lib-name=your_custom_lib.so cuda-process=true ! 'video/x-raw(memory:NVMM), format=RGBA' ! nvegltransform ! nveglglessink
#3

Nvivafilter only gives me RGBA and nv12, but like I stated I need to have BGR format in the end. Is it then possible to modify nvivfilter lib to get BGR format, thus avoiding using unreliable videoconvert?

#4

nvivafilter plugin is not open source. You can just build your own processing lib.

Rethinking to your use case, you don’t have to use nvvidconv. You may just try:

cv::VideoCapture("v4l2src device=/dev/video1 ! video/x-raw, width=(int)3840, height=(int)2160,format=(string)I420, framerate=(fraction)30/1 ! videoconvert ! video/x-raw, format=(string)BGR ! appsink")

You would also boost your TX2

sudo nvpmodel -m0
sudo /home/nvidia/jetson_clocks.sh

You can also build a recent version of opencv that accepts I420 as input (checked 3.3.1 works).

#5

Unfortunately this project requires that the stream is sent down to the gpu, otherwise the perfomance drop drasticaly. So I can’t get away with videoconvert

#6

Hi Forevermore,
On OpenCV 3.3, it supports I420. Please refer to
https://devtalk.nvidia.com/default/topic/1024245/jetson-tx2/opencv-3-3-and-integrated-camera-problems-/post/5210735/#5210735

#7

Did you ever find a solution? I am in the same boat as you, I need BGR using the GPU, otherwise performance suffers.