Nvcompositor to appsink for OpenCV

Autonomous Machines Jetson & Embedded Systems Jetson Xavier NX
,
1

Hello,

So I have this pipeline to merge two videos

gst-launch-1.0 nvcompositor name=comp \
 sink_0::width=1920 sink_0::height=1080 \
 sink_1::xpos=1280 sink_1::width=640 sink_1::height=480 \
 ! nv3dsink \
 v4l2src device=/dev/video0 io-mode=2 ! 'video/x-raw,format=YUY2,width=1920,height=1080,framerate=30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,width=1600,height=900' ! comp. \
 videotestsrc pattern=0  ! 'video/x-raw,format=YUY2,width=640,height=480,framerate=30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! comp. -e

While I am able to display the result in nv3dsink, I cannot make the pipeline work in Python/OpenCV
As I know OpenCV requires the pipeline to use appsink so I changed the pipeline accordingly

gst-launch-1.0 nvcompositor name=comp \
 sink_0::width=1920 sink_0::height=1080 \
 sink_1::xpos=1280 sink_1::width=640 sink_1::height=480 \
 ! nvvidconv ! 'video/x-raw,format=BGRx' ! queue ! videoconvert ! queue ! xvimagesink \
 v4l2src device=/dev/video0 io-mode=2 ! 'video/x-raw,format=YUY2,width=1920,height=1080,framerate=30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,width=1600,height=900' ! comp. \
 videotestsrc pattern=0  ! 'video/x-raw,format=YUY2,width=640,height=480,framerate=30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! comp. -e

I get an error could not link nvvconv1 to comp, comp can’t handle caps video/x-raw(memory:NVMM), format=(string)RGBA, width=(int)1600, height=(int)900
How do I make it work ?

3

Hi,
Please try

$ gst-launch-1.0 nvcompositor name=comp  sink_0::width=1920 sink_0::height=1080  sink_1::xpos=1280 sink_1::width=640 sink_1::height=480  ! 'video/x-raw(memory:NVMM)' ! nvvidconv ! video/x-raw ! videoconvert ! xvimagesink  v4l2src device=/dev/video0 io-mode=2 ! 'video/x-raw,format=UYVY,width=1920,height=1080,framerate=30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,width=1600,height=900' ! comp.  videotestsrc pattern=0  ! 'video/x-raw,format=UYVY,width=640,height=480,framerate=30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! comp. -e
1 Like
4

Hi @DaneLLL

Thank you! the pipeline is working in as command line but I am stuck yet again on getting a proper image in Python/OpenCV. Running the pipeline through the CLI gives a proper colorful image while OpenCV disorders everything for no reason

Running GStreamer pipeline from command line
Running GStreamer pipeline from command line1722×968 539 KB

Using the pipelien to open camera from OpenCV
Using the pipelien to open camera from OpenCV1722×971 398 KB

Here is a piece of my code

gst_str="nvcompositor name=comp sink_0::width=1920 sink_0::height=1080 sink_1::xpos=1280 sink_1::width=640 sink_1::height=480 ! video/x-raw(memory:NVMM) ! nvvidconv ! video/x-raw,width=1366,height=768 ! appsink v4l2src device=/dev/video0 io-mode=2 ! video/x-raw,format=YUY2,width=1920,height=1080,framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM) ! comp. videotestsrc pattern=0 ! video/x-raw,format=YUY2,width=640,height=480,framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM) ! comp."
cap = cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER)
...
ret, frame = cap.read()
cv2.imshow('frame', frame)

Any clue on how to fix this ?

5

You would provide BGR format to opencv. Try this:

gst_str="nvcompositor name=comp sink_0::width=1920 sink_0::height=1080 sink_1::xpos=1280 sink_1::width=640 sink_1::height=480 ! 'video/x-raw(memory:NVMM)' ! nvvidconv ! video/x-raw,format=BGRx,width=1366,height=768 ! videoconvert ! video/x-raw, format=BGR ! appsink   ... "
6

Hi @Honey_Patouceul

Unfortunately this did not change anything…

7

I remember now that I had seen this once. It was because of the width. For some reason (reading 3 bytes pixels with 32 bits words), it may fail when the width is not a multiple of 4. Try resizing to 1368x768.

2 Likes
8

You’re right, thanks! I would have never guessed.

9

Did not know setting width value is tricky. Thanks, Honey Patouceul.