Nvcompositor with opencv not work

v.lambert · January 5, 2023, 11:44pm

I use the below gst-launch:

nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvcomp.  nvarguscamerasrc sensor-id=1 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvcomp. nvcompositor name=nvcomp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_1::xpos=1920 sink_1::ypos=0 sink_1::width=1920 sink_1::height=1080 ! 'video/x-raw(memory:NVMM)' ! nvvidconv ! 'video/x-raw,format=RGBA' ! videoconvert ! 'video/x-raw,format=BGR' ! appsink

it work on terminal but when I try the below code for python it wont work :

cv2.VideoCapture("nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvcomp.  nvarguscamerasrc sensor-id=1 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvcomp. nvcompositor name=nvcomp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_1::xpos=1920 sink_1::ypos=0 sink_1::width=1920 sink_1::height=1080 ! 'video/x-raw(memory:NVMM)' ! nvvidconv ! 'video/x-raw,format=RGBA' ! videoconvert ! 'video/x-raw,format=BGR' ! appsink", cv2.CAP_GSTREAMER)

i have this error on execution :

(python3:2227): GStreamer-CRITICAL **: 00:43:01.790: gst_element_make_from_uri: assertion 'gst_uri_is_valid (uri)' failed

(python3:2227): GStreamer-CRITICAL **: 00:43:01.792: gst_element_make_from_uri: assertion 'gst_uri_is_valid (uri)' failed

(python3:2227): GStreamer-CRITICAL **: 00:43:01.792: gst_element_make_from_uri: assertion 'gst_uri_is_valid (uri)' failed

(python3:2227): GStreamer-CRITICAL **: 00:43:01.792: gst_element_make_from_uri: assertion 'gst_uri_is_valid (uri)' failed

(python3:2227): GStreamer-CRITICAL **: 00:43:01.795: gst_element_make_from_uri: assertion 'gst_uri_is_valid (uri)' failed

(python3:2227): GStreamer-CRITICAL **: 00:43:01.795: gst_element_make_from_uri: assertion 'gst_uri_is_valid (uri)' failed

(python3:2227): GStreamer-CRITICAL **: 00:43:01.797: gst_element_make_from_uri: assertion 'gst_uri_is_valid (uri)' failed
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (711) open OpenCV | GStreamer warning: Error opening bin: syntax error
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created

and the result of print(cv.getBuildInformation())is :

...
Video I/O:
    FFMPEG:                      YES
      avcodec:                   YES (57.107.100)
      avformat:                  YES (57.83.100)
      avutil:                    YES (55.78.100)
      swscale:                   YES (4.8.100)
      avresample:                NO
    GStreamer:                   YES (1.14.5)
    v4l/v4l2:                    YES (linux/videodev2.h)
...

Honey_Patouceul · January 5, 2023, 11:54pm

You may try explictly setting the video sink inputs such as:

cv2.VideoCapture("nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvcomp.sink_0  nvarguscamerasrc sensor-id=1 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvcomp.sink_1  nvcompositor name=nvcomp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_1::xpos=1920 sink_1::ypos=0 sink_1::width=1920 sink_1::height=1080 ! 'video/x-raw(memory:NVMM)' ! nvvidconv ! 'video/x-raw,format=RGBA' ! videoconvert ! 'video/x-raw,format=BGR' ! appsink", cv2.CAP_GSTREAMER)

v.lambert · January 6, 2023, 9:47am

when I run this command in terminal it work :

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvcomp.sink_0  nvarguscamerasrc sensor-id=1 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvcomp.sink_1  nvcompositor name=nvcomp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_1::xpos=1920 sink_1::ypos=0 sink_1::width=1920 sink_1::height=1080 ! 'video/x-raw(memory:NVMM)' ! nvvidconv ! 'video/x-raw,format=RGBA' ! videoconvert ! 'video/x-raw,format=BGR' ! appsink

But with the python code it dont work :

import cv2 as cv

cap = cv.VideoCapture("nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvcomp.sink_0  nvarguscamerasrc sensor-id=1 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvcomp.sink_1  nvcompositor name=nvcomp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_1::xpos=1920 sink_1::ypos=0 sink_1::width=1920 sink_1::height=1080 ! 'video/x-raw(memory:NVMM)' ! nvvidconv ! 'video/x-raw,format=RGBA' ! videoconvert ! 'video/x-raw,format=BGR' ! appsink", cv.CAP_GSTREAMER)


ret, frame = cap.read()

cv.imwrite("test_2_cam_image" + ".jpg", frame)

Error on terminal :

Honey_Patouceul · January 6, 2023, 9:52am

Sorry that I missed that, the single quotes are only required in shell command. Try:

cap = cv.VideoCapture("nvarguscamerasrc sensor-id=0 ! video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! nvcomp.sink_0  nvarguscamerasrc sensor-id=1 ! video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! nvcomp.sink_1  nvcompositor name=nvcomp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_1::xpos=1920 sink_1::ypos=0 sink_1::width=1920 sink_1::height=1080 ! video/x-raw(memory:NVMM) ! nvvidconv ! video/x-raw,format=RGBA ! videoconvert ! video/x-raw,format=BGR ! appsink", cv.CAP_GSTREAMER)

v.lambert · January 6, 2023, 9:57am

Yes your code work thanks :)
But the picture is black :/ I supposed the color conversion doesn’t work.

v.lambert · January 6, 2023, 10:05am

Honey_Patouceul:

cap = cv.VideoCapture("nvarguscamerasrc sensor-id=0 ! video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! nvcomp.sink_0  nvarguscamerasrc sensor-id=1 ! video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! nvcomp.sink_1  nvcompositor name=nvcomp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_1::xpos=1920 sink_1::ypos=0 sink_1::width=1920 sink_1::height=1080 ! video/x-raw(memory:NVMM) ! nvvidconv ! video/x-raw,format=RGBA ! videoconvert ! video/x-raw,format=BGR ! appsink", cv.CAP_GSTREAMER)

The last probleme is that the nvcompositor must be done in RGBA but opencv wants BGR. Could you advise something to make this conversion?

The result of execution of this code :

GST_ARGUS: Creating output stream
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 1 
   Camera mode  = 2 
   Output Stream W = 1920 H = 1080 
   seconds to Run    = 0 
   Frame Rate = 29.999999 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

CONSUMER: Producer has connected; continuing.
GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 2 
   Output Stream W = 1920 H = 1080 
   seconds to Run    = 0 
   Frame Rate = 29.999999 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (933) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success

The generated picture :

Honey_Patouceul · January 6, 2023, 10:20am

Try:

cap = cv2.VideoCapture("nvarguscamerasrc sensor-id=0 ! video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! queue ! nvcomp.sink_0  nvarguscamerasrc sensor-id=1 ! video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! queue ! nvcomp.sink_1  nvcompositor name=nvcomp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_1::xpos=1920 sink_1::ypos=0 sink_1::width=1920 sink_1::height=1080 ! video/x-raw(memory:NVMM),format=RGBA,width=3840,height=1080  ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink drop=1", cv2.CAP_GSTREAMER)

But it may be slow with this resolution on Nano.
Also note that for 1920x1080, the native mode has framerate 30 fps.

v.lambert · January 6, 2023, 10:38am

The generated picture is still black :/
I have a question, why do you add the “queue” module and “drop=1” at the end ?

Honey_Patouceul · January 6, 2023, 10:45am

I added queue because this is something to always do when you have several subpipelines feeding a mux or compositor, it prevents synchronization issues that can lead to deadlock.
I added drop=1 because, at least in some older JetPack releases, it helps nvarguscamerasrc to cleanly terminate.

When you say the output is black, how did you display it ?
Try:

import cv2

cap = cv2.VideoCapture("nvarguscamerasrc sensor-id=0 ! video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! queue ! nvcomp.sink_0  nvarguscamerasrc sensor-id=1 ! video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! queue ! nvcomp.sink_1  nvcompositor name=nvcomp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_1::xpos=1920 sink_1::ypos=0 sink_1::width=1920 sink_1::height=1080 ! video/x-raw(memory:NVMM),format=RGBA,width=3840,height=1080  ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink drop=1", cv2.CAP_GSTREAMER)
if not cap.isOpened():
	print('Failed to open capture')
	exit(-1)
	
w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
fps = float(cap.get(cv2.CAP_PROP_FPS))
print('capture opened, framing %dx%d@%f fps' % (w,h,fps))	
	
while True:
	ret,frame=cap.read()
	if not ret:
		print('Failed to read from capture')
		exit(-2)
	
	cv2.imshow('Test', frame)
	cv2.waitKey(1)

I just tested with videotestsrc and it works, although slow:

cap = cv2.VideoCapture("videotestsrc ! video/x-raw,width=320,height=240 ! nvvidconv ! video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! queue ! nvcomp.sink_0  videotestsrc ! video/x-raw,width=320,height=240 ! nvvidconv ! video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! queue ! nvcomp.sink_1  nvcompositor name=nvcomp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_1::xpos=1920 sink_1::ypos=0 sink_1::width=1920 sink_1::height=1080 ! video/x-raw(memory:NVMM),format=RGBA,width=3840,height=1080 ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink drop=1", cv2.CAP_GSTREAMER)

v.lambert · January 6, 2023, 11:01am

Its work thanks a lot !

My finaly code :

import cv2
import utility

cap = cv2.VideoCapture("nvarguscamerasrc sensor-id=0 ! video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! queue ! nvcomp.sink_0  nvarguscamerasrc sensor-id=1 ! video/x-raw(memory:NVMM),width=1920,height=1080,framerate=21/1,format=NV12 ! nvvidconv ! video/x-raw(memory:NVMM),format=RGBA ! queue ! nvcomp.sink_1  nvcompositor name=nvcomp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_1::xpos=1920 sink_1::ypos=0 sink_1::width=1920 sink_1::height=1080 ! video/x-raw(memory:NVMM),format=RGBA,width=3840,height=1080  ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink drop=1", cv2.CAP_GSTREAMER)
if not cap.isOpened():
    print('Failed to open capture')
    exit(-1)

w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
fps = float(cap.get(cv2.CAP_PROP_FPS))
print('capture opened, framing %dx%d@%f fps' % (w, h, fps))

while True:
    ret, frame = cap.read()
    if not ret:
        print('Failed to read from capture')
        exit(-2)

    # cv2.imshow('Test', frame)
    cv2.imwrite("manual_photos_maker/"+"test_2_cam_image_" +
                utility.get_current_time()+".jpg", frame)
    cv2.waitKey(1)

I dont have access to a screen connected to the jetson so I write generated image in file ans it’s work !

I noticed that the first images are black then after ~5 images the cameras appear ;)

Honey_Patouceul · January 6, 2023, 11:06pm

Yes, Argus may try to auto-tune gains, exposure, white-balance… this may take a few frames to set up… If you know the setup values, you may set these as nvarguscamerasrc properties and get rid of this.

As you dont’have a local display, you may either try to save each image as jpeg with:

out=cv2.VideoWriter('appsrc ! video/x-raw,format=BGR ! queue ! videoconvert ! video/x-raw,format=BGRx ! nvvidconv ! video/x-raw(memory:NVMM),format=I420 ! nvjpegenc ! image/jpeg, format=MJPG ! jpegparse ! multifilesink location=img_%06d.jpg

Or :

# streaming RTP/JPG with static payload 26:
out=cv2.VideoWriter('appsrc ! video/x-raw,format=BGR ! queue ! videoconvert ! video/x-raw,format=BGRx ! nvvidconv ! video/x-raw(memory:NVMM),format=I420 ! nvjpegenc ! image/jpeg, format=MJPG ! rtpjpegpay ! udpsink port=5004', cv2.CAP_GSTREAMER)

# Or streaming as H264/MP2T with static payload 33 (may be ok on host receiver for ffmpeg ot VLC without SDP file):
out=cv2.VideoWriter('appsrc ! video/x-raw,format=BGR ! queue ! videoconvert ! video/x-raw,format=BGRx ! nvvidconv ! video/x-raw(memory:NVMM),format=NV12 ! nv4l2h264enc ! h264parse ! rtpmp2tpay ! udpsink port=5004', cv2.CAP_GSTREAMER)

check that out.isOpened() and add out.write(frame) in the loop instead of imshow/waitKey.

system · January 20, 2023, 11:07pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Problem with nvcompositor + nvarguscamerasrc Jetson AGX Xavier camera , gstreamer , sony-imx390	27	4359	October 18, 2021
Nvarguscamerasrc issues, related to OpenCV Jetson Nano camera	36	6985	October 12, 2022
nvarguscamerasrc OpenCV (Solved) Jetson AGX Xavier opencv	52	30348	October 18, 2021
openCV video writer to gstreamer appsrc Jetson Nano jetpack	22	14472	October 15, 2021
Jetson Nano 2gb RPI Camera v2.1 Jetson Nano camera , opencv , gstreamer	4	1333	July 20, 2022
Access to Raspberry Cam ( nvargus-daemon ) from docker container Jetson Nano	20	8510	October 14, 2021
Nvcompositor does not work with NVMM attribute on jetpack 6 Jetson Orin NX gstreamer	6	53	September 19, 2024
nvvidconv plugin and v4l2 camera problem :( Jetson TX1	42	12561	September 26, 2016
Nvcompositor segfaults Jetson AGX Orin gstreamer	7	85	December 5, 2024
RTSP streaming of two CSI cameras by using Gstreamer Jetson Nano rtsp , nvbugs	5	2362	October 15, 2021

Nvcompositor with opencv not work

Related topics