to use onboard camera

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

I tried to run yolov3 but I failed. I don’t know what I should do :(

==========================================================================================================================

cae@cae-desktop:~/Downloads/darknet$ ./darknet detector demo cfg/coco.data cfg/yolov3.cfg yolov3.weights “nvarguscamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink”

==========================================================================================================================

returns

==========================================================================================================================
Loading weights from yolov3.weights…Done!
video file: nvarguscamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 2592 x 1944 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 34000, max 550385000;

GST_ARGUS: 2592 x 1458 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 34000, max 550385000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 22000, max 358733000;

GST_ARGUS: Running with following settings:
Camera index = 0
Camera mode = 1
Output Stream W = 2592 H = 1458
seconds to Run = 0
Frame Rate = 29.999999
GST_ARGUS: PowerService: requested_clock_Hz=27216000
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
GST_ARGUS: Cleaning up
GST_ARGUS:
PowerServiceHwVic::cleanupResources
CONSUMER: Done Success
GST_ARGUS: Done Success
OpenCV Error: Unspecified error (GStreamer: unable to start pipeline
) in cvCaptureFromCAM_GStreamer, file /home/nvidia/build_opencv/opencv/modules/videoio/src/cap_gstreamer.cpp, line 887
VIDEOIO(cvCreateCapture_GStreamer (CV_CAP_GSTREAMER_FILE, filename)): raised OpenCV exception:

/home/nvidia/build_opencv/opencv/modules/videoio/src/cap_gstreamer.cpp:887: error: (-2) GStreamer: unable to start pipeline
in function cvCaptureFromCAM_GStreamer

Couldn’t connect to webcam.
: Resource temporarily unavailable
darknet: ./src/utils.c:256: error: Assertion `0’ failed.
Aborted (core dumped)

==========================================================================================================================

Actually, I could use onboard camera with this code(below).

==========================================================================================================================

gst-launch-1.0 nvarguscamerasrc ! ‘video/x-raw(memory:NVMM), width=640, height=480, framerate=30/1, format=NV12’ ! nvvidconv flip-method=2 ! nvegltransform ! nveglglessink -e

==========================================================================================================================

so I applied the code in yolov3 model. But…

==========================================================================================================================

cae@cae-desktop:~/Downloads/darknet$ ./darknet detector demo cfg/coco.data cfg/yolov3.cfg yolov3.weights “gst-launch-1.0 nvarguscamerasrc ! ‘video/x-raw(memory:NVMM), width=640, height=480, framerate=30/1, format=NV12’ ! nvvidconv flip-method=2 ! nvegltransform ! nveglglessink -e”

==========================================================================================================================

returns

==========================================================================================================================

Loading weights from yolov3.weights…Done!
video file: gst-launch-1.0 nvarguscamerasrc ! ‘video/x-raw(memory:NVMM), width=640, height=480, framerate=30/1, format=NV12’ ! nvvidconv flip-method=2 ! nvegltransform ! nveglglessink -e

(darknet:11513): GStreamer-CRITICAL **: 19:53:49.037: gst_element_make_from_uri: assertion ‘gst_uri_is_valid (uri)’ failed
OpenCV Error: Unspecified error (GStreamer: cannot find appsink in manual pipeline
) in cvCaptureFromCAM_GStreamer, file /home/nvidia/build_opencv/opencv/modules/videoio/src/cap_gstreamer.cpp, line 802
VIDEOIO(cvCreateCapture_GStreamer (CV_CAP_GSTREAMER_FILE, filename)): raised OpenCV exception:

/home/nvidia/build_opencv/opencv/modules/videoio/src/cap_gstreamer.cpp:802: error: (-2) GStreamer: cannot find appsink in manual pipeline
in function cvCaptureFromCAM_GStreamer

Couldn’t connect to webcam.
: No such file or directory
darknet: ./src/utils.c:256: error: Assertion `0’ failed.
Aborted (core dumped)

===========================================================================================================================
what’s wrong?

2

Hi,

Have you built OpenCV from the source?
The default package in JetPack doesn’t enable the GStreamer support.

This requires you to build it from the source.
Here is the script for your reference:
[url]https://github.com/AastaNV/JEP/blob/master/script/install_opencv3.4.0_TX2.sh[/url]

By the way, you can also follow this tutorial to enable the darknet on TX2:
[url]https://github.com/Alro10/YOLO-darknet-on-Jetson-TX2[/url]

Thanks.

3

I meet such issue too. When I run cmd: gst-launch-1.0 nvarguscamerasrc ! ‘video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1’ ! nvoverlaysink -e, it works fine. But I use opencv:

VideoCapture cap;
cap.open("nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1' ! nvoverlaysink -e");

It’s fail as below:

(trt-yolo-app:15437): GStreamer-CRITICAL **: 13:57:51.264: gst_element_make_from_uri: assertion ‘gst_uri_is_valid (uri)’ failed

(trt-yolo-app:15437): GLib-GObject-WARNING **: 13:57:51.270: invalid cast from ‘GstNvOverlaySink-nvoverlaysink’ to ‘GstBin’

(trt-yolo-app:15437): GStreamer-CRITICAL **: 13:57:51.270: gst_bin_iterate_elements: assertion ‘GST_IS_BIN (bin)’ failed

(trt-yolo-app:15437): GStreamer-CRITICAL **: 13:57:51.270: gst_iterator_next: assertion ‘it != NULL’ failed

(trt-yolo-app:15437): GStreamer-CRITICAL **: 13:57:51.270: gst_iterator_free: assertion ‘it != NULL’ failed
OpenCV Error: Unspecified error (GStreamer: cannot find appsink in manual pipeline
) in cvCaptureFromCAM_GStreamer, file /home/nvidia/build_opencv/opencv/modules/videoio/src/cap_gstreamer.cpp, line 802
VIDEOIO(cvCreateCapture_GStreamer (CV_CAP_GSTREAMER_FILE, filename)): raised OpenCV exception:

/home/nvidia/build_opencv/opencv/modules/videoio/src/cap_gstreamer.cpp:802: error: (-2) GStreamer: cannot find appsink in manual pipeline
in function cvCaptureFromCAM_GStreamer

Could not initialize capturing…

4

hello @duanlinlin, have you found a solution to use the tx2’s onboard camera with the cv2.VideoCapture method? it would be very helpful to me to see your working pipeline stream.

i have currently this pipeline stream but it fails:

nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=640, height=480, framerate=30/1, format=NV12' ! nvvidconv flip-method=2 ! nvegltransform ! nveglglessink ! appsink

The error is:

Unable to query number of channels

do you have a hint?

thanks in advance, tobi

5

@duanlinlin and @tobilab,

You would need a recent opencv version built with gstreamer support, which may not be available from opencv version in JetPack or SDM.

Your pipeline has to end with appsink in suitable format such as BGR for opencv application to read frames.

You may try to remove quotes. These are required with gst-launch for preventing shell interpretation with parenthesis, but not suitable to give a pipeline string to gstreamer from opencv or else.

If you are running R31 or later, you may try to give this pipeline string:

nvarguscamerasrc ! video/x-raw(memory:NVMM), width=640, height=480, framerate=30/1, format=NV12 ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink

On earlier versions up to R28.2, use nvcamerasrc instead of nvarguscamerasrc gstreamer plugin.

6

dear @Honey_Patouceul,
thanks for your supply.

I changed the pipeline stream for the opencv VideoCapture method to:

cv2.VideoCapture("nvarguscamerasrc ! video/x-raw(memory:NVMM), width=(int)640, height=(int)480, format=(string)NV12, framerate=(fraction)30/1 ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink")

The cam image is displayed in a new window, but in terminal there comes a error message:

VIDEOIO ERROR: V4L: device nvarguscamerasrc ! video/x-raw(memory:NVMM), width=(int)640, height=(int)480, format=(string)NV12, framerate=(fraction)30/1 ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink: Unable to query number of channels

I do not know what to change on the pipeline stream because i dont know what is meant with “unable to query number of channels” ? What channels are meant?

Do you have any ideas or hints?

Thanks for your help, tobi

7

This is a harmless error message. You can get rid of this by adding a second argument cv2.CAP_GSTREAMER after your pipeline string in cv2.videoCapture.

cv2.VideoCapture("nvarguscamerasrc ... appsink", cv2.CAP_GSTREAMER)

When no mode is specified, opencv may try to interpret the provided string for various kind of sources: V4L device, file if the string ends with an extension (has a dot) which is not your case, or gstreamer pipeline.

When it tries as V4L device, if you have built your opencv version with libv4l/libv4l2 support, then this message is issued.

8

Thank you @Honey_Patouceul for the explaination. I will edit the cv2.CAP_GSTREAMER argument.

Can i ask you an other question in generel. Do you know how i can make sure that the pipeline stream and the classification algorithms are timed correctly? As i see now, the only possibility that i have to adjust ist the fps setting in the pipeline stream or? Because at the moment the onboard camera → classification works but if i change and use a network camera it seems that the camera can not “deliver” the images as fast as the pipeline/algorithm needs the images. That causes the script to crash. Am i right that the only changeable parameter for me is the fps setting or do you have another hint to adjust that and keep the camera/script parts timed correctly?

thanks in advice, tobi