Optimizing opencv gstreamer with mipi and usb uvc cameras

warpstar22 · May 18, 2020, 10:56pm

I am trying to use MIPI cameras and USB UVC cameras at the same time. To test, I’m running a simple program that streams the cameras to an OpenCV gui window. I noticed though that there is considerable lag with the MIPI port. The UVC camera seems to be operating properly.

My pipeline for the mipi camera is:
nvarguscamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, framerate=(fraction)60/1, format=(string)NV12 ! nvvidconv flip-method=0 ! video/x-raw, width=(int)1280, height=(int)720, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink
I just put that in the VideoCapture constructor.

I open the UVC camera with VideoCapture cap("/dev/video1", cv::CAP_V4L2)

Even if I turn down the MIPI camera to its lowest resolution, I can still see noticeable lag with fast moving objects. I’m hoping I can minimize that performance difference. Is there a way I can optimize the gstreamer pipeline or do something else? Or is there something else I should do to get decent performance when using both MIPI and USB UVC cameras?

Also, not sure if this is important but I get a gstreamer warning that it cannot query video position. I was able to remove that for the UVC camera by adding the cv::CAP_V4L2 option but I don’t know what to do for the mipi camera.

DaneLLL · May 19, 2020, 2:37am

Hi,
For the USB camera, you may check if you can run

VideoCapture cap("v4l2src device=/dev/video1 ! video/x-raw,width=1920,height=1080,format=UYVY,framerate=30/1 ! videoconvert ! video/x-raw,format=BGR ! appsink");

OpenCV is CPU based frameworks and please execute sudo jetson_clocks to get max performance on Jetson Nano.

If OpenCV is not must-have in your implementation, you can consider to use gstreamer or jetson_multimedia_api, which use hardware DMA buffers to get optimal performance.

warpstar22 · May 19, 2020, 9:32pm

Thank you for your response.

I tried using jetson_clocks and didn’t see too much improvement. But now I’d like to restore the original settings. I saw in a post that all I needed to do was reboot the device. But after a reboot and running sudo jetson_clocks --show, I see that all 4 CurrentFreq’s are 1479000 which is the MaxFreq. Is it different for Jetson Nano?

I can’t get this to run either. If I run it with gst-launch-1.0, I get streaming stopped, reason not-negotiated (-4)

Just to clarify things, why does using the DMA affect the performance of the MIPI? The MIPI port performs very well in OpenCV when its by itself. The delay problem only occurs when I use it in conjunction with the USB camera.

Also, just a note. It would probably be nice if on this page or somewhere, it was mentioned that Jetson Nano has the samples stored in /usr/src/jetson_multimedia_api/samples. It took me a while to find that.

DaneLLL · May 20, 2020, 1:08am

Hi,
Probably your camera does not support UYVY format. You may refer to FAQ
Q: I have a USB camera. How can I lauch it on Jetson Nano?
to configure the source correctly.

We don’t indicate path of the samples.My apology for this. If you consider to use jetson_multimedia_api, you can check 12_camera_v4l2_cuda for USB cameras, and 10_camera_recording for Bayer sensors(through Argus).

warpstar22 · May 21, 2020, 1:12am

Hi,
Thank you for the FAQ. I was able to use gst-launch-1.0 with a gstreamer pipeline for the USB camera. I used
v4l2src device=/dev/video1 ! video/x-raw, format=YUY2, width=640, height=480, framerate=30/1 ! nvvidconv ! 'video/x-raw(memory:NVMM), format=NV12' ! nvoverlaysink

But if I change the pipeline to
v4l2src device=/dev/video1 ! video/x-raw, format=YUY2, width=640, height=480, framerate=30/1 ! videoconvert ! video/x-raw, format=BGR ! appsink
and try to use it in OpenCV I am unable to use the camera. If I open it with option cv::CAP_V4L2, I am unable to open the camera. If I open it with option cv::CAP_GSTREAMER, I get the following:

[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (1757) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src0 reported: Internal data stream error.
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (515) startPipeline OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (1055) setProperty OpenCV | GStreamer warning: no pipeline

If this is an OpenCV error, I can go ask on their forums. Thank you

Honey_Patouceul · May 22, 2020, 12:48am

Are you using local X server or headless ?

Also, it is unclear if your source is outputting YUYV or UYVY. You may show the output of:

v4l2-ctl -d1 --list-formats-ext

If using local X server and display, you would first try:

gst-launch-1.0 v4l2src device=/dev/video1 ! video/x-raw, format=YUY2, width=640, height=480, framerate=30/1 ! videoconvert ! xvimagesink

If this works, you would try this in python opencv (or translate to C++):

cap=cv2.videoCapture('v4l2src device=/dev/video1 ! video/x-raw, format=YUY2, width=640, height=480, framerate=30/1 ! videoconvert ! video/x-raw, format=BGR ! appsink', cv2.CAP_GSTREAMER)

#or try HW conversion
cap=cv2.videoCapture('v4l2src device=/dev/video1 ! video/x-raw, format=YUY2, width=640, height=480, framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM) ! nvvidconv ! video/x-raw, format=BGRx !  videoconvert ! video/x-raw, format=BGR ! appsink', cv2.CAP_GSTREAMER)

warpstar22 · May 22, 2020, 1:08am

Thanks for the response. I actually figured it out today. Not sure why it wasn’t working but I am currently using this pipeline to use the MJPG data so I can get faster framerates at higher resolutions:
v4l2src device=/dev/video1 ! image/jpeg, width=1280, height=1024, framerate=30/1 ! jpegparse ! jpegdec ! videoconvert ! video/x-raw, format=BGR ! appsink

The capture speeds are great now. But I still have the issue with the delay in how the video shows up. My guess is I’ve done all I can with gstreamer and what the Jetson is capable here.

If someone could confirm this, it seems like my options are to figure out an issue with OpenCV or to use the jetson_multimedia_api

warpstar22 · May 22, 2020, 1:13am

Actually, scratch that. I found the solution for at least one UVC camera and one MIPI camera. After appsink, you can specify drop=true and the delay goes away. The problem is individual buffer sizes for grabbing frames. OpenCV would take the earliest frame in the buffer and that could be a few ms behind and could vary when having different framerates.

Honey_Patouceul · May 22, 2020, 1:15am

On recent L4T releases, you may also try nvv4l2decoder for mjpeg decoding.

gst-launch-1.0 v4l2src device=/dev/video1 ! image/jpeg, width=1280, height=1024, framerate=30/1 ! jpegparse ! nvv4l2decoder mjpeg=true ! nvvidconv ! videoconvert ! xvimagesink

cap=cv2.videoCapture('v4l2src device=/dev/video1 ! image/jpeg, width=1280, height=1024, framerate=30/1 ! jpegparse ! nvv4l2decoder mjpeg=true ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink', cv2.CAP_GSTREAMER)

#or
cap=cv2.videoCapture('v4l2src device=/dev/video1 ! image/jpeg, width=1280, height=1024, framerate=30/1, format=MJPG ! nvv4l2decoder mjpeg=true ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink', cv2.CAP_GSTREAMER)

warpstar22 · May 22, 2020, 9:35pm

Hi Honey_Patouceul,

Thanks for the recommendation. Is there a specific benefit to using nvv4l2decoder such as faster captures? Using jpegdec seems to be working well.

DaneLLL · May 25, 2020, 12:00am

Hi,

It used hardware engine NVJPG in nvv4l2decoder. The jpegdec plugin is software decoder. You may run sudo tegrastats in comparing CPU usage. If the CPU usage is fine and acceptable, it is good to use jpegdec.

warpstar22 · May 25, 2020, 10:00pm

Hi,
So I tried the nvv4l2decoder to see what sort of performance advantage it may have but the image is not being display properly. Below is a sample.

The color is off and the rest of the image is grayish. My pipeline is as follows:
gst-launch-1.0 v4l2src device=/dev/video2 ! image/jpeg, width=1280, height=1024, framerate=30/1 ! jpegparse ! nvv4l2decoder mjpeg=true ! nvvidconv ! videoconvert ! xvimagesink sync=false -e

I had to add sync=false to the pipeline as I was getting some delay issue and the debugs said “There may be a timestamping problem, or this computer is too slow.”

Honey_Patouceul · May 25, 2020, 10:07pm

You may try to remove jpegparse from the pipeline and tell if it works better without it?

gst-launch-1.0 -e v4l2src device=/dev/video2 ! image/jpeg, width=1280, height=1024, framerate=30/1, format=MJPG ! nvv4l2decoder mjpeg=true ! nvvidconv ! videoconvert ! xvimagesink

warpstar22 · May 25, 2020, 10:42pm

I still see the same issue after removing jpegparse.

warpstar22 · May 25, 2020, 10:48pm

I appreciate the responses. I was wondering if I could also get help in figuring out how to get good performance with different framerates. My MIPI camera at its highest resolution has 21fps and the UVC camera at the highest resolution is at 15fps. The delay is similar to what I had before adding drop=true to appsink. I don’t think the delay is due to a higher resolution. I can see an issue occuring with the threads not in sync because of the slightly different rate. I thought using videorate might help, but I haven’t seen any noticeable improvement. If there’s a solution that would be great. Otherwise, I guess I have to limit the resolution to have a common fps for all the cameras I want to use.

DaneLLL · May 26, 2020, 1:49am

Hi,
Please run

gst-launch-1.0 v4l2src device=/dev/video2 num-buffers=300 ! image/jpeg, width=1280, height=1024, framerate=30/1 ! filesink location=test.mjpg

And share test.mjpg so that we can use the stream to reproduce the issue.

warpstar22 · May 26, 2020, 7:47pm

Hi,
It says that mjpg is not allowed for file upload. Here is a google drive link.

warpstar22 · June 1, 2020, 9:29pm

Hi @DaneLLL,

Have you found a solution to my problem?

DaneLLL · June 2, 2020, 1:23am

Hi,
We can observe the issue on r32.4.2. Certain YUV422 is not handled well in nvv4l2decoder. Will investigate further.

warpstar22 · June 5, 2020, 11:49pm

Hi @DaneLLL

I encountered a weird problem with using OpenCV and GStreamer. If I try to grab one frame at a time between 2 MIPI and 2 UVC Cameras, I get really good performance on grabbing frames. It’s around 5e-05 seconds. If I change the following:

cam1.grab() //mipi1
cam2.grab() //mipi2
cam3.grab() //uvc1
cam4.grab() //uvc2

to:

cam1.grab()
cam2.grab()
cam3.grab()
cam3.grab()
cam4.grab()
cam4.grab()

I get much worse performance. I’m getting anywhere from 50 to 150ms. This is even with sudo jetson_clocks added. A similar thing happens with the MIPI cameras. Do you know what may be going wrong here?