OpenCV GStreamer Capture really slow

Hi,

I am using this command to get a picture from my CSI camera. It is an IMX219. This works, but the cam.read(frame) function takes 5 Milliseconds. This is too much for a camera that should support 120FPS at 720P. Because I’m doing some processing I only have 1-2ms for the cam read.

I am using OpenCV4.3.

This is my code. I suspect a faulty GStreamer command.

VideoCapture cam;
cam.open("nvarguscamerasrc exposuretimerange=\"1 1\" gainrange=\"1 1\" ! video/x-raw(memory:NVMM), width=1280, height=720, framerate=120/1 ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink");

Then I read the frame in a shared CPU/GPU pointer. This means I could directly read to the GPU frame if needed. Later I convert the frame to HSV. So if there is a way to directly convert to the HSV color space or to save one conversion step, it would be great.

I hope you can give me some suggestions and some help,

Greetings,
Fabio

You may check if your camera is really able to perform 1280x720 @ 120 fps. RPi’s cam is not able to do that AFAIK, although the sensor itself can be faster.
1280x720 @ 120 fps is supported on TX2 for OV5693 sensor in devkit onboard camera. Be sure you’re not seeing this mode because of wrong or uncorrectly adapter driver.

If you indeed have a camera able to do so and correct driver for it, the following command would display it overlaying GUI, so better maximize your terminal before, or don’t click with mouse so that you’ll be able to stop pipeline with a Ctrl-C that goes to right terminal window:

gst-launch-1.0 -v nvarguscamerasrc exposuretimerange="1 1" gainrange="1 1" ! 'video/x-raw(memory:NVMM), format=NV12,width=1280, height=720, framerate=120/1' ! nvoverlaysink

If it works fine, you may measure framerate with:

gst-launch-1.0 -v nvarguscamerasrc exposuretimerange="1 1" gainrange="1 1" ! 'video/x-raw(memory:NVMM), format=NV12,width=1280, height=720, framerate=120/1' ! fpsdisplaysink video-sink=fakesink text-overlay=false

If all good so far, the problem is probably with either videoconvert, or in opencv videoio. 120 fps is a very high framerate for opencv application on jetson CPU. From your description I understand you get the frame read from camera into CPU Mat and copy into a pinned or unified memory GpuMat. This is not so efficient for high framerates unless very low resolution.

I don’t know what kind of processing you’re expecting to perform, but I’d suggest to try the nvivafilter method.
This allows to use cv::cuda functions easily. Of course you need a build with opencv_contrib CUDA enabled, but this should already be your case.
The following example is doing the following conversions in CUDA for a binary threshold on Hue: RGBA->RGB->HSV->H->threshold->HSV->RGB-RGBA. This is just an example.
For trying, you would create a directory:

mkdir test-opencv-cuda

where you would save the following files (attached here with extra .txt extension because of the forum restrictions), then edit Makefile for adjusting your opencv installed path in OPENCV_DIR definition.

gst-custom-opencv_cudaprocess.cu

gst-custom-opencv_cudaprocess.cu.txt (9.1 KB)

Makefile

Makefile.txt (5.2 KB)

You would check that cuda libs and opencv libs paths are in LD_LIBRARY_PATH, if not set it, and build with make and if no error try to display or fps measurement:

cd test-opencv-cuda
echo $LD_LIBRARY_PATH
#If not in, this would be a minimal way:
export OPENCV_DIR=<where_you_have_installed_it>
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$OPENCV_DIR/lib
make

#Better boost your jetson now for better perf
sudo nvpmodel -m0
sudo jetson_clocks

#Try to display result:
gst-launch-1.0 -v nvarguscamerasrc exposuretimerange="1 1" gainrange="1 1" ! 'video/x-raw(memory:NVMM), format=NV12,width=1280, height=720, framerate=120/1' ! nvvidconv ! nvivafilter cuda-process=true customer-lib-name=lib-gst-custom-opencv_cudaprocess.so ! 'video/x-raw(memory:NVMM), format=RGBA' ! nvoverlaysink

#Measure fps
gst-launch-1.0 -v nvarguscamerasrc exposuretimerange="1 1" gainrange="1 1" ! 'video/x-raw(memory:NVMM), format=NV12,width=1280, height=720, framerate=120/1' ! nvvidconv ! nvivafilter cuda-process=true customer-lib-name=lib-gst-custom-opencv_cudaprocess.so ! 'video/x-raw(memory:NVMM), format=RGBA' ! fpsdisplaysink video-sink=fakesink text-overlay=false

On AGX Xavier R32.4.2 with OV5693 onboard camera from TX2 and opencv-4.3.0-dev, I get 90 fps without boosting clocks, and a solid 120 fps after boosting.

PS: If you’re setting manual exposure and gains, you may be interested in disabling AWB and setting digitalgain as well.

Thank you for your fast and detailed answer. I am only replying now, because I first wanted to get more information about the topic.
The hint to not use unified memory was excellent. It brought my whole program to 120fps, capped by the cam :)
Also I disabled AWB, which I didn’t know was possible. Thank you for that.

I was able to compile your examples and they run with 120fps. But my program is a bit too complicated to put in a Gstreamer pipeline. But what I wanted to try is to get a HSV image into OpenCV as fast as possible with a nvivafilter, without doing dozens of conversions. Sadly I was not able to use your pipline or mine, which is based on your example, to work with opencv using appsink. When I launch it as you descriped it did work. Can you please give me an example VideoCapture Gstreamer command to use your example? I’m trying to get more performance headroom for future features.

Kind regards,
Fabio

Hi,
For hit frame rate case, it would be better to run pure gstreamer pipeline.

Since OpenCV takess significant CPU loading, please execute sudo nvpmodel -m 2 and sudo jetson_clocks to enable 6 cores in max clocks(for detail, please check document).
And you can run sudo tegrastats to get runtime system loading.