You may check if your camera is really able to perform 1280x720 @ 120 fps. RPi’s cam is not able to do that AFAIK, although the sensor itself can be faster.
1280x720 @ 120 fps is supported on TX2 for OV5693 sensor in devkit onboard camera. Be sure you’re not seeing this mode because of wrong or uncorrectly adapter driver.
If you indeed have a camera able to do so and correct driver for it, the following command would display it overlaying GUI, so better maximize your terminal before, or don’t click with mouse so that you’ll be able to stop pipeline with a Ctrl-C that goes to right terminal window:
gst-launch-1.0 -v nvarguscamerasrc exposuretimerange="1 1" gainrange="1 1" ! 'video/x-raw(memory:NVMM), format=NV12,width=1280, height=720, framerate=120/1' ! nvoverlaysink
If it works fine, you may measure framerate with:
gst-launch-1.0 -v nvarguscamerasrc exposuretimerange="1 1" gainrange="1 1" ! 'video/x-raw(memory:NVMM), format=NV12,width=1280, height=720, framerate=120/1' ! fpsdisplaysink video-sink=fakesink text-overlay=false
If all good so far, the problem is probably with either videoconvert, or in opencv videoio. 120 fps is a very high framerate for opencv application on jetson CPU. From your description I understand you get the frame read from camera into CPU Mat and copy into a pinned or unified memory GpuMat. This is not so efficient for high framerates unless very low resolution.
I don’t know what kind of processing you’re expecting to perform, but I’d suggest to try the nvivafilter method.
This allows to use cv::cuda functions easily. Of course you need a build with opencv_contrib CUDA enabled, but this should already be your case.
The following example is doing the following conversions in CUDA for a binary threshold on Hue: RGBA->RGB->HSV->H->threshold->HSV->RGB-RGBA. This is just an example.
For trying, you would create a directory:
mkdir test-opencv-cuda
where you would save the following files (attached here with extra .txt extension because of the forum restrictions), then edit Makefile for adjusting your opencv installed path in OPENCV_DIR definition.
gst-custom-opencv_cudaprocess.cu
gst-custom-opencv_cudaprocess.cu.txt (9.1 KB)
Makefile
Makefile.txt (5.2 KB)
You would check that cuda libs and opencv libs paths are in LD_LIBRARY_PATH, if not set it, and build with make and if no error try to display or fps measurement:
cd test-opencv-cuda
echo $LD_LIBRARY_PATH
#If not in, this would be a minimal way:
export OPENCV_DIR=<where_you_have_installed_it>
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$OPENCV_DIR/lib
make
#Better boost your jetson now for better perf
sudo nvpmodel -m0
sudo jetson_clocks
#Try to display result:
gst-launch-1.0 -v nvarguscamerasrc exposuretimerange="1 1" gainrange="1 1" ! 'video/x-raw(memory:NVMM), format=NV12,width=1280, height=720, framerate=120/1' ! nvvidconv ! nvivafilter cuda-process=true customer-lib-name=lib-gst-custom-opencv_cudaprocess.so ! 'video/x-raw(memory:NVMM), format=RGBA' ! nvoverlaysink
#Measure fps
gst-launch-1.0 -v nvarguscamerasrc exposuretimerange="1 1" gainrange="1 1" ! 'video/x-raw(memory:NVMM), format=NV12,width=1280, height=720, framerate=120/1' ! nvvidconv ! nvivafilter cuda-process=true customer-lib-name=lib-gst-custom-opencv_cudaprocess.so ! 'video/x-raw(memory:NVMM), format=RGBA' ! fpsdisplaysink video-sink=fakesink text-overlay=false
On AGX Xavier R32.4.2 with OV5693 onboard camera from TX2 and opencv-4.3.0-dev, I get 90 fps without boosting clocks, and a solid 120 fps after boosting.
PS: If you’re setting manual exposure and gains, you may be interested in disabling AWB and setting digitalgain as well.