Nano not using GPU with gstreamer/python. Slow FPS, dropped frames

I am using the a typical pipeline (see below) to feed my Opencv/Python program frames. My CPU is running at 100% , while my GPU is only 25%. Is there anything I can do to put more work on the GPU?

def gstreamer_pipeline(
capture_width=3264,
capture_height=1848,
display_width=3264,
display_height=1848,
framerate=28,
flip_method=0,
):
return (
"nvarguscamerasrc ! "
"video/x-raw(memory:NVMM), "
"width=(int)%d, height=(int)%d, "
"format=(string)NV12, framerate=(fraction)%d/1 ! "
"nvvidconv flip-method=%d ! "
"video/x-raw, width=(int)%d, height=(int)%d, format=(string)BGRx ! "
"videoconvert ! "
“video/x-raw, format=(string)BGR ! appsink”
% (
capture_width,
capture_height,
framerate,
flip_method,
display_width,
display_height,
)
)

Hi,
The optimal-performance solution is to pass NVMM[video/x-raw(memory:NVMM)] buffers in the pipelines. However, with python, it only accepts CPU[video/x-raw,format=BGR] buffers. This requires copying NVMM buffers to CPU buffers and takes certain CPU loading.

On Jetson Nano with ‘sudo jetson_clcoks’ being executed, it might be good for 1920x1080p30. But the resolution of your usecase is close to 4K, which shall exceed hardware capability.

Suggest you try pure gstreamer or tegra_multimedia_api.

That makes sense.

Is there any way for me to be able to do opencv processing on the device with the CSI camera? I am trying to make a live-streaming system. I need to do a little bit of panning/zooming and text overlays on the stream

Hi,
There is a sample of gstreamer + cuda::gpuMat. Please refer to
https://devtalk.nvidia.com/default/topic/1022543/jetson-tx2/gstreamer-nvmm-lt-gt-opencv-gpumat/post/5311027/#5311027

Can I use gpuMat with C++ only? or can I run gpuMat with python?

Hi,
Not sure but it looks like python uses appsink to get CPU[video/x-raw] buffer. It may not be supported to use gpuMat.

Ok. Thanks for the help. I’m going to use pure gstreamer as you suggested.

Hi,
Attached a sample for reference. Please follow the steps:
0. Install CUDA, tegra_multimedia_api samples through sdkmanager

  1. Down the script https://github.com/AastaNV/JEP/blob/master/script/install_opencv4.1.1_Jetson.sh
  2. Add the option and execute the script to build OCV4.1.1
-D OPENCV_GENERATE_PKGCONFIG=YES
  1. Execute
$ sudo ldconfig -v
  1. Build and run the sample
~/gst_cv_gpumat$ CUDA_VER=10.0 make
Compiling: gst_cv_gpumat.cpp
g++ -I/usr/src/tegra_multimedia_api/include -I/usr/local/cuda-10.0/include `pkg-config --cflags gstreamer-1.0 opencv4)` -c gst_cv_gpumat.cpp -o gst_cv_gpumat.o
Linking: gst_cv_gpumat
g++ -o gst_cv_gpumat gst_cv_gpumat.o -I/usr/src/tegra_multimedia_api/include -I/usr/local/cuda-10.0/include `pkg-config --cflags gstreamer-1.0 opencv4)` -Wall -std=c++11 -L/usr/lib/aarch64-linux-gnu/tegra/ -lEGL -lGLESv2 -L/usr/lib/aarch64-linux-gnu/tegra/ -lcuda -lnvbuf_utils -L/usr/local/cuda-10.0/lib64/ -lcudart  `pkg-config --libs gstreamer-1.0 opencv4`
~/gst_cv_gpumat$ ./gst_cv_gpumat

gst_cv_gpumat.zip (3.05 KB)

1 Like
Improving GSTreamer and OpenCV performance in Nvidia Jetson TX2
How can I get gpu memory buffer from gstreamer?
NvMMLiteOpen : Block : BlockType = 261 , appsink never preroll
Is busy CPU usage when jetson nano hardware decoding is doing task?
Can’t find library file for cudaEGL.h
GPU Acceleration Support for OpenCV Gstreamer Pipeline
Leopard Imaging CSI Cameras and Aruco detection through OpenCV not using CUDA
Low-Latency CSI Camera Stream
Some question about jetson nano/xavier-nx and deep stream
Is there a way to crop an image with gpu
OpenCV - gstreamer use 100% CPU and 0% GPU
What is the meaning of "memory:NVMM"?
Best widgets API for use with XAVIER / CUDA / OpenCV (e.g. Qt, GTK)
Can gstreamer and VideoCapture share the same image data memory?
Help needed: RTSP stream from raspberry pi to jetson nano
How to get a Cuda GpuMat into Gstreamer?
How can I transform CUeglFrame to OpenCV Mat object?
NV Multimedia API with OpenCV
NVIDIA Gstreamer nvvidconv question
How to reduce CPU load factor in opencv + gstreamer
Gstreamer udp decoding pipeline in Jetson TX2
Nvivafilter: different input and output buffers
Nvvidconv colorspace conversion difficulties
Gstreamer CUDA Implementation Low FPS, cudaDeviceSynchronize Load
Retrieve buffer from gstreamer flow when using video/x-raw(memory:NVMM)
Industry camera and opencv program on TX2
Issues decoding RTSP stream using nvv4l2decoder with Jetpack 4.4
Getting frames for mipi camera for further processing
LibArgus EGLStream to nvivafilter

Hi @DaneLLL
Is this link use pure gstreamer for decoding? Don’t have this code bottleneck of gstreamer+opencv(copying NVMM buffer to CPU buffer)? I see in this code in lines 123,125 use numpy and opencv lib, Aren’t these lines of code bottleneck like gstreamer+opencv?

and tegra_multimedia_api has python code api?

Hi @LoveNvidia
Your questions look different from this topic. For clearness, please make a new post.

The SH link is not working. Anyway you can provide new instructions for jetpack 4.4?
I am having similar issues, the Gstreamer is using only CPU.
OpenCV 4.4.0

Hi,
Please check this link:

It is cuda 10.2 in JP4.4, so please set CUDA_VER=10.2

License for attached code?

Hi

Do you mean you have license concern about using the script?

Sorry for not being clear. I noticed gst_cv_gpumat.cpp had no license header, so I was wondering what the license was. My understanding is without such an explicit license, it’s “all rights reserved” in which case I can’t use it. There is a permissive license on the Makefile, but to me it’s not explicit this applies to the .cpp.

@DaneLLL I modified the code above for my camera, but I see poorer latency under GPUmat usage than pure CPU. I have a separate post containing the details here Gstreamer CUDA Implementation Low FPS, cudaDeviceSynchronize Load

If you had some time to take a look I would appreciate it.

bump? @DaneLLL

Working on something RN and an answer on this would be very helpful.

Bump? Very much appreciate clarification on the license for gst_cv_gpumat.cpp. I’m using the Argus/MMAPI example code instead at the moment, but that’s more work.

Hi,
Except the part of using cuda filters in OpenCV, the permission is same as jetson_multimedia_api samples:

/*
 * Copyright (c) 2016-2020, NVIDIA CORPORATION. All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 *  * Redistributions of source code must retain the above copyright
 *    notice, this list of conditions and the following disclaimer.
 *  * Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the distribution.
 *  * Neither the name of NVIDIA CORPORATION nor the names of its
 *    contributors may be used to endorse or promote products derived
 *    from this software without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
 * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
 * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
 * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
 * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
 * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */

In the sample it utilizes cuda filter in OpenCV for demonstration purpose. For other purpose, you probably need to check the license, too.

1 Like

Thanks, @DaneLLL. I will use this license if I use this code!