Nano not using GPU with gstreamer/python. Slow FPS, dropped frames

jacobrichards · November 12, 2019, 3:27pm

I am using the a typical pipeline (see below) to feed my Opencv/Python program frames. My CPU is running at 100% , while my GPU is only 25%. Is there anything I can do to put more work on the GPU?

def gstreamer_pipeline(
capture_width=3264,
capture_height=1848,
display_width=3264,
display_height=1848,
framerate=28,
flip_method=0,
):
return (
"nvarguscamerasrc ! "
"video/x-raw(memory:NVMM), "
"width=(int)%d, height=(int)%d, "
"format=(string)NV12, framerate=(fraction)%d/1 ! "
"nvvidconv flip-method=%d ! "
"video/x-raw, width=(int)%d, height=(int)%d, format=(string)BGRx ! "
"videoconvert ! "
“video/x-raw, format=(string)BGR ! appsink”
% (
capture_width,
capture_height,
framerate,
flip_method,
display_width,
display_height,
)
)

DaneLLL · November 13, 2019, 4:24am

Hi,
The optimal-performance solution is to pass NVMM[video/x-raw(memory:NVMM)] buffers in the pipelines. However, with python, it only accepts CPU[video/x-raw,format=BGR] buffers. This requires copying NVMM buffers to CPU buffers and takes certain CPU loading.

On Jetson Nano with ‘sudo jetson_clcoks’ being executed, it might be good for 1920x1080p30. But the resolution of your usecase is close to 4K, which shall exceed hardware capability.

Suggest you try pure gstreamer or tegra_multimedia_api.

jacobrichards · November 13, 2019, 4:44am

That makes sense.

Is there any way for me to be able to do opencv processing on the device with the CSI camera? I am trying to make a live-streaming system. I need to do a little bit of panning/zooming and text overlays on the stream

DaneLLL · November 13, 2019, 5:44am

Hi,
There is a sample of gstreamer + cuda::gpuMat. Please refer to
https://devtalk.nvidia.com/default/topic/1022543/jetson-tx2/gstreamer-nvmm-lt-gt-opencv-gpumat/post/5311027/#5311027

jacobrichards · November 13, 2019, 2:27pm

Can I use gpuMat with C++ only? or can I run gpuMat with python?

DaneLLL · November 14, 2019, 1:56am

Hi,
Not sure but it looks like python uses appsink to get CPU[video/x-raw] buffer. It may not be supported to use gpuMat.

jacobrichards · November 14, 2019, 2:09am

Ok. Thanks for the help. I’m going to use pure gstreamer as you suggested.

DaneLLL · November 21, 2019, 4:21am

Hi,
Attached a sample for reference. Please follow the steps:
0. Install CUDA, tegra_multimedia_api samples through sdkmanager

Down the script [s]https://github.com/AastaNV/JEP/blob/master/script/install_opencv4.1.1_Jetson.sh[/s]
JEP/install_opencv4.5.0_Jetson.sh at master · AastaNV/JEP · GitHub
Add the option and execute the script to build OCV4.1.1

-D OPENCV_GENERATE_PKGCONFIG=YES

Execute

$ sudo ldconfig -v

Build and run the sample

~/gst_cv_gpumat$ CUDA_VER=10.0 make
Compiling: gst_cv_gpumat.cpp
g++ -I/usr/src/tegra_multimedia_api/include -I/usr/local/cuda-10.0/include `pkg-config --cflags gstreamer-1.0 opencv4)` -c gst_cv_gpumat.cpp -o gst_cv_gpumat.o
Linking: gst_cv_gpumat
g++ -o gst_cv_gpumat gst_cv_gpumat.o -I/usr/src/tegra_multimedia_api/include -I/usr/local/cuda-10.0/include `pkg-config --cflags gstreamer-1.0 opencv4)` -Wall -std=c++11 -L/usr/lib/aarch64-linux-gnu/tegra/ -lEGL -lGLESv2 -L/usr/lib/aarch64-linux-gnu/tegra/ -lcuda -lnvbuf_utils -L/usr/local/cuda-10.0/lib64/ -lcudart  `pkg-config --libs gstreamer-1.0 opencv4`
~/gst_cv_gpumat$ ./gst_cv_gpumat

gst_cv_gpumat.zip (3.05 KB)

LoveNvidia · August 4, 2020, 3:54pm

Hi @DaneLLL
Is this link use pure gstreamer for decoding? Don’t have this code bottleneck of gstreamer+opencv(copying NVMM buffer to CPU buffer)? I see in this code in lines 123,125 use numpy and opencv lib, Aren’t these lines of code bottleneck like gstreamer+opencv?

and tegra_multimedia_api has python code api?

DaneLLL · August 4, 2020, 11:37pm

Hi @LoveNvidia
Your questions look different from this topic. For clearness, please make a new post.

gui.steffen · August 26, 2020, 9:04am

The SH link is not working. Anyway you can provide new instructions for jetpack 4.4?
I am having similar issues, the Gstreamer is using only CPU.
OpenCV 4.4.0

DaneLLL · August 27, 2020, 5:11am

Hi,
Please check this link:
https://github.com/AastaNV/JEP/blob/master/script/install_opencv4.3.0_Jetson.sh

It is cuda 10.2 in JP4.4, so please set CUDA_VER=10.2

mdegans · February 9, 2021, 2:15am

License for attached code?

DaneLLL · February 9, 2021, 7:57am

Hi

Do you mean you have license concern about using the script?

mdegans · February 9, 2021, 7:05pm

Sorry for not being clear. I noticed gst_cv_gpumat.cpp had no license header, so I was wondering what the license was. My understanding is without such an explicit license, it’s “all rights reserved” in which case I can’t use it. There is a permissive license on the Makefile, but to me it’s not explicit this applies to the .cpp.

spacebaseone · February 9, 2021, 10:01pm

@DaneLLL I modified the code above for my camera, but I see poorer latency under GPUmat usage than pure CPU. I have a separate post containing the details here Gstreamer CUDA Implementation Low FPS, cudaDeviceSynchronize Load

If you had some time to take a look I would appreciate it.

mdegans · February 11, 2021, 8:31pm

bump? @DaneLLL

Working on something RN and an answer on this would be very helpful.

mdegans · February 14, 2021, 9:45pm

Bump? Very much appreciate clarification on the license for gst_cv_gpumat.cpp. I’m using the Argus/MMAPI example code instead at the moment, but that’s more work.

DaneLLL · February 17, 2021, 6:02am

Hi,
Except the part of using cuda filters in OpenCV, the permission is same as jetson_multimedia_api samples:

/*
 * Copyright (c) 2016-2020, NVIDIA CORPORATION. All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 *  * Redistributions of source code must retain the above copyright
 *    notice, this list of conditions and the following disclaimer.
 *  * Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the distribution.
 *  * Neither the name of NVIDIA CORPORATION nor the names of its
 *    contributors may be used to endorse or promote products derived
 *    from this software without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
 * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
 * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
 * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
 * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
 * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */

In the sample it utilizes cuda filter in OpenCV for demonstration purpose. For other purpose, you probably need to check the license, too.

mdegans · February 20, 2021, 6:26am

Thanks, @DaneLLL. I will use this license if I use this code!