OpenCV VideoWriter to nvdrmvideosink fps drops

Robotics & Edge Computing Jetson & Embedded Systems Jetson Nano
1

I am trying to develope a “live-video-stabilizer” application on the jetson nano.
The requirements are, that I need to push 720p50 on the HDMI port in “console mode”.

So for letting the nano boot without the graphical target, i set

systemctl set-default multi-user.target

which just works fine.

Trying out without stabilization, I used the following g-streamer-1.0 pipeline:

gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=50/1' ! nvdrmvideosink

which is also working as intended.

For implementing the stabilization, I want to use OpenCV (Version 4).

For the first part, I wanted to test if my gstreamer pipeline is able to play using the OpenCV Classes VideoCapture and VideoWriter.

My code is the following:

#include <iostream>
#include <chrono>
#include <opencv2/opencv.hpp>
#include <opencv2/videoio.hpp>
#include <opencv2/highgui.hpp>

#include <gst/gst.h>
#include <glib.h>

using namespace std;
using namespace cv;

int main()
{
        const char* gst_in_pipe = "nvarguscamerasrc ! video/x-raw(memory:NVMM), width=1280, height=720, format=NV12, framerate=50/1 ! nvvidconv ! video/x-raw, format=I420 ! appsink";
        VideoCapture cap(gst_in_pipe, CAP_GSTREAMER);

        const char* gst_out_pipe = "appsrc ! video/x-raw, format=BGR, framerate=50/1, width=1280, height=720 ! videoconvert ! video/x-raw, format=BGRx ! nvvidconv ! video/x-raw(memory:NVMM), format=NV12 ! nvdrmvideosink";
        VideoWriter out(gst_out_pipe, CAP_GSTREAMER, 0, 50, Size(1280,720));

        if(!cap.isOpened()) {
                cout << "Failed to open camera." << endl;
                return (-1);
        }
        Mat frame, bgr;
        for(;;) {
                auto start = chrono::high_resolution_clock::now();
                cap >> frame;
                cvtColor(frame, bgr, COLOR_YUV2BGR_I420);
                out << bgr;
                waitKey(1);
                auto finish = chrono::high_resolution_clock::now();
                chrono::duration<double> elapsed = finish - start;
                cout << "Elapsed time: " << elapsed.count() << "s\n";
        }

        cap.release();
        out.release();
        return 0;
}

When I start the application, the first ~2 seconds are working fine, but after that the framerate drops to ~(0.5 - 1) fps instead of the desired 50fps.

Using jtop (the nano is in MAX mode) when my application runs, it shows that all cpu cores are running at ~25-35%, the GPU is even at 0%.

Anyone has a clue what causes this behavior?

2

Hi,
The code has two pipelines and involves extra memcpy(). You may run sudo jetson_clocks and see if you can achieve acceptable performance.

An optimal solution is to run single pipeline and add prob callback. Please take a look at
https://devtalk.nvidia.com/default/topic/1066465/jetson-nano/nano-not-using-gpu-with-gstreamer-python-slow-fps-dropped-frames/post/5403975/#5403975