Multi-stream rtsp GStreamer + Opencv

Autonomous Machines Jetson & Embedded Systems Jetson Nano
, ,
1

I want to decode multi-stream rtsp with h264 HW of jetson nano and opencv.
I use jetpack 4.2.2 and opencv-3.4.6 build from source with gstreamer support.

when I use this elements, I get correct results, but the problem is that memory usage gradually increased and I get reasoable cpu usage and Actived NVDEC.

    gstream_elemets = (
        'rtspsrc location=RTSP latency=300 !'
        'rtph264depay !'
        'h264parse !' 
        'omxh264dec enable-max-performance=1 enable-low-outbuffer=1 !'
        'nvvidconv ! video/x-raw, format=(string)BGRx!'
        'videoconvert !'
        'appsink').

cv2.VideoCapture(gstream_elemets, cv2.CAP_GSTREAMER)

The problem caused bucause I input rtsp frame_rate = 25 and I limited to 5-1 FPS in the while loop like this :

While cap.isOpened():
          framer = cap.read()[1]
          sleep(0.1)

I decide to define frame_rate in the gstreamer elemets like this whitout sleep in while loop:

gstream_elemets = (
    'rtspsrc location=RTSP latency=300 !'
    'rtph264depay !'
    'h264parse !' 
    'omxh264dec enable-max-performance=1 enable-low-outbuffer=1 !'
    'nvvidconv ! video/x-raw, format=(string)BGRx!'
    'videorate ! video/x-raw, framerate=(fraction)10/1'
    'videoconvert !'
    'appsink')
cv2.VideoCapture(gstream_elemets, cv2.CAP_GSTREAMER)

This is also corectly work and ram is fixed but CPU usage is about 2x increased.

My goal is that:
if I input frame_rate is variable like 25, I want to used best pipeline in opencv to get frames in the spesific frame_rate in the output like 10.
at least cpu usage and without memory leakage.

1 Like
5

hi LoveNvidia:
how about set the frame timer ,then select the num you want process?

frame_count = 0
frame_base_num = 2#adjust it to process_rate
While cap.isOpened():

      framer = cap.read()[1]
      frame_count += 1

      if(frame_count % frame_base_num== 0 )
      #to process
6

I used this way and has same problem again. The problem is in the gstreamer, because the rate on input is more than output, and I guess the gstreamer doesn’t release the buffer, and the gradually fill up the buffer.

7

seems correctly. input>output, buffer overflow
try
1)remove enable-low-outbuffer=1
2)maybe only way is to set framerate as actual use like you set, though CPU usage is 2X increased

8

enable-low-outbuffer=1 launch the stream with low memory usage, why I do remove this element?

9

enable-low-outbuffer is property of the gst-omx decoder plugin which is removed will increase the decode buffer, I guess. so maybe can alleviate the frame-rate mismatch(input>output).
for higher cpu usage ,see here

11

In your opinion, the enable-low-outbuffer is like drop frames operation?
If we want to reach from 25 FPS input stream to 5 FPS in output stream. Is it better to use enable-low-outbuffer?
If enable-low-outbuffer is like drop frame operation, so what’s difference between enable-low-outbuffer and queue leaky=2?

12

not such meaning, I guess the enable-low-outbuffer is decreased the internal buffer for decoding, so ,if not set it, maybe more buffers for decoding which will alleviate the frame-rate mismatch, just try ,if no use , the better way is set the same rate for input/output as your experiment

13

Opencv can work with different framerates, but you would have to set these accordingly.
Decreasing framerate is not difficult. You can do both ways:

  1. Use gstreamer plugin videorate:
#include <iostream>
#include <opencv2/core.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/videoio.hpp>

int main(void)
{
 	const char *gst_cap = "videotestsrc ! video/x-raw, format=BGR, width=1280, height=720, framerate=30/1"
                               " ! videorate ! video/x-raw, format=BGR, framerate=5/1"
                               " ! videoconvert ! queue ! video/x-raw, format=BGR ! appsink ";

        cv::VideoCapture cap(gst_cap, cv::CAP_GSTREAMER);
        if( !cap.isOpened() )
        {
            std::cout << "Error: Cv::VideoCapture.open() failed" << std::endl;
            return 1;
        }
	else
	    std::cout << "Cam opened" << std::endl;

    	unsigned int width = cap.get(cv::CAP_PROP_FRAME_WIDTH); 
    	unsigned int height = cap.get(cv::CAP_PROP_FRAME_HEIGHT); 
    	unsigned int pixels = width*height;
        float fps    = cap.get(cv::CAP_PROP_FPS);
        std::cout <<" Frame size : "<<width<<" x "<<height<<", "<<pixels<<" Pixels "<<fps<<" FPS"<<std::endl;

        const char *gst_out = "appsrc ! video/x-raw, format=BGR ! queue"
			      " ! videoconvert ! fpsdisplaysink video-sink=xvimagesink ";

        cv::VideoWriter out(gst_out, cv::CAP_GSTREAMER, 0, fps, cv::Size(width, height));
        if( !out.isOpened() )
        {
            std::cout << "Error: Cv::VideoWriter.open() failed" << std::endl;
            return 2;
        }
	else
	    std::cout << "Writer opened" << std::endl;

   	cv::Mat frame_in(width, height, CV_8UC3);
        for(;;)
        {
		if (!cap.read(frame_in)) {
			std::cout<<"Capture read error"<<std::endl;
			break;
		}
		
  		out.write(frame_in);
		cv::waitKey(1); 	
        }

	cap.release();
        return 0;
}
  1. As suggested by @Jeffli, read N frames and only process last one:
#include <iostream>
#include <opencv2/core.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/videoio.hpp>

int main(void)
{
 	const char *gst_cap = "videotestsrc ! video/x-raw, format=BGR, width=1280, height=720, framerate=30/1"
                               " ! videoconvert ! queue ! video/x-raw, format=BGR ! appsink ";

        cv::VideoCapture cap(gst_cap, cv::CAP_GSTREAMER);
        if( !cap.isOpened() )
        {
            std::cout << "Error: Cv::VideoCapture.open() failed" << std::endl;
            return 1;
        }
	else
	    std::cout << "Cam opened" << std::endl;

    	unsigned int width = cap.get(cv::CAP_PROP_FRAME_WIDTH); 
    	unsigned int height = cap.get(cv::CAP_PROP_FRAME_HEIGHT); 
    	unsigned int pixels = width*height;
        float fps    = cap.get(cv::CAP_PROP_FPS);
        std::cout <<" Frame size : "<<width<<" x "<<height<<", "<<pixels<<" Pixels "<<fps<<" FPS"<<std::endl;

        const char *gst_out = "appsrc ! video/x-raw, format=BGR, framerate=5/1 ! queue"
			      " ! videoconvert ! fpsdisplaysink video-sink=xvimagesink ";

        cv::VideoWriter out(gst_out, cv::CAP_GSTREAMER, 0, 5, cv::Size(width, height));
        if( !out.isOpened() )
        {
            std::cout << "Error: Cv::VideoWriter.open() failed" << std::endl;
            return 2;
        }
	else
	    std::cout << "Writer opened" << std::endl;

   	cv::Mat frame_in(width, height, CV_8UC3);
        for(;;)
        {
		int div=6;
                for(; div>0; --div) {
		     if (!cap.read(frame_in)) {
			  std::cout<<"Capture read error"<<std::endl;
			  break;
		     }
		}
                if(div >0)
                     break;

  		out.write(frame_in);
		cv::waitKey(1); 	
        }

	cap.release();
        return 0;
}
14

If enable-low-outbuffer is decreased the internal buffer for decoding, Is it better set this option to zero?

the better way is set the same rate for input/output as your experiment

for input rate I can’t access for changing to as low-rate as output.

15

Yes you can. Check example 1 just above, using videorate for changing from 30 fps source into 5 fps for opencv.

16

I tested the below gstreamer before, but I don’w know why cpu usage is increased two times.

gstream_elemets = (
‘rtspsrc location=RTSP latency=300 !’
‘rtph264depay !’
‘h264parse !’
‘omxh264dec enable-max-performance=1 enable-low-outbuffer=1 !’
‘nvvidconv ! video/x-raw, format=(string)BGRx!’
‘videorate ! video/x-raw, framerate=(fraction)5/1’
‘videoconvert !’
‘appsink’)
cv2.VideoCapture(gstream_elemets, cv2.CAP_GSTREAMER)

Is it maybe due to I set incorrct order of gstreamer elements, righ?
what’s the rtph264depay? what does it do? Is it needed?
in your opinion, Is it better to use appsink sync=0?