Ask for help about the delay of video hardware decoding

Autonomous Machines Jetson & Embedded Systems Jetson AGX Orin
,
1

I used the following Gstreamer pipeline on Orin to perform hard and soft decoding of RTSP separately, and found that the delay is about the same, around 260ms; And I tried to use ffmpeg decoding on x86 notebook win10 system, and the result was that the delay was only 140ms, which is a big difference.

soft decoding：gst-launch-1.0 rtspsrc location=rtsp://192.9.200.94:554/live.sdp protocols=udp latency=100 ! rtph264depay ! h264parse ! avdec_h264 ! xvimagesink
hard decoding ：gst-launch-1.0 rtspsrc location=rtsp://192.9.200.94:554/live.sdp protocols=udp latency=100 ! rtph264depay ! h264parse ! nvv4l2decoder ! nv3dsink -e

Question: Why is the delay of hard and soft decoding similar on Orin? How to optimize the hardware decoding to reduce the latency to the same level as the x86 notebook?

3

Hi,
You may set latency=0 to rtspsrc for a try. And can set xvimagesink sync=0 or nv3dsink sync=0 to disable synchronization mechanism of gstreamer.

Can also try this setup for comparison:

Jetson AGX Orin FAQ
Q: Is there an example for running UDP streaming?

1 Like
4

Setting xvimagesink sync=0 reduces the soft decoding delay to about 170ms, but setting nv3dsink sync=0 has no significant change

5

Hi,
Please set disable-dpb to nvv4l2decoder and check again.

7

When the above pipeline is opened with opencv on orin, the delay is very large (tens of seconds), and it will accumulate over time, but this does not happen on an x86 notebook. What should I do?

include “opencv2/imgcodecs.hpp”
include “opencv2/highgui.hpp”
include “opencv2/stitching.hpp”

include

using namespace std;
using namespace cv;

int main(int argc, char* argv)
{

    std::stringstream rtsp_gststr;
    std::string url_ = "rtsp://192.9.200.94:554/live.sdp";
    size_t latency = 20;
    size_t frame_width = 1920;
    size_t frame_height = 1080;
    size_t framerate  = 50;


    rtsp_gststr << "rtspsrc location=" << url_ << " latency=" << latency
     << " ! rtph264depay ! video/x-h264, clock-rate=90000" << ", framerate=" << framerate << "/1" 
     << " ! h264parse ! avdec_h264  ! videoconvert ! video/x-raw, format=BGR ! appsink";

    std::cout << rtsp_gststr.str() << std::endl;
    cv::VideoCapture capture_;
    // cv::VideoCapture cap = cv::VideoCapture(rtsp_gststr.str(), cv::CAP_GSTREAMER);
    bool ret = capture_.open(rtsp_gststr.str(), cv::CAP_GSTREAMER);        
    // bool ret = capture_.open(url_);
    if(!ret){
        std::cout << "open failed" << std::endl;
        return -1;
    }
     

    for(;;)
    {
        cv::Mat image;
        capture_ >> image;
        if (image.empty()){
            std::cout << "image.empty";
            continue;
        }
        cv::imshow("image", image);
        cv::waitKey(1);

    }

return 0;

}

8

Hi @1443660560 , you could try using the nvidia gstreamer elements for decoding and video convert instead of avdec_h264 ! videoconvert you could use nvv4l2decoder ! nvvideoconvert. Also you could add a queue with the properties max-size-buffer=1 and leaky=2 before the appsink