Question about video decode in Jetson tx2

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

Hi,
I want to use the part of hard-decode in tx2. Therefore, I choose to use opencv with gstreamer plugin to decode video file. There is my question.

I run the following order in Terminal. Everything is fine. I can see the parsed video is playing.

gst-launch-1.0 filesrc location=test.mp4 ! qtdemux name=demux demux.video_0 ! queue ! h264parse ! omxh264dec ! nveglglessink -e

However, when I type this order into opencv, like below, I only can see the first frame. It seems that the video stream stucks. Is my order right? Thanks in advance.

std::string gstStr = "filesrc location=test.mp4 ! qtdemux name=demux demux.video_0 ! queue ! h264parse ! omxh264dec ! nveglglessink -e ! appsink"
cv::VideoCapture videoCap(gstStr);
if(! videoCap.isOpened())
{
    std::cout<<" read video failed !"<<std::endl;
    return -1;
}
while(1)
{
    videoCap >> frame;
    ++idx; 
    cv::imshow("test", frame);
    if(cv::waitKey(1) == 'q')
        break;
}
2

Hi,
You may try

filesrc location=test.mp4 ! qtdemux name=demux demux.video_0 ! queue ! h264parse ! omxh264dec ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink

From OpenCV 3.3, I420 should also work:

filesrc location=test.mp4 ! qtdemux name=demux demux.video_0 ! queue ! h264parse ! omxh264dec ! nvvidconv ! video/x-raw, format=(string)I420 ! appsink

https://devtalk.nvidia.com/default/topic/1024245/jetson-tx2/opencv-3-3-and-integrated-camera-problems-/post/5210735/#5210735

3

Thanks for quick reply! It works as your suggestion does. However, I’ve found another question.

When I use the following order, my decode speed is 25 fps.

filesrc location=test.mp4 ! qtdemux name=demux demux.video_0 ! queue ! h264parse ! omxh264dec ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink

When I use the usual order in opencv to decode video, my decode speed is 146 fps.

cv::VideoCapture videoCap("test.mp4");

It’s so weird, since the first approach should get a larger speed theoretically. I check the activity of gpu in both conditions by using sudo tegrastates. And gpu is activated in both conditions.

The opencv used is the original opencv in tx2, which version is 3.3.1. And the jetpack in tx2 is 4.2.

4

I finally find where the problem is. The opencv used here was compiled with the support of both gstreamer and ffmpeg.
When I use the usual order videoCap(“test.mp4”) to decode, ffmpeg is activated.

So, can I draw a conclusion that the capacity of ffmpeg is more powerful than gstreamer in video hard-decode in tx2 ? If so, is there any material to use ffmpeg in hard-decode?

Thanks in advance !

5

Hi,
Most GStreamer sink elements synchronize the stream using timestamps unless configured otherwise. Please try adding the option sync=false to check if it improves your framerate:

filesrc location=test.mp4 ! qtdemux name=demux demux.video_0 ! queue ! h264parse ! omxh264dec ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink sync=false

You can also improve performance by adding queue elements to separate sections of the pipeline that use CPU, in order to execute them on separate threads. On my personal experience, the element that limits GStreamer performance the most on TX2 is videoconvert. Try adding queues before and after the element:

filesrc location=test.mp4 ! qtdemux name=demux demux.video_0 ! queue ! h264parse ! omxh264dec ! nvvidconv ! video/x-raw, format=(string)BGRx ! queue ! videoconvert ! queue ! video/x-raw, format=(string)BGR ! appsink sync=false

Supporting RGBx on you application would allow you to remove the videoconvert and further improve performance.

6

Execellent ! Both methods mentioned above can improve the performance in tx2. Those are great ways to do further improvement.
I really want to know where I can find such material and what the differences between RGBx and RGB are.
Many thanks !

7

Hi,

We are an embedded software company that works a lot with GStreamer and NVIDIA embedded platforms. Most of the knowledge is acquired by experience. However, we document our findings on a developer wiki:

https://developer.ridgerun.com/wiki/index.php?title=Main_Page

We also provide an open-source GStreamer profiler named GstShark. This tool can help you determine which element is limiting performance:

https://developer.ridgerun.com/wiki/index.php?title=GstShark

Basically RGBx is more efficient for memory access, since each pixel is stored in a word (32b) instead of 3/4 of a word. You can use it as RGB, just make sure to change your code to ignore the “x”.