First frame decoding taking longer time than all other frames

[bibeksharma.sharma]

Hi, I am working with Jetson Xavier AGX for video processing purposes. I have created a pipeline that would decode a given sample.mp4 file and send it to a fakesink to observe the latency I would get. I have used the GST_SHARK Interlatency tracer that gives processing time of each buffer from source to each element of the pipeline. The pipeline is as follows:
GST_DEBUG=" GST_TRACER:7" GST_TRACERS=“interlatency” gst-launch-1.0 filesrc location=sample.mp4 num-buffers=8 ! qtdemux ! h264parse ! nvv4l2h264decoder enable-max-peroformance=1 ! fakesink

I have used the number of buffers to be 8 as it results in 2 video frames being decoded. The first video frame decoding takes around 112.9mS as observed in the below screenshot where we can see the interlatency between the filesrc and fakesink to be 00.112943095 seconds. However, we can check that the second frame only needs 3.947mS as observed from the screenshot below.

So my question is why am i getting a processing time so large for the first frame where as for the second frame the processing time is extremely low. What are the ways in which I will be able to get the processing time of the first frame to be lower?

[DaneLLL]

Hi,
Please also set this property:

  disable-dpb         : Set to disable DPB buffer for low latency
                        flags: readable, writable
                        Boolean. Default: false

It shall reduce certain latency but some steam may not work in disable-dpb=true. Please set it and give it a try.

[bibeksharma.sharma]

Even after setting the disable-dpb to true there is no change in the processing time of the first frame and still is 112mS. What is the main reason that is causing the first frame to take so long ?

[DaneLLL]

Hi,
It would take some time in initializing decoder. And depending on the stream, need to keep more than one frame for decoding initial frames. For example, if the stream is encoded in IDR P P P …, decoder has to decode the first IDR and keep it for decoding the second P frame. Also decoding IDR frames takes more time than P frames.

[bibeksharma.sharma]

I see when there is no operation going on the decoder and the encoder engines are OFFLINE. Is there a way to initialize the decoder/encoder engine to be ONLINE always so, I do not have to spend the time initializing the decoder/encoder, hence saving some time that would reduce the latency of my pipeline.

[DaneLLL]

Hi,
We test the use-cases that encoder/decoder is initialized and destroyed for each encoding/decoding tasks. If encoder/decoder is not destroyed and re-initialized for changing from this task to next task, there might be potential issues, such as memory leak. Would suggest do initialization/termination for each task.

[bibeksharma.sharma]

Okay it makes sense. But I want to try and see what is the latency if I have the decoder initialized. What are the steps to keep the decoder/encoder initialized ? Is there any documentation that would help me accomplish the same ?

[DaneLLL]

Hi,
The implementation in gst-v4l2 is based on jetson_multimedia_api. So you can check the samples:

/usr/src/jetson_multimedia_api/samples/00_video_decode

And try to port to gst-v4l2.

In 00_video_decode, once the resolution changes, the event is sent V4L2_EVENT_RESOLUTION_CHANGE and query_and_set_capture() is called. Ideally it should work if the same function is applied to gst-v4l2.

Or you may consider use jetson_multimedia_api for development.