Seeking is not frame-exact when using nvv4l2decoder

Please provide complete information as applicable to your setup.

• Hardware Platform: GPU
• DeepStream Version: 6.1
• TensorRT Version:
• NVIDIA GPU Driver Version: 510.47.03
• Issue Type: Bug
• How to reproduce the issue? See description below.

To make it possible to run inference for a specific section of a media file we are using seeking as described in the GStreamer Pipeline Manipulation Documentation in the section Play a section of a media file. When using the nvv4l2decoder the result of the seek is always one frame off. This wouldn’t be a big issue for us if we could compensate for the error, but to make matters worse the segment event sent is also one frame off, making the calculated streaming time one frame before the actual streaming time. This behaviour occurs both for H264 and H265 and we have tested with Deepstream 6.0, 6.0.1 and 6.1 with the same result. If we switch from nvv4l2decoder to avdec_h264 (or avdec_h265 for H265) the seeking is frame exact. Doing so is however not a good option for us as we would lose the hardware accelerated decoding and no longer be utilizing NVENC with that work-around.

The way we have verified that seeking is actually one frame off is by using test media generated with ffmpeg with the time stamp written in all frames.

We have a repro for this issue that we can send if needed. The code is however a couple of 100 lines even though we made out best effort to make it as short as possible for demonstrating this issue.

We are looking into this issue.

Can you provide your reproducing steps?

Thanks! I have attached a test program which performs a seek to an interval of the input media and jpeg encodes all frames in that interval. The test program also writes GStreamer debug graphs for all state changes if the environment variable GST_DEBUG_DUMP_DOT_DIR is set.

Generate 25 FPS H264 test media with a timestamp in each frame

ffmpeg -y -f lavfi -i testsrc=duration=300:decimals=3:rate=25:size=1280x720 -pix_fmt yuv420p 5_minutes_h264.mp4

Build the test program

g++ -std=c++17 cut_frames.cpp -o cut_frames `pkg-config --cflags --libs gstreamer-1.0`

Run a test using avdec_h264 with the expected result of frames at 5.00s and 5.04s

./cut_frames --verbose 5000 5040 <full path to test file>

The result is that the expected frames are are written to the result directory (/result).

Run the same test using nvv4l2decoder

./cut_frames --verbose --use-nvv4l2decoder 5000 5040 <full path to test file>

The result is that frames at 4.96s and 5.00s are written to the result directory (/result). The segment event and buffer PTS of the frames in this test are identical to the test with avdec_h264, but the frames returned are 0.04 seconds earlier than reported, so the result is off by one frame.

When isolating this behaviour to the attached test program we have started to suspect that the explanation for this result might not only be due to how nvv4l2decoder handles timestamps and seeking but instead is caused by a combination of events and messages between different GStreamer elements. When using uridecodebin the seeking with avdec_h264 returns the expected frames while the exact same pipeline with avdec_h264 replaced by nvv4l2decoder does not. We have however also done testing where we did not use uridecodebin and instead created the required elements manually. The initial results are that nvv4ldecoder actually gives the correct result with that pipeline.

cut_frames.cpp (12.5 KB)

We are investigating the issue. Will be back once there is any progress

1 Like