For some reason, sometimes, video encoded with using vp9 format can not be
propertly decoded by nvv4l2decoder. If be exact: there is a problem with frame
timestamping. For example: imagine that we decode one video frame-by-frame
with two gst pipelines - first uses vp9dec, second one - nvv4l2decoder. We
expect that timestamp for same frames will be the same. But it is not always
what we see. Sometimes we can see that frame, timestamp of which we know
exactly, is timestamped by time, that should have previous or next frame in
I’m not sure what exactly causes this problem, but I was able to reproduce this
issue only with using irregular timestamps (during reading video from cameras
we have not perfect framerate) and with duplicating some frames (one duplicate
in 2 seconds in average).
For reproducing this problem I created minimal example (see attached archive), that simulate this
problem. At first run it creates a video for the check, that can be used
several times. I am not sure that the problem will be reproduced with newely
generated video, so here you can also find a sample directory, that contains
video for reproducing this problem. After video generation you can run the
program again and it will decode the video (by HW_PIPELINE define you can
control what decoder to use) and will print each frame PTS and frame order
number. Normally, both outputs, from cpu and gpu pipelines, must be the same,
but it is not always true. Moreover, even output of gpu pipeline for same video
can be different from time to time. Just check it by diff min-ex.zip (296.5 KB)
According to videotestsrc - I actually tried to reproduce this problem with videotestsrc, but I was unable to do it. It works stable with videotestsrc. Probably because there are no duplicates and framerate is stable. As I mentioned before: only if framerate is fluctuating and there are some duplicated frames - only in this conditions I was able to reproduce this problem. Maybe it can happen in different circumstances as well, but I was not able reproduce it.
According to using vp9 encoder - encoder works well (on production we use nvv encoder), problem is definitely in decoder. Cpu decoder (vp9dec) always works well - it produce exactly same output over same video, but nvv decoder - for case that I provided it can be different from time to time over same video. So I absolutely sure that problem somewhere in nvv decoder and somehow connected to frame timestamps
We would expect camera source to generate frames in order. If the source generates frames not in continuous order, it is possible the encoder does not work properly. For this use-case, one possible solution is to buffer the frames in appsrc, and feed frames to encoder in continuous order.
Frames are always in continuous order, if I understand it correctly (next frame has timestamp not less then previous). But frame intervals are not always same (non monotonic). Also there can be some duplicated frames
I have looked deeper into: why we have duplicated timestamps in our pipeline - initially I thought that there was a problem with camera, but I was wrong. Problem is in nvv encoder (vp9) - there is some mess with PTS and DTS in pipeline after this element. Unfortunately I can’t provide reproducable example, because for some reason I can reproduce it only when use our cameras on release device. But I see that the problem is connected to changing PTS in gst buffers. I also found very similar issue: PTS values altered by nvv4l2h64enc - we do the same thing - change PTS - and have same problem - some mess with PTS/DTS after nvv encoder. So I assume that there is an same issue behind it
Please check gst_v4l2_video_dec_find_nearest_frame() in gstv4l2videodec.c. It is not expected if there are two frames with identical timestamp, so it is possible the function does not pick frames in order. For encoding, please check if you can encode two frames into different timestamps when the timestamp is identical. Or for decoding, you may customize gst_v4l2_video_dec_find_nearest_frame() to handle the identical timestamp.
As I mentioned in previous message, with encoder there is a different problem. There are no duplicated frames during encoding, but during decoding result will look like we have them. Please check this issue PTS values altered by nvv4l2h64enc - unfortunately I can not provide reproducable example of this problem, but what we observe during encoding by nvv4l2vp9enc is the same