Incorrect timestamping during decoding vp9 video by nvv4l2decoder

For some reason, sometimes, video encoded with using vp9 format can not be
propertly decoded by nvv4l2decoder. If be exact: there is a problem with frame
timestamping. For example: imagine that we decode one video frame-by-frame
with two gst pipelines - first uses vp9dec, second one - nvv4l2decoder. We
expect that timestamp for same frames will be the same. But it is not always
what we see. Sometimes we can see that frame, timestamp of which we know
exactly, is timestamped by time, that should have previous or next frame in
the video.

I’m not sure what exactly causes this problem, but I was able to reproduce this
issue only with using irregular timestamps (during reading video from cameras
we have not perfect framerate) and with duplicating some frames (one duplicate
in 2 seconds in average).

For reproducing this problem I created minimal example (see attached archive), that simulate this
problem. At first run it creates a video for the check, that can be used
several times. I am not sure that the problem will be reproduced with newely
generated video, so here you can also find a sample directory, that contains
video for reproducing this problem. After video generation you can run the
program again and it will decode the video (by HW_PIPELINE define you can
control what decoder to use) and will print each frame PTS and frame order
number. Normally, both outputs, from cpu and gpu pipelines, must be the same,
but it is not always true. Moreover, even output of gpu pipeline for same video
can be different from time to time. Just check it by diff
min-ex.zip (296.5 KB)

Hi,
Please share which Jetpack release you use. And does it work if you run the pipeline like:

videotestsrc is-live=1 ! nvvidconv ! nvv4l2vp9enc ! matroskamux ! filesink location=vp9.mkv

Seems like you cannot use hardware vp9 encoder. Please try with videotestsrc and see if it works.

Here is output of jetson_release command on device that I use:

 - NVIDIA Jetson Xavier NX (Developer Kit Version)
   * Jetpack 4.4.1 [L4T 32.4.4]
   * NV Power Mode: MODE_10W_2CORE - Type: 3
   * jetson_stats.service: active
 - Libraries:
   * CUDA: 10.2.89
   * cuDNN: 8.0.0.180
   * TensorRT: 7.1.3.0
   * Visionworks: 1.6.0.501
   * OpenCV: 4.1.1 compiled CUDA: NO
   * VPI: 0.4.4
   * Vulkan: 1.2.70

According to videotestsrc - I actually tried to reproduce this problem with videotestsrc, but I was unable to do it. It works stable with videotestsrc. Probably because there are no duplicates and framerate is stable. As I mentioned before: only if framerate is fluctuating and there are some duplicated frames - only in this conditions I was able to reproduce this problem. Maybe it can happen in different circumstances as well, but I was not able reproduce it.
According to using vp9 encoder - encoder works well (on production we use nvv encoder), problem is definitely in decoder. Cpu decoder (vp9dec) always works well - it produce exactly same output over same video, but nvv decoder - for case that I provided it can be different from time to time over same video. So I absolutely sure that problem somewhere in nvv decoder and somehow connected to frame timestamps

Hi,
The latest Jetpack 4 release is 4.6.3. Are you able to upgrade to the release and try again? We also have Jetpack 5.1 release with kernel 5.10. May also consider use 5.1.

Unfortunately currently we are unable to migrate our devices to newest version. Also, are you sure that this problem doesn’t appear in new versions?

Hi,
Please try to save vp9 stream and decode with jetson_multimedia_api sample:

/usr/src/jetson_multimedia_api/samples/00_video_decode/

Would like to clarify if it is also present in jetson_multimedia_api sample. Or specific to gstreamer.

Unfortunately I am unable to same raw vp9 stream to file. When I save it, for example by demuxing video from my minimal example by:

gst-launch-1.0 filesrc location=video.mkv ! matroskademux ! filesink location=out.vp9

then I can not decode the output file with vp9dec (unuspported bitstream) or with nvv decoder (crashes). Could you provide example how to save raw vp9 stream to file?

Hi,
Please use ffmpeg to mux it into ivf file:

$ ffmpeg -i test.mkv -c copy test.ivf

And decode it through 00_video_decode :

00_video_decode$ ./video_decode VP9 test.ivf

1 Like

Doesn’t work, ffmpeg throw error about non monotonically increasing dts to muxer - what makes sense, because there is irregular timestamps

Hi,
The command works if we generate the mkv file with videotestsrc. Do you set PTS to each GstBuffer in appsrc? Probably the timestamps are not set correctly.

but I explained it above, in the issue: timestamps produced by our cameras are non monotonic. Could you take a look on minimal example that I provided?

Hi,
We would expect camera source to generate frames in order. If the source generates frames not in continuous order, it is possible the encoder does not work properly. For this use-case, one possible solution is to buffer the frames in appsrc, and feed frames to encoder in continuous order.

Frames are always in continuous order, if I understand it correctly (next frame has timestamp not less then previous). But frame intervals are not always same (non monotonic). Also there can be some duplicated frames

I have looked deeper into: why we have duplicated timestamps in our pipeline - initially I thought that there was a problem with camera, but I was wrong. Problem is in nvv encoder (vp9) - there is some mess with PTS and DTS in pipeline after this element. Unfortunately I can’t provide reproducable example, because for some reason I can reproduce it only when use our cameras on release device. But I see that the problem is connected to changing PTS in gst buffers. I also found very similar issue: PTS values altered by nvv4l2h64enc - we do the same thing - change PTS - and have same problem - some mess with PTS/DTS after nvv encoder. So I assume that there is an same issue behind it

Hi,
Please check gst_v4l2_video_dec_find_nearest_frame() in gstv4l2videodec.c. It is not expected if there are two frames with identical timestamp, so it is possible the function does not pick frames in order. For encoding, please check if you can encode two frames into different timestamps when the timestamp is identical. Or for decoding, you may customize gst_v4l2_video_dec_find_nearest_frame() to handle the identical timestamp.

As I mentioned in previous message, with encoder there is a different problem. There are no duplicated frames during encoding, but during decoding result will look like we have them. Please check this issue PTS values altered by nvv4l2h64enc - unfortunately I can not provide reproducable example of this problem, but what we observe during encoding by nvv4l2vp9enc is the same