Incorrect DTS during encoding via nvv encoder

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
,
1

I found out that there is some mess with DTS during encoding not perfect framerate via nvv encoder. For example, for perfect 10 fps we expect frame timestamps like:

0.0
0.1
0.2
0.3
0.4
0.5
...

But because in our case frames are produced and timestamped by camera they have not perfect framerate. Also framerate of our camera is higher then we need (imagine that it is 25 fps instead of 10), so we use videorate drop-only=TRUE for have 10 or less frames per second. As result we have timestamps like here:

0.99
1.08
1.17
1.26
1.35
1.44
1.63
1.72
1.81
1.9

Between 1.44 and 1.63 is bigger difference then we can expect, but it is absolutely fine (it can be caused by some delay in camera or videorate dropped some extra frame here). if we compare this values to prefect framerate, then we will see next picture:

1.0  0.99
1.1  1.08
1.2  1.17
1.3  1.26
1.4  1.35
1.5  1.44
1.6  1.63

And nvv encoder fails with value 1.44 - this value is closer to 1.4 then 1.35, so nvv encoder pushes frame that timestamped with 1.44 instead of frame that timestamped with 1.35. Moreover, it uses 1.35 as DTS for frame that has PTS 1.44. See logs bellow (identity0 is before encoder and __idenitity1 is after):

/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain   ******* (identity0:sink) (2080000 bytes, dts: none, pts: 0:00:01.350000000, duration: none, offset: -1, offset_end: -1, flags: 00000000 , meta: none) 0x5588039500
/GstPipeline:pipeline0/GstIdentity:identity1: last-message = chain   ******* (identity1:sink) (567 bytes, dts: 0:00:01.170000000, pts: 0:00:01.170000000, duration: none, offset: 0, offset_end:  1, flags: 00002000 delta-unit , meta: none) 0x7f94157a00
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain   ******* (identity0:sink) (2080000 bytes, dts: none, pts: 0:00:01.440000000, duration: none, offset: -1, offset_end: -1, flags: 00000000 , meta: none) 0x5588039610
/GstPipeline:pipeline0/GstIdentity:identity1: last-message = chain   ******* (identity1:sink) (549 bytes, dts: 0:00:01.260000000, pts: 0:00:01.260000000, duration: none, offset: 0, offset_end:  1, flags: 00002000 delta-unit , meta: none) 0x7f94157c20
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain   ******* (identity0:sink) (2080000 bytes, dts: none, pts: 0:00:01.630000000, duration: none, offset: -1, offset_end: -1, flags: 00000000 , meta: none) 0x5588039720
/GstPipeline:pipeline0/GstIdentity:identity1: last-message = chain   ******* (identity1:sink) (547 bytes, dts: 0:00:01.350000000, pts: 0:00:01.440000000, duration: none, offset: 0, offset_end:  1, flags: 00002000 delta-unit , meta: none) 0x7f94157070
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain   ******* (identity0:sink) (2080000 bytes, dts: none, pts: 0:00:01.720000000, duration: none, offset: -1, offset_end: -1, flags: 00000000 , meta: none) 0x5588039830
/GstPipeline:pipeline0/GstIdentity:identity1: last-message = chain   ******* (identity1:sink) (567 bytes, dts: 0:00:01.440000000, pts: 0:00:01.630000000, duration: none, offset: 0, offset_end:  1, flags: 00002000 delta-unit , meta: none) 0x55880390c0
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain   ******* (identity0:sink) (2080000 bytes, dts: none, pts: 0:00:01.810000000, duration: none, offset: -1, offset_end: -1, flags: 00000000 , meta: none) 0x5587a44080
/GstPipeline:pipeline0/GstIdentity:identity1: last-message = chain   ******* (identity1:sink) (549 bytes, dts: 0:00:01.630000000, pts: 0:00:01.720000000, duration: none, offset: 0, offset_end:  1, flags: 00002000 delta-unit , meta: none) 0x5587a444c0
/GstPipeline:pipeline0/GstIdentity:identity1: last-message = chain   ******* (identity1:sink) (547 bytes, dts: 0:00:01.720000000, pts: 0:00:01.350000000, duration: none, offset: 0, offset_end:  1, flags: 00002000 delta-unit , meta: none) 0x5587a442a0
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain   ******* (identity0:sink) (2080000 bytes, dts: none, pts: 0:00:01.900000000, duration: none, offset: -1, offset_end: -1, flags: 00000000 , meta: none) 0x5587a44190
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain   ******* (identity0:sink) (2080000 bytes, dts: none, pts: 0:00:02.000000000, duration: none, offset: -1, offset_end: -1, flags: 00000000 , meta: none) 0x5587a44e50
/GstPipeline:pipeline0/GstIdentity:identity1: last-message = chain   ******* (identity1:sink) (567 bytes, dts: 0:00:01.810000000, pts: 0:00:01.810000000, duration: none, offset: 0, offset_end:  1, flags: 00002000 delta-unit , meta: none) 0x7f94157d30

As result: it produces problem during decoding - cpu decoder (vp9dec) just produces frames with duplicated timestamps for that case, but nvv decoder produces different results each time (see my issue about that: Incorrect timestamping during decoding vp9 video by nvv4l2decoder). Also looks like same problem is described here: PTS values altered by nvv4l2h64enc

jetson_release output on my device:

 - NVIDIA Jetson Xavier NX (Developer Kit Version)
   * Jetpack 4.4.1 [L4T 32.4.4]
   * NV Power Mode: MODE_10W_2CORE - Type: 3
   * jetson_stats.service: active
 - Libraries:
   * CUDA: 10.2.89
   * cuDNN: 8.0.0.180
   * TensorRT: 7.1.3.0
   * Visionworks: 1.6.0.501
   * OpenCV: 4.1.1 compiled CUDA: NO
   * VPI: 0.4.4
   * Vulkan: 1.2.70

Please see attached minimal example for reproduce the problem

Is there any possibility to fix that?
min-exemple.zip (2.1 KB)

3

Hi,
Please check if you observe the issue with steady timestamp in 33000000ns(30fps). To confirm it works in steady frame date and the issue is specific to unique timestamps of the camera sources. May also try h264/h265 encoding and see if it works.

4

I never was able to reproduce this issue with steady fps, including 30fps. According to using h264/h265 - we can’t use them, we are restricted by vp9 format

5

Hi,
Yo may consider use jetson_multimedia_api. It is low-level interface and can eliminate possible issue in gstreamer frameworks. For encoding, please refer to

/usr/src/jetson_multimedia_api/samples/01_video_encode/

Certain corner cases may not be well handled in gstreamer. In low-level interface, you can set timestamp to each frame manually. If two frames have identical timestamp, please set the next frame to have an offset to identify the two frames.

6

Certain corner cases may not be well handled in gstreamer

Problem is not in default gstreamer elements or pipeline, because it can not be reproduced with cpu encoder, only with nvv* encoder. Check minimal example that I provided - if you remove HW_PIPELINE define, then it will use cpu pipeline instead and problem will not appear

7

Hi,
We will try Jetpack 5.1 and see if the issue is present. This may take some time since it is specific to the condition that source is not stable. For general use-case, we would expect the source generates frames in steady rate.

9

Hi,
We try the sample on Xavier/Jetpack 4.6.3 and 5.1. The timestamps shown in identity1 are in order. So it may be an issue in previous release. Please check if you can upgrade to later release and give it a try.