I found out that there is some mess with DTS during encoding not perfect framerate via nvv encoder. For example, for perfect 10 fps we expect frame timestamps like:
0.0
0.1
0.2
0.3
0.4
0.5
...
But because in our case frames are produced and timestamped by camera they have not perfect framerate. Also framerate of our camera is higher then we need (imagine that it is 25 fps instead of 10), so we use videorate drop-only=TRUE
for have 10 or less frames per second. As result we have timestamps like here:
0.99
1.08
1.17
1.26
1.35
1.44
1.63
1.72
1.81
1.9
Between 1.44 and 1.63 is bigger difference then we can expect, but it is absolutely fine (it can be caused by some delay in camera or videorate dropped some extra frame here). if we compare this values to prefect framerate, then we will see next picture:
1.0 0.99
1.1 1.08
1.2 1.17
1.3 1.26
1.4 1.35
1.5 1.44
1.6 1.63
And nvv encoder fails with value 1.44 - this value is closer to 1.4 then 1.35, so nvv encoder pushes frame that timestamped with 1.44 instead of frame that timestamped with 1.35. Moreover, it uses 1.35 as DTS for frame that has PTS 1.44. See logs bellow (identity0 is before encoder and __idenitity1 is after):
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain ******* (identity0:sink) (2080000 bytes, dts: none, pts: 0:00:01.350000000, duration: none, offset: -1, offset_end: -1, flags: 00000000 , meta: none) 0x5588039500
/GstPipeline:pipeline0/GstIdentity:identity1: last-message = chain ******* (identity1:sink) (567 bytes, dts: 0:00:01.170000000, pts: 0:00:01.170000000, duration: none, offset: 0, offset_end: 1, flags: 00002000 delta-unit , meta: none) 0x7f94157a00
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain ******* (identity0:sink) (2080000 bytes, dts: none, pts: 0:00:01.440000000, duration: none, offset: -1, offset_end: -1, flags: 00000000 , meta: none) 0x5588039610
/GstPipeline:pipeline0/GstIdentity:identity1: last-message = chain ******* (identity1:sink) (549 bytes, dts: 0:00:01.260000000, pts: 0:00:01.260000000, duration: none, offset: 0, offset_end: 1, flags: 00002000 delta-unit , meta: none) 0x7f94157c20
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain ******* (identity0:sink) (2080000 bytes, dts: none, pts: 0:00:01.630000000, duration: none, offset: -1, offset_end: -1, flags: 00000000 , meta: none) 0x5588039720
/GstPipeline:pipeline0/GstIdentity:identity1: last-message = chain ******* (identity1:sink) (547 bytes, dts: 0:00:01.350000000, pts: 0:00:01.440000000, duration: none, offset: 0, offset_end: 1, flags: 00002000 delta-unit , meta: none) 0x7f94157070
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain ******* (identity0:sink) (2080000 bytes, dts: none, pts: 0:00:01.720000000, duration: none, offset: -1, offset_end: -1, flags: 00000000 , meta: none) 0x5588039830
/GstPipeline:pipeline0/GstIdentity:identity1: last-message = chain ******* (identity1:sink) (567 bytes, dts: 0:00:01.440000000, pts: 0:00:01.630000000, duration: none, offset: 0, offset_end: 1, flags: 00002000 delta-unit , meta: none) 0x55880390c0
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain ******* (identity0:sink) (2080000 bytes, dts: none, pts: 0:00:01.810000000, duration: none, offset: -1, offset_end: -1, flags: 00000000 , meta: none) 0x5587a44080
/GstPipeline:pipeline0/GstIdentity:identity1: last-message = chain ******* (identity1:sink) (549 bytes, dts: 0:00:01.630000000, pts: 0:00:01.720000000, duration: none, offset: 0, offset_end: 1, flags: 00002000 delta-unit , meta: none) 0x5587a444c0
/GstPipeline:pipeline0/GstIdentity:identity1: last-message = chain ******* (identity1:sink) (547 bytes, dts: 0:00:01.720000000, pts: 0:00:01.350000000, duration: none, offset: 0, offset_end: 1, flags: 00002000 delta-unit , meta: none) 0x5587a442a0
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain ******* (identity0:sink) (2080000 bytes, dts: none, pts: 0:00:01.900000000, duration: none, offset: -1, offset_end: -1, flags: 00000000 , meta: none) 0x5587a44190
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain ******* (identity0:sink) (2080000 bytes, dts: none, pts: 0:00:02.000000000, duration: none, offset: -1, offset_end: -1, flags: 00000000 , meta: none) 0x5587a44e50
/GstPipeline:pipeline0/GstIdentity:identity1: last-message = chain ******* (identity1:sink) (567 bytes, dts: 0:00:01.810000000, pts: 0:00:01.810000000, duration: none, offset: 0, offset_end: 1, flags: 00002000 delta-unit , meta: none) 0x7f94157d30
As result: it produces problem during decoding - cpu decoder (vp9dec) just produces frames with duplicated timestamps for that case, but nvv decoder produces different results each time (see my issue about that: Incorrect timestamping during decoding vp9 video by nvv4l2decoder). Also looks like same problem is described here: PTS values altered by nvv4l2h64enc
jetson_release output on my device:
- NVIDIA Jetson Xavier NX (Developer Kit Version)
* Jetpack 4.4.1 [L4T 32.4.4]
* NV Power Mode: MODE_10W_2CORE - Type: 3
* jetson_stats.service: active
- Libraries:
* CUDA: 10.2.89
* cuDNN: 8.0.0.180
* TensorRT: 7.1.3.0
* Visionworks: 1.6.0.501
* OpenCV: 4.1.1 compiled CUDA: NO
* VPI: 0.4.4
* Vulkan: 1.2.70
Please see attached minimal example for reproduce the problem
Is there any possibility to fix that?
min-exemple.zip (2.1 KB)