[Jetson Orin] How to retrieve frame timestamp during the AV1 decoding process

Autonomous Machines Jetson & Embedded Systems Jetson AGX Orin
, ,
1

Hello,
Sorry to disturb you, I am trying to integrate the AV1 encoding/decoding codec to my embedded application (by using the benefit of AV1 hardware encoder/decoder present in jetson Orin). I am able to encode/decode frames successfully but I can not retrieve timestamp of decoded frame. The decoding process is asynchronous and I need the decoded frame timestamp for synchronizing my output frames (the v4l2_buf.timestamp obtained after dequeue the capture_plane).

I reproduced the issue with the samples provided in Jetpack. In our application we are using similar approach of copy-timestamp from your sample and it works properly with H264/H265. Using the same approach, I downloaded the same video from your blog (the jockey video) and I ran the following commands:

Encoding:
./video_encode ./Jockey_3840x2160.yuv 3840 2160 AV1 ./Jockey.av1 --copy-timestamp 0

Decoding:
./video_decode AV1 --disable-rendering --stats --copy-timestamp 0 120 --input-nalu -o test.yuyv ../01_video_encode/Jockey.av1

I can’t retrieve the timestamp of decoded frame with your sample using AV1:

Creating decoder in blocking mode 
Opening in BLOCKING MODE 
Set governor to performance before enabling profiler
NvMMLiteOpen : Block : BlockType = 282 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 282 
Setting frame input mode to 0 
It's a valid IVF file
Starting decoder capture loop thread
Video Resolution: 3840x2160
Video SAR width: 0 SAR height: 0
Decoder colorspace ITU-R BT.601 with standard range luma (16-235)
Query and set capture successful
[0]dec capture plane dqB timestamp [0s0us]
[1]dec capture plane dqB timestamp [0s0us]
[2]dec capture plane dqB timestamp [0s0us]
[3]dec capture plane dqB timestamp [0s0us]
[4]dec capture plane dqB timestamp [0s0us]
[5]dec capture plane dqB timestamp [0s0us]
[6]dec capture plane dqB timestamp [0s0us]
[7]dec capture plane dqB timestamp [0s0us]
[8]dec capture plane dqB timestamp [0s0us]
[9]dec capture plane dqB timestamp [0s0us]
[10]dec capture plane dqB timestamp [0s0us]
[11]dec capture plane dqB timestamp [0s0us]
[12]dec capture plane dqB timestamp [0s0us]
[0]dec capture plane dqB timestamp [0s0us]
[1]dec capture plane dqB timestamp [0s0us]
[2]dec capture plane dqB timestamp [0s0us]
[3]dec capture plane dqB timestamp [0s0us]
[4]dec capture plane dqB timestamp [0s0us]
[5]dec capture plane dqB timestamp [0s0us]
[6]dec capture plane dqB timestamp [0s0us]
[7]dec capture plane dqB timestamp [0s0us]
[8]dec capture plane dqB timestamp [0s0us]
[9]dec capture plane dqB timestamp [0s0us]
[10]dec capture plane dqB timestamp [0s0us]
[11]dec capture plane dqB timestamp [0s0us]
[12]dec capture plane dqB timestamp [0s0us]
[0]dec capture plane dqB timestamp [0s0us]
[1]dec capture plane dqB timestamp [0s0us]
[2]dec capture plane dqB timestamp [0s0us]
[3]dec capture plane dqB timestamp [0s0us]
[4]dec capture plane dqB timestamp [0s0us]
[5]dec capture plane dqB timestamp [0s0us]
[6]dec capture plane dqB timestamp [0s0us]
[7]dec capture plane dqB timestamp [0s0us]
[8]dec capture plane dqB timestamp [0s0us]
[9]dec capture plane dqB timestamp [0s0us]
[10]dec capture plane dqB timestamp [0s0us]
[11]dec capture plane dqB timestamp [0s0us]
[12]dec capture plane dqB timestamp [0s0us]
[0]dec capture plane dqB timestamp [0s0us]
[1]dec capture plane dqB timestamp [0s0us]
[2]dec capture plane dqB timestamp [0s0us]
[3]dec capture plane dqB timestamp [0s0us]
[4]dec capture plane dqB timestamp [0s0us]
[5]dec capture plane dqB timestamp [0s0us]
[6]dec capture plane dqB timestamp [0s0us]
[7]dec capture plane dqB timestamp [0s0us]
[8]dec capture plane dqB timestamp [0s0us]
[9]dec capture plane dqB timestamp [0s0us]
[10]dec capture plane dqB timestamp [0s0us]
[11]dec capture plane dqB timestamp [0s0us]
[12]dec capture plane dqB timestamp [0s0us]
[0]dec capture plane dqB timestamp [0s0us]
[1]dec capture plane dqB timestamp [0s0us]
[2]dec capture plane dqB timestamp [0s0us]
[3]dec capture plane dqB timestamp [0s0us]
[4]dec capture plane dqB timestamp [0s0us]
[5]dec capture plane dqB timestamp [0s0us]
[6]dec capture plane dqB timestamp [0s0us]
[7]dec capture plane dqB timestamp [0s0us]
[8]dec capture plane dqB timestamp [0s0us]
[9]dec capture plane dqB timestamp [0s0us]
[10]dec capture plane dqB timestamp [0s0us]
[11]dec capture plane dqB timestamp [0s0us]
[12]dec capture plane dqB timestamp [0s0us]
[0]dec capture plane dqB timestamp [0s0us]
[1]dec capture plane dqB timestamp [0s0us]
[2]dec capture plane dqB timestamp [0s0us]
[3]dec capture plane dqB timestamp [0s0us]
[4]dec capture plane dqB timestamp [0s0us]
[5]dec capture plane dqB timestamp [0s0us]
[6]dec capture plane dqB timestamp [0s0us]
[7]dec capture plane dqB timestamp [0s0us]
[8]dec capture plane dqB timestamp [0s0us]
[9]dec capture plane dqB timestamp [0s0us]
[10]dec capture plane dqB timestamp [0s0us]
[11]dec capture plane dqB timestamp [0s0us]
[12]dec capture plane dqB timestamp [0s0us]
[0]dec capture plane dqB timestamp [0s0us]
[1]dec capture plane dqB timestamp [0s0us]
[2]dec capture plane dqB timestamp [0s0us]
[3]dec capture plane dqB timestamp [0s0us]
[4]dec capture plane dqB timestamp [0s0us]
[5]dec capture plane dqB timestamp [0s0us]
[6]dec capture plane dqB timestamp [0s0us]
[7]dec capture plane dqB timestamp [0s0us]
[8]dec capture plane dqB timestamp [0s0us]
[9]dec capture plane dqB timestamp [0s0us]
[10]dec capture plane dqB timestamp [0s0us]
[11]dec capture plane dqB timestamp [0s0us]
[12]dec capture plane dqB timestamp [0s0us]
[0]dec capture plane dqB timestamp [0s0us]
[1]dec capture plane dqB timestamp [0s0us]
[2]dec capture plane dqB timestamp [0s0us]
[3]dec capture plane dqB timestamp [0s0us]
[4]dec capture plane dqB timestamp [0s0us]
[5]dec capture plane dqB timestamp [0s0us]
[6]dec capture plane dqB timestamp [0s0us]
[7]dec capture plane dqB timestamp [0s0us]
[8]dec capture plane dqB timestamp [0s0us]
[9]dec capture plane dqB timestamp [0s0us]
[10]dec capture plane dqB timestamp [0s0us]
[11]dec capture plane dqB timestamp [0s0us]
[12]dec capture plane dqB timestamp [0s0us]
[0]dec capture plane dqB timestamp [0s0us]
[1]dec capture plane dqB timestamp [0s0us]
[2]dec capture plane dqB timestamp [0s0us]
[3]dec capture plane dqB timestamp [0s0us]
[4]dec capture plane dqB timestamp [0s0us]
[5]dec capture plane dqB timestamp [0s0us]
[6]dec capture plane dqB timestamp [0s0us]
[7]dec capture plane dqB timestamp [0s0us]
[8]dec capture plane dqB timestamp [0s0us]
[9]dec capture plane dqB timestamp [0s0us]
[10]dec capture plane dqB timestamp [0s0us]
[11]dec capture plane dqB timestamp [0s0us]
[12]dec capture plane dqB timestamp [0s0us]
[0]dec capture plane dqB timestamp [0s0us]
[1]dec capture plane dqB timestamp [0s0us]
[2]dec capture plane dqB timestamp [0s0us]
[3]dec capture plane dqB timestamp [0s0us]
[4]dec capture plane dqB timestamp [0s0us]
[5]dec capture plane dqB timestamp [0s0us]
[6]dec capture plane dqB timestamp [0s0us]
[7]dec capture plane dqB timestamp [0s0us]
[8]dec capture plane dqB timestamp [0s0us]
[9]dec capture plane dqB timestamp [0s0us]
[10]dec capture plane dqB timestamp [0s0us]
[11]dec capture plane dqB timestamp [0s0us]
[12]dec capture plane dqB timestamp [0s0us]
[0]dec capture plane dqB timestamp [0s0us]
[1]dec capture plane dqB timestamp [0s0us]
[2]dec capture plane dqB timestamp [0s0us]
[3]dec capture plane dqB timestamp [0s0us]
[4]dec capture plane dqB timestamp [0s0us]
[5]dec capture plane dqB timestamp [0s0us]
[6]dec capture plane dqB timestamp [0s0us]
[7]dec capture plane dqB timestamp [0s0us]
[8]dec capture plane dqB timestamp [0s0us]
[9]dec capture plane dqB timestamp [0s0us]
[10]dec capture plane dqB timestamp [0s0us]
[11]dec capture plane dqB timestamp [0s0us]
[12]dec capture plane dqB timestamp [0s0us]
[0]dec capture plane dqB timestamp [0s0us]
[1]dec capture plane dqB timestamp [0s0us]
[2]dec capture plane dqB timestamp [0s0us]
[3]dec capture plane dqB timestamp [0s0us]
[4]dec capture plane dqB timestamp [0s0us]
[5]dec capture plane dqB timestamp [0s0us]
[6]dec capture plane dqB timestamp [0s0us]
[7]dec capture plane dqB timestamp [0s0us]
[8]dec capture plane dqB timestamp [0s0us]
[9]dec capture plane dqB timestamp [0s0us]
[10]dec capture plane dqB timestamp [0s0us]
[11]dec capture plane dqB timestamp [0s0us]
[12]dec capture plane dqB timestamp [0s0us]
[0]dec capture plane dqB timestamp [0s0us]
[1]dec capture plane dqB timestamp [0s0us]
[2]dec capture plane dqB timestamp [0s0us]
[3]dec capture plane dqB timestamp [0s0us]
[4]dec capture plane dqB timestamp [0s0us]
[5]dec capture plane dqB timestamp [0s0us]
[6]dec capture plane dqB timestamp [0s0us]
[7]dec capture plane dqB timestamp [0s0us]
[8]dec capture plane dqB timestamp [0s0us]
[9]dec capture plane dqB timestamp [0s0us]
[10]dec capture plane dqB timestamp [0s0us]
[11]dec capture plane dqB timestamp [0s0us]
[12]dec capture plane dqB timestamp [0s0us]
[0]dec capture plane dqB timestamp [0s0us]
[1]dec capture plane dqB timestamp [0s0us]
[2]dec capture plane dqB timestamp [0s0us]
[3]dec capture plane dqB timestamp [0s0us]
[4]dec capture plane dqB timestamp [0s0us]
[5]dec capture plane dqB timestamp [0s0us]
[6]dec capture plane dqB timestamp [0s0us]
[7]dec capture plane dqB timestamp [0s0us]
[8]dec capture plane dqB timestamp [0s0us]
[9]dec capture plane dqB timestamp [0s0us]
[10]dec capture plane dqB timestamp [0s0us]
[11]dec capture plane dqB timestamp [0s0us]
[12]dec capture plane dqB timestamp [0s0us]
[0]dec capture plane dqB timestamp [0s0us]
[1]dec capture plane dqB timestamp [0s0us]
[2]dec capture plane dqB timestamp [0s0us]
[3]dec capture plane dqB timestamp [0s0us]
[4]dec capture plane dqB timestamp [0s0us]
[5]dec capture plane dqB timestamp [0s0us]
[6]dec capture plane dqB timestamp [0s0us]
[7]dec capture plane dqB timestamp [0s0us]
[8]dec capture plane dqB timestamp [0s0us]
[9]dec capture plane dqB timestamp [0s0us]
[10]dec capture plane dqB timestamp [0s0us]
[11]dec capture plane dqB timestamp [0s0us]
[12]dec capture plane dqB timestamp [0s0us]
[0]dec capture plane dqB timestamp [0s0us]
[1]dec capture plane dqB timestamp [0s0us]
[2]dec capture plane dqB timestamp [0s0us]
[3]dec capture plane dqB timestamp [0s0us]
End of stream
Input file read complete
[4]dec capture plane dqB timestamp [0s0us]
Got EoS at output plane
[5]dec capture plane dqB timestamp [0s0us]
Got EoS at capture plane
Exiting decoder capture loop thread
----------- Element = dec0 -----------
Total Profiling time = 3.06821
Average FPS = 65.1846
Total units processed = 201
-------------------------------------
************************************
Total Profiling Time = 0 sec
************************************
App run was successful

I saw from samples help that copy-timestamp is only supported for H264/H265 but I am looking for a way to also use it with AV1. Is there a way to associate timestamp to frame during the AV1 decoding process by using copy-timestamp or other tricks ?

Thank you very much in advance,
Tanguy

3

Hi,
The codec structure of H264/H265 is similar and we can parse NALs to get each frame. AV1 looks to have different structure and not sure if it is possible. Will check with our teams and update.

4

Hello,
Thanks a lot for your quick response ! I am looking forward to your reply.

5

Hello,
Do you have any update from your technical team ?

Thanks in advance,
Tanguy

7

Hi,
We are checking if we can support this in next release.

8

Hello,
Thanks for your update. I am really looking forward to this. Do not hesitate to send me a pre-release version for testing if possible !
Thanks a lot,
Tanguy

9

Hello,
Do you have any update or pre-release version allowing timestamp retrieval from AV1 decoded frame ? If not do you know when this important aspect will be available ?
Thanks,
Tanguy

10

Hi,
We shall have this supported in next Jetpack 5.1.

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.