Jetson Video Encoder: unreliable metadata output with Slice Level Encode

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
,
1

Hello

I am working on realtime streaming application for my client. I am using Jetson (AGX and Nano), NvVideoEncoder C++ API and I am encoding h.265. Recently I started using Slice Level Encode to retrieve slices of the frame being encoded ASAP, even before the encoding process for the remainder of frame ends.

My code also makes use of encoder output metadata. I’ve noticed that while that metadata is fine when Slice Level Encode is disabled, it is unreliable, at times almost zeroed out when Slice Level Encode is on. In order to demonstrate the problem I used the 01_video_encode sample from Jetson multimedia API samples:

/usr/src/jetson_multimedia_api/samples/01_video_encode/video_encode ../videos/nv12_sample.yuv 1920 1080 H265 experiment.h265 --sp -slt 2 -slen 1000 --report-metadata
This produces the following output:

Frame 0: isKeyFrame=1 AvgQP=29 MinQP=27 MaxQP=30 EncodedBits=468472
Frame 1: isKeyFrame=0 AvgQP=30 MinQP=29 MaxQP=32 EncodedBits=99704
Frame 2: isKeyFrame=0 AvgQP=31 MinQP=30 MaxQP=32 EncodedBits=67448
Frame 3: isKeyFrame=0 AvgQP=31 MinQP=30 MaxQP=32 EncodedBits=63672
Frame 4: isKeyFrame=0 AvgQP=32 MinQP=32 MaxQP=32 EncodedBits=57456
Frame 5: isKeyFrame=0 AvgQP=31 MinQP=31 MaxQP=32 EncodedBits=72800
Frame 6: isKeyFrame=0 AvgQP=30 MinQP=30 MaxQP=32 EncodedBits=164912
Frame 7: isKeyFrame=0 AvgQP=29 MinQP=29 MaxQP=31 EncodedBits=209096
Frame 8: isKeyFrame=0 AvgQP=29 MinQP=29 MaxQP=30 EncodedBits=204808
Frame 9: isKeyFrame=0 AvgQP=29 MinQP=29 MaxQP=29 EncodedBits=230704
Frame 10: isKeyFrame=0 AvgQP=30 MinQP=30 MaxQP=30 EncodedBits=189872
Frame 11: isKeyFrame=0 AvgQP=31 MinQP=31 MaxQP=31 EncodedBits=178832
Frame 12: isKeyFrame=0 AvgQP=31 MinQP=31 MaxQP=31 EncodedBits=193624
Frame 13: isKeyFrame=0 AvgQP=32 MinQP=32 MaxQP=32 EncodedBits=121288
Frame 14: isKeyFrame=0 AvgQP=32 MinQP=32 MaxQP=32 EncodedBits=179456
Frame 15: isKeyFrame=0 AvgQP=32 MinQP=32 MaxQP=32 EncodedBits=163544
Frame 16: isKeyFrame=0 AvgQP=32 MinQP=32 MaxQP=32 EncodedBits=154256
Frame 17: isKeyFrame=0 AvgQP=32 MinQP=32 MaxQP=32 EncodedBits=161848
Frame 18: isKeyFrame=0 AvgQP=33 MinQP=33 MaxQP=33 EncodedBits=120120
Frame 19: isKeyFrame=0 AvgQP=33 MinQP=33 MaxQP=33 EncodedBits=129904
Frame 20: isKeyFrame=0 AvgQP=33 MinQP=33 MaxQP=33 EncodedBits=131976

and so on. Nothing to see here, everything is fine. But once I enable Slice Level Encode, things change. Command in use is:
/usr/src/jetson_multimedia_api/samples/01_video_encode/video_encode ../videos/nv12_sample.yuv 1920 1080 H265 experiment.h265 --sp -slt 2 -slen 1000 --report-metadata --sle
Here I can see very different picture:

Frame 0: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 1: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 2: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 3: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 4: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 5: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 6: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 7: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 8: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 9: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 10: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 11: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 12: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 13: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 14: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 15: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 16: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 17: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 18: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 19: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 20: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0

This is clear nonsense. First frame IS a keyframe, it’s QPs aren’t all equal to zero (and this is not lossless encoding), EncodedBits can’t be zero either (now that would be amazing compression ;-) ) and so on…let’s wait for awhile, and some reasonable information will actually appear:

Frame 58: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 59: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 60: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 61: isKeyFrame=1 AvgQP=29 MinQP=27 MaxQP=30 EncodedBits=468472
Frame 62: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 63: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 64: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0

and then after a looong time output is more like

Frame 1450: isKeyFrame=0 AvgQP=30 MinQP=30 MaxQP=30 EncodedBits=104352
Frame 1451: isKeyFrame=0 AvgQP=30 MinQP=30 MaxQP=30 EncodedBits=100784
Frame 1452: isKeyFrame=0 AvgQP=33 MinQP=33 MaxQP=33 EncodedBits=113240
Frame 1453: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 1454: isKeyFrame=0 AvgQP=32 MinQP=32 MaxQP=32 EncodedBits=89728
Frame 1455: isKeyFrame=0 AvgQP=29 MinQP=29 MaxQP=30 EncodedBits=138264
Frame 1456: isKeyFrame=0 AvgQP=31 MinQP=31 MaxQP=31 EncodedBits=103352
Frame 1457: isKeyFrame=0 AvgQP=30 MinQP=30 MaxQP=30 EncodedBits=96384
Frame 1458: isKeyFrame=0 AvgQP=0 MinQP=0 MaxQP=0 EncodedBits=0
Frame 1459: isKeyFrame=0 AvgQP=29 MinQP=29 MaxQP=30 EncodedBits=135808
Frame 1460: isKeyFrame=0 AvgQP=30 MinQP=30 MaxQP=30 EncodedBits=109136
Frame 1461: isKeyFrame=0 AvgQP=30 MinQP=28 MaxQP=31 EncodedBits=99464
Frame 1462: isKeyFrame=0 AvgQP=30 MinQP=30 MaxQP=31 EncodedBits=586184

So ALMOST all entries make sense, but not all of them. I think it is a bug.

Regards
Michal

3

Hi,
It looks to be a bug. Please share which release you use. Jetpack 4.6.2 or 5.0.2?

4

I have access to two machines, Nano and AGX. Both machines exhibit the same problem. I tried to find out how to get Jetpack version (I am fairly new to Jetsons family), and only found that dpkg-query --show nvidia-l4t-core produces the output that can be correlated with Jetpack version.

So on the Nano, the output is:
nvidia-l4t-core 32.7.1-20220219090432
While on the AGX:
nvidia-l4t-core 32.5.2-20210709090156

Hope that helps, otherwise please tell me how to check Jetson versions in a better way!
Regards
Michal

5

Hi,
You can execute

cat /etc/nv_tegra_release

6

Hello!

The output of that is:
# R32 (release), REVISION: 7.1, GCID: 29818004, BOARD: t210ref, EABI: aarch64, DATE: Sat Feb 19 17:05:08 UTC 2022 on the Nano
# R32 (release), REVISION: 5.2, GCID: 27767740, BOARD: t186ref, EABI: aarch64, DATE: Fri Jul 9 16:05:07 UTC 2021 on the AGX

Best
Michal

7

Hi,
Please share how many YUV420 frames are in nv12_sample.yuv. Would like to generate the YUV input with same frame number.

8

Hello

Thanks for working on my issue, @DaneLLL! The video I use most often is a Full HD sample with exactly 1631 frames. But I have seen the effect in question with other videos, including when encoding live camera stream.

Oh, and BTW I only just noticed - metadata is reported as “Frames” whereas it should be reported as slices in this case. With -slt 2 -slen 1000 encoder will create sliced of 1kB or less, and for reasonable quality Full HD stream even small P frame will have several such slices. So it should be something like: “Frame n, slice 1: metadata1, Frame n, slice 2: metadata2, …”

Best
Michal