Jetson Video Encoder: problems with GDR, slices and Slice Level Encode

[michal34]

Hello!

I am working on low-latency real time streaming product for my client, using Jetson hardware (Nano and AGX). I am encoding live camera input into h.265 stream. My code is written in C++ and uses Argus for camera input, and NvVideoEncoder class for the video encoding.

Stream features that I use are slicing (setting slice sizes to 1kB) and GDR (every now and then, instead of sending I frames). My code is working more or less fine, except that I’ve noticed that GDR always subdivides frame into 2 or 3 slices, ignoring slicing settings. Please consider the plot below:

One can clearly see that while non-GDR frames are subdivided into 1kB slices (short bars of alternating shades), GDR frames are subdivided into either I/P, P/I/P or P/I slices. But that’s a minor problem.

The bigger problem is, I’d like to use slice level encoding on top of GDR and slicing. The reason being - we want lowest latency possible. I want to send the first slices of the frame ASAP, well before the encode ends. Unfortunately GDR, slicing and slice level encode don’t work together. My code crashes on the first GDR frame.

To avoid copyright issues, as well as to save everybody’s time, I worked out the command line switches for the 01_video_encode sample that demonstrate the problem. So this is the one that fails:
/usr/src/jetson_multimedia_api/samples/01_video_encode/video_encode ../videos/nv12_video.yuv 1920 1080 H265 experiment.h265 --sp --egdr -gdrf gdr.txt --input-metadata -gdrof gdr_stream.h265 -slt 2 -slen 1000 --sle
I get the following messages:

Creating Encoder in blocking mode
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 8
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 8
892744264
842091854
NVMEDIA: H265 : Profile : 1
NVMEDIA_ENC: bBlitMode is set to TRUE
NvVideoEncTransferOutputBufferToBlock: DoWork failed line# 674
NvVideoEnc: NvVideoEncTransferOutputBufferToBlock TransferBufferToBlock failed Line=685

Now, if I remove either --sle (meaning that I use GDR and slicing, but not Slice Level Encode) or -slt 2 -slen 1000 (meaning that I use GDR and Slice Level Encode but not slicing) everything works just fine.

Unless I am missing something, this looks like a bug to me. I’ll be grateful for any help. I had the idea of disabling the slicing just for the GDR periods, but unfortunately this is not possible, as the setSliceLength() can only be called before requestBuffers on the encoder planes.

Regards
Michal

[kayccc]

Your topic was posted in the wrong category. I am moving this to the Jetson AGX Xavier category for visibility.

[DaneLLL]

Hi,
This looks expected. Please refer to explanation in
GDR in H265 encoding not working as expected - #9 by DaneLLL

[michal34]

Sorry about posting in wrong forum, I am new here. But no, thread that you linked to, @DaneLLL is not the answer to my question. That thread explains how to trigger GDR and how GDR computes number of frames it will spread on.

My points were:

1. That the GDR frames disobey slice size settings, instead always dividing into 2 or 3 slices (minor problem, really)
2. That the GDR, slicing and Slice Level Encode, when used together, cause encoder to crash on the first GDR frame. Unfortunately, this is a major problem for my client.

It is easy to replicate it. Just have a gdr.txt with (for example) “4 11”. Then issue a following command:
/usr/src/jetson_multimedia_api/samples/01_video_encode/video_encode ../videos/nv12_video.yuv 1920 1080 H265 experiment.h265 --sp --egdr -gdrf gdr.txt --input-metadata -gdrof gdr_stream.h265 -slt 2 -slen 1000 --sle
The encoder will crash on the first GDR frame. I will be grateful for any help.

Best regards
Michal

[DaneLLL]

Hi,
GDR + slice level encode is not supported. Please enable either GDR or slice level encode. It may not work properly if both are enabled simultaneously.

[michal34]

Thanks for the explanation @DaneLLL. So I can’t do GDR and setSliceLevelEncode(true) at the same time, understood.

What about using GDR with slices enabled? I mean when calling setSliceLength(V4L2_ENC_SLICE_LENGTH_TYPE_BITS, 1024) or similar. I found out that works but only on non-GDR slices:

As you can see, very first frame (IDR frame), as well as any following non GDR frame is subdivided into 1k slices. But GDR frames are only divided into 2/3/2 (I/P, P/I/P, P/I) slices. Is this the only behaviour supported?

Can you please also take a quick look at my other thread “unreliable metadata output with Slice Level Encode”? Just as this one, it was posted to the wrong category, unfortunately I don’t know how to move it.
Thanks
Michal

[DaneLLL]

Hi,
Please check if it works if you encode each frame into single slice. We have tried it in the post:
GDR in H265 encoding not working as expected - #7 by DaneLLL

This mode should work as expected.

[michal34]

Hello!

I really appreciate your help, @DaneLLL! I did as instructed:

1. To encode with GDR and slicing (1kB slices here) I ran the following:
   /usr/src/jetson_multimedia_api/samples/01_video_encode/video.nv12.yuv 1920 1080 H265 experiment.h265 --sp --egdr -gdrf gdr.txt --input-metadata -gdrof gdr_1k_sliced.h265 -slt 2 -slen 1000

This is how the stream structure looks like:

2. To encode with GDR but without slicing I did this:
   /usr/src/jetson_multimedia_api/samples/01_video_encode/video_encode video.nv12.yuv 1920 1080 H265 experiment.h265 --sp --egdr -gdrf gdr.txt --input-metadata -gdrof gdr_1k_sliced.h265

In this case video stream looks like this:

Now, one can easily see that on the first plot, all frames are sliced. For example I frames are sliced (alternating shades of red), P frames are sliced (alternating shades of grey). GDR frames are sliced as well, but only into 2/3/2 sliced (I slice then P slice, P/I/P slice, P then I slice).

On the second plot however, I frames are not sliced (one shade of red), P frames are not sliced either (one shade of gray). The only frames that exhibit slicing are the GDR frames, and they look very similar to previous plot, again 2/3/2 (I/P, P/I/P, P/I).

So it looks as if GDR is taking precedence over “normal” slicing and produces the 2/3/2 slices, ignoring the finer slicing setting. This is not ideal, because fine slicing can be for example used to subdivide the frame exactly into network MTU-sized units. And this ceases to work with GDR. Whether it should be considered a “bug”, or “lack of feature”, I am not sure, but GDR could easily emit series of P slices/series of I slices/series of P slices again.

Now for the thread that you mentioned. I read it many times before even posting here. In my opinion, it deals with two things:
a) That at some point GDR changed from “permanently enabled” to “pulses”, so that one has to reenable GDR periodically if one wishes to have GDR all the time.
b) How GDR divides a frame. BTW I think the original thread is mistaken there. In my experiments, I was able to subdivide Full HD frame (so 1080 pels high) into up to 17 slices. If what original thread says were true, I should be able to subdivide into 34 slices (because 1080/32 is 33.75, so ceil of that is 34). But I can only get 17 slices, tops. In my opinion original explanation rings true, but if one assumed CTB size of 64x64 - which is BTW the maximum size that the h.265 standard allows. Then 1080/64 is 16.875, ceil of that is 17, and that’s what I am getting.

In any case, neither a) nor b) really deals with GDR frame subdivision into slices (how many slices are there in GDR frame). b) deals with how many GDR frames there will be in GDR “sequence”. This is not the same.

Yes, I am new to this forum and fairly new to Jetson HW. But I have years of experience with h.264, h.265 and hardware encoders from various vendors. I really like the Jetson video encoders, they are perhaps most powerful and easy to use products I ever worked with. Still, they could be made a bit better!

Again, thanks for your help!
Michal