AV1 encoder - ultra low latency still has delay of 4 frames. Is there a "low delay" setting?

When I choose the ultra low latency tuning info for HEVC, it emits a stream that can be decoded very quickly with little latency. i.e. you give it a compressed packet, and it gives you back a decoded frame shortly afterwards.

The same setting with AV1 does not appear to do so.

I am decoding using the dav1d library, and I can see that the decoder needs to be given 4 video frame packets, and only then will it suddenly emit all 4 decoded frames with each call to the decoder. Then the pattern repeats, and it needs another 4 packets before outputting any further decoded frames.

I get the same when encoding using the VIDEO CODEC SDK on desktop platforms.

Is this normal behaviour for the ultra low latency tuning info? Or have I missed a setting somewhere that can reduce this latency?

Hi,
We don’t support it in AV1 encoding. Please share how you configure the ultra low latency tuning info to H265. We will check and see if it can be enabled in AV1.

Yes, this is what works for HEVC:

    struct v4l2_ext_control control;
    memset(&control, 0, sizeof(control));
    control.id = V4L2_CID_MPEG_VIDEOENC_CUDA_TUNING_INFO;
    control.value = V4L2_ENC_TUNING_INFO_ULTRA_LOW_LATENCY;
    struct v4l2_ext_controls controls;
    memset(&controls, 0, sizeof(controls));
    controls.count = 1;
    controls.controls = &control;
    controls.ctrl_class = V4L2_CTRL_CLASS_MPEG;
    v4l2_ioctl(m_video_encoder_fd, VIDIOC_S_EXT_CTRLS, &controls);

Hi,
The control looks specific to using GPU as encoder but we use NVENC on Jetson platforms. So you set it on Orin NX and it works? Would like to confirm this.

I’ll double check this and get back to you.

Yes, I think this control is having no effect. I had ported my code for Jetson from the desktop Video SDK and I probably picked this one as what I assumed was the equivalent to the Video SDK’s tuning info.

I think I am getting the illusion of low-latency from HEVC because I am also explicitly setting B-frames to zero.

It would be nice to have this tuning info option available to Jetson though, especially as not all the fine-tuning options are available (lookahead depth, etc).

As for AV1, it would be nice to be able to control the latency too, something like SVT’s “low delay mode”.

I have mistakenly assumed the 4-frame delay was encoder-related; actually it turns out that this is a dav1d decoder setting. By default dav1d uses the number of decode threads to enforce a delay of a certain number of frames in the decoder. If I decode using Videotoolbox on macOS there is no such delay and it is low-latency.

Sorry for the confusion.

For consistency it would still be good to have tuning info as we do with the Video SDK, but it looks like I can control the latency by controlling the number of B-frames on both HEVC and AV1.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.