NvMediaIEP AV1 encoder, raw obu stream error

Autonomous Vehicles DRIVE AGX Orin DRIVE AGX Orin General
1

Software Version
DRIVE OS Linux 6.0.8.1 and DriveWorks 5.10

Hardware Platform
DRIVE AGX Orin

Hi, in our use-case we transfer the raw encoded stream through rtp. Sometimes, for debugging reasons we save the bitstream to raw .265 files that can be played back with ffmpeg/gstreamer. We are about to introduce AV1 encoding. The code is almost identical to the H265 implementation, only the AV1 specific config structs are changed, the base logic remained the same.

By default we set repeatSeqHdr to NVMEDIA_ENCODE_SEQUENCEHDR_REPEAT_IDR_FRAMES, that somewhat works but whenever there is a resolution change the decoder (av1dec) stops:

0:00:03.902082202  5247 0xffff68002360 WARN        codecparsers_av1 gstav1parser.c:4235:gst_av1_parse_uncompressed_frame_header: parse uncompressed frame header error 3
0:00:03.902087194  5247     0x187e8000 WARN        codecparsers_av1 gstav1parser.c:4235:gst_av1_parse_uncompressed_frame_header: parse uncompressed frame header error 3
0:00:03.902123034  5247 0xffff68002360 WARN                av1parse gstav1parse.c:1865:gst_av1_parse_handle_to_big_align:<av1parse0> Parse obu error, discard whole buffer 20884.
0:00:03.902160249  5247     0x187e8000 WARN                av1parse gstav1parse.c:1865:gst_av1_parse_handle_to_big_align:<video_parse> Parse obu error, discard whole buffer 20884.

If we set repeatSeqHdr to NVMEDIA_ENCODE_SEQUENCEHDR_REPEAT_INTRA_FRAMES, the decoder immediately stops:

[ INFO] [2025-06-04T15:08:27.678+0000] [] [vmaf-pipeline-front] state changed for [vmaf-pipeline-front] element from READY to PAUSED, pending PLAYING
0:00:03.951794079  5247 0xffff68002360 WARN                  av1dec gstav1dec.c:447:gst_av1_dec_handle_frame:<av1dec0> error: Failed to decode frame
0:00:03.951831838  5247 0xffff68002360 WARN                  av1dec gstav1dec.c:447:gst_av1_dec_handle_frame:<av1dec0> error: 
[ERROR] [2025-06-04T15:08:27.680+0000] [] [vmaf-pipeline-front] Error received from element av1dec0: Failed to decode frame. Debug info: ../subprojects/gst-plugins-bad/ext/aom/gstav1dec.c(447): gst_av1_dec_handle_frame (): /GstPipeline:vmaf-pipeline-front/GstBin:encoder_bin_front_main_stream/GstKpiAV1VmafBin:kpiav1vmafbin_front_main_stream/GstBin:bin0/GstAV1Dec:av1dec0

I’ve found the following, super similar, Jetson specific thread

Is it possible, that the Drive AGX Orin AV1 implementation suffers from the same error? Please advise how to fix the problem.

Thank you in advance,
Adam

3

Dear @AdamBalazsVay ,
Is it possible to provide simple repro code and steps to verify and debug on our side.

4

Is this still an issue to support? Any result can be shared?

5

Dear @kayccc and @SivaRamaKrishnaNV,

I’m sorry for the late reply but we are still working on this issue. We have checked the source of the nvm_iep_sci sample application and saw that you wrap the raw encoded byte stream to IVF container when using AV1:

if((args->videoCodec == CODEC_VP9) || (args->videoCodec == CODEC_AV1)) {
    AddIVFPrefix(args, pOutputThreadArgs->outputFile, bytesAvailable, frameCounter-1, framesNum);
}

This container is not used for H264 and H265. Should IVF container be used when streaming directly to av1parse and av1dec GStreamer elements? Or only when we store the raw encoded stream to files?

Right now our test pipeline looks something like the following:

nvmediaencoder ! av1parse ! av1dec

The output of our nvmediaencoder element is the same as the byte output of nvm_iep_sci sample app without the IVF container.

Could you please clarify the exact bitstream format your AV1 encoder puts the data? Is it standard obu stream? I don’t know why av1parse raise those parse uncompressed frame header error and Parse obu error, discard whole buffer warnings.

6

Dear @kayccc and @SivaRamaKrishnaNV,

I think we have found the root of the issue. In nvm_iep_sci with gopLength and idrPeriod set to NVMEDIA_ENCODE_INFINITE_GOPLENGTH and with GOP pattern set to 1 (IPPP…) you never request IDR frame with specifying NvMediaEncodePicParamsAV1’s pictureType to NVMEDIA_ENCODE_PIC_TYPE_IDR and encodePicFlags to NVMEDIA_ENCODE_PIC_FLAG_OUTPUT_SPSPPS. When we run the AV1 encoder without manually requesting IDR frames and/or SPSPPS everything works fine.

Once we request NVMEDIA_ENCODE_PIC_TYPE_IDR frame with NVMEDIA_ENCODE_PIC_FLAG_OUTPUT_SPSPPS, GStreamer’s av1parse element raise the OBU stream errors mentioned above.

To test this just try setting the pictureType in the nvm_iep_sci app to NVMEDIA_ENCODE_PIC_TYPE_IDR once in a while and add NVMEDIA_ENCODE_PIC_FLAG_OUTPUT_SPSPPS to picParams->encodePicFlags. Then try to play back the file you got with the following pipeline:
GST_DEBUG=3 gst-launch-1.0 filesrc location=./test.av1 ! ivfparse ! av1parse ! av1dec ! fakesink and see the errors generated.

Kind regards,
Adam

7

So the issue is noticed with IEP sample as well? If so, could you share the modified sample? Note that gstreamer is not officially supported or tested on DRIVE.

8

Dear @SivaRamaKrishnaNV,

I think I’ve found the problem.
Please find an archive containing the image_encoder.c patchfile and av1.cfg (2.8 KB) files attached. Please apply the patchfile on /drive/drive-linux/samples/nvmedia_6x/iep and run the sample app with sudo ./nvm_iep_sci -cf ./av1.cfg. The directory should contain a 2896x1876.yuv file with the raw frames to encode.

If you compile the nvm_iep_sci app right after the patchfile is applied (with //picParams->pictureType = NVMEDIA_ENCODE_PIC_TYPE_IDR; and //pictureType = NVMEDIA_ENCODE_PIC_TYPE_IDR; commented out we get a .av1 file all the players can play without any issues or warnings.

Here is the frame header structure of the file without manual IDR frame request on the first frame:

good_packets
good_packets2056×1063 195 KB

If you un-comment the manual NVMEDIA_ENCODE_PIC_TYPE_IDR requests so they take effect, something interesting happens:

bad_packets
bad_packets2056×1063 194 KB

Without manual NVMEDIA_ENCODE_PIC_TYPE_IDR frame request the encoder inserts a KEY_FRAME (type id: 0) automatically, then all the following frames are P frames.

With manual NVMEDIA_ENCODE_PIC_TYPE_IDR request the very first packet has an INTRA_ONLY_FRAME (type id: 2) frame type, the second frame is a KEY_FRAME (type id: 0) - we didn’t request that, it got automatically inserted - and just then all the following frames are P frames.

In my opinion requesting a NVMEDIA_ENCODE_PIC_TYPE_IDR should make the encoder to put a KEY_FRAME (type id: 0) into the stream, but instead it puts a INTRA_ONLY_FRAME (type id: 2).

The API supports requesting NVMEDIA_ENCODE_PIC_TYPE_IDR and NVMEDIA_ENCODE_PIC_TYPE_I, I think requesting an I-frame should end up with a INTRA_ONLY_FRAME (type id: 2) frame and requesting an IDR frame should end up with KEY_FRAME (type id: 0).

Could you please explain why requesting an NVMEDIA_ENCODE_PIC_TYPE_IDR frame right after the encoder is created end up with getting first an INTRA_ONLY_FRAME, then with the second frame KEY_FRAME (without requesting it) and then with P frames? Is it a bug?

As the encoder decides to put an INTRA_ONLY_FRAME frame first, av1parse GStreamer element drops that frame as on resolution changes it expect a KEY_FRAME that invalidates all of the previous references, but it gets an INTRA_ONLY_FRAME instead.