Fail to use nvv4l2dec in ffmpeg

I used below steps to install nvidia ffmpeg.

$ echo “deb https://repo.download.nvidia.com/jetson/ffmpeg main main” | sudo tee -a /etc/apt/sources.list
$ echo “deb-src https://repo.download.nvidia.com/jetson/ffmpeg main main” | sudo tee -a /etc/apt/sources.list
$ sudo apt update
$ sudo apt install ffmpeg

In our code, we use ffmpeg API to decode h264 stream. avcodec_find_decoder_by_name(“h264_nvv4l2dec”) to get and use h264_nvv4l2dec decoder.

But it did not work. We got below output on screen.

Opening in BLOCKING MODE
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
[h264_nvv4l2dec @ 0x55a41c3d60] non-existing PPS 0 referenced
[h264_nvv4l2dec @ 0x55a41c3d60] SEI type 1 size 904 truncated at 848
[h264_nvv4l2dec @ 0x55a41c3d60] non-existing PPS 0 referenced
[h264_nvv4l2dec @ 0x55a41c3d60] SEI type 1 size 904 truncated at 848
[h264_nvv4l2dec @ 0x55a41c3d60] non-existing PPS 0 referenced
[h264_nvv4l2dec @ 0x55a41c3d60] SEI type 1 size 904 truncated at 848
[h264_nvv4l2dec @ 0x55a41c3d60] non-existing PPS 0 referenced
[h264_nvv4l2dec @ 0x55a41c3d60] SEI type 1 size 904 truncated at 848
[h264_nvv4l2dec @ 0x55a41c3d60] non-existing PPS 0 referenced
[h264_nvv4l2dec @ 0x55a41c3d60] SEI type 1 size 904 truncated at 848
[h264_nvv4l2dec @ 0x55a41c3d60] non-existing PPS 0 referenced
[h264_nvv4l2dec @ 0x55a41c3d60] SEI type 1 size 904 truncated at 848
[h264_nvv4l2dec @ 0x55a41c3d60] non-existing PPS 0 referenced
[h264_nvv4l2dec @ 0x55a41c3d60] SEI type 1 size 904 truncated at 848
[h264_nvv4l2dec @ 0x55a41c3d60] non-existing PPS 0 referenced
[h264_nvv4l2dec @ 0x55a41c3d60] SEI type 1 size 904 truncated at 848
[h264_nvv4l2dec @ 0x55a41c3d60] non-existing PPS 0 referenced
[h264_nvv4l2dec @ 0x55a41c3d60] SEI type 1 size 904 truncated at 848
[h264_nvv4l2dec @ 0x55a41c3d60] non-existing PPS 0 referenced
[h264_nvv4l2dec @ 0x55a41c3d60] SEI type 1 size 904 truncated at 848
[h264_nvv4l2dec @ 0x55a41c3d60] non-existing PPS 0 referenced
[h264_nvv4l2dec @ 0x55a41c3d60] SEI type 1 size 904 truncated at 848
[h264_nvv4l2dec @ 0x55a41c3d60] non-existing PPS 0 referenced
[h264_nvv4l2dec @ 0x55a41c3d60] SEI type 1 size 904 truncated at 848
[h264_nvv4l2dec @ 0x55a41c3d60] non-existing PPS 0 referenced
[h264_nvv4l2dec @ 0x55a41c3d60] SEI type 1 size 904 truncated at 848
[h264_nvv4l2dec @ 0x55a41c3d60] non-existing PPS 0 referenced

Is there anything else we need do to use nvv4l2dec decoder in ffmpeg?

Hi,
Please check if it is similar to this:
Libavcodec.so: undefined reference to `v4l2_open’ - #6 by ch3c00h

You can run ffmpeg command but have issues in C sample.

Not this issue. We can run ffmpeg command to decoder.

Our issue is when using ffmpeg API to decoder in our C++ program, we can’t decoder H264 stream if avcodec_find_decoder_by_name(“h264_nvv4l2dec”). On jtop, we can see NVDEC is on when start our program, then NVDEC is OFF.

Hi,
Please check if this helps:
Libavcodec.so: undefined reference to `v4l2_open’ - #9 by DaneLLL

If you still observe the issue with self-built libs, please share a simple sample and we can set up and try to reproduce the issue.

Please see below code. I provided init and decodeBuffer functions.

bool init()
{
avcodec_register_all();
pCodecCtx = avcodec_alloc_context3(nullptr);
if (!pCodecCtx) {
return false;
}

pCodecCtx->thread_count = 4;

// pCodec = avcodec_find_decoder(AV_CODEC_ID_H264);
pCodec = avcodec_find_decoder_by_name(“h264_nvv4l2dec”);
if (!pCodec || avcodec_open2(pCodecCtx, pCodec, nullptr) < 0) {
return false;
}

pCodecParserCtx = av_parser_init(AV_CODEC_ID_H264);
if (!pCodecParserCtx) {
    return false;
}

pFrameYUV = av_frame_alloc();
if (!pFrameYUV) {
    return false;
}

pFrameRGB = av_frame_alloc();
if (!pFrameRGB) {
    return false;
}

pSwsCtx = nullptr;

pCodecCtx->flags2 |= AV_CODEC_FLAG2_SHOW_ALL;

return true;

}

void decodeBuffer(const uint8_t *buf, int bufLen)
{
const uint8_t *pData = buf;
int remainingLen = bufLen;
int processedLen = 0;

AVPacket pkt;
av_init_packet(&pkt);
pthread_mutex_lock(&decodemutex);
while (remainingLen > 0) {
    if (!pCodecParserCtx || !pCodecCtx) {
        //DSTATUS("Invalid decoder ctx.");
        break;
    }
    processedLen = av_parser_parse2(pCodecParserCtx, pCodecCtx,
                                    &pkt.data, &pkt.size,
                                    pData, remainingLen,
                                    AV_NOPTS_VALUE, AV_NOPTS_VALUE, AV_NOPTS_VALUE);
    remainingLen -= processedLen;
    pData += processedLen;

    if (pkt.size > 0) {
        int gotPicture = 0;
        avcodec_decode_video2(pCodecCtx, pFrameYUV, &gotPicture, &pkt);

        if (!gotPicture) {
            ////DSTATUS_PRIVATE("Got Frame, but no picture\n");
            continue;
        } else {
            int w = pFrameYUV->width;
            int h = pFrameYUV->height;
            ////DSTATUS_PRIVATE("Got picture! size=%dx%d\n", w, h);

            if (nullptr == pSwsCtx) {
                pSwsCtx = sws_getContext(w, h, pCodecCtx->pix_fmt,
                                         w, h, AV_PIX_FMT_RGB24,
                                         4, nullptr, nullptr, nullptr);
            }

            if (nullptr == rgbBuf) {
                bufSize = avpicture_get_size(AV_PIX_FMT_RGB24, w, h);
                rgbBuf = (uint8_t *) av_malloc(bufSize);
                avpicture_fill((AVPicture *) pFrameRGB, rgbBuf, AV_PIX_FMT_RGB24, w, h);
            }

            if (nullptr != pSwsCtx && nullptr != rgbBuf) {
                sws_scale(pSwsCtx,
                          (uint8_t const *const *) pFrameYUV->data, pFrameYUV->linesize, 0, pFrameYUV->height,
                          pFrameRGB->data, pFrameRGB->linesize);

                pFrameRGB->height = h;
                pFrameRGB->width = w;

                // below function is to copy image data and send signal
                decodedImageHandler.writeNewImageWithLock(pFrameRGB->data[0], bufSize, w, h);
            }
        }
    }
}
pthread_mutex_unlock(&decodemutex);
av_free_packet(&pkt);

}

Please note:
// pCodec = avcodec_find_decoder(AV_CODEC_ID_H264);
pCodec = avcodec_find_decoder_by_name(“h264_nvv4l2dec”);

Previously if we use “pCodec = avcodec_find_decoder(AV_CODEC_ID_H264);”, we can decode H264 stream. But it did not use NVDEC HW decoder. Then we use “pCodec = avcodec_find_decoder_by_name(“h264_nvv4l2dec”);” and expect to see NVDEC HW decoder working, but it can’t work.

I got below output on screen.

Opening in BLOCKING MODE
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
[h264_nvv4l2dec @ 0x557f2b04a0] Starting capture thread
[h264_nvv4l2dec @ 0x557f2b04a0] non-existing PPS 0 referenced
[h264_nvv4l2dec @ 0x557f2b04a0] SEI type 1 size 904 truncated at 848
[h264_nvv4l2dec @ 0x557f2b04a0] unknown SEI type 239
[h264_nvv4l2dec @ 0x557f2b04a0] unknown SEI type 257
[h264_nvv4l2dec @ 0x557f2b04a0] unknown SEI type 239
[h264_nvv4l2dec @ 0x557f2b04a0] unknown SEI type 240
[h264_nvv4l2dec @ 0x557f2b04a0] non-existing PPS 0 referenced
[h264_nvv4l2dec @ 0x557f2b04a0] SEI type 1 size 904 truncated at 848
[h264_nvv4l2dec @ 0x557f2b04a0] unknown SEI type 239
[h264_nvv4l2dec @ 0x557f2b04a0] unknown SEI type 257
[h264_nvv4l2dec @ 0x557f2b04a0] unknown SEI type 239
[h264_nvv4l2dec @ 0x557f2b04a0] unknown SEI type 240
[h264_nvv4l2dec @ 0x557f2b04a0] SEI type 1 size 904 truncated at 848
[h264_nvv4l2dec @ 0x557f2b04a0] unknown SEI type 239
[h264_nvv4l2dec @ 0x557f2b04a0] unknown SEI type 257
[h264_nvv4l2dec @ 0x557f2b04a0] unknown SEI type 239
[h264_nvv4l2dec @ 0x557f2b04a0] unknown SEI type 240
[h264_nvv4l2dec @ 0x557f2b04a0] Decoder colorspace ITU-R BT.601 with standard range luma (16-235)
[h264_nvv4l2dec @ 0x557f2b04a0] Query and set capture successful
[h264_nvv4l2dec @ 0x557f2b04a0] Resource unavailable
[h264_nvv4l2dec @ 0x557f2b04a0] SEI type 1 size 904 truncated at 848
[h264_nvv4l2dec @ 0x557f2b04a0] unknown SEI type 239
[h264_nvv4l2dec @ 0x557f2b04a0] unknown SEI type 257
[h264_nvv4l2dec @ 0x557f2b04a0] unknown SEI type 239
[h264_nvv4l2dec @ 0x557f2b04a0] unknown SEI type 240

Hi,
It looks like there’s no valid SPS/PPS in the bitstream. Could you try to decode sample_outdoor_car_1080p_10fps.h264 and check if the issue is still present?

If using “pCodec = avcodec_find_decoder(AV_CODEC_ID_H264);”, the stream can be decoded successfully. It should not be stream issue.

Hi,
The hardware decoder may not work identically as software decoder. From the prints it looks like there is no valid SPS/PPS in the stream. Would suggest you discard the stream before receiving valid SPS/PPS. May also try sample_outdoor_car_1080p_10fps.h264 and see if the sample stream is decode-able.

The decoder will parse stream and discard the data before receiving valid SPS/PPS since if there is no SPS/PPS information, decoder can’t decode stream. All those are done by decoder. We dont need discard the stream before receiving SPS/PPS.

Where can we get sample_outdoor_car_1080p_10fps.h264 file?

Hi,
The H264 stream is in

/usr/src/jetson_multimedia_api/data/Video

I debuged the nvv4l2 decoder in ffmpeg-4.2.2/libavcodec/nvv4l2_dec.c, found capture thread will wait for V4L2_EVENT_RESOLUTION_CHANGE event to continue caputre works. But it did not receive this event for long time. I suppose when first SPS/PPS are received, decoder should know the width&height, then send V4L2_EVENT_RESOLUTION_CHANGE event. But it did not happen.