H264_nvv4l2dec lost frames while decoding a small video file encoded with h264

Autonomous Machines Jetson & Embedded Systems Jetson Xavier NX
,
1

patch to ffmpeg example file: ffmpeg-4.2.7/doc/examples/filtering_video.c

To reproduce the issue, apply the patch from above link:
$patch filtering_video.c < h264_nvv4l2dec.patch
Compile the ffmpeg example file, and run the program (filtering_video) on jetson: (test.mp4 is my test video file, you can find the test.mp4 file following the link to the patch. I am not allowed to add more than one link to the post. )

With ffmpeg h264 decoder, 53 frames are produced.
$ ./filtering_video test.mp4
Number of frames decoded: 53

With h264_nvv4l2dec, only 37 frames are produced. The numbers of frames vary with different runs.
$ ./filtering_video test.mp4 nvdec
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
Number of frames decoded: 37

Here is the output for test.mp4:
ffprobe test.mp4
ffprobe version 4.2.7 Copyright (c) 2007-2022 the FFmpeg developers
built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
configuration: --enable-nvv4l2dec --enable-libv4l2 --enable-shared --extra-libs=‘-L/usr/lib/aarch64-linux-gnu/tegra -lnvbuf_utils’ --extra-cflags=-I/usr/src/jetson_multimedia_api/include/
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from ‘test.mp4’:
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf57.83.100
Duration: 00:00:02.31, start: 0.000000, bitrate: 213 kb/s
Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 240x160, 131 kb/s, 24 fps, 24 tbr, 12288 tbn, 48 tbc (default)
Metadata:
handler_name : VideoHandler
Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 12000 Hz, stereo, fltp, 74 kb/s (default)
Metadata:
handler_name : SoundHandler

System info from jtop:
VIDIA Jetson UNKNOWN - Jetpack UNKNOWN [L4T 34.1.0]

  • Up Time: 140 days 3:47:17 Version: 3.1.3
  • Jetpack: UNKNOWN [L4T 34.1.0] Author: Raffaello Bonghi
  • Board: e-mail: raffaello@rnext.it
    • Type: UNKNOWN
    • SOC Family: tegra194 ID:
    • Module: UNKNOWN Board: P2822-0000
    • Code Name: galen
    • Cuda ARCH: NONE
    • Serial Number: 1562821003202
  • Libraries: - Hostname: ubuntu
    • CUDA: NOT_INSTALLED - Interfaces:
    • OpenCV: 3.4.3 compiled CUDA: NO * eth0: x.x.x.x
    • TensorRT: 8.4.0.9
    • VPI: ii libnvvpi2 2.0.9 arm64 NVIDIA Vision Programming Interface library
    • VisionWorks: NOT_INSTALLED
    • Vulkan: 1.3.203
    • cuDNN: 8.3.2.49
3

Hi,
For ffmpeg software stack, it would need other users to check and share experience. IF you think it is more like an issue in our low-level sodtware stack, please try to reproduce it with either sample:

/usr/src/jetson_multimedia_api/samples/00_video_decode/
/usr/src/jetson_multimedia_api/samples/unittest_samples/decoder_unit_sample/

And share the steps.

4

DaneLLL, thanks for the reply.

We are using ffmpeg libs in our applications. We don’t use the low-level Nvidia API directly.
Is the following ffmpeg implementation supported by Nvidia?

$ echo “deb https://repo.download.nvidia.com/jetson/ffmpeg main main” | sudo tee -a /etc/apt/sources.list

$ sudo apt update
$ sudo apt install ffmpeg
$  ffmpeg -decoders | grep nvv4l
ffmpeg version 4.2.7 Copyright (c) 2000-2022 the FFmpeg developers
built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
configuration: --enable-nvv4l2dec --enable-libv4l2 --enable-shared --extra-libs=‘-L/usr/lib/aarch64-linux-gnu/tegra -lnvbuf_utils’ --extra-cflags=-I/usr/src/jetson_multimedia_api/include/
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
V… h264_nvv4l2dec h264 (nvv4l2dec) (codec h264)
V… hevc_nvv4l2dec hevc (nvv4l2dec) (codec hevc)
V… mpeg2_nvv4l2dec mpeg2 (nvv4l2dec) (codec mpeg2video)
V… mpeg4_nvv4l2dec mpeg4 (nvv4l2dec) (codec mpeg4)
V… vp8_nvv4l2dec vp8 (nvv4l2dec) (codec vp8)
V… vp9_nvv4l2dec vp9 (nvv4l2dec) (codec vp9)

5

I compile the samples you mentioned and run it:
both failed.

./video_decode H264 -o output.mp4 test.mp4
Set governor to performance before enabling profiler
Creating decoder in blocking mode
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
Setting frame input mode to 1
Input file read complete
Starting decoder capture loop thread
FMO streams not supported for T210 onwards
FMO streams not supported for T210 onwards
Got EoS at output plane
[ERROR] (NvV4l2Element.cpp:114) Error while DQing event: Invalid argument
Error in dequeueing decoder event
Exiting decoder capture loop thread
App run failed

./decode_sample test.mp4 output.mp4
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
Starting capture thread
Input file read complete
FMO streams not supported for T210 onwards
FMO streams not supported for T210 onwards
Error in dequeueing decoder event
Exiting decoder capture loop thread

6

Hi,
The input to the samples is h264/h265 stream. Please extract the stream from MP4 files and try.

7

Thanks for the suggestion, DaneLLL.
I extracted the h264 stream and got the following results, which matches my expectations of 55 frames.
./decode_sample test.h264 raw_output
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
Starting capture thread
Input file read complete
Resolution: 240x160
Decoder colorspace ITU-R BT.601 with standard range luma (16-235)
Query and set capture successful
Resource unavailable
Resource unavailable
Exiting decoder capture loop thread
Decoder Run is successful
num_frame_decoded 55

That indicates the low-level API works fine. The problem lies with the integration with ffmpeg decoding framework.
Can you respond to my question above about the support from Nvidia for ffmpeg?

“For ffmpeg software stack, it would need other users to check and share experience.”
I provided the patch for my testing, and it is very easy to reproduce the problem. Can you have someone from Nvidia to take a look?
Other users may not care a few frames lost at the end of the file. But it is critical for our application to get every frame.

Thanks,

8

Hi,
We do not dedicate to working in ffmpeg, so may not be able to suggest next. For the reason, the code is public so that users can check and share experience to each other.

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.