Hardware accelerated video playback with L4T ffmpeg

[allcoms]

I have read in the L4T docs that ffmpeg is supposed to support hardware accelerated decoding of specific video codecs but I have been unable to achieve smooth playback of the UHD and 4K h264 and h265 videos I have tested.

I have had success playing hw accelerated videos under L4T with jetson-ffmpeg using commands like these:

ffplay -vcodec hevc_nvmpi -i vid.mp4
ffplay -vcodec h264_nvmpi -i vid.mp4

What would be the equivalent commands to play h264 or h265 videos accelerated using the L4T ffmpeg build?

I’m running JetPack 4.4 on a Jetson Nano.

Thanks

[DaneLLL]

Hi,
Please refer to this post:

Accelerated HW Decode with ffmpeg not working，Decoder h264 does not support device type cuda

Hi, Please check the steps of downloading source code in And the steps to configure/make/run ffmpeg with hardware decoding in http://ffmpeg.org/pipermail/ffmpeg-devel/2020-June/263746.html

[allcoms]

Thanks for that DaneLLL!

I’ve not tried building ffmpeg with that patch applied yet but according to this page:

https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide%2Fmultimedia.html%23wwpID0E0JB0HA

I don’t need to and nvv4l2dec should be included in the standard ffmpeg package but it seems that it isn’t, at least not in Jetpack 4.4.

So:

Could that page be updated with (a link to) these instructions?

Is this patch going to get upstreamed and become an official part of ffmpeg?

Are nvidia going to submit another patch to ffmpeg for encoding too or will we have to use jetson-ffmpeg for that?

Thanks

[allcoms]

I should also note that the ffmpeg page is linked to in the Jetson Nano FAQ, which also neglects to mention having to rebuild with this patch.

Jetson Nano FAQ

Q: I have a 256GB sd card, but it only shows 14 GB after bootup. Any idea? A: Please use Etcher tool to re-size SD card image: Download https://www.balena.io/etcher/ Execute Etcher tool and select “sd-blob.img” image -> select SDcard -> Flash After completed flash, plug-in SD Card to boot up Jetson-Nano Using SDK manager to install JetPack components Q: When I try to Install SDK components, the SDK Manager says “Could not detect Nvidia Jeson device connected to USB”. Does anyone also hit th…

[allcoms]

I’ve given this a quick test now, but only with one h264 file.

It seems to work when run with sudo but the patch fails to apply to both the latest git ffmpeg and the latest ffmpeg stable release. Instead you have to checkout the ffmpeg git commit from the end of May, right before this patch was released, to get it to apply cleanly.

Since my last post I have also checked out the ffmpeg devs responses to this patch on the ffmpeg-devel ml and they aren’t happy with how its been implemented. It sounds like it won’t be upstreamed without a pretty major revision.

[DaneLLL]

Hi,
The steps should work fine without building the source manually:

$ echo "deb https://repo.download.nvidia.com/jetson/ffmpeg main main" |  sudo tee -a /etc/apt/sources.list
$ echo "deb-src https://repo.download.nvidia.com/jetson/ffmpeg main main" |  sudo tee -a /etc/apt/sources.list
$ sudo apt update
$ sudo apt install ffmpeg

After executing the steps, you should see the hardware decoders:

nvidia@nvidia-desktop:/$ ffmpeg -decoders | grep nvv4l2
ffmpeg version n4.2.2-15-g6878ea5a44 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04)
  configuration: --prefix=/usr --enable-nvv4l2dec --enable-libv4l2 --enable-shared --extra-libs='-L/usr/lib/aarch64-linux-gnu/tegra -lnvbuf_utils' --extra-cflags='-I /usr/src/jetson_multimedia_api/include/'
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
 V..... h264_nvv4l2dec       h264 (nvv4l2dec) (codec h264)
 V..... hevc_nvv4l2dec       hevc (nvv4l2dec) (codec hevc)
 V..... mpeg2_nvv4l2dec      mpeg2 (nvv4l2dec) (codec mpeg2video)
 V..... mpeg4_nvv4l2dec      mpeg4 (nvv4l2dec) (codec mpeg4)
 V..... vp8_nvv4l2dec        vp8 (nvv4l2dec) (codec vp8)
 V..... vp9_nvv4l2dec        vp9 (nvv4l2dec) (codec vp9)

[allcoms]

Thanks DaneLLL!

The instructions at:

https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide%2Fmultimedia.html%23wwpID0E0JB0HA

Need to be corrected. The page is titled ‘Accelerated decode with ffmpeg’ but if you follow the instructions on there as-is users will end up with the non-accelerated build.

Unfortunately ffplay isn’t included in the nvidia ffmpeg package. If you build ffmpeg from source ffplay gets built in a standard make and ffplay is also included in the normal Ubuntu ffmpeg package so please include ffplay in future updates of ffmpeg.

I’m also disappointed that I don’t see any nvv4l2dec decoder options when I run mpv --vd=help after installing the nvidia ffmpeg. I’m going to try rebuilding mpv to see if that fixes it.

EDIT

Actually, it’s mpv --hwdec=help to show the list of hardware decoding options but yes, still no nvv4l2dec. Looks like ffmpeg isn’t as closely tied to mpv as I hoped.

[allcoms]

I’ve had a quick look at the latest git mpv source and there is no link between ffmpeg and mpv as far as the hardware decoding is concerned, which makes me wonder which would be easier to port to mpv , nvv4l2dec or nvmpi, the jetson-ffmpeg decoder/encoder?

Here are all of the currently supported mpv hardware decoding backends:

wscript_build.py:        ( "video/out/d3d11/hwdec_d3d11va.c",     "d3d11 && d3d-hwaccel" ),
wscript_build.py:        ( "video/out/d3d11/hwdec_dxva2dxgi.c",   "d3d11 && d3d9-hwaccel" ),
wscript_build.py:        ( "video/out/gpu/hwdec.c" ),
wscript_build.py:        ( "video/out/hwdec/hwdec_cuda.c",        "cuda-interop" ),
wscript_build.py:        ( "video/out/hwdec/hwdec_cuda_gl.c",     "cuda-interop && gl" ),
wscript_build.py:        ( "video/out/hwdec/hwdec_cuda_vk.c",     "cuda-interop && vulkan" ),
wscript_build.py:        ( "video/out/hwdec/hwdec_vaapi.c",       "vaapi-egl || vaapi-vulkan" ),
wscript_build.py:        ( "video/out/hwdec/hwdec_vaapi_gl.c",    "vaapi-egl" ),
wscript_build.py:        ( "video/out/hwdec/hwdec_vaapi_vk.c",    "vaapi-vulkan" ),
wscript_build.py:        ( "video/out/opengl/hwdec_d3d11egl.c",   "d3d-hwaccel && egl-angle" ),
wscript_build.py:        ( "video/out/opengl/hwdec_drmprime_drm.c","drm" ),
wscript_build.py:        ( "video/out/opengl/hwdec_dxva2egl.c",   "d3d9-hwaccel && egl-angle" ),
wscript_build.py:        ( "video/out/opengl/hwdec_dxva2gldx.c",  "gl-dxinterop-d3d9" ),
wscript_build.py:        ( "video/out/opengl/hwdec_ios.m",        "ios-gl" ),
wscript_build.py:        ( "video/out/opengl/hwdec_osx.c",        "videotoolbox-gl" ),
wscript_build.py:        ( "video/out/opengl/hwdec_rpi.c",        "rpi-mmal" ),
wscript_build.py:        ( "video/out/opengl/hwdec_vdpau.c",      "vdpau-gl-x11" ),

[rreddy78]

@allcoms

ffplay -vcodec hevc_nvmpi -i vid.mp4
ffplay -vcodec h264_nvmpi -i vid.mp4

Sorry for an offtopic. But how did you build the ffplay. When I install

sudo apt install ffmpeg

I see only ffmpeg and ffprobe in the /usr/bin

Even when I build from sources I can only see those two binaries built.

[allcoms]

Hi rreddy78

When you build jetson-ffmpeg or the regular ubuntu ffmpeg ffplay gets built, not the nvidia ffmpeg.

[rreddy78]

Hi @allcoms

I didnt quite get what you say. I have a followed the instructions in this topic. And I get the results fine:

rreddy78@jetson:~/ffmpeg-4.2.2$ ./ffmpeg -decoders | grep nvv4l2
V… h264_nvv4l2dec h264 (nvv4l2dec) (codec h264)
V… hevc_nvv4l2dec hevc (nvv4l2dec) (codec hevc)
V… mpeg2_nvv4l2dec mpeg2 (nvv4l2dec) (codec mpeg2video)
V… mpeg4_nvv4l2dec mpeg4 (nvv4l2dec) (codec mpeg4)
V… vp8_nvv4l2dec vp8 (nvv4l2dec) (codec vp8)
V… vp9_nvv4l2dec vp9 (nvv4l2dec) (codec vp9)

I think I have the jetson-ffmpeg built, but I am not sure why ffplay does not get built.

I have a hunch it does not get built because SDL is NOT available (on which ffplay depends in suppose…?)

[allcoms]

sudo apt build-dep ffmpeg

[rreddy78]

Solved!

FFMPEG: FFPLAY binary not getting generated on compilation

It was because I did not have libsdl2-dev package.

sudo apt-get install libsdl2-dev and now building again…

Thanks a lot…

[rreddy78]

Hi @DaneLLL

I have the ffmpeg working. I checked with a 1080p 30fps h264 encoded file:

rreddy78@jetson:/media/96747D21747D0571/Documentaries/HD Video$ ffprobe gives

Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 3259 kb/s, 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc (default)
Metadata:
handler_name : VideoHandler

sudo ffplay -vcodec h264_nvv4l2dec is working well an video plays smoothly.

I see this:

Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
59.01 A-V: -0.010 fd= 5 aq= 20KB vq= 122KB sq= 0B f=0/5

Does this mean HW decoded is being used ?

Because I see that all the CPU cores are occupied at 40%. Pity we dont have a Windows TaskManager like GPU usage visualizer…

[DaneLLL]

Hi,

@rreddy78

Does this mean HW decoded is being used ?

Yes, this shows hardware decoder i srunning. For confirmation, you can run sudo tegrastats and check if NVDEC is present.

[rreddy78]

@DaneLLL

Yes, this shows hardware decoder i srunning. For confirmation, you can run sudo tegrastats and check if NVDEC is present.

Thanks @DaneLLL,
I see that accelerated multimedia is possible with ffmpeg and gstreamer on the jetson devices

I just had a basic question:
If the ffmpeg and gstreamer are accelerated, are the applications based on ffmpeg (https://github.com/amiaopensource/ffmpeg-amia-wiki/wiki/3)-Graphical-User-Interface-Applications-using-FFmpeg) and gstreamer (https://gstreamer.freedesktop.org/apps/) are also accelerated if I build those applications from source and use the NVIDIA ffmpeg and gstreamer libraries ?

Thanks

[DaneLLL]

Hi,

@rreddy78

If the ffmpeg and gstreamer are accelerated, are the applications based on ffmpeg (https://github.com/amiaopensource/ffmpeg-amia-wiki/wiki/3)-Graphical-User-Interface-Applications-using-FFmpeg) and gstreamer (https://gstreamer.freedesktop.org/apps/) are also accelerated if I build those applications from source and use the NVIDIA ffmpeg and gstreamer libraries ?

In ffmpeg, you would need to check if the decoder(such as h264_nvv4l2dec) is picked by the applications.
In gstreamer, you would need to use NVIDIA plugins such as nvv4l2decoder. Please take a look at gstreamer user guide.