Hardware accelerated Ffmpeg on Jetson

Hello, there are multiple posts mentioning only hardware accelerated decoding is supported by Jetson products but encoding will be supported in a later release https://forums.developer.nvidia.com/t/the-nvidia-ffmpeg-package-supports-hardware-accelerated-decode-on-jetson-platforms/123694/7

I have tried jocover’s oss alternative but no luck with it.

Is there any official roadmap for hardware accelerated encoding for ffmpeg?

Thanks

Are you asking for which Jetson platform?

Hi,
It is not planned yet. Please use the community contribution.
Jetson Nano FAQ

Q: Is hardware acceleration enabled in ffmpeg?

Or please consider use jetson_multimedia_api. For hooking with ffmpeg, copying decoded YUVs to CPU buffer is done in the frameworks. In jetson_multiemdia_api, you can have frame data in hardware DMA buffers and use hardware engines, without additional memory copy.

It should be possible to get some H264 encoding support

Not extensively tested with JP-5.0.2 (and also faced errors with H264 decoding), but seems ok with AGX Orin (not for Xavier-NX). Try this :

sudo apt update
sudo apt-get install -y nvidia-l4t-jetson-multimedia-api

mkdir ffmpeg_top
cd ffmpeg_top

# Build jocover's nvmpi lib
git clone https://github.com/jocover/jetson-ffmpeg.git
cd jetson-ffmpeg
cp /usr/src/jetson_multimedia_api/include/nvbuf_utils.h include
cd ..
mkdir build-jetson-ffmpeg-jocover
cd build-jetson-ffmpeg-jocover
cmake -D CMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=/usr/local/jetson-jocover-nvmpi ../jetson-ffmpeg
#cmake -D CMAKE_BUILD_TYPE=DEBUG -DCMAKE_INSTALL_PREFIX=/usr/local/jetson-jocover-nvmpi ../jetson-ffmpeg
make -j$(nproc)
sudo make install
sudo ldconfig
cd ..
export PKG_CONFIG_PATH=/usr/local/jetson-jocover-nvmpi/share/pkgconfig:$PKG_CONFIG_PATH
export LD_LIBRARY_PATH=/usr/local/jetson-jocover-nvmpi/lib:$LD_LIBRARY_PATH


# Build ffmpeg 4.2
sudo apt-get install -y libcanberra-gtk-module libopenmpt-dev libxvidcore-dev libsdl2-dev libx264-dev

git clone git://source.ffmpeg.org/ffmpeg.git -b release/4.2 --depth=1
cd ffmpeg

# jocover's nvmpi patch
wget https://github.com/jocover/jetson-ffmpeg/raw/master/ffmpeg_nvmpi.patch
git apply ffmpeg_nvmpi.patch
export PKG_CONFIG_PATH=/usr/local/share/pkgconfig:$PKG_CONFIG_PATH

mkdir build

# Simple debug build enabling nvmpi
./configure --extra-libs='-L/usr/lib/aarch64-linux-gnu/tegra -lnvbuf_utils' --extra-cflags='-I /usr/src/jetson_multimedia_api/include/' --enable-nvmpi --prefix=/usr/local/ffmpeg --pkgconfigdir=/usr/local/ffmpeg/lib/pkgconfig --enable-debug

# For JP5 new jetson mmapi:
#./configure --extra-libs='-L/usr/lib/aarch64-linux-gnu/tegra -lnvbuf_utils -lnvbufsurface' --extra-cflags='-I /usr/src/jetson_multimedia_api/include/' --enable-nvmpi --prefix=/usr/local/ffmpeg --pkgconfigdir=/usr/local/ffmpeg/lib/pkgconfig --enable-debug

# A more featured release build
#./configure --toolchain=hardened --extra-libs='-L/usr/lib/aarch64-linux-gnu/tegra -lnvbuf_utils -lnvbufsurface' --extra-cflags='-I /usr/src/jetson_multimedia_api/include/' --enable-nvmpi --enable-gpl --disable-stripping --enable-opencl --enable-avisynth --enable-libfontconfig --enable-libfreetype --enable-libopenmpt --enable-libpulse --enable-libx264 --enable-libxml2 --enable-libxvid --enable-libv4l2 --enable-opengl --enable-sdl2 --enable-libdrm --enable-shared --prefix=/usr/local/ffmpeg --pkgconfigdir=/usr/local/ffmpeg/lib/pkgconfig

make -j$(nproc)
sudo make install
cd ../..
export LD_LIBRARY_PATH=/usr/local/ffmpeg/lib:$LD_LIBRARY_PATH
export PATH=/usr/local/ffmpeg/bin:$PATH
export PKG_CONFIG_PATH=/usr/local/ffmpeg/lib/pkgconfig:$PKG_CONFIG_PATH

# Test
# Launch another terminal for monitoring with tegrastats... When running, NVENC should appear and display rate
ffmpeg -v verbose -i <your_source_file> -c:v h264_nvmpi test_out.mkv

For H264 decoding with JP-5.0.2

On both AGX Orin and Xavier-NX, there is a segmentation fault in libtegrav4l2.
@DaneLLL , @jocover, may you check if the error is in jetson-ffmpeg or in libtegrav4l2 ? Details below:

gdb ffplay 
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ffplay...
(No debugging symbols found in ffplay)
(gdb) set args ../../Videos/YachtRide_1920x1080_30fps_420_8bit_AVC_MP4.mp4 -vcodec h264_nvmpi
(gdb) run
Starting program: /usr/local/ffmpeg/bin/ffplay ../../Videos/YachtRide_1920x1080_30fps_420_8bit_AVC_MP4.mp4 -vcodec h264_nvmpi
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
ffplay version a1b534b Copyright (c) 2003-2022 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --extra-libs='-L/usr/lib/aarch64-linux-gnu/tegra -lnvbuf_utils -lnvbufsurface' --extra-cflags='-I /usr/src/jetson_multimedia_api/include/' --enable-nvmpi --prefix=/usr/local/ffmpeg --pkgconfigdir=/usr/local/ffmpeg/lib/pkgconfig --enable-debug
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
[New Thread 0xfffff47a3900 (LWP 99702)]
[New Thread 0xfffff1abf900 (LWP 99705)]
[New Thread 0xffffe17e5900 (LWP 99706)]
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '../../Videos/YachtRide_1920x1080_30fps_420_8bit_AVC_MP4.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf55.9.100
  Duration: 00:00:20.00, start: 0.000000, bitrate: 3122 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080, 3119 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)
    Metadata:
      handler_name    : VideoHandler
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
[New Thread 0xffffdba64900 (LWP 99707)]
[New Thread 0xffffd9872900 (LWP 99708)]
[New Thread 0xffffd8e70900 (LWP 99709)]
NvMMLiteBlockCreate : Block : BlockType = 261 
[New Thread 0xffffcbda3900 (LWP 99710)]
[New Thread 0xffffcb5a2900 (LWP 99711)]
[New Thread 0xffffcada1900 (LWP 99712)]
    nan M-V:    nan fd=   0 aq=    0KB vq=  367KB sq=    0B f=0/0   
Thread 9 "read_thread" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xffffcb5a2900 (LWP 99711)]
0x0000ffffe0fad438 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libtegrav4l2.so
(gdb) bt
#0  0x0000ffffe0fad438 in  () at /usr/lib/aarch64-linux-gnu/tegra/libtegrav4l2.so
#1  0x0000ffffe0fa7dd0 in TegraV4L2_Ioctl () at /usr/lib/aarch64-linux-gnu/tegra/libtegrav4l2.so
#2  0x0000fffff1a700a8 in plugin_ioctl () at /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvideocodec.so
#3  0x0000fffff73ea390 in v4l2_ioctl () at /lib/aarch64-linux-gnu/libv4l2.so.0
#4  0x0000fffff79a678c in NvV4l2ElementPlane::qBuffer(v4l2_buffer&, NvBuffer*) (this=0xffffd4011948, v4l2_buf=..., shared_buffer=0x0) at /usr/src/jetson_multimedia_api/samples/common/classes/NvV4l2ElementPlane.cpp:252
#5  0x0000fffff79990b0 in respondToResolutionEvent(v4l2_format&, v4l2_crop&, nvmpictx*) (format=..., crop=..., ctx=0xffffd4012570) at /home/nvidia/Desktop/ffmpeg/jetson-ffmpeg/nvmpi_dec.cpp:186
#6  0x0000fffff799920c in dec_capture_loop_fcn(void*) (arg=0xffffd4012570) at /home/nvidia/Desktop/ffmpeg/jetson-ffmpeg/nvmpi_dec.cpp:210
#7  0x0000fffff799cc20 in std::__invoke_impl<void*, void* (*)(void*), nvmpictx*>(std::__invoke_other, void* (*&&)(void*), nvmpictx*&&) (__f=@0xffffd41c9b60: 0xfffff799911c <dec_capture_loop_fcn(void*)>) at /usr/include/c++/9/bits/invoke.h:60
#8  0x0000fffff799cb58 in std::__invoke<void* (*)(void*), nvmpictx*>(void* (*&&)(void*), nvmpictx*&&) (__fn=@0xffffd41c9b60: 0xfffff799911c <dec_capture_loop_fcn(void*)>) at /usr/include/c++/9/bits/invoke.h:95
#9  0x0000fffff799caac in std::thread::_Invoker<std::tuple<void* (*)(void*), nvmpictx*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=0xffffd41c9b58) at /usr/include/c++/9/thread:244
#10 0x0000fffff799ca40 in std::thread::_Invoker<std::tuple<void* (*)(void*), nvmpictx*> >::operator()() (this=0xffffd41c9b58) at /usr/include/c++/9/thread:251
#11 0x0000fffff799ca0c in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void* (*)(void*), nvmpictx*> > >::_M_run() (this=0xffffd41c9b50) at /usr/include/c++/9/thread:195
#12 0x0000fffff72cdfac in  () at /lib/aarch64-linux-gnu/libstdc++.so.6
#13 0x0000fffff787d624 in start_thread (arg=0xfffff72cdf90) at pthread_create.c:477
#14 0x0000fffff77d449c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Hi,
We don’t support TX1 on Jetpack 5, so the error may not be hit on Jetpack 4. Ideally jocover’s implementation should work fine on Jetpack 4 releases.