I was able to solve this. Before explaining details, I will briefly describe the solution and how to install mpv, vlc and ffmpeg with nvmpi and nvv4l2dec support. I have shared precompiled debs so it would be easy for anyone.
By the way, totem practically does not work on Jetson Nano, for me it was much slower than mpv with software decoding, and it does not come even close to mpv in therms of features anyway; gst-play-1.0 technically works but it does not support landscape mode on display with native portrait orientation which made it unusable for me (since it rotates all videos 90 degrees and I found no way to force it to landscape orientation) and lacks even more features than totem. This is exactly why I could not give up on this: mpv has a lot of features I actually need. VLC has a lot of features too, but I prefer mpv. But I got VLC working with hardware video acceleration too and made precompiled debs in case somebody prefers VLC.
==== mpv and ffmpeg with nvmpi and nvv4l2dec ====
To install, run the following in the directory where you saved mpv-nvmpi-nvv4l2dec.zip (copy and paste all three lines in the terminal):
wget http://Dragon.Studio/2021/02/mpv-nvmpi-nvv4l2dec.zip &&
unzip -n mpv-nvmpi-nvv4l2dec.zip &&
sudo apt -y --allow-change-held-packages --allow-downgrades install ./mpv-nvmpi-nvv4l2dec/*.deb &&
sudo apt-mark hold $(find mpv-nvmpi-nvv4l2dec | grep deb | xargs -n1 basename | cut -d_ -f1)
Last line is important otherwise Ubuntu may offer to “upgrade” ffmpeg or mpv with versions without nvmpi and nvv4l2dec.
Then in ~/.config/mpv/mpv.conf add the following line (create the file if it does not exist):
hwdec=nvmpi-copy
You can also try nvmpi
, nvv4l2dec
and nvv4l2dec-copy
as hwdec
but with nvmpi
I got the best performance. VLC will use NVMPI by default, no need to configure anything.
You can also use ffmpeg with hardware acceleration for both decoding and encoding, some examples:
ffmpeg -y -c:v h264_nvmpi -i input.mp4 -c:v hevc_nvmpi output.mp4 # Decode input.mp4 with H264 and encode it HEVC
ffmpeg -y -c:v hevc_nvmpi -i input.mp4 -c:v h264_nvmpi output.mp4 # Decode input.mp4 with HEVC and encode it to H264
For decoding, you can also use hevc_nvv4l2dec
, h264_nvv4l2dec
. Besides h264
or hevc
, you can also use vp9
, vp8
, mpeg2
or mpeg4
for decoding, just change prefix accordingly, for example: vp9_nvmpi
.
==== vlc 4.0.0-dev with nvmpi and qt 5.15 + ffmpeg with nvmpi and nvv4l2dec ====
This is how to install vlc with ffmpeg, compiled with nvmpi support (ffmpeg also includes nvv4l2dec support). This allows to use hardware accelerated video encoding and decoding in Jetson Nano. Tested with Ubuntu 18.04 and JetPack 4.5.
First, uninstall vlc if you have it installed (usually by running sudo apt remove vlc
).
To install, run the following in the directory where you saved mpv-nvmpi-nvv4l2dec.zip (copy and paste all three lines in the terminal):
wget http://Dragon.Studio/2021/02/vlc-nvmpi-qt515.zip &&
unzip -n vlc-nvmpi-qt515.zip &&
sudo apt -y --allow-change-held-packages --allow-downgrades install ./vlc-nvmpi-qt515/*.deb &&
sudo apt-mark hold $(find vlc-nvmpi-qt515 | grep deb | xargs -n1 basename | cut -d_ -f1)
Last line is important otherwise Ubuntu may offer to “upgrade” vlc or ffmpeg with versions without nvmpi. I had to use vlc version newer than included in Ubuntu 18.04 due to issues I encountered (I was unable to get it working with nvmpi).
Qt 5.15 is necessary for vlc 4.0.0-dev. I compiled all included Qt debs myself from sources provided by this repository Qt 5.15.2 for /opt Bionic : Stephan Binner since id did not include binaries for arm64 architecture, it took a lot of time and effort.
==== ffmpeg with nvmpi and nvv4l2dec ====
It is already included in mpv and vlc packages above, so if you installed debs from one of them, you already have it. But in case somebody needs it without mpv or vlc, I made a separate zip file. To install, run:
wget http://Dragon.Studio/2021/02/ffmpeg-3.4.8-nvmpi-nvv4l2dec.zip &&
unzip -n ffmpeg-3.4.8-nvmpi-nvv4l2dec.zip &&
sudo apt -y --allow-change-held-packages --allow-downgrades install ./ffmpeg-nvmpi-nvv4l2dec/*.deb &&
sudo apt-mark hold $(find ffmpeg-nvmpi-nvv4l2dec | grep deb | xargs -n1 basename | cut -d_ -f1)
This will install ffmpeg-3.4.8 with nvmpi and nvv4l2 support and will hold
installed packages to prevent Ubuntu 18.04 from overwriting them with versions without nvmpi and nvv4l2 support during system update. In the zip file, ffmpeg is of the same version as ffmpeg included in Ubuntu 18.04 at the time of writing. Using this version provided the best compatibility. I had issues when trying to use newer versions of ffmpeg in Ubuntu 18.04, this is why I ended up backporting nvmpi and nvv4l2dec patches to it.
==== Performance ====
FullHD and 2.5K at 60FPS plays smoothly. Both mpv and VLC have performance good enough to watch 4K@30FPS at stock NVDEC frequency. But 4K@60FPS in mpv is jerky und unwatchable and in VLC does not play at all because of too many dropped frames. Even with overclocked NVDEC I managed at most 4K@50FPS in mpv. This is because media players have some overhead. But for anything below and including 4K@50FPS hardware accelerated video decoding worked great for me.
gst-play-1.0
can play 4K@60FPS. But gst-play-1.0 is useless in my case since it does not support landscape mode on monitors with native portrait orientation so I have to use gst-launch-1.0 with custom pipeline:
gst-launch-1.0 filesrc location=video.mp4 ! decodebin ! nvvidconv flip-method=1 ! nvoverlaysink`
You need to use flip-method=0
if using display with native landscape orientation (or just use gst-play-1.0
). This command achieves the best video decoding performance, but obviously lacks mpv and VLC features, so I only resort to it if I encounter 4K@60FPS video. 4K@50FPS and below plays smoothly for me. Without overclocking, I guess up to 4K@40-45FPS should play smoothly in mpv, which is useful for watching 4K@30FPS at 1.5x speed (at which FPS effectively becomes 45). VLC performance is somewhat worse than mpv: VLC cannot play 4K@30FPS smoothly at 1.5x speed even with overclocking.
==== Low-latency playback in mpv and gstreamer ====
If lowest latency is needed (for example when watching real-time video from IP camera), since mpv version in Ubuntu 18.04 does not support low-latency profile, I had to use the following options to achieve the best best latency (within 0.1-0.2s range depending on IP camera and stream resolution; replace IP address with actual address of your IP camera):
mpv --no-cache --audio-buffer=0 --vd-lavc-threads=1 --cache-pause=no --no-audio --demuxer-lavf-probe-info=no --demuxer-lavf-analyzeduration=0.0 --video-sync=audio --interpolation=no --keep-open-pause=no --untimed --rtsp-transport=tcp rtsp://192.168.1.10/1
Another example of low-latency playback, this time with with gst-launch-1.0 to play video in landscape on mode on monitor with native portrait orientation (remove flip-method=1
or set it to 0
if your monitor has native landscape orientation; also adjust overlay width and height as needed, or remove all overlay-* parameters to use default values for fullscreen playback):
gst-launch-1.0 rtspsrc location=rtsp://169.254.160.104:554/av0_0 latency=0 drop-on-latency=true max-size-buffers=0 ! decodebin ! nvvidconv flip-method=1 ! nvoverlaysink overlay-x=0 overlay-y=0 overlay-w=1440 overlay-h=1920 sync=false -e
VLC is not useful for low latency playback as far as I can tell.
==== How I got nvmpi and nvv4l2dec working with mpv and vlc ====
The rest of this post is about is how I got ffmpeg, mpv and VLC to use video hardware acceleration. If you are not a programmer or not interested to know the details, feel free to skip the rest of this post.
I rebased NVidia ffmpeg patch to git version of ffmpeg, and also added on top the patch from jocover. As a result, I have ffmpeg with both nvv4l2dec (decoding only) and nvmpi (both encoding and decoding).
Unfortunately, at first, when trying to play videos with mpv, I get the following error (at the time I was working on this in January, the patch https://github.com/jocover/jetson-ffmpeg/pull/70 did not exist yet so I had no clue where to start):
mpv --vo=null --hwdec=nvmpi-copy /home/lissanro/Videos/jellyfish-10-mbps-hd-hevc.mkv
...
[ffmpeg/video] hevc_nvmpi: video_get_buffer: image parameters invalid
[ffmpeg/video] hevc_nvmpi: get_buffer() failed
Error while decoding frame (hardware decoding)!
...
This error is not specific to mpv. Somebody here Trying to add support for NVMPI ffmpeg decoders - The VideoLAN Forums attempted to add nvmpi surrport in VLC and they got the same issue. It is not nvmpi specific either, using nvv4l2dec produces almost the same error:
[ffmpeg/video] hevc_nvv4l2dec: video_get_buffer: image parameters invalid
[ffmpeg/video] hevc_nvv4l2dec: get_buffer() failed
Error while decoding frame (hardware decoding)!
Good news, ffplay does play video with hardware acceleration with both nvmpi and nvv4l2dec, and some software which depends on ffmpeg and its video acceleration also works flawlessly (for example, moonlight-qt).
But why not mpv and VLC? First, I decided to add some more debug information. The error happens in ffmpeg itself, this is why it is not specific to any particular player. This is the code where error happens (libavcodec/decode.c):
if (avctx->codec_type == AVMEDIA_TYPE_VIDEO) {
if ((unsigned)avctx->width > INT_MAX - STRIDE_ALIGN ||
(ret = av_image_check_size2(FFALIGN(avctx->width, STRIDE_ALIGN), avctx->height, avctx->max_pixels, AV_PIX_FMT_NONE, 0, avctx)) < 0 || avctx->pix_fmt<0) {
av_log(avctx, AV_LOG_ERROR, "video_get_buffer: image parameters invalid\n");
ret = AVERROR(EINVAL);
goto fail;
}
avctx->width
= 1920, so it cannot be bigger than INT_MAX
(=2147483647) - STRIDE_ALIGN
(=16), and av_image_check_size2()
returns 0 (which is not < 0
so it is good return value). All these avctx
values are good… except one - avctx->pix_fmt turned out to be -1 (AV_PIX_FMT_NONE
) which is not correct, and this is why this error is produced. It is -1 only few first frames but mpv decides not to use hardware decoding after the first error, then it becomes 0.
I tried to a hack to force avctx->pix_fmt = 0
right before the check. This worked well with --vo=null
- jtop
shows that hardware video decoding being used my mpv. So hardware acceleration can function in mpv. Unfortunately, this hack breaks with --vo=gpu
(at the time when I was debugging this, I did not care about deb packaging and I have used newer mpv version compiled from git; older mpv for Ubuntu 18.04 which comes in precompiled deb files does not support --vo=gpu
):
mpv --vo=gpu --hwdec=nvmpi-copy /home/lissanro/Videos/jellyfish-10-mbps-hd-hevc.mkv
(+) Video --vid=1 (*) (hevc 1920x1080 29.970fps)
[ffmpeg] swscaler: No accelerated colorspace conversion found from yuv420p to rgba.
[ffmpeg] swscaler: No accelerated colorspace conversion found from yuv420p to rgba.
double free or corruption (!prev)
zsh: abort (core dumped) mpv --vo=gpu --hwdec=nvmpi-copy
This is the point where I felt like I am on right track and closer to the solution. I remembered that on Raspberry Pi 3 B+ mpv did not work with hardware acceleration out of the box either but after building ffmpeg and mpv it was possible to make it work. It used MMAL decoder, so what it does differently with avctx->pix_fmt
? In ffmmal_init_decoder()
it has these two lines of code in libavcodec/mmaldec.c
:
int ret = 0;
...
avctx->pix_fmt = ret;
But libavcodec/nvmpi_dec.c
and libavcodec/nvv4l2_dec.c
lack pix_fmt
initialization. As soon as I added it, they both started to work in mpv and vlc too!
So it was never an issue with mpv. It was a bug in both ffmpeg decoders, this is why many other programs were affected including vlc.
When I found time to work on backporting to ffmpeg-3.4.8 and making deb packages, I updated to the latest patch from jetson-ffmpeg project, and I noticed some issues which did not exist before in nvmpi decoder. It turns out that later in January somebody independently tried to develop a similar fix for nvmpi decoder but it breaks decoding of some video streams. In pull requests there is a patch (which is not yet accepted) which supposed to make things better but it still does not make it as reliable as just initializing avctx->pix_fmt
to 0
- some videos which could decode before with nvmpi still fail to decode. This is why in the ffmpeg package provided by me, avctx->pix_fmt
is always initialized to 0
in NVMPI decoder so it does not have these issues (after all, MMAL decoder for Raspberry Pi does similar thing, and it is known to work well with mpv). Perhaps I create a pull request or report an issue to jetson-ffmpeg project later when I have time.
All zip files with debs I have provided also include patches I have used in case somebody wants to compile themselves.