FFmpeg NVENC CUDA Encoding Fails: "Failed to query nvenc max version"

Gaming and Visualization Technologies Video Processing & Optical Flow
, ,
1

I am developing a video processing application that uses FFmpeg + CUDA + NVENC for hardware-accelerated encoding.
However, when I try to encode a video using h264_nvenc, I encounter the following error:

Command:

ffmpeg -y -vsync 0 -hwaccel cuda -i input.mp4 -c:a copy -c:v h264_nvenc output.mp4

Error Message:

[h264 @ 0x55e18ff71000] decoder->cvdl->cuvidGetDecoderCaps(&caps) failed
[h264 @ 0x55e18ff71000] Failed setup for format cuda: hwaccel initialisation returned error.
[h264_nvenc @ 0x55e18f441300] Failed to query nvenc max version: unknown error (402681824): (no details)

System Information:

  • OS: Ubuntu 20.04
  • GPU: NVIDIA RTX 4090
  • CUDA Version: 12.2
  • NVIDIA Driver Version: 535.161.07
  • FFmpeg Version: N-118453-ga28dc06869 (custom-built)

FFmpeg Configuration:

I built FFmpeg with the following options:

./configure --enable-gpl --enable-libx264 --enable-libx265 --enable-nvenc --enable-cuda \
--enable-cuvid --enable-nonfree --enable-cuda-nvcc --enable-libnpp \
--extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64
make -j$(nproc)
make install

Steps I Have Taken:

1. Updated nv-codec-headers to match my driver version
git clone https://git.videolan.org/git/ffmpeg/nv-codec-headers.git
cd nv-codec-headers
git checkout sdk/12.0
sudo make install
2. Recompiled FFmpeg using CUDA 12.2 and NVENC
./configure --enable-gpl --enable-libx264 --enable-libx265 --enable-nvenc --enable-cuda \
--enable-cuvid --enable-nonfree --enable-cuda-nvcc --enable-libnpp \
--extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64
make -j$(nproc)
make install
3. Installed Video Codec SDK 12.0.16 and manually copied
cp Video_Codec_SDK_12.0.16/libnvcuvid.so /usr/lib/x86_64-linux-gnu/
ln -s /usr/lib/x86_64-linux-gnu/libnvcuvid.so /usr/lib/x86_64-linux-gnu/libnvcuvid.so.1
cp Video_Codec_SDK_12.0.16/libnvidia-encode.so /usr/lib/x86_64-linux-gnu/
ln -s /usr/lib/x86_64-linux-gnu/libnvidia-encode.so /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.1
4. Verified CUDA is detected in FFmpeg
ffmpeg -hwaccels
Output:
Hardware acceleration methods:
cuda

But this error still occurs

2

May I ask if you have solved it? I also encountered this problem

3

I haven’t solved it yet

4

Oof, I’ve hit something similar before but never figured out the exact reason. Sometimes it was just a driver mismatch, other times FFmpeg wasn’t built with proper NVENC support.

Just curious — are you running this in a container or straight on the OS? I’ve seen this error pop up more in Docker setups. Also, which GPU are you using?

Would be super helpful if someone has a checklist for when this happens. Seems like a pretty common pain point.

5

Hi @594764633, you do not need the step “3”. The NVIDIA driver has all the needed libraries. Please remove the manually copied files from step 3 and perform a clean driver installation.

Observation: If you do not use FFmpeg NPP-based filters, you do not need CUDA. Your FFmpeg configure command line can be: ./configure --enable-gpl --enable-libx264 --enable-libx265