We suspect use of some uninitialized memory within the nvcuvid library resulting in an invalid level error being returned at initialization of the nvenc video encoder.
Test environment
| OS | Ubuntu 24.04.3 LTS |
| Linux version | Linux 6.8.0-85-generic #85-Ubuntu SMP PREEMPT_DYNAMIC Thu Sep 18 15:26:59 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| GPU | NVIDIA GeForce RTX 5050 |
| Driver | 580.126.09 (also 580.76.05) |
| Docker OS | Ubuntu 22.04 |
We want to encode the following video stream:
| Codec | hevc |
| Frame size | 3840x2160 |
| Scan type | Progressive |
| Pixel format | yuv 422 10 bit |
| Profile | Main 4:2:2 10 (Rext) |
| Level | 5.1 |
| Tier | High |
| Bitrate | 80 Mbps |
| B-frame count | 2 |
We’re using ffmpeg build based on tag 8.0.1 with nvenc enabled:
ffmpeg version n8.0.1 Copyright (c) 2000-2025 the FFmpeg developers
built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04.2)
configuration: --enable-nvenc
libavutil 60. 8.100 / 60. 8.100
libavcodec 62. 11.100 / 62. 11.100
libavformat 62. 3.100 / 62. 3.100
libavdevice 62. 1.100 / 62. 1.100
libavfilter 11. 4.100 / 11. 4.100
libswscale 9. 1.100 / 9. 1.100
libswresample 6. 1.100 / 6. 1.100
We use the following (simplified) ffmpeg command line:
./ffmpeg -f lavfi -i testsrc2=size=3840x2160:rate=50 \
-vf format=p210le \
-t 10 \
-c:v hevc_nvenc \
-profile:v rext \
-level:v 5.1 \
-tier:v high \
-pix_fmt p210le \
-b:v 80M \
-g 50 \
-bf 2 \
-preset p5 \
-r 50 \
-f mpegts output.ts \
-y
When use the above command we get varying results.
Most of the times the nvenc InitializeEnoder returns an error:
[hevc_nvenc @ 0x4e42340] InitializeEncoder failed: invalid param (8): Invalid Level.
According to us the level 5.1 should be valid for this combination of parameters and it looks like a false report.
Sometimes it encodes without any problems.
We have also run the ffmpeg command via valgrind and forcing the allocated memory to be filled with 0xFF. In that case we get 100% occurance of the error.
valgrind \
--tool=memcheck \
--leak-check=no \
--track-origins=no \
--show-reachable=no \
--undef-value-errors=no \
--num-callers=4 \
--malloc-fill=0xFF \
--free-fill=0xFF \
When we force the memory to be set to 0x00 it will run succesfully every time.
With the valgrind memory check it also looks whether it has conditional jumps on uninitialized memory (allocated and used by libnvcuvid.so):
==136501== Conditional jump or move depends on uninitialised value(s)
==136501== at 0x31751D3E: ??? (in /usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.126.09)
==136501== by 0x3174D977: ??? (in /usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.126.09)
==136501== by 0x317B557E: ??? (in /usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.126.09)
==136501== by 0x316C5D96: ??? (in /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.580.126.09)
==136501== Uninitialised value was created by a stack allocation
==136501== at 0x317516AF: ??? (in /usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.126.09)
==136501==
==136501== Conditional jump or move depends on uninitialised value(s)
==136501== at 0x31751D45: ??? (in /usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.126.09)
==136501== by 0x3174D977: ??? (in /usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.126.09)
==136501== by 0x317B557E: ??? (in /usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.126.09)
==136501== by 0x316C5D96: ??? (in /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.580.126.09)
==136501== Uninitialised value was created by a stack allocation
==136501== at 0x317516AF: ??? (in /usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.126.09)
==136501==
==136501== Conditional jump or move depends on uninitialised value(s)
==136501== at 0x31751D6E: ??? (in /usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.126.09)
==136501== by 0x3174D977: ??? (in /usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.126.09)
==136501== by 0x317B557E: ??? (in /usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.126.09)
==136501== by 0x316C5D96: ??? (in /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.580.126.09)
==136501== Uninitialised value was created by a stack allocation
==136501== at 0x317516AF: ??? (in /usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.126.09)
==136501==
==136501== Conditional jump or move depends on uninitialised value(s)
==136501== at 0x31751D76: ??? (in /usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.126.09)
==136501== by 0x3174D977: ??? (in /usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.126.09)
==136501== by 0x317B557E: ??? (in /usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.126.09)
==136501== by 0x316C5D96: ??? (in /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.580.126.09)
==136501== Uninitialised value was created by a stack allocation
==136501== at 0x317516AF: ??? (in /usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.126.09)
==136501==
==136501== Conditional jump or move depends on uninitialised value(s)
==136501== at 0x3180446F: ??? (in /usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.126.09)
==136501== by 0x3180BABB: ??? (in /usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.126.09)
==136501== by 0x317BA28F: ??? (in /usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.126.09)
==136501== by 0x316D283F: ??? (in /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.580.126.09)
==136501== Uninitialised value was created by a heap allocation
==136501== at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==136501== by 0x317C9089: ??? (in /usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.126.09)
==136501== by 0x317BA2C4: ??? (in /usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.126.09)
==136501== by 0x316D283F: ??? (in /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.580.126.09)
Additional findings:
- When we disable b-frames (-bf 0) then we don’t get this error.
- When we set the level to auto the encoded output sometimes has level 6.2 and sometimes 5.1.
(Cautious) Conclusion:
- It looks like the libnvcuvid.so library makes use of uninitialized memory. The video level calculation done by the library is affected by this calculation
Please let us know whether additional information is required.