Summary
We have isolated a severe, linear memory leak inside the NVENC driver wrapper when calling nvEncReconfigureEncoder repeatedly. This leak occurs exclusively when combining the H.264 codec with the D3D12 backend.
To ensure this is not caused by application-level mismanagement, we have written a zero-dependency, standalone C++ reproduction tool (attached as a ZIP) that completely eliminates frame encoding, input/output resource registration, or texture allocations. It purely spins up an NVENC session, initializes the codec, and repeatedly triggers the reconfigure API.
Core Findings
- H.264 + D3D12: Process private bytes grow linearly by ~3.1 to 3.3 MiB per
nvEncReconfigureEncodercall. - H.264 + D3D11 / CUDA: Perfectly stable (0 MiB leak over thousands of iterations).
- HEVC/H.265 + D3D12: Perfectly stable (0 MiB leak), confirming the issue is specific to the H.264 implementation layer over D3D12.
- Orphaned Memory: Calling
nvEncDestroyEncoderand releasing the D3D12 device does not free the leaked private bytes, leaving massive orphaned allocations in the heap.
Tested Environment
- GPU: NVIDIA GeForce RTX 4070 SUPER
- NVIDIA Driver: 610.47
- NVENC SDK Header: 13.0
- OS: Windows 10 (10.0.19045)
- Base Configuration: 1920x1080 @ 60 FPS, 15 Mbps, Preset P4 + Low Latency Tuning.
Reproduction Matrix & Empirical Data
Below is the automated summary output from running our validation matrix for 1,000 iterations per test scenario.
| Ablation / Test Case | Total Leak (MiB) | Per-Iter Leak (KiB) | Leak Remaining After Destroy (MiB) | Status |
|---|---|---|---|---|
baseline (H264+D3D12) |
+3302 | 3382 | +3296 | ❌ LEAKING |
bitrate_only (H264+D3D12) |
+3302 | 3381 | +3296 | ❌ LEAKING |
cbr (H264+D3D12) |
+3302 | 3382 | +3296 | ❌ LEAKING |
d3d11 (H264+D3D11) |
+0 | 0 | +0 | Stable |
cuda (H264+CUDA) |
+0 | 0 | +0 | Stable |
hevc_d3d12 (HEVC+D3D12) |
+67 | 0 | -49 | Stable |
Note: For hevc_d3d12, the transient +67 MiB is initial workspace allocation, which successfully gets cleaned up upon session destruction (returning to baseline), whereas H.264 permanently abandons ~3.3 GiB of memory.
Repro Steps & Attached Tool
I have attached a full reproduction package nvenc_d3d12_h264_leak_repro.zip containing:
nvenc_reconfigure_probe.cpp- The standalone minimal reproducible C++ source code.run_nvenc_reconfigure_probe.ps1- An automated PowerShell script that auto-locates the VS environment (vswhere), compiles, and runs the entire validation suite seamlessly.
How to run:
$env:NVENC_SDK_INCLUDE = "C:\Path\To\Your\NVENC\SDK\include"
.\run_nvenc_reconfigure_probe.ps1 -Iterations 1000 -LogEvery 50
nvenc_reconfigure_probe.zip (11.2 KB)