What is the recommended way to deal with NV_ENC_ERR_OUT_OF_MEMORY

Gaming and Visualization Technologies Video Processing & Optical Flow
1

Hi,

We’re experiencing NV_ENC_ERR_OUT_OF_MEMORY errors when calling nvEncEncodePicture on VMs with an A10 GPU using vGPU profiles of 1GB and 2GB VRAM. The issue occurs more frequently at 4K resolution but is also reproducible at 1080p when concurrent GPU-intensive applications (e.g., Unigine Benchmark) are running. On a 1GB profile, we’ve observed multiple occurrences per day during 4K stream encoding.

Currently, our only mitigation strategy is encoder re-initialization, which introduces unacceptable latency for real-time encoding scenarios; because of this, we have a software encoding fallback.

Some additional info:

  • Codec: H.264
  • Input source: DirectX texture (derived from desktop duplication API)
  • Tested implementations: Both libavcodec NVENC and our custom NVENC implementation exhibit identical behavior
  • Issue appears mitigated on 4GB profiles, suggesting exponential reliability improvement with increased VRAM allocation
  • Trying to ignore the error and encode again gives NV_ENC_ERR_INVALID_PARAM

Specific areas where I’m seeking guidance:

  1. Memory management best practices for low VRAM profiles
  2. Efficient error recovery strategies that don’t require full encoder reconstruction
  3. Potential configuration optimizations to reduce memory pressure

I understand the inherent limitations of 1GB VRAM profiles but I’m still interested in any possible optimization strategies or architectural recommendations.

If this is more of a driver issue or if I should move this post then please let me know.

Thanks,
Félix