H.264 encoder fails with NV_ENC_ERR_UNSUPPORTED_PARAM with Insider builds of Windows

Gaming and Visualization Technologies Video Processing & Optical Flow
1

We have our software using hardware H.264 encoder with various NVIDIA adapters. Recently we started getting reports from users on unexpected failures, and the are all on insider builds of Windows (e.g. 19033 and higher).

At this moment it is hard to tell more details even though we have some minidumps on hands because debug symbols for insider builds are not published. So far we know:

  • we are using Direct3D 11 interface of the encoder
  • the code is built against Video Codec SDK 6.0.1 headers
  • the error is NV_ENC_ERR_UNSUPPORTED_PARAM and more rarily NV_ENC_ERR_OUT_OF_MEMORY
  • the problem takes place with variety of GPUs, RTX 2080 Ti and GTX 970 just for examples
  • encoder API DLL versions vary, to give some reference: 27.21.14.5012

If there are any known problem or workaround, please share. I will update as I have more details.

2

Also one of the users reported that he has side by side another video encoding software which seemingly works, so I am guessing it might be something like this https://devtalk.nvidia.com/default/topic/1069890/video-codec-and-optical-flow-sdk/possible-bug-forceidr-to-resend-sps-pps-does-not-work-in-d3d11-encoder-api-but-works-in-cuda/ related, or there is a composition of problems.

3

OK, so for whatever reason NVIDIA Video Encoder API started enumerating 7 new presets (whether these are new yet not announced and documented ones, or it’s a math problem with number of elements in enumeration). When preset details are requested the API responds with a failure as I mentioned above.

And yeah I only managed to reproduce the problem once I upgraded a rig to fast ring insider build 19555.