nvEncOpenEncodeSessionEx results in NV_ENC_ERR_GENERIC in drivers after 537.58

[bas13]

(moved from nvEncOpenEncodeSessionEx results in NV_ENC_ERR_GENERIC in drivers after537.58)

Hi all,

We are working on an application for rendering video and it has been working for years now. We use NVidia encoding api 10.0 and after an update to a new driver around november/december openening an encoding session fails. Strange thing is that the same thing happens with the example application provided in the Video Codec SDK and I can’t figure out what’s the cause of this error.

What happens is that an error gets thrown in the following constructor:
NvEncoder::NvEncoder(NV_ENC_DEVICE_TYPE eDeviceType, void *pDevice, uint32_t nWidth, uint32_t nHeight, NV_ENC_BUFFER_FORMAT eBufferFormat,
uint32_t nExtraOutputDelay, bool bMotionEstimationOnly, bool bOutputInVideoMemory) :
m_pDevice(pDevice),
m_eDeviceType(eDeviceType),
m_nWidth(nWidth),
m_nHeight(nHeight),
m_nMaxEncodeWidth(nWidth),
m_nMaxEncodeHeight(nHeight),
m_eBufferFormat(eBufferFormat),
m_bMotionEstimationOnly(bMotionEstimationOnly),
m_bOutputInVideoMemory(bOutputInVideoMemory),
m_nExtraOutputDelay(nExtraOutputDelay),
m_hEncoder(nullptr)
{
LoadNvEncApi();

if (!m_nvenc.nvEncOpenEncodeSession) 
{
    m_nEncoderBuffer = 0;
    NVENC_THROW_ERROR("EncodeAPI not found", NV_ENC_ERR_NO_ENCODE_DEVICE);
}

NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS encodeSessionExParams = { NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS_VER };
encodeSessionExParams.device = m_pDevice;
encodeSessionExParams.deviceType = NV_ENC_DEVICE_TYPE_CUDA;// m_eDeviceType;
encodeSessionExParams.apiVersion = NVENCAPI_VERSION;
void *hEncoder = NULL;
NVENC_API_CALL(m_nvenc.nvEncOpenEncodeSessionEx(&encodeSessionExParams, &hEncoder));
m_hEncoder = hEncoder;

}

Calling NVENC_API_CALL(m_nvenc.nvEncOpenEncodeSessionEx(&encodeSessionExParams, &hEncoder)); will result in an exception with NV_ENC_ERR_GENERIC as error code.

I’ve also tried the 12.1 SDK resulting in the same issue. We are using NvEncoderD3D11.cpp as basis for our encoder code.
I’ve tested it with Windows 10 and 11 using different graphical cards like:
GTX 1060 Super
RTX 5000
RTX 4060
GTX 1080

Downgrading the driver makes the encoder work again.

Can anyone help me finding out whats going wrong? What can I do to solve this?

[bas13]

I’m trying to get this to work for a few days now. It works with drivers up to 537.58, not after. Is there somewhere where I can try to find more detailed information on what causes the NVENC error 20?

[mandar_godse]

Hi.
Are you able to reproduce this issue with SDK sample application? It will be helpful if you provide us repro application with exact steps to reproduce this issue.
Thanks.

[bas13]

Thank you for your reply. Sadly I can’t just share the repo due to legal restrains. I’m looking into what I can do.

I just build the examples. Those work. So the strange thing is that basically the same code works in one project and not in the other. That being said, it works in the other with older drivers not with newer. I’m looking at the differences trying to find out what it could be. My bet is that it is something in the compiler or linker options that causes this issue, but maybe I’m staring in the wrong direction.

[bas13]

After a lot of puzzling I’ve solved it. Thanks again for your reaction. Somehow older nvcuvid and nvEncodeAPI64 dll’s ended up in the resulting .Net directory causing the strange behavior.