NVEnc Issue with Delphi since API V12.1

Gaming and Visualization Technologies Video Processing & Optical Flow
1

Hello, I am an a programmer for VSO SOFTWARE, an independant software vendor. We use NVEnc in our Blu-ray or video conversion solution (ConvertXtoHD, ConvertXtoVideo)

Our programming language is Delphi. We use NVEnc since 2014 in our code (windows 32 bits).
Our code was based on API 8.1 and worked very well up to API V11.

Starting from driver V 522 and API V12.1, our code fail to work.

Everything is working : initialization, opening encoding session, allocation of surfaces, getting extradata for global header. But calling nvEncEncodePicture() function now systematically return an error NV_ENC_ERR_GENERIC.
Calling after the function nvEncGetLastErrorString() return a unaligned pointer to a memory filled with a repetition of [F0 AD BA 0D] bytes (a 0xBAADF00D equivalent)

After updating the API to V 12.1 - we still have exactly the same issue. Everything work fine up to the call to nvEncEncodePicture()

Furthermore, after a carefull check / comparison with a 32 bits build of FFMPEG, we use exactly the same init sequence, the same values passed to all function - but in FFMpeg the return is either NV_ENC_SUCCESS) or NV_ENC_ERR_NEED_MORE_INPUT - and in our code always NV_ENC_ERR_GENERIC

How can we find out what could be the cause of this issue ?

NOTE: We are using 2 computers for testing: One with Driver V546.33 and card GeForce GTX 1080 Ti, and one with Drivers V546.33 and card GeForce RTX 4060 Ti

2

Hi there @wesson and welcome to the NVIDIA developer forums.

I took the liberty of moving your post into the correct category.

There might have been some changes in v12 that broke backwards compatibility, did you check in the SDK release notes if you can find something?

I try to get some of the owners to comment here as well.

Thanks!

4

Hi, thank you for moving it to the correct category it’s true that’s my 1st post and I was completely unaware of were it must be placed.

Yes, I read and found nothing.

But also because this issue was reported by our users after updating their NVIDIA drivers, we have updated our code to API to V12.1 (the latest available) and it’s still not working.

As mentionned earlier we also did a side by side comparison with the latest FFMpeg (that we compile under MinGW32), and our code. We have dumped the binary content of what’s feeding

nvEncOpenEncodeSessionEx() (NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS)
nvEncInitializeEncoder() (NV_ENC_INITIALIZE_PARAMS and NV_ENC_CONFIG)
nvEncEncodePicture() (NV_ENC_PIC_PARAMS)

So, if we binary compare the dumps, they are strictly identical (but the dynamic values such as pic_params.inputBuffer or pic_params.outputBitstream which are CUDA surfaces allocated by API)

Yet, the call to nvEncEncodePicture() fail in our code while it goes without error in FFMpeg code.

So that’s really a very strange issue - and that’s why we are really puzzled with it.

With some ASM debugging we have found that the error code is set in the nvcuvid.dll, in a function located in 5f8f27e0 with the following call stack :

:5f8eab67 ; C:\Windows\SysWOW64\nvcuvid.dll
:5f8a5bfa ; C:\Windows\SysWOW64\nvcuvid.dll
:21295e49 ; C:\Windows\SysWOW64\nvEncodeAPI.dll
:2129632f ; C:\Windows\SysWOW64\nvEncodeAPI.dll
:21295463 ; C:\Windows\SysWOW64\nvEncodeAPI.dll
:212910b1 ; C:\Windows\SysWOW64\nvEncodeAPI.dll
:212adcdf ; C:\Windows\SysWOW64\nvEncodeAPI.dll ← nvEncEncodePicture()

This function is probably getting a resource - check if the resource is NULL and if it is it returns 2, which then cause the NV_ENC_ERR_GENERIC to be returned.
So it’s grabbing something that is not there but we really don’t know what. We have reproduced the exact same sequences of calls than FFMpeg to either Cuda and NvEnc functions, to no avail. So we are left with no clue.

Obviously if that can help we can provide the software build that reproduce the issue (with API V12.1) Just ask.

5

Dear all (?),
here is our progress on the issue: Our code and implementation seems correct, and we have found that API 12.1 is working for us up to driver V 537.58.

We did some test on 2 different computers: one with RTX 4060 Ti, and one with GEForce 1070

Failure start to occur on driver 546.01 Nov. 1, 2023 * - and all the drivers following (tested 546.01, 546.17, 546.35 also fail with the same issue: every steps to initialize the encoder work without returning an error, and call to nvEncEncodePicture() systematically fail from 1st call.

So it is obviously a problem introduced in the 546 series of the drivers.

As no one even seem to bother of that case I assume that we will have to live with this problem for an extended period of time. So be it !

  • Edit: My 1st post incorrectly mentioned failure since driver 522 which was not the case.
6

Hi wesson, please be a bit patient, we are still here.

The additional information is very helpful and it is passed on, I assure you.

One more piece of information we will need is which (Windows) operating system you are using, if you could share that.

Thanks.

7

Hello,
Thank you for your answer.
I currently do some test under 4 different computers,

3 running Windows 11 64 bits with the auto updates turned on

  • Windows 11 Professionnel French version, version 21H2, system build 22000.2538
    For our test, we do them with 3 different video cards:
    1 RTX 4060 Ti
    1 GTX 1080 Ti
    1 GTX 1070 Ti

and a older one running Windows 10 64 bits, version 22H2, system build 19045.3930
with a GTX 980 NVidia card.

the behaviour is strictly identical in the 4 computers: when driver is 546+ the H264 HW encoding no longer work (and we don’t use yet HEVC or AV-1 so it’s untested so far)

Side note: If available, we also use CUDA hardware decoding to feed the encoder - yet I don’t think this is related to our issue.

If any want to try, here is a link to download the latest build of our software. It can be used without registration. Just make sure hardware encoding is turned on in the settings and go ! if it’s using NVEnc it’s displayed.

https://download.vso-software.fr/archives/cxhd/v3/vso_convertxtohd_setup_3.0.0.78.exe

Should you need any other information feel free to ask.

8

Thanks for providing the reproducer. I have filed an internal tracker 4478133. We will try to reproduce this issue internally and get back in case need any additional information.

9

Hello,

I might have same or at least similar problem where nvEncEncodePicture() was returning NV_ENC_ERR_GENERIC. After long investigation finally I have found out the reason for this: it turns out it was related to runtime loading of CUDA library - and more specifically to loading some of the library symbols. When using dlsym() I was loading symbols like cuCtxCreate, cuCtxDestroy and couple more. And this was working up to Nvidia drivers version 535. After that I had this error. And to fix this I had to load functions from CUDA library with ‘_v2’ postfix, so cuCtxCreate_v2, cuCtxDestroy_v2, etc.

So, I’m not sure if this is the same problem, but it looks very similar. In any case I hope this might help other people.