I have a bug in my integration of NVENC where locking the bitstream for the first frame will sometimes return with NV_ENC_ERR_INVALID_PARAM. Most of the time the code works fine.
If I let this error occur a couple times, it can break the system; making any CUDA call block indefinitely. This can only be solved by rebooting the machine.
The problem happens using driver 378.13 and Video SDK 7.1.9 on CentOS 7.3 hosts, one with a Quadro M6000, the other with a Titan X.
The error is not very verbose, I suppose the bitstream buffer is the cause somehow, but I’ve checked the handles and they’re all valid. Maybe a bug in the driver itself?