We are using the NVIDIA driver (Version 513.46) to encode the Windows desktop into an H.264 stream captured either using the DXGI Windows capture interface or the NVFBC interface. Our test windows server 2022 instance is running on GCP on the Tesla P4 GPU (n1-standard-4 instance type with P4 GPU).
Issue: We noticed that the H.264 stream freezes every now and then and has to be reset by reinitializing the encoder. After further analysis, we noticed that, in the encoding pipeline, the NvEncLockBitstream API returns NV_ENC_ERR_INVALID_PARAM causing us to abandon the encoding process. We have noticed this especially happens as soon as the Chrome browser is launched on the Windows desktop. We speculate that there is a contention for some GPU resources between the Chrome browser and our encoding process. We haven’t seen this issue on any other NVIDIA GPU versions than the Tesla P4. Is this a driver bug?