Nvidia driver randomly unloading or failing on Ubuntu?

Graphics / Linux Linux
,
1

Driver version: 525.60.13
OS: Ubuntu 20.04.5 LTS
GPU: Nvidia T4

We have tried a couple of different driver versions, but none of them resolve these issues.

  1. Driver randomly unloads, nvidia-smi is then telling us that NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.. Only solution is to reinstall the driver.

  2. No capable devices found message from ffmpeg. This boils down to checking capabilities of the card using cuDeviceComputeCapability and counting the NVENC capable devices. This number randomly turns out to be 0 but on the next launch it can be non-zero and work just fine without needing to make any changes.

Not sure if these two issues are related, but is this a known bug and if so does there exist some fix for this issue? I’ve searched around on the forums but nothing conclusive has come up.