Unfortunately soon after CUDA started failing to initialize reliably, all GPUs fell off the bus due to Linux Nvidia driver bug (unlike hardware causes that may cause GPU fall off the bus, this one happens only in Linux but not with Windows Nvidia driver). Also, I think CUDA initialization errors prior to GPUs falling off the bus happened before on another driver version, just the other time I did not collect bug report info. So it may be related issue that may reveal itself as a precursor, but I am not sure.
Here is additional bug report log - the one in the previous message was during CUDA failing to initialize issues, and this one after GPUs fell off the bus:
nvidia-bug-report.log.gz (2.8 MB)
I would like to reiterate that this issue is specific to Linux Nvidia driver. In the main thread about GPU fell off the bus issue, I already provided links to multiple independent reports (1, 2) of people having the GPU falling off the bus issue in Linux, but working fine in Windows.
Can someone from Nvidia investigate this please? And let me know if I can provide more debug information.