Unfortunately, newer driver or these workaround did not help, and all GPUs fell of the bus once again.
I noticed that some hours before they did, CUDA started failing to initialize often when starting applications, but starting to work after some tries, then beginning to fail to initialize again. Here I provided bug report when CUDA was failing to initialize and additional details: With latest 575.51.02 driver, after working for some time, CUDA started to fail to initialize after a day of uptime
Then, GPUs fell off the bus soon after that. Here is new bug report, with latest 375.51.02 driver:
nvidia-bug-report.log.gz (2.8 MB)
I would like to reiterate that this issue is specific to Linux Nvidia driver. In above messages I already provided links to multiple independent reports (1, 2) of people having the GPU falling off the bus issue in Linux, but working fine in Windows.
Can someone from Nvidia investigate this please? And me know if I can provide more debug information.