GPU crashing randomly for nvidia version 390.48 on Quadro 4000

Hey guys,

I have an Linux Arch system with a Quadro 4000 GPU that is crashing. I have had this Arch installation for a number of years and don’t recall ever having this issue. In the past few months the random crashes seem to be happening more often. I was hoping that perhaps whatever was wrong would be fixed as new versions of the driver came out, however after a month or so always updating to the latest available, the issue is still there.

When the crash occurs the keyboard, mouse, and monitor all seem to go dead, however I can still SSH into the machine, and all my long running services like docker containers, network mounts, etc… all work fine. I ultimately have to issue a sudo reboot to get the machine back up and running where I can use the console again.

Ultimately I would like to figure out why it is crashing and get it fixed, but perhaps someone knows of a way to restart whatever is crashing so the console would work again without forcing me to reboot?

Attached is the output from running sudo nvidia-bug-report.sh.

Let me know if there is anything more I can provide or if anyone has any ideas what might be happening?
thank you!
nvidia-bug-report.log.gz (138 KB)