GPU crashing randomly for nvidia version 390.48 on Quadro 4000

Hey guys,

I have an Linux Arch system with a Quadro 4000 GPU that is crashing. I have had this Arch installation for a number of years and don’t recall ever having this issue. In the past few months the random crashes seem to be happening more often. I was hoping that perhaps whatever was wrong would be fixed as new versions of the driver came out, however after a month or so always updating to the latest available, the issue is still there.

When the crash occurs the keyboard, mouse, and monitor all seem to go dead, however I can still SSH into the machine, and all my long running services like docker containers, network mounts, etc… all work fine. I ultimately have to issue a sudo reboot to get the machine back up and running where I can use the console again.

Ultimately I would like to figure out why it is crashing and get it fixed, but perhaps someone knows of a way to restart whatever is crashing so the console would work again without forcing me to reboot?

Attached is the output from running sudo

Let me know if there is anything more I can provide or if anyone has any ideas what might be happening?
thank you!
nvidia-bug-report.log.gz (138 KB)