I have 2 times quad GTX 780 running on X79 boards for CUDA on Ubuntu 12.04. I experience on the GTX 780 a dropout of the driver after about 30 hours of runtime. System reports that I don’t have Nvidia drivers running an longer. I have read something about a 36 hour TDR bug that has been fixed on Windows. Is that related? What driver do you recommend for Linux? I have no problems with my 6 Titans running in 319.32. All 319 drivers don’t see 4 780 cards, 325.08 worked for some time (< 30 hours), 325.15 crashed once - trying to reproduce.
I don’t have a nvidia-bug-report.log for this yet, I re-installed drivers after the crash.
EDIT: I have now attached the bug report and logfile.
I’m in the same situation. In my case it occurs with 2 GTX 780 running on Ubuntu 12.04. Each graphic card is running on separated PCs as secondary video card but when nvidia-smi reports errors and wrong fan speeds it also stops the X, even though they are not connected to the monitor (they are only used for CUDA computations).
Thanks sandipt for responding, but I’ve resolved the problem installing the drivers 319.60. Previously, I had tried with the previous versions of 319 without succces. In my case, the problem was caused without any apparent execution of programs on the graphic cards. On the PCs installed, they were configured as a secondary video card, and prepared for CUDA computing, but non execution was needed to make them crash. Those PCs were running Ubuntu 12.10, CPU i7-3820, 16GB RAM, etc…