Can you check with a more current driver? If it still fails, please run nvidia-bug-report.sh as root and attach the resulting .gz file to your post. Hovering the mouse over an existing post of yours will reveal a paperclip icon.
I have uploaded the nvidia-bug-report.log.gz file. Please help check it. Other lab members are using the current driver. If the problem comes with the driver, we can update it then. nvidia-bug-report.log.gz (307 KB)
Looks like it starts when the driver hits an XID 62 and a display engine timeout. This might be due to the Titan is also used to run X while running high cuda loads. Try disabling the Xserver after reboot and check if the issue still appears.