Have you tried putting the kernel driver into persistent mode by invoking
nvidia-smi -pm 1
directly after startup?
Please note that Ubuntu 12.04 is not suppported by the currently shipping version of CUDA, which is 5.0. Encountering various problems on an unsupported platform is not entirely unexpected. A list of supported platforms can be found in the CUDA 5.0 release notes:
It end up that after re-boot both GPU’s show up in the DeviceQuery and ,for a brief window of time, I am able to successfully use the Tesla GPU through Cuda code. But soon after that ( but not as a result of using it for computation) it ‘disappears’ from the DeviceQuery and only the NVS is listed as a CUDA capable device. It usually lasts for about 5 minutes then it is gone.
So I am wondering what may come in and disable/override that GPU? Is there anyway of tracing through a log file what may have happened?