Different results with the same algorithm [GeForce GTX780Ti]


I’m developing an algorithm using CUDA 6.0 with a GeForce GTX780Ti card running on Ubuntu 12.04 LTS. The problem, as I said, is that I obtain different results when I executed the algorithm twice.

For the record, I have executed the same algorithm twice using CUDA 6.0 with a GeForce GTX480 card running also on Ubuntu 12.04 LTS and I obtain SAME results, so I do not know what is happening because aparently the algorithm is Ok, but not for the GTX780Ti…

Any ideas?

Thanks a lot.

PS: The results are actually files. I compare the files generated by the algorithm (running it twice) and I see if they are the same or not.

When there is un-intended non-determinism in running a CUDA program, the first thing you would want to do is check the basics. It is possible that the software just happened to work on a particular platform, although there is a latent bug hidden somewhere that is exposed by moving to a different platform.

(1) Make sure return status is checked for all API calls (CUDA, CUBLAS, CUFFT, etc)
(2) Likewise, make sure the status of all kernel launches is checked. A kernel may fail to launch, or encounter an error, eliminating a particular step in the data processing pipeline
(3) Use cuda-memcheck to check that there are no race conditions and no out-of-bounds accesses in CUDA kernels
(4) Check data integrity before it is sent from the host to the device. Use the maximum warning level for host compilation to increase static checks, and use a tool like valgrind to check for out-of-bounds access at runtime.

If you cannot find any software issues and you have a second GTX 780Ti available, you could check whether the problem is tied to the one particular card. Consumer cards may be factory overclocked, meaning they run at higher clocks than the original NVIDIA specifications. If there is a means to operate the card at standard clocks, you may want to try that to see whether it makes a difference. You can find the NVIDIA specified clocks for the GTX 780 Ti here: