Kernel crashed due to "unspecified launch failure" on CUDA 9 but not CUDA 8

Accelerated Computing CUDA CUDA Programming and Performance
1

Might the watchdog timer have changed? If so, how do I revert this setting on Linux/Ubuntu? This GPU (P100) is not attached to a display.

2

deviceQuery will tell you if the watchdog is active on a particular GPU

under linux, to avoid the watchdog, my usual suggestion is to make sure the GPU in question is not enumerated for X (e.g. is not in xorg.conf)

3

Run time limit is off. What else coudl cause this?

4

The most frequent reason for “unspecified launch failure” is a memory access out of bounds. Your code may contain a latent bug (e.g. such as a race condition, or invoking undefined C++ behavior) that got exposed by differences in code generation between CUDA 8 and CUDA 9. Another possibility (much less likely, I think) is that there is a code generation bug introduced with CUDA 9.

I would suggest running the code under control of cuda-memcheck, with all checkers enabled, to see whether it has any complaints about the device code.