Need to remove timeouts and the "launch timed out and was terminated" message

[avidday]

I ported my C code and got it running using CUDA but I keep getting this error after about 5 or 10 seconds:

“cudaSafeCall() Runtime API error : the launch timed out and was terminated.”

How do I turn off timeouts? I want the thing to run, no matter how long it takes. Ideally, I’d like to poll my kernel every 60 seconds to make sure it’s still going, but it should never time out.

You cannot turn them off. If your card runs a display manager, it will have a 5 second watchdog timer on any kernel. The only work-arounds are to use a dedicated card for computation or not use a display manager at all (both are possible under linux), or break the work up into small, short running kernels and run them many times. It seems that the second strategy should be possible, just provide a subset of the search space to each kernel and run lots of kernel executions. You can keep the accumulating results in GPU memory and periodically copy them back and write them to disk. Depending on what card you are using, you can even overlap the copying and kernel execution so that the copying and disk IO is effectively free.