I want to debug the core file using cuda-gdb. I set this in my .bashrc:
After crashing, a cpu core dump and a gpu core dump are generated. I load them into cuda-gdb this way:
(cuda-gdb) target core core.cpu core.cuda
But I found I can only backtrace to the kernel in *.cu file. I want to backtrace deeper to the function calling the kernel in *.cpp file for debugging. How can I do this using cuda-gdb?
Unfortunately this use case is not supported. CUDA kernel launches are asynchronous (CPU doesn’t wait for GPU to finish the job). For your case, this would mean that the exception happens on GPU, while CPU continues the execution (not waiting for GPU code to complete).
Thanks @AKravets . Is there any way to turn off the asynchronous to help for debugging?
You can make CUDA launches synchronous by setting the
CUDA_LAUNCH_BLOCKING=1 environment variable, but this would not make cuda-gdb produce combined backtraces (the required functionality is not implemented by
On the other hand, you can get such backtraces using compute-sanitizer: Compute Sanitizer User Manual :: Compute Sanitizer Documentation
You can check the backtrace documentation section for details: Compute Sanitizer User Manual :: Compute Sanitizer Documentation