Can cuda-gdb backtrace to a function in *.cpp when debugging core file?

Developer Tools CUDA Developer Tools CUDA-GDB
1

I want to debug the core file using cuda-gdb. I set this in my .bashrc:

export CUDA_ENABLE_COREDUMP_ON_EXCEPTION=1
export CUDA_ENABLE_CPU_COREDUMP_ON_EXCEPTION=1

After crashing, a cpu core dump and a gpu core dump are generated. I load them into cuda-gdb this way:

(cuda-gdb) target core core.cpu core.cuda

But I found I can only backtrace to the kernel in *.cu file. I want to backtrace deeper to the function calling the kernel in *.cpp file for debugging. How can I do this using cuda-gdb?

2

Hi @umiswing
Unfortunately this use case is not supported. CUDA kernel launches are asynchronous (CPU doesn’t wait for GPU to finish the job). For your case, this would mean that the exception happens on GPU, while CPU continues the execution (not waiting for GPU code to complete).

3

Thanks @AKravets . Is there any way to turn off the asynchronous to help for debugging?

4

Hi @umiswing
You can make CUDA launches synchronous by setting the CUDA_LAUNCH_BLOCKING=1 environment variable, but this would not make cuda-gdb produce combined backtraces (the required functionality is not implemented by cuda-gdb).

On the other hand, you can get such backtraces using compute-sanitizer: Compute Sanitizer User Manual :: Compute Sanitizer Documentation

You can check the backtrace documentation section for details: Compute Sanitizer User Manual :: Compute Sanitizer Documentation