When using cuda-gdb for debugging in CLion, setting a breakpoint within a kernel function causes CLion to freeze. However, when using cuda-gdb directly from the command line for debugging, everything works as expected without any issues.
1,Environment
Ubuntu 22.04
gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
2,CLion:2023.3.5
**3,Cuda driver:**12.2
thinkpad@thinkpad-ThinkPad-P53:~$ nvidia-smi
Wed Jul 24 12:41:30 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01 Driver Version: 535.183.01 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Quadro T1000 Off | 00000000:01:00.0 On | N/A |
| N/A 45C P0 13W / 50W | 792MiB / 4096MiB | 10% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
4,Cuda toolkit: 12.0
thinkpad@thinkpad-ThinkPad-P53:~$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Mon_Oct_24_19:12:58_PDT_2022
Cuda compilation tools, release 12.0, V12.0.76
Build cuda_12.0.r12.0/compiler.31968024_0
thinkpad@thinkpad-ThinkPad-P53:~$ cuda-gdb -v
NVIDIA (R) CUDA Debugger
12.0 release
Portions Copyright (C) 2007-2022 NVIDIA Corporation
GNU gdb (GDB) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
5,Program(main.cu):
#include <iostream>
#include <cuda_runtime.h>
//
__global__ void multiplyArrays(float *dev_a, float *dev_b, float *dev_c, int N) {
int index = blockIdx.x * blockDim.x + threadIdx.x;
if (index < N) {
dev_c[index] = dev_a[index] + dev_b[index];
}
}
int main() {
//
int N = 100;
//
float *h_a = new float[N];
float *h_b = new float[N];
float *h_c = new float[N];
//
for (int i = 0; i < N; i++) {
h_a[i] = 1.0f * i / N;
h_b[i] = 1.0f * i / N;
}
//
float *d_a, *d_b, *d_c;
cudaMalloc((void**)&d_a, N * sizeof(float));
cudaMalloc((void**)&d_b, N * sizeof(float));
cudaMalloc((void**)&d_c, N * sizeof(float));
//
cudaMemcpy(d_a, h_a, N * sizeof(float), cudaMemcpyHostToDevice);
cudaMemcpy(d_b, h_b, N * sizeof(float), cudaMemcpyHostToDevice);
//
int threadsPerBlock = 1;
int blocksPerGrid = (N + threadsPerBlock - 1) / threadsPerBlock;
//
multiplyArrays<<<blocksPerGrid, threadsPerBlock>>>(d_a, d_b, d_c, N);
//
cudaDeviceSynchronize();
cudaError_t error = cudaGetLastError();
if (error != cudaSuccess) {
std::cerr << "CUDA kernel failed: " << cudaGetErrorString(error) << std::endl;
return 1;
}
//
cudaMemcpy(h_c, d_c, N * sizeof(float), cudaMemcpyDeviceToHost);
//
for (int i = 0; i < N; i++) {
std::cout << "h_c[" << i << "] = " << h_c[i] << std::endl;
}
//
cudaFree(d_a);
cudaFree(d_b);
cudaFree(d_c);
delete[] h_a;
delete[] h_b;
delete[] h_c;
return 0;
}
6.When set no breakpoint£ just run in debug£ program run successful.
/opt/cuda/hello/cmake-build-debug/hello
h_c[0] = 0
h_c[1] = 0.02
h_c[2] = 0.04
h_c[3] = 0.06
h_c[4] = 0.08
....
7,When using cuda-gdb direct and set a breakpoint on line 6 output
thinkpad@thinkpad-ThinkPad-P53:/opt/cuda/hello/cmake-build-debug$ cuda-gdb hello
NVIDIA (R) CUDA Debugger
12.0 release
Portions Copyright (C) 2007-2022 NVIDIA Corporation
GNU gdb (GDB) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from hello...
(cuda-gdb) b 6
Breakpoint 1 at 0xb1e1: file /opt/cuda/hello/main.cu, line 10.
(cuda-gdb) r
Starting program: /opt/cuda/hello/cmake-build-debug/hello
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff4dff000 (LWP 3763)]
[Detaching after fork from child process 3764]
[New Thread 0x7fffe9fff000 (LWP 3774)]
[New Thread 0x7fffe97fe000 (LWP 3775)]
[New Thread 0x7fffe8ffd000 (LWP 3776)]
[Switching focus to CUDA kernel 0, grid 1, block (0,0,0), thread (0,0,0), device 0, sm 0, warp 0, lane 0]
Thread 1 "hello" hit Breakpoint 1, multiplyArrays<<<(100,1,1),(1,1,1)>>> (dev_a=0x7fffc7200000, dev_b=0x7fffc7200200, dev_c=0x7fffc7200400, N=100) at /opt/cuda/hello/main.cu:6
6 int index = blockIdx.x * blockDim.x + threadIdx.x;
(cuda-gdb)
8,When set breakpoint(line 6 too) Clion output these and hang forever
8.1:in consel windows:
/opt/cuda/hello/cmake-build-debug/hello
8.2:in Debugger->GDB window:
NVIDIA (R) CUDA Debugger
12.0 release
Portions Copyright (C) 2007-2022 NVIDIA Corporation
GNU gdb (GDB) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff4dff000 (LWP 5833)]
[Detaching after fork from child process 5834]
[New Thread 0x7fffe9fff000 (LWP 5844)]
[New Thread 0x7fffe97fe000 (LWP 5845)]
[New Thread 0x7fffe8ffd000 (LWP 5846)]
Thread 1 "hello" hit Breakpoint 1, multiplyArrays<<<(100,1,1),(1,1,1)>>> (dev_a=0x7fffc7200000, dev_b=0x7fffc7200200, dev_c=0x7fffc7200400, N=100) at /opt/cuda/hello/main.cu:6
6 int index = blockIdx.x * blockDim.x + threadIdx.x;
At this point, it’s impossible to terminate the debugging session; even closing CLion results in a notification that “the process ‘hello’ is still running”. Choosing to abort leads to CLion continuously displaying a message saying ‘waiting for process detach.’ The only way to close CLion seems to be by using the ‘kill’ command in the terminal.
It appears that cuda-gdb is running normally, but Cion is unable to properly interface with cuda-gdb.