Kernel code execution causes segmentation fault in cuda-dbg but not when executed standalone

[darren.puckey]

Hi,
I’m trying to setup my development environment and being able to step through code is clearly an imperative.
I have a 64GB Jetson AGX Orin Development Kit with Jetpack 6.0 installed running LT4 Ubuntu 22.04. I believe the installation is standard as it’s what ships with the dev kit.
I’ve tried to run two of the samples, matrixMul and vectorAdd and get the same result with both. I haven’t modified anything in the code.
They compile without error and I can run them as executables without error. However, when I try to debug them, either through vscode or standalone via cuda-gdb from the command line I get a segmentation fault as soon as I step into the kernel code. If the executable didn’t run I’d assume it was a code or compiler/linker issue but I don’t see how that can be given the executable works.
I have tried the CUDBG_USE_LEGACY_DEBUGGER=1 option, this made no difference.
I feel as if it’s something to do with the aarch64 architecture as I can find almost nothing relating to using Jetson in my searches for an answer.
Results from cuda-dbg below:

kds@ubuntu:~/Documents/cuda-samples-working/Samples/0_Introduction/vectorAdd$ /usr/local/cuda-12.2/bin/cuda-gdb vectorAdd
NVIDIA (R) CUDA Debugger
CUDA Toolkit 12.2 release
Portions Copyright (C) 2007-2023 NVIDIA Corporation
GNU gdb (GDB) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type “show copying” and “show warranty” for details.
This GDB was configured as “aarch64-elf-linux-gnu”.
Type “show configuration” for configuration details.
For bug reporting instructions, please see:
https://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.

For help, type “help”.
Type “apropos word” to search for commands related to “word”…
Reading symbols from vectorAdd…
(cuda-gdb) run
Starting program: /home/kds/Documents/cuda-samples-working/Samples/0_Introduction/vectorAdd/vectorAdd
[Thread debugging using libthread_db enabled]
Using host libthread_db library “/lib/aarch64-linux-gnu/libthread_db.so.1”.
[Vector addition of 50000 elements]
[New Thread 0xfffff536c840 (LWP 54752)]
[Detaching after fork from child process 54753]
[New Thread 0xfffff4a6b840 (LWP 54760)]
[New Thread 0xffffe9ffc840 (LWP 54761)]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads

Thread 1 “vectorAdd” received signal SIGSEGV, Segmentation fault.
0x0000fffff54a9580 in ?? () from /lib/aarch64-linux-gnu/libcudadebugger.so.1
(cuda-gdb)

Running standalone:
kds@ubuntu:~/Documents/cuda-samples-working/Samples/0_Introduction/vectorAdd$ ./vectorAdd
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

ANY suggestions would be greatly appreciated, I’ve lost so much time searching and trying solutions…
Thanks,
Darren

[AKravets]

Hi @darren.puckey
Thank you for you report! Could you please try the following:

• Make sure your user (kds) is in the debug group:

sudo usermod -a -G debug kds

and re-login

• Could you also try running the debug session as root?

[darren.puckey]

Thanks you so much.
The ‘sudo usermod -a -G debug kds’ and reboot option worked.
Have I missed this in the user documentation?
If not, it would be really helpful if it was made clear this was required to save other new users like myself the same pain!

Finally I feel as if I can start to move forwards on the exciting stuff.
Regards,
Darren

[AKravets]

Hi @darren.puckey
Glad it worked for you! I will mark the topic as resolved.

Have I missed this in the user documentation?

It’s mentioned here: CUDA-GDB