Cannot Debug Jetson AGX Xavier with JetPack SDK 5.0 Developer Preview

I have a Jetson AGX Xavier developer kit that I recently flashed (using the SDK manager) with JetPack 5.0 Developer Preview. I have otherwise done no custom setup.

I cannot seem to debug (either remotely or using cuda-gdb locally/directly on the developer kit) at all, even with simple examples.

When I try to use cuda-gdb locally/directly on the developer kit from the terminal to run and debug a program, this is what happens:

test@ubuntu:~$ cd Desktop/
test@ubuntu:~/Desktop$ 
test@ubuntu:~/Desktop$ 
test@ubuntu:~/Desktop$ cuda-gdb blah_blah 
NVIDIA (R) CUDA Debugger
11.4 release
Portions Copyright (C) 2007-2021 NVIDIA Corporation
GNU gdb (GDB) 10.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-elf-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from blah_blah...
(cuda-gdb) run
Starting program: /home/test/Desktop/blah_blah 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
[Detaching after fork from child process 23554]
fatal:  One or more CUDA devices cannot be used for debugging. Please consult the list of supported CUDA devices for more details. (error code = CUDBG_ERROR_INVALID_DEVICE(0xb)
(cuda-gdb)

When I try to remotely debug, this is what I see in the Debugger Console in Eclipse:

Coalescing of the CUDA commands output is off.
NVIDIA (R) CUDA Debugger
11.4 release
Portions Copyright (C) 2007-2021 NVIDIA Corporation
GNU gdb (GDB) 10.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Reading /lib/ld-linux-aarch64.so.1 from remote target...
warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.
Reading /lib/ld-linux-aarch64.so.1 from remote target...
Reading /lib/ld-2.31.so from remote target...
Reading /lib/.debug/ld-2.31.so from remote target...
Reading no//lib/ld-2.31.so from remote target...
0x0000fffff7fcd0c0 in ?? () from target:/lib/ld-linux-aarch64.so.1
Reading /lib/aarch64-linux-gnu/librt.so.1 from remote target...
Reading /lib/aarch64-linux-gnu/libpthread.so.0 from remote target...
Reading /lib/aarch64-linux-gnu/libdl.so.2 from remote target...
Reading /lib/aarch64-linux-gnu/libstdc++.so.6 from remote target...
Reading /lib/aarch64-linux-gnu/libm.so.6 from remote target...
Reading /lib/aarch64-linux-gnu/libgcc_s.so.1 from remote target...
Reading /lib/aarch64-linux-gnu/libc.so.6 from remote target...
Reading /lib/aarch64-linux-gnu/librt-2.31.so from remote target...
Reading /lib/aarch64-linux-gnu/.debug/librt-2.31.so from remote target...
Reading no//lib/aarch64-linux-gnu/librt-2.31.so from remote target...
Reading /lib/aarch64-linux-gnu/8a626d3e3b0b4edee63830da2a9d373585a858.debug from remote target...
Reading /lib/aarch64-linux-gnu/.debug/8a626d3e3b0b4edee63830da2a9d373585a858.debug from remote target...
Reading no//lib/aarch64-linux-gnu/8a626d3e3b0b4edee63830da2a9d373585a858.debug from remote target...
Reading /lib/aarch64-linux-gnu/libdl-2.31.so from remote target...
Reading /lib/aarch64-linux-gnu/.debug/libdl-2.31.so from remote target...
Reading no//lib/aarch64-linux-gnu/libdl-2.31.so from remote target...
Reading /lib/aarch64-linux-gnu/c118da2026b956ca08055b04e86081d446cb6a.debug from remote target...
Reading /lib/aarch64-linux-gnu/.debug/c118da2026b956ca08055b04e86081d446cb6a.debug from remote target...
Reading no//lib/aarch64-linux-gnu/c118da2026b956ca08055b04e86081d446cb6a.debug from remote target...
Reading /lib/aarch64-linux-gnu/libm-2.31.so from remote target...
Reading /lib/aarch64-linux-gnu/.debug/libm-2.31.so from remote target...
Reading no//lib/aarch64-linux-gnu/libm-2.31.so from remote target...
Reading /lib/aarch64-linux-gnu/f1ff7964d00e3857e0c08d512e490af56450ae.debug from remote target...
Reading /lib/aarch64-linux-gnu/.debug/f1ff7964d00e3857e0c08d512e490af56450ae.debug from remote target...
Reading no//lib/aarch64-linux-gnu/f1ff7964d00e3857e0c08d512e490af56450ae.debug from remote target...
Reading /lib/aarch64-linux-gnu/libc-2.31.so from remote target...
Reading /lib/aarch64-linux-gnu/.debug/libc-2.31.so from remote target...
Reading no//lib/aarch64-linux-gnu/libc-2.31.so from remote target...

Temporary breakpoint 1, main (argc=1, argv=0xfffffffff498) at ../src/blah_blah.cu:349
349	int main(int argc, char **argv) {
Reading /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1 from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_gpu.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_mem.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_sync.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_host1x.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvos.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvsocsys.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvsciipc.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_chip.so from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libcuda.so.1.1.debug from remote target...
Reading no//usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_gpu.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvrm_gpu.so.debug from remote target...
Reading no//usr/lib/aarch64-linux-gnu/tegra/libnvrm_gpu.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_mem.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvrm_mem.so.debug from remote target...
Reading no//usr/lib/aarch64-linux-gnu/tegra/libnvrm_mem.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_sync.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvrm_sync.so.debug from remote target...
Reading no//usr/lib/aarch64-linux-gnu/tegra/libnvrm_sync.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_host1x.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvrm_host1x.so.debug from remote target...
Reading no//usr/lib/aarch64-linux-gnu/tegra/libnvrm_host1x.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvos.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvos.so.debug from remote target...
Reading no//usr/lib/aarch64-linux-gnu/tegra/libnvos.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvsocsys.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvsocsys.so.debug from remote target...
Reading no//usr/lib/aarch64-linux-gnu/tegra/libnvsocsys.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvsciipc.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvsciipc.so.debug from remote target...
Reading no//usr/lib/aarch64-linux-gnu/tegra/libnvsciipc.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/libnvrm_chip.so.debug from remote target...
Reading /usr/lib/aarch64-linux-gnu/tegra/.debug/libnvrm_chip.so.debug from remote target...
Reading no//usr/lib/aarch64-linux-gnu/tegra/libnvrm_chip.so.debug from remote target...
fatal:  One or more CUDA devices cannot be used for debugging. Please consult the list of supported CUDA devices for more details. (error code = CUDBG_ERROR_INVALID_DEVICE(0xb)

Related to the above remote debugging output, if I look on the developer kit filesystem for one of the mentioned debug .so files (e.g. /usr/lib/aarch64-linux-gnu/tegra/.debug/libcuda.so.1.1.debug), I cannot find such a file nor the .debug directory.

Prior to this I was using version 4.6.1 and did not encounter any debugging issues.

Any help would be appreciated, thanks.

Hi,

Could you try it with the root authority?
For example, we can run it normally with the following command:

$ cd /usr/local/cuda/samples/0_Simple/vectorAdd
$ sudo make
$ sudo /usr/local/cuda/bin/cuda-gdb vectorAdd
$ sudo /usr/local/cuda/bin/cuda-gdb vectorAdd
NVIDIA (R) CUDA Debugger
11.4 release
Portions Copyright (C) 2007-2021 NVIDIA Corporation
GNU gdb (GDB) 10.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-elf-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from vectorAdd...
(No debugging symbols found in vectorAdd)
(cuda-gdb) run
Starting program: /usr/local/cuda-11.4/samples/0_Simple/vectorAdd/vectorAdd
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
[Vector addition of 50000 elements]
[Detaching after fork from child process 22977]
[New Thread 0xffffdfffc900 (LWP 22980)]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done
[Thread 0xffffdfffc900 (LWP 22980) exited]
[Inferior 1 (process 22973) exited normally]
(cuda-gdb)

Thanks.

That does indeed work. I was also able to verify that I can do the same sort of thing with running cuda-gdbserver using sudo and can then remotely debug properly.

Should this be this way? I don’t remember encountering this when using SDK v4.6.1, and this doesn’t really make sense to me.

Thanks for your help!

Hi,

You should also need the authority to profile CUDA on JetPack 4.6.
Thanks.