Using VScode debugger: One or more CUDA devices cannot be used for debugging(Jetson agx Orin 64Gb developer kit)

Developer Tools Nsight Visual Studio Code Edition
, ,
1

Hey All,

we recently got the Jetson AGX Orin 64GB developer kit, we immediately followed the getting started manual
up to a point that we wanted to test the debugging process with the cuda samples, but we keep getting:
One or more CUDA devices cannot be used for debugging

nvidia1@nvidia1-desktop:~/cuda_samples/3_Imaging/convolutionTexture$ lspci | grep -i nvidia
0001:00:00.0 PCI bridge: NVIDIA Corporation Device 229e (rev a1)
nvidia1@nvidia1-desktop:~/cuda_samples/3_Imaging/convolutionTexture$ uname -m && cat /etc/*release
aarch64
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"
# R35 (release), REVISION: 2.1, GCID: 32413640, BOARD: t186ref, EABI: aarch64, DATE: Tue Jan 24 23:38:33 UTC 2023
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
nvidia1@nvidia1-desktop:~/cuda_samples/3_Imaging/convolutionTexture$ gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

nvidia1@nvidia1-desktop:~/cuda_samples/3_Imaging/convolutionTexture$ uname -r
5.10.104-tegra

My json files:

c_cpp_properties.json:
{
    "configurations": [
        {
            "name": "Linux",
            "includePath": [
                "${workspaceFolder}/**",
                "${workspaceFolder}/../../common",
                "/home/nvidia1/cuda_samples/common/inc",
                "/usr/include/**",
                "/usr/include/c++",
                "/usr/include/aarch64-linux-gnu",
                "/home/nvidia1/cuda_samples/common/inc",
                "/usr/local/include",
                "/lib/gcc/aarch64-linux-gnu/9/include",
                "/usr/include/c++/9"
            ],
            "defines": [],
            "compilerPath": "/usr/local/cuda/bin/nvcc",
            "cStandard": "gnu17",
            "cppStandard": "gnu++14",
            "intelliSenseMode": "linux-gcc-x64",
            "configurationProvider": "ms-vscode.makefile-tools"
        }
    ],
    "version": 4
}

launch.json:
{
    "configurations": [
        {
            "name": "CUDA C++: Launch",
            "type": "cuda-gdb",
            "request": "launch",
            "program": "/home/nvidia1/cuda_samples/3_Imaging/convolutionTexture/convolutionTexture"//"${workspaceFolder}/convolutionTexture"
        }
    ]
}

extensions.json:
{
    "recommendations": [
        "nvidia.nsight-vscode-edition",
        "ms-vscode.cpptools",
        "ms-vscode.makefile-tools"
    ]
}

tasks.json:
{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "sample",
            "type": "shell",
            "command": "make dbg=1",
            "problemMatcher": ["$nvcc"],
            "group": {
                "kind": "build",
                "isDefault": true
            }
        }
    ]
}

output of the debugger while tryng to debug:

Type "show configuration" for configuration details.

For bug reporting instructions, please see:

<https://www.gnu.org/software/gdb/bugs/>.

Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".

Type "apropos word" to search for commands related to "word".

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".

[Detaching after fork from child process 15662]

fatal: One or more CUDA devices cannot be used for debugging. Please consult the list of supported CUDA devices for more details. (error code = CUDBG_ERROR_INVALID_DEVICE(0xb)

Please help me solve this issue
Thanks,
Liron

2

Can you confirm that you’ve enabled debug permissions?

Refer to section ’ 3.1.2. Using the CUDA-GDB debugger on Jetson and Drive Tegra devices’

3

Hey thx for the reply.

Yes I enabled debug permissions:

nvidia1@nvidia1-desktop:~/cuda_samples/0_Simple/matrixMul$ sudo usermod -a -G debug $USER
[sudo] password for nvidia1: 
nvidia1@nvidia1-desktop:~/cuda_samples/0_Simple/matrixMul$ make clean
>>> GCC Version is greater or equal to 4.7.0 <<<
rm -f matrixMul matrixMul.o
rm -rf ../../bin/aarch64/linux/release/matrixMul
nvidia1@nvidia1-desktop:~/cuda_samples/0_Simple/matrixMul$ sudo make dbg=1
>>> GCC Version is greater or equal to 4.7.0 <<<
/usr/local/cuda-11.4/bin/nvcc -ccbin g++ -I../../common/inc  -m64 -g -G    --threads 0 --std=c++11 -gencode arch=compute_53,code=sm_53 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_62,code=sm_62 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_72,code=sm_72 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_87,code=sm_87 -gencode arch=compute_87,code=compute_87 -o matrixMul.o -c matrixMul.cu
/usr/local/cuda-11.4/bin/nvcc -ccbin g++   -m64 -g -G      -gencode arch=compute_53,code=sm_53 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_62,code=sm_62 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_72,code=sm_72 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_87,code=sm_87 -gencode arch=compute_87,code=compute_87 -o matrixMul matrixMul.o 
mkdir -p ../../bin/aarch64/linux/debug
cp matrixMul ../../bin/aarch64/linux/debug
nvidia1@nvidia1-desktop:~/cuda_samples/0_Simple/matrixMul$

but i still get the same error:

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-elf-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".

Breakpoint 1, main (argc=1, argv=0xffffffffed58) at matrixMul.cu:275
275	  printf("[Matrix Multiply Using CUDA] - Starting...\n");
[Detaching after fork from child process 18373]
fatal:  One or more CUDA devices cannot be used for debugging. Please consult the list of supported CUDA devices for more details. (error code = CUDBG_ERROR_INVALID_DEVICE(0xb)
4

I met this error as well on my newly setup platforms (both AGX Xavier with JP5.1.1 and AGX Orin with JP5.0.2).

It mysteriously disapeared after I’ve once successfully done a remote debug session from my Linux x86 host via vscode (1.77.3 at the moment). I’ve updated my x86 host gpu driver to this:

NVIDIA-SMI 535.86.05 Driver Version: 535.86.05 CUDA Version: 12.2

Yes, it is wiered that it showed cuda v12.2 (acutally I’m using v11.4 on x86 host as well. I had before installed v12.2). Not sure exactly what had fixed the issue. But this is what I got.