Using VScode debugger: One or more CUDA devices cannot be used for debugging(Jetson agx Orin 64Gb developer kit)

Hey All,

we recently got the Jetson AGX Orin 64GB developer kit, we immediately followed the getting started manual
up to a point that we wanted to test the debugging process with the cuda samples, but we keep getting:
One or more CUDA devices cannot be used for debugging

nvidia1@nvidia1-desktop:~/cuda_samples/3_Imaging/convolutionTexture$ lspci | grep -i nvidia
0001:00:00.0 PCI bridge: NVIDIA Corporation Device 229e (rev a1)
nvidia1@nvidia1-desktop:~/cuda_samples/3_Imaging/convolutionTexture$ uname -m && cat /etc/*release
aarch64
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"
# R35 (release), REVISION: 2.1, GCID: 32413640, BOARD: t186ref, EABI: aarch64, DATE: Tue Jan 24 23:38:33 UTC 2023
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
nvidia1@nvidia1-desktop:~/cuda_samples/3_Imaging/convolutionTexture$ gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

nvidia1@nvidia1-desktop:~/cuda_samples/3_Imaging/convolutionTexture$ uname -r
5.10.104-tegra

My json files:

c_cpp_properties.json:
{
    "configurations": [
        {
            "name": "Linux",
            "includePath": [
                "${workspaceFolder}/**",
                "${workspaceFolder}/../../common",
                "/home/nvidia1/cuda_samples/common/inc",
                "/usr/include/**",
                "/usr/include/c++",
                "/usr/include/aarch64-linux-gnu",
                "/home/nvidia1/cuda_samples/common/inc",
                "/usr/local/include",
                "/lib/gcc/aarch64-linux-gnu/9/include",
                "/usr/include/c++/9"
            ],
            "defines": [],
            "compilerPath": "/usr/local/cuda/bin/nvcc",
            "cStandard": "gnu17",
            "cppStandard": "gnu++14",
            "intelliSenseMode": "linux-gcc-x64",
            "configurationProvider": "ms-vscode.makefile-tools"
        }
    ],
    "version": 4
}

launch.json:
{
    "configurations": [
        {
            "name": "CUDA C++: Launch",
            "type": "cuda-gdb",
            "request": "launch",
            "program": "/home/nvidia1/cuda_samples/3_Imaging/convolutionTexture/convolutionTexture"//"${workspaceFolder}/convolutionTexture"
        }
    ]
}

extensions.json:
{
    "recommendations": [
        "nvidia.nsight-vscode-edition",
        "ms-vscode.cpptools",
        "ms-vscode.makefile-tools"
    ]
}

tasks.json:
{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "sample",
            "type": "shell",
            "command": "make dbg=1",
            "problemMatcher": ["$nvcc"],
            "group": {
                "kind": "build",
                "isDefault": true
            }
        }
    ]
}

output of the debugger while tryng to debug:

Type "show configuration" for configuration details.

For bug reporting instructions, please see:

<https://www.gnu.org/software/gdb/bugs/>.

Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".

Type "apropos word" to search for commands related to "word".

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".

[Detaching after fork from child process 15662]

fatal: One or more CUDA devices cannot be used for debugging. Please consult the list of supported CUDA devices for more details. (error code = CUDBG_ERROR_INVALID_DEVICE(0xb)

Please help me solve this issue
Thanks,
Liron

Can you confirm that you’ve enabled debug permissions?

Refer to section ’ 3.1.2. Using the CUDA-GDB debugger on Jetson and Drive Tegra devices’

1 Like

Hey thx for the reply.

Yes I enabled debug permissions:

nvidia1@nvidia1-desktop:~/cuda_samples/0_Simple/matrixMul$ sudo usermod -a -G debug $USER
[sudo] password for nvidia1: 
nvidia1@nvidia1-desktop:~/cuda_samples/0_Simple/matrixMul$ make clean
>>> GCC Version is greater or equal to 4.7.0 <<<
rm -f matrixMul matrixMul.o
rm -rf ../../bin/aarch64/linux/release/matrixMul
nvidia1@nvidia1-desktop:~/cuda_samples/0_Simple/matrixMul$ sudo make dbg=1
>>> GCC Version is greater or equal to 4.7.0 <<<
/usr/local/cuda-11.4/bin/nvcc -ccbin g++ -I../../common/inc  -m64 -g -G    --threads 0 --std=c++11 -gencode arch=compute_53,code=sm_53 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_62,code=sm_62 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_72,code=sm_72 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_87,code=sm_87 -gencode arch=compute_87,code=compute_87 -o matrixMul.o -c matrixMul.cu
/usr/local/cuda-11.4/bin/nvcc -ccbin g++   -m64 -g -G      -gencode arch=compute_53,code=sm_53 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_62,code=sm_62 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_72,code=sm_72 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_87,code=sm_87 -gencode arch=compute_87,code=compute_87 -o matrixMul matrixMul.o 
mkdir -p ../../bin/aarch64/linux/debug
cp matrixMul ../../bin/aarch64/linux/debug
nvidia1@nvidia1-desktop:~/cuda_samples/0_Simple/matrixMul$ 

but i still get the same error:

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-elf-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".

Breakpoint 1, main (argc=1, argv=0xffffffffed58) at matrixMul.cu:275
275	  printf("[Matrix Multiply Using CUDA] - Starting...\n");
[Detaching after fork from child process 18373]
fatal:  One or more CUDA devices cannot be used for debugging. Please consult the list of supported CUDA devices for more details. (error code = CUDBG_ERROR_INVALID_DEVICE(0xb)

I met this error as well on my newly setup platforms (both AGX Xavier with JP5.1.1 and AGX Orin with JP5.0.2).

It mysteriously disapeared after I’ve once successfully done a remote debug session from my Linux x86 host via vscode (1.77.3 at the moment). I’ve updated my x86 host gpu driver to this:

NVIDIA-SMI 535.86.05 Driver Version: 535.86.05 CUDA Version: 12.2

Yes, it is wiered that it showed cuda v12.2 (acutally I’m using v11.4 on x86 host as well. I had before installed v12.2). Not sure exactly what had fixed the issue. But this is what I got.

Hi,

I have the same situation with a Jetson Orin AGX on Jetpack 5.0.2 (exactly same computer, with uname -r:5.10.104-tegra). Like you have said, I made sure my user is in the debug group. I also tried the remote steps, but i cant start the host part of gdb server.

(dreamvu_ws) gaus@tegra-ubuntu:~$ uname -m && cat /etc/*release
aarch64
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION=“Ubuntu 20.04.6 LTS”

R35 (release), REVISION: 1.0, GCID: 31346300, BOARD: t186ref, EABI: aarch64, DATE: Thu Aug 25 18:41:45 UTC 2022

NAME=“Ubuntu”
VERSION=“20.04.6 LTS (Focal Fossa)”
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME=“Ubuntu 20.04.6 LTS”
VERSION_ID=“20.04”
HOME_URL=“https://www.ubuntu.com/”
SUPPORT_URL=“https://help.ubuntu.com/”
BUG_REPORT_URL=“Bugs : Ubuntu”
PRIVACY_POLICY_URL=“https://www.ubuntu.com/legal/terms-and-policies/privacy-policy”
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Here is the console output :

Correctly config I2S ports!
– Configuring done
– Generating done
– Build files have been written to: /home/gaus/Documents/Antennerie-acoustique3D/Antennerie-acoustique3D
[ 85%] Built target lib_antenneAcoustique3D
[100%] Built target main
NVIDIA (R) CUDA Debugger
11.4 release
Portions Copyright (C) 2007-2021 NVIDIA Corporation
GNU gdb (GDB) 10.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type “show copying” and “show warranty” for details.
This GDB was configured as “aarch64-elf-linux-gnu”.
Type “show configuration” for configuration details.
For bug reporting instructions, please see:
https://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.

For help, type “help”.
Type “apropos word” to search for commands related to “word”…
Reading symbols from ./bin/main…
(cuda-gdb) r
Starting program: /home/gaus/Documents/Antennerie-acoustique3D/Antennerie-acoustique3D/bin/main config/default_config.config
[Thread debugging using libthread_db enabled]
Using host libthread_db library “/lib/aarch64-linux-gnu/libthread_db.so.1”.
[New Thread 0xffffcdf07900 (LWP 12771)]
[New Thread 0xffffcd706900 (LWP 12772)]
[New Thread 0xffffcbf05900 (LWP 12773)]
[New Thread 0xffffca704900 (LWP 12774)]
[New Thread 0xffffc7f03900 (LWP 12775)]
[New Thread 0xffffc6702900 (LWP 12776)]
[New Thread 0xffffc5f01900 (LWP 12777)]
[New Thread 0xffffc4700900 (LWP 12778)]
[New Thread 0xffffc1eff900 (LWP 12779)]
[New Thread 0xffffc06fe900 (LWP 12780)]
[New Thread 0xffffbfefd900 (LWP 12781)]
[0ms]
----- PARAMETERS -----

Audio par

    device_name: 4p16
    n_frames: 64
    n_channels: 64                       
    selec_mic: {1 ... 64}                                 
    samplingFreq: 32000
    client_n_samples: 1024                      
    n_new_data: -1                              

Video par

    n_phi: 672   
    n_theta: 224 
    sim_video: true

Imagery par

    mic_pos_fname: config/mic_coords_31cm.txt   
    algorithm: N1
    freqMin: 50
    freqMax: 0

GUI par

    ip_address: 192.168.1.134
    control_port: 65535
    fps_default: 30

Logging parameters (for categories add an # to remove)

    log_level: LOG_DEBUG
    log_categories: CAT_GENERAL CAT_AUDIO CAT_GPU CAT_VIDEO CAT_SERVER CAT_MSG 
    log_timestamp: true
    max_lines_saved: 1000

[98ms] Audio/4p16: Opening the audio stream was SUCCESSFULL
[123ms] Audio/4p16: Device SETUP was SUCCESSFULL
[New Thread 0xffffb9531900 (LWP 12783)]
[127ms] Audio/4p16/producer: → Thread started!
[New Thread 0xffffb8d30900 (LWP 12784)]
[148ms] Audio/4p16/consummer: → Thread started!
[1023ms] Video: Video sim is setup
[Thread 0xffffbfefd900 (LWP 12781) exited]
[Thread 0xffffc4700900 (LWP 12778) exited]
[Thread 0xffffc5f01900 (LWP 12777) exited]
[Thread 0xffffcd706900 (LWP 12772) exited]
[Thread 0xffffcbf05900 (LWP 12773) exited]
[Thread 0xffffc1eff900 (LWP 12779) exited]
[Thread 0xffffc7f03900 (LWP 12775) exited]
[Thread 0xffffcdf07900 (LWP 12771) exited]
[Thread 0xffffc6702900 (LWP 12776) exited]
[Thread 0xffffc06fe900 (LWP 12780) exited]
[Thread 0xffffca704900 (LWP 12774) exited]
[Detaching after fork from child process 12786]
fatal: One or more CUDA devices cannot be used for debugging. Please consult the list of supported CUDA devices for more details. (error code = CUDBG_ERROR_INVALID_DEVICE(0xb)
(cuda-gdb) [Thread 0xffffb8d30900 (LWP 12784) exited]
[Thread 0xffffb9531900 (LWP 12783) exited]

Also added the debug flag in compiler : set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} -g -G")

It is not possible for us to update the jetpack version as it is a custom version of the kernels on a custom Carrier board.

Thank you,

Loic

Hi, @loic.boileau

Sorry for the issue you met.
Can you clarify

  1. Are you using VScode to remote debug
    or 2) using cuda-gdb directly on Orin ?

Can you try run “sudo cuda-gdb ./sample” directly to see if debug works ?

Close the topic as it is originally from 2023.5.
Please submit another post if any issue remained.