Unable to debug without software_preemption

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX1
1

Starting with R24.2, I’ve noticed that cuda-gdb cannot debug CUDA applications unless you turn on set cuda software_preemption on. This is true even if you’re not running an X server, which used to be the typical condition requiring it.

The X server is disabled with commands like this:

systemctl disable lightdm
systemctl stop lightdm

In R24.2, the response is:

/usr/local/cuda/bin/cuda-gdb $Ccuda/vectorAdd
NVIDIA (R) CUDA Debugger
8.0 release
Portions Copyright (C) 2007-2016 NVIDIA Corporation
GNU gdb (GDB) 7.6.2
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-elf-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /coyote/coy1/todd/test/cuda/vectorAdd...done.
(cuda-gdb) r
Starting program: /coy1/todd/test/cuda/vectorAdd 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
fatal:  All CUDA devices are used for display and cannot be used while debugging. (error code = CUDBG_ERROR_ALL_DEVICES_WATCHDOGGED(0x18)
(cuda-gdb)

I see this with any CUDA application that I try.

You can make some progress by turning software_preemption. But, as I understand it, that comes with pretty heavy overhead. And it disables features, like attach.

2

Hi,
Did you manage to resole this? I am having the same problem. Even worth, I only can run targets as root or sudo. Could you please tell me how to turn on software_preemption here.
Thanks much,
Nik

3

Found it: CUDA_DEBUGGER_SOFTWARE_PREEMPTION=1

4

There also is a command in cuda-gdb:

set cuda software_preemption on

5

Hi Todd Allen and NYP,

You can make some progress by turning software_preemption. But, as I understand it, that comes with pretty heavy overhead.
Correct, enabling software_preemption has never been tested on mobile platforms with will most likely fail on many cases.

We’re investigating the proper solution for this issue, and will include the fix in coming release.
Once any clear schedule, I will post to you.

Thanks

6

I want to be sure you don’t have the issue backward. I’m saying if you don’t use software_preemption, then you cannot debug at all. If you do use it, then some things work.