Hello! I have a debug build of TensorFlow built with symbols for Cuda as well as Tensorflow. My system has 2 Nvidia Quadro M2000s.
I am trying to debug a Python script I made, using Cuda GDB, however I keep getting the error :
fatal: All CUDA devices are used for display and cannot be used while debugging. (error code = CUDBG_ERROR_ALL_DEVICES_WATCHDOGGED(0x18)
or: Attaching to process running on watchdogged GPU is not possible. Please repeat the attempt in console mode or restart the process with CUDA_VISIBLE_DEVICES environment variable set.
For the first error I am simply running my program like:
cuda-gdb --args python3.6 script.py
The second error I get when I use PyCharm, set a breakpoint at a line, print the process ID then attach cuda-gdb to that process.
I tried following some of the advice online such as setting software preemption mode etc, but to no avail. New to cuda/cuda-gdb so sorry if I missed something obvious.