Calling functions within cuda-gdb

Cuda gdb is working pretty well tracing the program I want it to. However, when trying to sometimes print values/or results from a function call. Ie.
variable->value() where variable is some pointer, cuda-gdb just prints:
Python not initialized

I can confirm that when printing variable I get a meaningful output that I can follow to get value, but when I do just: *variable
I still get Python not initialized
It doesnt always do this which is strange to me. I found this thread [PATCH] Make "backtrace" doesn't print python stack if init python dir get fail and tried finding where gdb_python_initialized is initialized or used and just grepping /usr/local/cuda-10.1/share/gdb (which has the /python directory) for that name produces no results.

Any assistance would be appreciated!

Hi @davido,

Can you try using the cuda-gdb from the latest 11.5 CTK release? There are several overhauls to how we initialize python and should support your use case. Note that cuda-gdb only supports python 3+ integration as of CTK 11.5.

Can I even use cuda-gdb from that release if I am using Cuda 10.1 for the specific version of TensorFlow I am working with?

Latest debug build version with cuda symbols seems to be TF 2.1 ?
Appreciate the fast response.

Hi @davido to use CUDA Toolkit 11.5 you need minimal CUDA driver version 450.80.02 (see Release Notes :: CUDA Toolkit Documentation) for more details.

If you need help with getting TF running on recent CUDA versions, you might try CUDA Setup and Installation - NVIDIA Developer Forums forum.

This does not solve the problem. The latest version of TF with debug symbols for both itself and CUDA seems to be TF 2.1 which does not use CUDA 11, and works only with CUDA 10.1. Is there anything to do to fix this issue while remaining with an older version of CUDA because that is the only version compatible with builds that are actually important for debugging?

@davido can you try the following to disable python auto-loading:

(cuda-gdb) set auto-load python-scripts off

The python not initialized error for cuda-gdb bundled with CUDA Toolkit 10.1 is likely related to cuda-gdb locating pretty printers for the type you are trying to print. It is running into a failure due to python not being initialized. With cuda-gdb, we use a different approach to enabling python support via a dlopen mechanism. This is different than upstream gdb. With upstream gdb, if the python libraries are not found on the system the debugger will fail to start with a runtime link failure. Our approach is to provide as much support as possible.

You could try using the newer cuda-gdb with an older CTK. Do you know which version of the driver you are using? You can obtain this via the nvidia-smi command.

auto-load doesnt seem to have the option python-scripts in the version I am using. I get the error:
(cuda-gdb) set auto-load python-scripts off
Valid is only global ‘set auto-load no’; otherwise check the auto-load sub-commands.

And when checking info for auto-load python-scripts is not an option.
Also I believe I am using driver 470.74 though it says my Cuda version is 11.4 which is not true, I have a second installation of Cuda which would be 11.4, but I made sure to set environment variables such that 10.1 is used. Is this driver version still being used for 10.1?