is there any possibility to completely turn off those optimizations, that produce the “value optimized out” errors when printing a variable while debugging with cuda-gdb? I tried
an I wrote “volatile” before every variable declaration in the code but I’m still getting “value optimized out” in the debugger. Is it correct, that this message appears because the variable value isn’t “live” (= stored in registers)? And how can I prevent the compiler from storing variable values in registers then?
Currently it is not possible to avoid those messages with compiler switches. You may use prints at the end of your kernel to print the value - this way the compiler will be forced to keep the variable around for longer.
thanks for your reply. Yes, I’m aware of the fact that you can use some workarounds to get the variable values (printf or some useless assignments that prevent the compiler from optimizing these values out). But I was hoping that there’s a possibility to see every variable in the debugger without changing the code (because that’s what debuggers are for, right?). If I have to use printf anyway, I don’t need a debugger…
I have a similar problem. I was trying to run merge sort which is from CUDA SDK in the debug mode. There was a function called binarySearch in that kernel thats been called many times. The fact is, whenever I try to read value of the variables in that function, I got value optimized out. For example, I try to see what the return value of the binary search is but I can never see it through the debugger. In the Makefile if I understand correctly that if I use -g -G, there would be any O options added.
This is a common problem. The compiler could save all the variables to memory all the time to make sure no variable is optimized out, but, by doing so, some applications could run out of memory. Those applications would run properly at -O2 but fail at -G -g. Moreover, the applications would be noticeably slower.
The next release of the CUDA toolkit (after 5.5) will include several changes to mitigate the issue though. The number of “liveness holes” should be reduced.
Has this issue been addressed? I use Cuda 7.5 and have put the -g -G parameters in my Makefile.
I see a SIGSEGV in a printf () statement in the kernel and it crashes there while I attempt to dereference a pointer stored in a local variable. cuda-gdb says variables are optimized out.
This is still happening on CUDA 11.4. Are there any new things we can try to make those variables show up? It would be great to have a statement from nvidia :)
Hi @tobias.fischer1
Can you share a repro case (e.g. you host/kernel code)?
Also can you check whether the same issue is reproducible with CUDA 11.8? Since the issue is likely shared between the compiler and the debugger, please compile your application using nvcc from CUDA Toolkit 11.8 and use CUDA debugger from the same toolkit.
Thank you for the response. Sadly I don’t have a working example anymore. I am on the Jetson Platform, stuck with CUDA 11.4. Also CUDA 11.8 does not seem to be out yet.
There is a known issue with “optimized out” variables with the cuda-gdb from CUDA 11.4 which is based on gdb 8.3.1. Can you try using a cuda-gdb from a newer CUDA Toolkit release to see if the issue goes away? You may see “unavailable” with newer cuda-gdb releases. This is expected and is due to the PC being outside the live range of the register. When compiling with -G, the compiler may still allocate a variable to a register only. In that case, when we are outside the live range of the register we prevent the user from viewing the value of the variable to avoid observing incorrect values.
I need to correct my statement. I verified that the problem has been resolved in CUDA 11.7. There was a bug in our container $LD_LIBRARY_PATH so that the actual library used was CUDA 11.4. We fixed the bug and tested cuda-gdb with debug build. Now everything seems to be fine. Sorry for the confusion.
I have something else to ask. I would like to write a script for cuda-gdb to automate debugging of a CUDA program. I used something below:
set pagination off
set logging file SOME_OUTPUT_FILE
set logging off
set cuda break_on_launch application
break SOME_FILE_PATH: SOME_LINE_NUMBER
cuda kernel 0 block (0,0,0) thread (0,0,0)
DO_SOMETHING
end
run
and I got the following error message as I ran the script with cuda-gdb -x MY_SCRIPT MY_APP: Invalid coordinates. CUDA focus unchanged. Could you point me to the correct direction?
Seems that this command only works in command line. As I used it in a script under break struct as follows, the same message showed up.
set pagination off
set logging file gdb.output
set logging off
set cuda break_on_launch application
break /home/zfan/sandbox/Virgo-Algo-Container-3.0/Blazer/MercuryImageComputer/KT/leaf/Virgo/src/ops/RefGen/ArrayMPA/src/mpa_utility.cu:1134
cuda kernel 0 block (0,0,0) thread (0,0,0)
#info cuda blocks
end
run
Reading symbols from ./TestMPA.x86-64...done.
Breakpoint 1 at 0xfc2c6: file ../ArrayMPA/src/mpa_utility.cu, line 1134.
../mpa_debug_script:8: Error in sourced command file:
Invalid coordinates. CUDA focus unchanged.
set pagination off
set logging file gdb.output
set logging off
set cuda break_on_launch application
run
break /home/zfan/sandbox/Virgo-Algo-Container-3.0/Blazer/MercuryImageComputer/KT/leaf/Virgo/src/ops/RefGen/ArrayMPA/src/mpa_utility.cu:1134
cuda kernel 0 block (0,0,0) thread (0,0,0)
#info cuda blocks
end
We don’t support “just in time” evaluation when switching cuda focus, its immediate. Since there are no CUDA threads running when evaluated, the error message is printed. You might be able to achieve what you are trying to accomplish by using conditional breakpoints: