Const changing while moving out of function in debugger

Developer Tools CUDA Developer Tools CUDA-GDB
,
1

While attempting to debug a CUDA program I was inspecting a specific function call inside a kernel. I was inspecting the sudden changing of a variable value which was marked as a const. Now, I noticed that when I continued using stepping over until the return statement was hit and it went back to the function call the value did not change at all as it should.
However, if I use the Step Out(F11) then the value does actually change when it should not. Does anyone know why this happens, and how I can prevent it from happening when using step out?
I am using VSCode to debug.

2

Hi, @l.e.v.d.wal

Can you please provide a mini repro for us to investigate ? Thanks !

3

gpuexplore-debug-issue.zip (791.7 KB)

Hi veraj,

I don’t really have an easy way of sharing it, as the issue is part of a bigger project, this was the smallest version I could share that encountered the issue.

It is about the value of ‘node_index‘ during the ‘sync_explore_p5_release_both____p5_release_both_____p5_release_both‘ call, specifically during the call to ‘matrix_store‘. In here if I put a breakpoint at the ‘matrix_store‘ call, step into the call, and then step out, the value of node_index sometimes changes somehow, which I cannot explain as to why that would happen.

Is there any additional information you might need?

4

Hi, @l.e.v.d.wal

Thanks for providing the source file.

I checked internally and indeed can observe the value change. But I also note the function call also changed. So it is the variable with the same name in another function. Would you please check the call stack to confirm ? Thanks !

5

Hello,

Yes, this is a known quirk with CUDA debugging. When you execute Step Out (F11), it basically tells the whole warp to finish running. This often causes the debugger to lose track of your particular thread, and will display a stale register value from some other thread, hence the apparent modification of a const variable.

A reliable way to work around this is to use Continue (F5) by placing a breakpoint on the very line after the function call, instead of Stepping Out (F11). I find that this is often more stable with the VSCode Nsight extension, and allows the debugger to get correctly synchronized.

Also, try compiling your code with no optimization, ie-O0, and also with -g -G so the compiler is not playing with register assignments.

Hope this helps!

6

Ah, that makes a lot of sense Martin, that does explain it. Thanks for both your help and interest!