Not unique to CUDA - I know. But is there anything specific to the CUDA printf?
I have this situation where adding a printf somewhere in a loop would make the program behave as expected but hangs otherwise. The code itself is quite bloated so I’d rather not post it.
On a high level, what are some potential causes that come to mind?
Edit : my bug has been identified - it was a race condition. But the question still open for some insights regarding effect of the printf function in general
in-kernel printf is always a function call, which is one of the more “disruptive” things in terms of compiler instruction scheduling (ordering of compiled code). A function call may impact register usage, for example.
One of the things that can happen is that it may affect the load and store pattern that the compiler implements, and this can affect the “visibility” of global data. global data always has some global visibility, but without other information, the compiler may “optimize” loads and stores in such a way as to “limit” visibility. And using (or not) a function call dropped into your code can affect this.
Suppose you have a “producer” loop in one thread that is setting a piece of global data, and then looping on some other piece of global data, to see if it is consumed. Unfortunately, without doing something special, merely “writing” global data in one thread may not actually make it visible to another thread.
So then you have another “consumer” thread that is looking for that global data, and never sees it, and never acknowledges it, and both threads spin forever - deadlock/hang.
The usage of printf in this scenario can cause a change in compiler behavior - perhaps writing that global data from a register out to global memory “earlier” than it otherwise would. And voila, your program begins to “work”.
(The underlying problem here is a bug in the code, of course. Thread-to-thread communication must be done carefully to avoid these things.)
I’m just sketching out something based on my experience. I don’t have any idea if its relevant to your case.
Adding printf() impedes code motion by the compiler, and this can easily cause behavioral change in the presence of a race condition. So whenever you see this sort of “Heisenbug”, where you try to observe incorrect behavior by adding a printf(), but the incorrect behavior goes away when you do that, suspecting a race condition as the root cause is an excellent initial hypothesis.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.