Hi all,
Has anybody seen this error, and if so what did you do to fix it?
Its showing up after approximately ~15,000+ calls to (varying from 12,000+ to 25,000+ iterations). The cudaResource is attached to a Qt 4.8.5 QGLBuffer unpacked pixel buffer that is allocated at program startup, and then re-used - no resizing is done as I allocate the maximum possible buffer size on startup as well.
I’ve thought of two options (1) Is to upgrade to Qt 5.1 since the QGLBuffer class is deprecated in that version, and (2) try deleting, re-initializing and re-mapping the buffer if the error is encountered. Option 2 is a last resort though.
Thanks!!
Throwing this out there … I just saw the following thread - See for example: Multithreaded graphics app, intermittent deadlock with Quadro FX3800, Driver 319.32, libGL.so.1 impl - Linux - NVIDIA Developer Forums for what I’m seeing a deadlock in the bowels of an NVIDIA OpenGL library makes sense.
Some additional information … it seems to only occur lately when recovering the computer after it goes into a sleep mode…
I ran into this error once before in a different context. I had a set of VBOs that where being used in a Cuda kernel for computation and then in an OpenGL shader for rendering. This error occurred when I changed my shader for debugging so that one of the VBOs wasn’t being used. Apparently there is a lazy allocation of the VBO so that if it isn’t used by OpenGL it isn’t actually created. Then it cannot be mapped.
Perhaps your pixel buffer is not being reallocated when returning from sleep mode.
Thanks for the suggestion… let me take a look at that. Should be simple enough… :-)
The fundamental issue ended up being an interaction with the gnome power management. Despite explicitly turning it off, the screen saver turned it back on, then after the time out value was exceeded the monitors were placed in a de-activated idle state. we hypothesize that some how this caused a reset or power down of the Quadro 4000 graphics card. I am testing a solution to force the dpms off via my .cshrc file through Thanksgiving week.
The work around solutions works. I ran successfully for over a week with out any issues.
We’ve included the following command in our .cshrc files.
xset -dpms
See the following link: Display Power Management Signaling - ArchWiki
Our system admins believe there may be a bug in the NVIDIA driver when recovering CUDA apps from sleep mode, not sure how to prove it though.