NSight 4.7 trouble with Titan Z

Hello,

I am having trouble using the NSight Visual Studio debugging tool with my Titan Z.
I am following the tutorial in the NSight 4.7 User Guide. After having set the proper breakpoints in the kernel, I start CUDA Debugging via the Nsight menu in VS2013 Community.
My screen goes black for about 3 seconds and then comes back. (For what it’s worth, I have tried cudaSetDevice(1) to force the second GPU on the Titan Z to do the work and leave the first for the display, but it made no difference)

The command window output:

VS2013 opened up a new tab titled “Source Not Available”, saying “Code not running - The current thread is not currently running code or the call stack could not be obtained.”

There are no local or auto variables. Continuing to the next breakpoint results in termination of the debug process.
However, the NSight monitor showed an active connection while debugging.

Here is NSight output:

Could someone point me in the right direction? I am unsure of what to do about this problem.
Thank you.

PS: I am using the latest 353.30 driver, CUDA Toolkit 7.5 and NSight 4.7.

Hi Matt,

Seems like your TDR is too short to launch debugger. Could you check this thing? Right click on the Nsight Monitor icon (which in the task bar)-> choose Options → General → WDDM TDR Delay. Its default value is 2 seconds, which is too short for debugging. You could extend it and reboot the machine to take effect your change.

Of cause you could ues preemption mode debugging, which is not affected by TDR. Also in Monitor options → CUDA. Set the first three options to True.

Qian

Thank you for your help, Qian.

I need to add another detail: CUDA debugging results in my display becoming unresponsive until the WDDM TDR watchdog runs out.

I have tried setting this to a higher number per your suggestion (I tried 5 seconds) but nothing changed, other than the time between starting debugging and the display going black, resetting the display driver.

I have also tried software pre-emption. Again, no change. I suspected that perhaps trying to debug an active kernel on an also active display unit could be problematic. The Titan-Z is a dual-gpu card, so I changed the kernel to run on the second, headless GPU. Again, no change.

The list of current GPU’s supported by NSight CUDA Debugging is suspiciously lacking an entry for the Titan-Z. The Titan X and Titan Black are there. I know the X architecture is quite different, but wasn’t the Black just ‘half’ a Z?
My worry is that the Titan Z is not currently supported by CUDA debugging, as somewhat supported by the documentation.

Hi Matt,

Are you running in SLI mode? I can get a similar situation as yours in SLI mode. To check this, right click on your desktop and choose NVIDIA Control Panel → 3D Settings → Configure Multi-GPU, Surround, PhysX. In the left Multi-GPU configuratoin box, is Maximize 3D performance checked? If it is, you may see there is a green bridge between two GPUs of TitanZ in the below graph. Check Disable multi-GPU mode and try again your project. Hope that works.

Cuda Debugging doesn’t work on SLI system is a known issue. Sorry for the inconvenience caused to you. Thanks for using Nsight.

Qian

Qian,

Again, thank you for your help! I was indeed running in SLI mode. I turned that off and CUDA debugging now works without problems. Thank you again!

-Matt