Setup:
GPU0: gtx 780 ti - Display
GPU1: Quadro K6000 - Headless < program explicitly calls cudaSetDevice for this card.
GPU2: gtx 690 a - Headless
GPU3: gtx 690 b - Headless
Windows 7 x64
Cuda Version: 7.0
Driver: 353.06
Nsight: 4.6
Prior to any of the below solutions, nsight would complain about dynamic parallelism not being supported by software preemption. But I didn’t think this setup would be considered software preemption since I’m using a headless GPU for computations. So it seems nsight is trying to debug all GPUs even though I’m only using the quadro card.
Currently in the NSight monitor Options > CUDA > Debugger, my settings are:
Desktop GPUs must use software preemption: False
Headless GPUs must use software preemption: False
TCC GPUs must use software preemption: False
Use this monitor for CUDA attach: True
In Visual Studio > NSIGHT > Options > CUDA, my settings are:
Preemption preference: Prefer No Software Preemption
I’ve followed the solution here:
http://stackoverflow.com/questions/17906390/cant-debug-cuda-cuda-dynamic-parallelism-debugging-is-not-supported-in-preempt
And disabled TDR as per here:
https://devtalk.nvidia.com/default/topic/535264/cuda-programming-and-performance/kernel-runs-fine-on-osx-linux-crashes-on-windows-/post/3762516/#3762516
Now when I launch nsight with no breakpoints my display driver crashes. If TDR is enabled it recovers but not completely. I can see an error stating “CUDA Dynamic Parallelism debugging is not supported in preemption mode. Breakpoints will be disabled”.
If TDR is disabled, then my display freezes and I have to hard reboot my computer.
On Linux there is an option to select which GPU’s to use for debugging. However I can’t seem to find that here. I feel like I remember seeing it in a previous version of nsight, but I’ve poked around in VS and in the nsight moitor and can’t find it.
What am I doing wrong? Is there something hidden I need to set to prevent nsight from seeing my display card?