Nsight Visual Studio crash with multi-gpu setup and dynamic parallelism.

GPU0: gtx 780 ti - Display
GPU1: Quadro K6000 - Headless < program explicitly calls cudaSetDevice for this card.
GPU2: gtx 690 a - Headless
GPU3: gtx 690 b - Headless

Windows 7 x64
Cuda Version: 7.0
Driver: 353.06
Nsight: 4.6

Prior to any of the below solutions, nsight would complain about dynamic parallelism not being supported by software preemption. But I didn’t think this setup would be considered software preemption since I’m using a headless GPU for computations. So it seems nsight is trying to debug all GPUs even though I’m only using the quadro card.

Currently in the NSight monitor Options > CUDA > Debugger, my settings are:
Desktop GPUs must use software preemption: False
Headless GPUs must use software preemption: False
TCC GPUs must use software preemption: False
Use this monitor for CUDA attach: True

In Visual Studio > NSIGHT > Options > CUDA, my settings are:
Preemption preference: Prefer No Software Preemption

I’ve followed the solution here:

And disabled TDR as per here:

Now when I launch nsight with no breakpoints my display driver crashes. If TDR is enabled it recovers but not completely. I can see an error stating “CUDA Dynamic Parallelism debugging is not supported in preemption mode. Breakpoints will be disabled”.
If TDR is disabled, then my display freezes and I have to hard reboot my computer.

On Linux there is an option to select which GPU’s to use for debugging. However I can’t seem to find that here. I feel like I remember seeing it in a previous version of nsight, but I’ve poked around in VS and in the nsight moitor and can’t find it.

What am I doing wrong? Is there something hidden I need to set to prevent nsight from seeing my display card?

You might want to post this question in the nsight visual studio forum:


If you only want to use the Quadro K6000 for CUDA, you could put that GPU in TCC mode.