NSight 4.6 + 347.88 flakiness?

Developer Tools Nsight Visual Studio Edition
1

I’m seeing a lot of mysterious behavior with Nsight 4.6, 347.88 and CUDA 7.0 on Win7/x64.

My primary debug target is a headless Maxwell 750 Ti.

Setting breakpoints in certain locations will hang the kernel. The debugger never resumes control at the breakpoint but you can break/stop out of the debugger.

Some kernels don’t seem to be debuggable no matter where the breakpoint is set.

This is in a very mature 24x7x365 codebase that was debugging fine with 4.5, CUDA 7 RC and the previous driver.

I hate rolling back… any suggestions?

2

Hi,

Could you check your Monitor’s setting? Right click on the Nsight Monitor’s icon → CUDA → Debugger. What’s your setting of “headless GPUs must use software preemption”? Can set it to True and try again? If the problem remains, would you mind send your project and let us to try locally? Give me your email address and I will give you a FTP site to upload. Thanks.

3

I’m back on Nsight 4.5 and with the “Prefer Software Preemption” option in the VS2013>Nsight>Options panel I can now hit a breakpoint in the problematic kernel:

648×533 45.7 KB

Thanks @qzhang! This will let me proceed.

I’m curious why this was necessary since it’s a headless GTX 750 Ti. I am on the latest 347.88 Geforce driver.

Last night I reverted from CUDA 7 to CUDA 7.0.18 RC + Nsight 4.5 because of a huge performance drop due to spilling on dozens of kernels. So I’ll wait until that’s fixed.

4

Hi,

Glad to hear it makes progress. Under current preemption mode, when the BP is hit, could you check which GPU it’s running on? Open CUDA Info window via Nsigh menu → Windows → CUDA Info → CUDA Info 1. Switch to Contexts page. Then you could see which GPU it’s running on and whether it’s SW Preemption mode. Is it indeed running on headless GTX 750 Ti? Thanks.

5

Here you go…

791×47 12.7 KB