The whole Windows system is dead when starting CUDA debugging

As title, the whole Windows system is dead when starting CUDA debugging in Visual Studio 2015. The Nsight Monitor displayed 50% translucent pop-up window saying connected to localhost. The CUDA project is created from the template “CUDA 9.2 Runtime” without any special configuration. I tried both Nsight 5.6 shipped with CUDA 9.2 and Night 5.3. It works if I debug step by step using VS’s own debugger. An msinfo file is here: https://www.dropbox.com/s/gxft9733vcq3sdi/msinfo32.nfo?dl=0. Some of the environment info is below:

  • Windows 10 1803
  • Quadro K600
  • Driver 397.44
  • CUDA 9.2
  • Visual Studio 2015 Enterprise Update 3 version 14.0.25431.01
  • Nsight 5.3 or 5.6.

So what part of my system is Nsight unhappy with? K600 is too old? How to fix? Thanks.

K600 is a GK107 card and it should be supported, it it your display card? If you got system hang when hit the gpu break point, you should use software preemption mode to debug it.

Thank you. So the software preemption can avoid GPU halt, right?

I have only the K600 inserted into the MOBO; the intel embedded gpu seems disabled so K600 is the display card. I checked all the options of Nsight 5.6 and found that, if I didn’t miss anything, the software preemption options are only available under:

  1. Monitor’s Options -> CUDA(Legacy) -> Desktop GPUs Must Use Software Preemption
  2. Monitor’s Options -> CUDA(Legacy) -> Headless GPUs Must Use Software Preemption
  3. Monitor’s Options -> CUDA(Legacy) -> TCC GPUs Must Use Software Preemption
  4. Option in Nsight menu -> CUDA -> CUDA Debugger (Legacy) -> Preemption Preference
  5. Option in Nsight menu -> CUDA -> CUDA Debugger (Legacy) -> Preemption Timeslicing
  6. Option in Nsight menu -> Graphics -> Shader Debugging -> Preferred Remote Shader Debugging

The settings of these options are shown in the screenshot. Because I have only one nVidia graphics card which is the display card, 2) and 3) are irrelevant. All other settings open the software preemption as much as possible. However, the Windows system is still dead when launching CUDA Debugging. I guess maybe software preemption was not implemented very well or even its support was terminated in recent versions of Nsight for Quadro K600. If that is the case, that’s a good news because that means the culpret is indeed GPU halt due to breakpoints. But a worse possibility is that there are bugs. My second Kepler graphics card is on the way. I will set up remote debugging when it arrives, when we’ll know if GPU halt is the cause. Let’s wait and see.

Actually nsight monitor prefers to use software preemption mode when you only have one GPU, could you try to set “visual studio -> Nsight\Options\CUDA\Preemption Preference” to “Prefer Software Preemption”, disable all break point in your app, then check the NSIGHT->Windows->CUDA Info->Contexts->SW Preemption to make sure that sw preemption is enable

In order to pause in CUDA Debugger, I added for (;;); before return 0 (I can’t add it in device code because that’s equivalent to a breakpoint). But there is no “Contexts->SW Preemption” menu item under “NSIGHT->Windows->CUDA Info”. I am using Nsight 5.6.

You just need to click the CUDA Info 1, there will be a pop up window, you can check if you’re under software preemption mode.

Anyway if your system freezes when nsight hits a bp, I’m quite sure it’s sw preemption issue, the driver stops at cuda kernel and cannot do the display job.

The CUDA Info 1 window is empty. I can see no info in it.

But thank you for the valuable hint. The second Kepler graphics card is on the way so I’ll soon find out what the matter is.

Um, sorry I forgot two things, the first one is cuda info only works when you hit a bp, the second one is we only fully test gk104 and gk106, officially gk107 is not supported, I hope your second kepler card is one of them.

My next kepler card is GK110 (GeForce GTX 780 Ti). I’m going to cry …

So does gk107 (and GK110) absolutely not work with CUDA Debugging (and perhaps Shader Debugging)? That is, you only didn’t test it, so there is little hope that GK107 and GK110 still happens to be working?

Take it easy, because your system freezes when nsight hits a bp on your gk107, at least it means the break point works but the software preemption doesn’t work, I think it should work if you have a second card, call me when your get the second card.

Just one more question: did you fully test gk104 and gk106 for both CUDA Debugging and Shader Debugging, or only for CUDA Debugging?

As Shader Debugging only supports kepler card, I think both of them are fully tested in our auto test system.

Though my next card is still in transit, I got another same Quadro K600. After trials, the CUDA debugging finally works. Both machines have CUDA 9.1 and Nsight 5.5 installed (Windows 7 cannot install CUDA 9.2. On Windows 10 when CUDA 9.2 is installed, the client would report error that the version of target’s Nsight is 5.6 when actually the version installed is 5.5. Bugs anywhere. Sigh~) Here is some experience:

  • Local CUDA Debugging for K600 only works in Windows 7. The GPU doesn’t halt on breakpoints therein. As mentioned in one of my previous post, the whole system freezes on Windows 10 version 1803, because the K600 GPU halts on breakpoints in Win10. So I guess the root cause lies in the difference of handling software preemption between the two Windows systems.

  • For remote CUDA Debugging for K600, I can only use Windows 10 as host and Windows 7 as target. If we change the role of the two machines, the host (Windows 7) will complain that it cannot synchronize to the target (Windows 10):

So where is the “Nsight Output Log”? I’m so curious what on earth the problem is. PS: I have turned off the firewalls of both Windows machines, so nothing can prevent file copy except bugs in Nsight.

  1. If you have two K600 on one computer, you just need to set CUDA_VISIBLE_DEVICES=1, then launch the nsight monitor, just make sure that the monitor can get this env variable, the cuda debugger will work on you second cuda card and you can debug you app on both win 10 and win 7 now, as gk107 is not officially supported and will be dropped in cuda 10, I think using a pascal card is a better choice.

  2. You can try to use admin privileges to launch the nsight monitor on you win 10, I guess there are some privilege issues on your system because the remote debugging will synchronize the binaries and source codes to the c:\ on your target machine, or you can override the default synchronization dir, please ref https://docs.nvidia.com/gameworks/index.html#developertools/desktop/nsight/synchronization.htm#kanchor266.

  1. I not only need to debug CUDA, but also want to debug GLSL. My previous graphics card is Pascal core GeForce GTX 1050 Ti. It plays very well with CUDA Debugging on Win10, but because only Kepler supports Shader Debugging, I sold it out.

  2. I tried launching Monitor on the target as Administrator just now, but of no avail. However, I noticed that some specific error messages are output to VS2015.
    In the Output window:

Nsight Debug
Synchronize error: IOException - The process cannot access the file 'Path_To_Project\testCUDA.VC.db' because it is being used by another process. - Path_To_Project\testCUDA.VC.db
Nsight Debug
Synchronize error: IOException - The process cannot access the file 'Path_To_Project\testCUDA.VC.VC.opendb' because it is being used by another process. - Path_To_Project\testCUDA.VC.VC.opendb

In the Error List window:

where the source code of kernel.cu around line 91 is:

// Launch a kernel on the GPU with one thread for each element.
addKernel<<<1, size>>>(dev_c, dev_a, dev_b);

The “<” immediately before 1 is underlined in red.

I don’t know what these info means. Do you have any idea? Thanks.

testCUDA.VC.db and testCUDA.VC.VC.opendb don’t need to be synced to the target machine, you can filter them

underlined red is excepted, visual studio cannot recognize <<< as it is a cuda not standard c++ operator, if the compiler doesn’t say anything wrong, that should be fine.

Yes! It is working now. Thank you so so much for your help, harryz_. It’s so nice and erudite of you!

May I ask you one more favor: could you please take a look at this question: https://devtalk.nvidia.com/default/topic/1035848/nsight-visual-studio-edition/shader-debugger-just-doesn-t-work-all-kinds-of-strange-errors/. Thank you again for your kind help.

It’s my pleasure to help you, but shader debugging isn’t my scope, I think some other guys will soon help you.

I got my second Kepler card – GeForce GTX 780 Ti - today and everything is working now. Thank you very much for the help!