Nsight 3.2 driver crash and can't step through shaders properly (DX11, HLSL)

Developer Tools Nsight Visual Studio Edition
1

First, the stats and specs:
Windows 7, 64bit
GTX470, running latest beta drivers (331.58)
Visual Studio 2010 ultimate
Using HLSL5.0
Running both the host and the monitor on the same computer

The problem: I’m trying to debug my pixel shader. So I start the graphics debugger, I pause the frame, I go to the shader list and select the proper shader. When put out a breakpoint, the device driver crashes, then in the bottom right it says that the driver has recovered from a crash.

After this, the breakpoint is active and I can look at the values, but when I try to step forward, it just loses focus and… keeps running, I guess? It’s usually at this point that I try to change focus back to any of the other windows, and that’s when visual studio freezes and dies.

I’m compiling the shaders using D3DX11CompileFromFile with these flags: D3DCOMPILE_ENABLE_STRICTNESS | D3DCOMPILE_DEBUG | D3DCOMPILE_SKIP_OPTIMIZATION

Not quite sure how relevant it is, but my shaders are using .psh/.vsh file extensions to get proper syntax coloring from my NShader plugin. I tried changing to .hlsl for one of the files, but it didn’t change anything.

This is the lastest crash log from when visual studio crashes:

Faulting application name: devenv.exe, version: 10.0.40219.1, time stamp: 0x4d5f2a73
Faulting module name: KERNELBASE.dll, version: 6.1.7601.18229, time stamp: 0x51fb1116
Exception code: 0xe0434352
Fault offset: 0x0000c41f
Faulting process id: 0x2b4
Faulting application start time: 0x01cecfe3fada2acd
Faulting application path: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\devenv.exe
Faulting module path: C:\Windows\syswow64\KERNELBASE.dll
Report Id: 43bf0837-3bd9-11e3-98dc-002522a0448c

All I can find regarding the actual display driver in the windows event viewer logs is this warning that keeps appearing after every crash:
Display driver nvlddmkm stopped responding and has successfully recovered.

I’m unsure if I can find more detailed crash logs for the graphics card somewhere? Either way, any help on this subject would be appreciated.

Edit: I’ve tried to do the same thing with Nsight 3.0 and 3.1, and I had the same problems. I’ve never been able to step through shaders, always had the same problem. Am I doing something fundamentally wrong? I have watched your dev videos on youtube when you’re debugging both OpenGL and DirectX applications, and I do exactly as they do, only difference is that when they step through the code it works like normal, but when I do it, it just loses focus/doesn’t step forward.

2

Hi,

Sorry about that trouble.
You are currently using Nsight 3.2 where you see this happen as well?

Let me clarify, it looks like there’s 2 separate issues. First (and most important) is that when you set a bp, you see the message about the driver recovering. This could be that by default, windows has a 2 second timeout (called TDR) if it detects that the GPU is unresponsive, and in your case (because of GPU, number of resources in the app, etc), whenever we hit the bp, we need to pull all that GPU state and it may be going beyond that 2 second timeout.
Could you bump the timeout to 10 seconds, please see the docs on how to change the TDR. Since you are on a local debug scenario, please keep “WDDM TDR enabled” set to True.
Please reboot your machine after this change and let me know how things go.

Second, you mentioned that devenv.exe crashes and see some of the info you pasted. Do you have more info on how/when this happens? Was it in use of Nsight functionality?

Thank you

3

Okay, first of all, the TDR change fixed it! Thanks a lot!

About the second problem, I’m sorry if I’m being vague, it’s a little hazy exactly what is going wrong, but I’ll try to describe it as clearly as possible.

I’ll begin with what I think is the root of the problem. When I try to step through the shader with too low TDR, everything becomes unresponsive. I then try to stop the debugging, (Shift+F5 or just pressing the button) and everything stops responding completely, the UI stops updating and everything. Then after a few seconds windows pop up that it has stopped working etcetc.

So yes, it does crash while using Nsight functionality, because I have the graphics inspector, shaders window and the graphics focus picker up in the background.

So to reproduce this error, I think you’d have to set the TDR too low, to make the shader debugging lose focus, and then try to stop the debugging through visual studio. I think that’d cause a crash or freeze.

I will try to set back the TDR to a low value and see if I can reproduce the crash for you. Will be back with results.

4

Okay! I’m a dummy, and my memory was slightly wrong. The crash happens when I first start the graphics debugging, press CTRL+Z and press the capture frame. And SOMETIMES, even if I do the exact thing every time, visual studio just freezes. It had nothing to do with breakpoints, it just sometimes freezes, with this error message in windows event viewer:

Faulting application name: devenv.exe, version: 10.0.40219.1, time stamp: 0x4d5f2a73
Faulting module name: KERNELBASE.dll, version: 6.1.7601.18229, time stamp: 0x51fb1116
Exception code: 0xe0434352
Fault offset: 0x0000c41f
Faulting process id: 0x144c
Faulting application start time: 0x01ced002e69187e0
Faulting application path: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\devenv.exe
Faulting module path: C:\Windows\syswow64\KERNELBASE.dll
Report Id: 5ea0f924-3bf6-11e3-a28b-002522a0448c

I tried for about 20 minutes to reproduce the crash with the breakpoint, using TDD==1, but everything went just fine. I’m terrible at this, and I’m really sorry that I can’t be of more assistance.

I did however find something interesting, although it is most likely unrelated. I wanted to fraps my process, to document exactly what I was doing, but it didnt work with my custom windows theme, so I set it back to Windows Aero or whatever the default theme in windows 7 is called, and suddenly I can’t get the API inspector window to show up anymore. It is greyed out. What’s up with that?

5

Good to hear you can shader debug now.

Moving to the VS hang issue, are you able to reproduce this with other applications (say any of the D3D SDK)?
Are you able to share your application? At least that I can run it on my side to reproduce the problem?
I don’t need the sources if it helps provide me with the reproducible.

As for the API Inspector issue, what was your custom windows theme details?
Is your reproducible:

  1. Set your windows theme to a custom theme (need to know what was your setting)
  2. Start debugging D3D app, and enter frame debugging, the API inspector comes up without a problem
  3. Stop debugging
  4. Without restarting VS, switch to the default Win7 (Aero) theme

At this stage, start debugging D3D app, enter frame debugging, it does NOT bring up the API inspector

What happens if you restart VS, does it work?

Also, are you using Nsight 3.2?

6

Quite busy right now, I’ll try to get back to you within a few days about the VS crash.

I can at least give you my source code: ftp://lemminas.dlinkddns.com/Public/Engine%20-%20Copy.7z

And yes, I am using NSight 3.2!

7

Ok, so! Apparently you can export windows 7 themes. Good to know. Here’s mine. It’s for Windows 7 64bit:

ftp://lemminas.dlinkddns.com/Public/CustomThemePack.themepack

If you don’t have a computer with w7 64bit, or for some reason can’t use it, the theme has a lot of changes done, all of them are made to improve performance and tweak away every single piece of fluff / graphics. All the UI is windows 98, for example. I’m not sure if I can even find a list of every single thing that I’ve changed.

I will have some time during monday, so I will try doing what you asked me to. Again, sorry for my tardiness.

8

This is what I did:

  1. Set windows theme to my custom theme.
  2. Start graphics debugging. → Graphics inspector shows up.
  3. Without restarting visual studio, I change theme to windows 7 default theme.
  4. Start graphics debugging. → Graphics inspector shows up again.

So everything works fine. I don’t really know what caused the problems last time. Just in case I also tried graphics debugging while I had fraps running (recording) in the background, just to see if it could have been fraps causing issues, but everything still worked fine.