We are developing a game in a custom modified Unity 2022.3.10f1 and our users with NVIDIA GPUs are reporting “device removed” errors. It happens only with D3D11 render API. D3D12 or Vulkan APIs are fine, but their implementation in Unity is significantly slower and less reliable (CPU and VRAM management instability), so we are still dependent on the old D3D11.
We first tried dxcpl, enabled debug layer and all validations and fixed any reported errors. That didn’t help. So I’m using Nsight Aftermath monitor active, but my detected GPU crashes have reported shader hash as “N/A”:
Based on that shader binary hash and hashes reported by my colleagues, I was able to pinpoint the crash to particle init. or particle simulation compute shader of the same VFX effect. If I make a GPU capture of the app in NVIDIA Nsight, I can find the hash:
In order to cause the crash, I need to often alt-tab away and return back to the app after a while or it crashed while I was live analyzing a GPU capture in NVIDIA Nsight:
You can also alt-enter to switch to windowed mode, so app can be minimized or moved.
The app is showing a couple of skinned characters. Each of them has a VFX on itself, reading its vertices and spawning particles. Those are not visible, because the movement is mild, but it shouldn’t have any effect on the actual issue.
My problem is that the Nsight Aftermath can’t show where exactly it’s crashing. D3D11 doesn’t have created .nvdbg files, like the D3D12 version has, but the D3D12 is never crashing.
Thanks for using Aftermath SDK and I am sorry that you meet such issue. There is something I am trying to get some confirmation, but I will let you know what I get.
“D3D11 doesn’t have created .nvdbg files”, that’s some kind of expected for Aftermath. According to Aftermath SDK’s Readme.md:
Nsight Aftermath is only fully supported for D3D12 devices. Only basic support
with a reduced feature set (no API resource tracking and no shader address
mapping) is available for D3D11 devices.
Hence you will not get the .nvdbg files when using D3D11 API.
By the way, it seems you are also using Aftermath monitor. The Aftermath monitor only supports D3D12 and Vulkan according to its document.
Hey An,
thanks for the replay. Do I have a chance to get to the crash details using anything else? It happens only on NVIDIA and only with D3D11 that we must keep using. Would integrated Aftermath SDK give us more info?
Also, when the GPU crash is caught by the Aftermath monitor, there is some kind of address reported. Can I somehow get the shader assembly line number out of that?
In the provided reproduction app, the problematic Compute Shaders used in the VFX are already compiled with disabled optimization and adding debug info. If I make a GPU capture, I can already see and debug their HLSL source. Here is a RenderDoc capture:
I rechecked the doc, there might be some issue if you want to see the shader source if you are using legacy Microsoft DirectX fxc.exe shader compiler, here is some words from the doc:
No IL-level or source-level shader mapping is supported for DX bytecode shaders generated by the legacy Microsoft DirectX fxc.exe shader compiler.
