NSight: gpu trace profiler fails to capture trace on 4070

Developer Tools Nsight Graphics
1

Hello!

GPU trace profiler fails to capture both Vulkan or DX12 traces.
Symptoms:

  • CPU-Memory usage goes up, until it reaches physical limit (32GB), and trace silently fails.
  • There is a message in log: Multiple GPU usage detected - collected data may not be complete.
  • Message is not true - application uses only single gpu. But system physically has 2: dedicated+iGPU

Issue happens since day 0 on this laptop, so can’t say was it update related or not.

My game also has it’s own crash-logger, here is the output:

---crashlog(SIGSEGV)---
GPU: NVIDIA GeForce RTX 4070 Laptop GPU
0x00007ff7df31164a: dbg::call_stack<64u>::collect(unsigned int) in C:\Programming\OpenGothic\build\x86_windows_msys_pe_64bit-Debug\opengothic\Gothic2Notr.exe
0x00007ff7df3112d0: CrashLog::dumpStack(char const*, char const*) in C:\Programming\OpenGothic\build\x86_windows_msys_pe_64bit-Debug\opengothic\Gothic2Notr.exe
0x00007ff7df310bb2: signalHandler(int) in C:\Programming\OpenGothic\build\x86_windows_msys_pe_64bit-Debug\opengothic\Gothic2Notr.exe
0x00007ff7df5c7290: _gnu_exception_handler in C:\Programming\OpenGothic\build\x86_windows_msys_pe_64bit-Debug\opengothic\Gothic2Notr.exe
0x00007ff9ef103dd3: _C_specific_handler in C:\Windows\SYSTEM32\ntdll.dll
0x00007ff9ef143f0f: _chkstk in C:\Windows\SYSTEM32\ntdll.dll
0x00007ff9eeff3b78: RtlWow64GetCurrentCpuArea in C:\Windows\SYSTEM32\ntdll.dll
0x00007ff9ef14384e: KiUserExceptionDispatcher in C:\Windows\SYSTEM32\ntdll.dll
0x00007ff919dab46b: QueryInjectionModuleClient in C:\Program Files\NVIDIA Corporation\Nsight Graphics 2025.1.0\target\windows-desktop-nomad-x64\WarpVizTarget.dll
0x00007ff919b88fd3: InitializeInjectionNvtx2 in C:\Program Files\NVIDIA Corporation\Nsight Graphics 2025.1.0\target\windows-desktop-nomad-x64\WarpVizTarget.dll
0x00007ff919b8a997: InitializeInjectionNvtx2 in C:\Program Files\NVIDIA Corporation\Nsight Graphics 2025.1.0\target\windows-desktop-nomad-x64\WarpVizTarget.dll
0x00007ff919b89911: InitializeInjectionNvtx2 in C:\Program Files\NVIDIA Corporation\Nsight Graphics 2025.1.0\target\windows-desktop-nomad-x64\WarpVizTarget.dll
0x00007ff9ee3e3b33: [unknown function] in C:\Windows\System32\sechost.dll
0x00007ff9ee403180: ProcessTrace in C:\Windows\System32\sechost.dll
0x00007ff919b899ca: InitializeInjectionNvtx2 in C:\Program Files\NVIDIA Corporation\Nsight Graphics 2025.1.0\target\windows-desktop-nomad-x64\WarpVizTarget.dll
0x00007ff919b50603: [unknown function] in C:\Program Files\NVIDIA Corporation\Nsight Graphics 2025.1.0\target\windows-desktop-nomad-x64\WarpVizTarget.dll
0x00007ff919b52d7a: [unknown function] in C:\Program Files\NVIDIA Corporation\Nsight Graphics 2025.1.0\target\windows-desktop-nomad-x64\WarpVizTarget.dll
0x00007ff919b4e9fe: [unknown function] in C:\Program Files\NVIDIA Corporation\Nsight Graphics 2025.1.0\target\windows-desktop-nomad-x64\WarpVizTarget.dll
0x00007ff919b6ba5c: [unknown function] in C:\Program Files\NVIDIA Corporation\Nsight Graphics 2025.1.0\target\windows-desktop-nomad-x64\WarpVizTarget.dll
0x00007ff919ba87be: DISPATCH_vkNegotiateLoaderLayerInterfaceVersion in C:\Program Files\NVIDIA Corporation\Nsight Graphics 2025.1.0\target\windows-desktop-nomad-x64\WarpVizTarget.dll
0x00007ff919b96e68: InitializeInjectionNvtx2 in C:\Program Files\NVIDIA Corporation\Nsight Graphics 2025.1.0\target\windows-desktop-nomad-x64\WarpVizTarget.dll
0x00007ff91b7ecc05: ZN7Tempest6Detail7VDevice5Queue7presentER16VkPresentInfoKHR in C:\Programming\OpenGothic\build\x86_windows_msys_pe_64bit-Debug\opengothic\libTempest.dll

Device: MSI Creator Z17 HX Studio
Driver: 572.83, game-ready
NSight: 2025.1.0.0

2

Hi,

Thank you for using Nsight Graphics and providing your feedback. We are sorry for any inconvenience you have encountered.

It’s hard for me to say anything right now. Could you please provide a simple example that would allow us to reproduce the issue? This will help us in investigating and resolving the problem more efficiently.

Thanks
An

3

On my end reproducible with anything, for example with vkcube application from VulkanSDK.

Here is screen-shoot of task manager, after pressing capture button(F11):

image
image1020×177 12.8 KB

After less than a minute it reached 99% memory usage and then crashed.

4

Hi Try,

I take some try on my local test machine, unfortunately, I can’t repro it. After press F11 on vkcube.exe which launched by Ngfx 2025.1.1’s GPUTrace activity, the memory usage just increases a little bit and keep the same.

It’s hard to say why, but can you do some try?

  • Does this happen on other machines?
  • Do you turn on/off some Vulkan layout? Please do some check by using Vulkan Configurator aka. vkconfig.exe (under your VulkanSDK’s installment folder)

Thanks
An

5

Maybe I’m wrong, but Vulkan Configurator should not affect anything, when not running.
Either way, nothing interesting there.

In RegEdit I have:

ExplicitLayers:

image
image1080×331 35.6 KB

Implicit layers:
image
image1093×328 38.5 KB

Does this happen on other machines?

I use to have another laptop before (it melted now :P) with 3070 + win10. There everything with same application, profiler use to work.

6

Hi Try,

Maybe I’m wrong, but Vulkan Configurator should not affect anything, when not running.

Just want to make sure the vkconfig.exe doesn’t run silently in system tray and mess something up.

Implicit layers

The layers also look OK.

I can’t repro this issue on my test machine. Some additional things you can try (Just brainstorm)

  • ‘Reset Activity’ button within GPUTrace activity
  • Reinstall the driver
  • Use some 3rd part tool to analysis the memory usage/leak?

By the way, Nsight Graphics 2025.2 is out, maybe you can update and take another try?

Thanks
An

7

I’ve updated driver to 572.83 and NSight to 2025.2. Nothing changed - same memory grow into a crash.

Also brainstorm:
Why NSight shows 2 windows for capture? There should be only a single one.

image
image471×114 2.86 KB

Why there is a warning Multiple GPU usage detected - collected data may not be complete.? In engine I’m not using mGPU, only single physical device.

Use some 3rd part tool to analysis the memory usage/leak?

I’m not really aware of any free and good one. If you can suggest some - can give it a go.

8

Hi Try,

Why NSight shows 2 windows for capture? There should be only a single one.

This feature allows you to profile certain windows, which means your app needs to use multi window.

Why there is a warning Multiple GPU usage detected - collected data may not be complete. ? In engine I’m not using mGPU, only single physical device.

Could you just disable your iGPU in the BIOS, and give it another try? If your BIOS doesn’t have such option, you can try to force your app to use dGPU by changing the settings in NVIDIA Control Panel. It’s hard to say the huge memory usage has some relationship with the warning, but we can take some try at least.

I’m not really aware of any free and good one. If you can suggest some - can give it a go.

I am not an expert on this topic, maybe you can give a try about Visual Studio, it has some built-in memory analysis feature ( Measure memory usage in your apps - Visual Studio (Windows) | Microsoft Learn) Just a personal suggestion though. :)

Thanks
An

9

This feature allows you to profile certain windows, which means your app needs to use multi window.

Possibly I was unclear on this part. NSight shows, as-if there are 2 windows for capture, while tracing a simple single-window application. While can definitely be unrelated and minor issue, but can also be side effect of bigger issue. For example, what if it collect samples in a loop, while waits for 2 present-events.

changing the settings in NVIDIA Control Pane

I’ve give that a go today. No changes - still shows multi-gpu for some reason. Done only via control panel, can’t do via BIOS, unfortunately.

10

Finally got time to build MSVC executable and run profiler.

Not much interesting unfortunately: it doesn’t do much due to lack of debug symbols of NSight, only this is shown:

image
image1209×890 24.1 KB

Basically, after 36 seconds since capture started, WarpViz.Injection.dll allocated about 13.9 GB of memory - that’s all unfortunately.
Let me know, if there are some setting to toggle, if it can help localize the issue.

11

Hi Try,

Based on your screenshot, it appears thatWarpViz.Injection.dll is consuming more memory than expected. Could you please provide a simple example that would allow us to reproduce the issue? This will help us in investigating and resolving the problem more efficiently.

Thanks
An

12

Hi, AYan!

As mentioned before, issue persist for any application on my system with both vulkan and dx12.
Here vkcube:

image
image1213×643 24.5 KB

13

Hi Try,

Sorry that I missed it. It’s weird that you meet such an issue.

  • While install driver, could you try a clean installation (Choose Custom and select Perform a clean installation)
  • Try to toggle some settings in GPUTrace activity.
  • Could you pick another machine and give it a try?

Thanks
An

14

Hello, again!

I’ve tested on other machine today: NSight 2024.2 works fine; 2025.2 - driver soft-crash.By soft-crash I mean black screen for a few second until windows recovers.
System was: win10; RTX3070; descktop. Not my machine, so couldn’t do much more.

Meanwhile locally:

  • Driver is cleanly installed
  • Both 2024.2.1 and 2025.2 not capturing anything, crashing with memory leak
  • Tweaking settings, lowering intervals and such has no effect it seems

A new log line I’ve notices in today’s testing:
NVIDIA Nsight Graphics,Sample duration = 2000ns; Warp State Sampling Interval = 16; GPU PMA Buffer Size (MB) = 4000; SOC PMA Buffer Size (MB) = 0

4000MB, essentially 4GB, appears as a bit much for a single frame. Also can’t find Sample duration in the settings, but 2000ns seem fine to me.

15

Hi Try,

Not much clue here, I take a try on a laptop (Win11/Ada GPU), and run the Nsight Graphics 2025.2 (download from the public web), launch the bundled sample (Help → Samples → vk_raytrace), do GPUTrace, the memory usage keeps ~500MB.

I also tried another desktop (Win10/Ampere GPU), the memory doesn’t leak when using vkcube or bundled vk_raytrace.

NVIDIA Nsight Graphics,Sample duration = 2000ns; Warp State Sampling Interval = 16; GPU PMA Buffer Size (MB) = 4000; SOC PMA Buffer Size (MB) = 0

It shows the same GPU PMA Buffer Size (MB) = 4000 here. The rest message also looks fine, except the Warp State Sampling Interval = 6, it seems you changed Warp State Samples Per PM Interval to a higher value.

Sorry that I have no clue about your issue.

Thanks
An

16

Yeah, I understand that it’s very difficult to debug such unpronounceable problems. I hope crash-logs that been generated over time can be of use for your developers.
Let me know, if there are any logs that I can share from my machine, or if there are new releases with extended logs.

Will use debugger as profiler for now :D

17

at the time you’re generating your crash-log, can you generate a minidump from a debugger and provide that to us? that might help us understand a bit better what is going on in the process.

18

Here you are:

Had to upload it to google drive - 375mb and about 27GB when unzipped.

19

can you add GPUTRACE_DISABLE_ETW=1 to the environment setting when launching through the GPU Trace activity ?
GPU Trace tries to collect ETW (event tracing for windows) events during collection and it looks like the failure happens during the parsing of those events. I’m not understanding why that would be the case, but that environment variable might at least unblock you.

(to put more color, ETW reports 3Billion contexts to GPU Trace in a CommandQueueStart event - which is non-sensical)

20

Hi,
GPUTRACE_DISABLE_ETW=1 works, thanks @AMAMODE !