Nsight System Crashes right before profiling finishes for a Vulkan Application

Developer Tools Nsight Systems Profiling x86 Windows Targets
1

When I try to profile a vulkan application, the application does launch without issue and the profiler does begin its process when the application shuts down but nsight crashes right before the profiling completes. The application always crashes when the “Samples processed” dataset around 72% and is always displays one of two messages before the Nsight freezes:

  1. New memory area at … /nvptxJitCompiler64.dll
  2. New memory area at … /nvrtum64.dll

In very rare occurances, I’ve managed to not encounter this error and the report is generated but this seems to happen more than 95% of the times so any indication on why this is happening would be really helpful.This is run on a laptop with a 2070 mobile GPU. I’ve tried to decipher the log file but had no luck so some help would be appreciated.

nsys-ui.log (19.4 MB)

2

When the profiler fails, is it successfully creating a .qdstrm or .qdrep file?

What all options are you using? I am assuming you are running from the GUI.

3

No, neither a .qdstrm or .qdrep file is created when the profiler fails. Yes, I’m running this from the GUI and the options I have enabled are:

  1. Sample target process
  2. Collect CPU context switch trace
  3. Collect GPU metrics
  4. Collect Vulkan Trace

Interestingly, I find that Nsight Systems also crashes when I try to open up some qdrep files that were generated when I ran the profiler a couple of months ago. I tried unintalling and reinstalling the program assuming it might have been an issue with my local build but that didn’t resolve any issues. I’ve attached the qdrep file that fails to open:
Report 1.qdrep (2.3 MB)

4

Are you running 2021.3?

Vulkan on Windows or Linux?

5

I’ve looped in one of my engineers to help as well. @ztasoulas

6

Hi adithyavenkatarao
Using NVIDIANsightSystems2021.3.1.54-ee9c30a on my Windows machine, the attached qdrep file opens without issues, I did not get a crash.

Regarding the crash during profiling, have you tried activating/deactivating the features you mentioned? Does the crash appear due to a specific feature?

7

I am running this on windows. And yes, I’ve tried enabling and disabling all of the features one by one to see if any particular one is causing this issue but no luck. Even running the profiler without any options leads to an application crash. @ztasoulas I am running the exact same version of NsightSystem.

8

Would it be possible to share the application? Using Vulkan samples I do no see any freeze/crash on Windows, with the same features enabled.

9

This issue seems to be application agnostic. For example, I see the same issue when I run the executables from GitHub - nvpro-samples/vk_mini_path_tracer: A beginner-friendly Vulkan path tracing tutorial in under 300 lines of C++. or GitHub - nvpro-samples/vk_raytracing_tutorial_KHR: Ray tracing examples and tutorials using VK_KHR_ray_tracing so I know that it’s not an issue with my application in particular. I was wondering whether you would know if there are background processes or something of that sort that might affect the execution of the profiler or if you see anything weird within the logs that might indicate what the problem might be.

10

I believe that the issue has fixed itself after I manually removed the Nsight Systems folder under C:/Program Files/NVIDIA Corporation. I noticed that the uninstaller did not remove some of the files from there and just tried removing it all before I reinstalled nsight systems and I’ve not been getting the issue ever since. To be honest, I don’t really understand why this would fix the issue but I can’t complain. If I start getting the same issue again, I’ll post it in this thread. Thanks again @ztasoulas and @hwilper for taking some time to look into this :)

1 Like
11

Well, glad the problem went away. Please do let us know if you hit anything else.

1 Like