Cupti - Crash in ToolsInjection64.dll

[FxR]

When closing my app it crashes in ToolsInjection64.dll. This occurs after I’ve set NVTX_INJECTION64_PATH to current tools path (Nsight Systems 2022.1.3).
In case I remove the injection path my app closes fine.

Any thaughts?

Regards,
Franz

[mjain]

Moved to the Nsight Systems forum.

[hwilper]

The first thing I am going to ask is if you can try with 2022.4 release available from the NVIDIA Nsight Systems | NVIDIA Developer.

Are you running through the GUI or the CLI, can you tell me what options you launched with?

[FxR]

Thanks for your answer. Using 2022.4 fixed it.
The crash occured while debugging in Visual Studio (Release build and debug build). My app does not use NVTX in release. I’ve seen that optix dll obviously uses NVTX.

Regards,
Franz

[FxR]

My stupid error: Hm, the crash is also here with 2022.4. Forgot to update the path, sorry.

[hwilper]

@afroger can you help out with this?

[FxR]

Is there any chance to fix this? Still crashing here on my machine…

[afroger]

Hi FxR,

I’m sorry you’re having trouble using Nsight Systems.

• Are you using Nsight Systems’ CLI or GUI?

  • If CLI, can you provide the exact command you are using?
  • If GUI, can you tell me roughly the features you have enabled? (e.g., CUDA tracing enabled, NVTX enabled, etc.)
  • If you’re using the GUI, could you give the CLI a try to see whether the problem goes away? If you’re not familiar with the Nsight Systems’ CLI, you can find the documentation here. You can start with nsys profile --trace cuda,nvtx --output test [application] [arguments].

• In your first comment, you mention that you’re setting NVTX_INJECTION64_PATH manually. You shouldn’t need to do this since the profiler will setup the environment for you when the application is launched by the tool (either through the GUI or CLI). Are there any specific reasons why you’re setting the environment variable yourself?

• From the context of this post, I’m guessing you’re trying to trace both CUDA and NVTX. Does the problem occur if you try to trace CUDA without NVTX? Conversely, does it occur if you try to trace NVTX without CUDA?

• Is the issue reproducible consistently on a wide variety of applications? If not, could should share steps so we can reproduce on our side?

Thanks for your help!

[FxR]

Hi afroger,

The crash occurs in Visual Studio while debugging. I’ve included nvToolsExt.h and I use NVTX interface in order to enable profiling for my opencl application.

The profiler is not started during my debug session, thus it was necessary to change the injection path. Accidentally I’ve had two installations of Nsight on my machine and the path pointed to the old one. This puzzled Nsight when used for profiling. So I changed the path manually.

I just use NVTX in an OpenCL application.

I’ve got two applications where this happens. One is the OpenCL app and the other one is using CUDA and NVTX.
Probably this is hard to reproduce as it happens only inside VS.
It is enough to call nvtxInitialize(0); in my app to cause the crash. Is there an nvtxClose?

Regards,
Franz

[afroger]

Hi FxR,

I apologize for the delayed response.

I just want to take a step back and explain a bit how NVTX works. NVTX is a header-only library that allows developers to annotate their code. Internally, it provides an interface for a tool to plug its implementation of the API. When no tool is running in the process, the NVTX implementation is a no-op. As you noticed, NVTX will look for a tool library provided through the NVTX_INJECTION64_PATH environment variable.

The crash occurs in Visual Studio while debugging. […] The profiler is not started during my debug session, thus it was necessary to change the injection path.

If I understand correctly, you’re not launching your application through the Nsight Systems’ GUI or CLI. You’re simply setting the NVTX_INJECTION64_PATH environment variable. Is my understanding correct?

Currently, Nsight Systems requires the application to be launched through its GUI or CLI. On Windows, you cannot simply set some environment variables when starting your process to enable the profiler. This is unsupported and untested.

In general, you don’t want to profile while debugging. Debuggers are very invasive tools that require compiling code without optimization. For profiling, you want to compile your code with optimizations to understand the performance characteristics of your application as it runs in a production-like environment. Profilers generally tend to be as noninvasive as possible to limit overhead.

To summarize:

• On Windows, you should never have to manually set the NVTX_INJECTION64_PATH environment variable. You should use the Nsight Systems GUI or CLI to launch your application
• Don’t try to run a debugger and profiler in the same process at the same time. Even if you could, the data returned by the profilers will be so heavily skewed because of the debugger overhead that it won’t be usable

It is enough to call nvtxInitialize(0); in my app to cause the crash. Is there an nvtxClose?

There is no nvtxClose.

Side note: except if your first call to NVTX is in a performance-critical section, my general recommendation is to let NVTX initialize lazily on the first call to any of its API. I rarely see nvtxInitialize being used.

[FxR]

Hi afroger,

sorry I thaught using the header file NVTX implementation is stable enough to be bound without GUI.
I understand that my lazy implementation is a bad solution. I’ll avoid using NVTX when no profiling is targeted.

Thanks for the details
Franz