Nsight Systems doesn't profile kernels

Developer Tools Nsight Systems Profiling Linux Targets
1

I have an application where multiple CUDA streams are used to achieve more concurrency.
Nsight Systems doesn’t provide any information about kernels running in streams (in default stream either), saying that memory operations (memset in my case) take 100% of time what is obviously wrong.

Nsight Systems shows the following (mind there are no warnings or errors):
External Media
The output is the same either I profile via Nsight Systems GUI or with “nsys profile -t cuda ./myapp” command and then import a report file in GUI.

Versions, hardware:
Ubuntu 18.04, GeForce RTX 2070 (the same situation is on Tesla V100), Driver Version: 418.67, CUDA Version: 10.1.

What’s wrong?

UPDATE: the same situation is with the app that uses default stream for all the calculations (one of older versions of the app). So, multiple streams are not the case, kernels are just not traced.

2

Found the reason of such behavior.

There was dynamic parallelism in the app, after I avoided it everything worked fine.
Question 1: Is it a bug?

Still, there were no warning, errors or any other messages indicating any issues or restrictions on profiling my app.
Question 2: How can I suggest an improvement, or file a bug in case it is a bug, to Nsight Systems?

3

Thank you for drawing our attention to this! It looks like Nsight Systems currently doesn’t trace CDP kernels correctly. We’ll get a bug filed internally, and will update this thread once we have more information.

4

Thank you! Wish you all the best with fixing it!

5

Hi!

This issue should be fixed in the latest Nsight Systems release 2019.5: NVIDIA Nsight Systems | NVIDIA Developer

Unfortunately due to another issue, you would need to create a file, named config.ini with the following line: “HandleInvalidEvents=true” in the directory, where you launch the nsys command.

For example

% cat config.ini
HandleInvalidEvents=true
% nsys profile -t cuda ./yourApp

6

Hi!

Downloaded the new version, created config.ini with the content you posted. Several other issues…

  1. When I run nsys with the app with dynamic parallelism present in the code, but it is not even run in it, I get “an illegal memory access was encountered” error. Without nsys it finished without any errors.
  2. After that Nsight Compute GUI doesn’t show anything that has some sense. Check out the screenshot. External Media
  3. When I comment the code with dynamic parallelism everything works as expected.
  4. I can’t say that the solution with creation of some additional file is convenient and user-friendly. How people could know that a file should be created without reading this post?
7

Hi Timofei, unfortunately the screenshot doesn’t seem to be available for us (410 Gone). Can you please attach it directly in the reply, so that we could see it. Thanks!

8

Sorry for that.
Don’t see any possibility to attach it directly to the message, so, here is the public link to it https://yadi.sk/i/UTd6T-nJViaeJQ

9

I am facing the same issue with kernels with dynamic parallelism.

I am using Nsight Systems 2019.6.1

Running the following command does not list my kernels at all.
nsys profile --stats=true ./app

I followed the advice regarding config.ini, but still cannot see the kernels.

Any help would be appreciated.

10

I’m going to ask Andrey to have a look at this jaideep…however he is OOO until 1/9

11

Thank you very much! Looking forward to Andrey’s comments.

12

jaideep777:

We looked into the issue with stats not printed correctly, and the upcoming release will contain a fix for this.

Piane_Ramso:

Unfortunately support for CDP has been lower on our priority list and we couldn’t get back to it yet.

13

Is the CDP support in yet or close to ready?

14

Unfortunately the answer is still no, trace for CDP is not supported. Thank you for asking.