PM sampling doesn't work

Developer Tools Nsight Compute
1

Hi,

does the PM sampling feature require a specific cuda version? I ran with ncu 2023.3 but cuda 12.2 version and the PM sampling graphs are empty? do I need cuda 12.3? or am I missing something?

2

Hi, @mustafaali
Sorry for the issue you met ! Can you clarify which GPU you are using and the driver version?

3

Hi @veraj

Thanks for the prompt response.

I am using 545.23.08 driver version, on A100 SXM GPU.

I am using pytorch 2.1 that uses cuda 12.2

the ncu version used to profile is 23.3.1

4

Hi, @mustafaali
The NCU version，driver version and GPU seems OK. Do you profile in container ?

5

Yes, I’m profiling using the latest pytorch docker container from here

6

Hi, @mustafaali

Thanks for reporting this ! We can reproduce this with PyTorch. Our dev will check this.
I will let you know if there is new update.

1 Like
8

Thanks!

On a separate note,
Do you think the driver version of 545 doesn’t support A100 SXM data center?

9

I am not sure about the data center release. If you are running cuda sample correctly, i think it is supported.

1 Like
10

I ran with ncu 2023.3 but cuda 12.2 version and the PM sampling graphs are empty?

This is a problem related to the context switch data that is collected at the same time to filter samples belonging to your process vs other processes. Context switch trace doesn’t currently work properly within containers, but your Nsight Compute version doesn’t handle this problem yet, resulting in the entire range of samples being filtered.

Since the filtering is reversible, as a workaround until a newer Nsight Compute version is available, please right click the timeline and select “Toggle Context Switch Filter”. This will disable the (broken) filtering, and you should be able to view your raw sampled data.

Note that you can view the same data for PM sampling metrics, as well as all other metrics, also in the Metric Details tool window. Open the window and then select a metric on the timeline, or on the Details page, or search for a metric directly within the window.

1 Like
11

Thanks!

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.