Nsys profile doesn't collect tensor core utilization and the metrics about tensor active/SM instructions are not shown in the GUI

Developer Tools Nsight Systems Profiling Linux Targets
1

I am using nsys profile --gpu-metrics-device all ./my_executable to trace the metrics of GPU, but I didn’t get anything about tensor core metrics as written in the nsight-systems UserGuide

image
image2418×668 87.7 KB

How can I get SM instructions/Tensor Active row in the nsight GUI?

The architecture of the GPU is Hopper.

2

@pkovalenko can you help out here?

3

Please let me know the following:

  1. NSys version
  2. GPU driver version
  3. Output of nsys profile --gpu-metrics-device help

Is there anything mentioning GPU Metrics in Diagnostic Summary page?

4

Hi, the environment is as follows:

  • NSys version is 24.1.1
  • Driver version is : 535.154.05
  • The output of nsys profile --gpu-metrics-device help is :
    Possible --gpu-metrics-device values are:
        0: NVIDIA H100 80GB HBM3 PCI[0000:18:00.0]
        1: NVIDIA H100 80GB HBM3 PCI[0000:2a:00.0]
        2: NVIDIA H100 80GB HBM3 PCI[0000:3a:00.0]
        3: NVIDIA H100 80GB HBM3 PCI[0000:5d:00.0]
        4: NVIDIA H100 80GB HBM3 PCI[0000:9a:00.0]
        5: NVIDIA H100 80GB HBM3 PCI[0000:ab:00.0]
        6: NVIDIA H100 80GB HBM3 PCI[0000:ba:00.0]
        7: NVIDIA H100 80GB HBM3 PCI[0000:db:00.0]
        all: Select all supported GPUs
        none: Disable GPU Metrics [Default]

The messages in diagnostic summary page are as follows:

image
image2042×1354 405 KB

5

2024.1.1 is our latest release. Are you sure you’re running that? The diagnostic summary page looks quite old. The GUI looks new, but nsys CLI that collected your report is definitely not 2024.1.1.

In any case, the error message explains the issue. You have to either run nsys as superuser, or allow access to perf counters to all users. Please refer to https://developer.nvidia.com/ERR_NVGPUCTRPERM for detailed guidance.

7

Sorry, the GUI version of nsys is 2024.1.1, while the nsys CLI version is 2023.2.3.1004-33186433v0.

I ran nsys as superuser and collected the the metrics about tensor core utilization. Thank you for your answer.