Time series data of metrics

Developer Tools Nsight Compute
1

Can anyone explain how to get time series data of metrics ? I want to collect all the counters at each time stamp. please suggest any documentation or commands to collect time series data of all counters.

Thanks and regards
Ramya

2

Nsight Systems supports periodic sampling of GPU metrics at as high as 200 kHz. There is a default configuration that has only a few SM counters + off chip interfaces (PCIe, NVLINK, DRAM). For GeForce/graphics focused GPUs there are also configurations that can collect the throughput (Speed of Light) metrics for most major units. The data can be visualized on the Nsight Sytem’s timeline but I don’t think it can be exported at this time.

3

what about Cupti? does it helps me to collect time series data?

Ubuntu 20.04.6 LTS (GNU/Linux 5.15.0-73-generic x86_64)
NVIDIA-SMI 530.30.02
Driver Version: 530.30.02
CUDA Version: 12.1

4

CUPTI does not have support for periodic sampling of GPU metrics.

The DCGM APIs support collection of GPU metrics. Refer DGCM profiling APIs documentation.

1 Like
5

image
image1112×515 14.3 KB

does above gpu version supports DCGM ?
Thanks and regards

6

No.
DCGM is supported only on NVIDIA datacenter GPUs. Refer Supported Platforms section of the DCGM document.

7

Can you please give information about the tool which helps me to collect time series data of all counters for nvidia rtx a5000 (screenshot having details of nvidia is already mentioned).

Thanks and regards

8

You can use Nsight Systems for RTX A5000 (as mentioned by Greg in the earlier reply). Refer the Nsight Systems Documentation.

9

I tried with Nsight compute cli before dcgm and cupti. I have collected metrics data with ncu but it is cumulative (such as sum,avg,max) for overall application. But counters data at each cycle during execution of application is required.
Thank you.

10

The suggestion is to use Nsight Systems (nsys) - not Nsight Compute (ncu).

1 Like
11

with ncu, so many metrics like dram, gpc, idc , etc., are supported. But nsys is not supported that many metrics?

12

Yes the list of metrics supported by Nsight Systems periodic sampling is small as in this mode only single pass metrics are supported.

Several of the Nsight Compute metrics require multiple passes (due to hardware constraints) for which the kernel is replayed multiple times.

1 Like