I am looking for a performance counter monitor that will monitor at runtime hardware metrics of an NVIDIA GPU. I need runtime monitoring (during the GPU application) and not monitoring after the end of the application like what nvidia visual profiler or nvprof do.
I have used nvidia-smi that monitors its metrics at runtime, but I need the metric of GPU utilization that refers to spatial percentage of total SMs/cores are used and not to time percentage (like what nvidia-smi refers to).
Is there any tool, that monitors spatial GPU utilization and other hardware metrics like cache misses during the CUDA application?
Nsight Compute and older versions of CUPTI are designed to monitoring single kernel launches. CUPTI has a continuous mode with a selection of counters that can be collected. The latest version of CUPTI is based upon Perfworks metrics API https://docs.nvidia.com/cupti/Cupti/r_main.html#r_host_metrics_api. The range profiling API may be able to satisfy your needs. Determining which metrics can be collected in single pass can be challenging.
The GPU is a collection of engines. The primary engine is the GR (graphics and compute) engine. Other engines include display, copy engines, NVENC (video encoder), NVDEC (video decoder), etc.
In hierarchical order I would recommend these counters
gr__cycles_active # of cycles where GR was active sm__cycles_active # of cycles with at least one warp in flight sm__warps_active cumulative # of warps in flight
with the metric
avg.pct_of_peak_sustained_elapsed
You can continue further down to look at instruction pipe active. Look for the counters named sm__pipe_{pipename>_cycles_active and add the same metric.
The Perfworks API through CUPTI provides 100s of raw counters with various roll-up and throughput metrics. For monitoring the key challenge is finding what can be collected in a single pass.