# How to get real-time SM occupancy

**URL:** <https://forums.developer.nvidia.com/t/how-to-get-real-time-sm-occupancy/222562>\
**Category:** Nsight Visual Studio Edition\
**Tags:** pytorch\
**Created:** [August 1, 2022, 2:16pm UTC](https://forums.developer.nvidia.com/t/how-to-get-real-time-sm-occupancy/222562 "2022-08-01T14:16:58Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![x-wang20](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@x-wang20](https://forums.developer.nvidia.com/u/x-wang20)\
**Post date:** [August 1, 2022, 2:16pm UTC](https://forums.developer.nvidia.com/t/how-to-get-real-time-sm-occupancy/222562/1 "2022-08-01T14:16:58Z")

</div>

![image](https://global.discourse-cdn.com/nvidia/original/3X/7/1/71ec3d81277e27fba66332c4c6d70b5aa1b0dda1.png)  
As shown above, when I make inference in pytorch and measure SM occupancy using **nvidia-smi --query-gpu=utilization.gpu,utilization.memory,memory.total,memory.free,memory.used --format=csv -l 1 -i 0** , I get following output:  
 ![image](https://global.discourse-cdn.com/nvidia/original/3X/e/d/ed53feaf6b4044af7847e6ffb229e236d78ac59b.png)  
That is: GPU utils is 100%  
However , when I profiled such code using ncu and parse ncu-rep file like this : **ncu --metrics sm\_\_throughput.avg.pct\_of\_peak\_sustained\_elapsed,gpu\_\_compute\_memory\_throughput.avg.pct\_of\_peak\_sustained\_elapsed --page raw --csv --import model.ncu-rep** I got SM occupancy for each kernel as following:  
 ![image](https://global.discourse-cdn.com/nvidia/original/3X/2/8/28a96edd6c4c8d152a02a6043e7bd740a953a301.jpeg)  
SM occupancies from ncu-rep files are not always consistent with nvidia-smi outputs and are smaller than 100%. I was confused and wondered how I could get real-time SM occupancy ?Thanks in advance

---

<div class="post-metadata">

**Author:** ![Greg](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/greg/32/14043_2.png) [@Greg](https://forums.developer.nvidia.com/u/Greg)\
**Post date:** [August 16, 2022, 3:18pm UTC](https://forums.developer.nvidia.com/t/how-to-get-real-time-sm-occupancy/222562/2 "2022-08-16T15:18:26Z")

</div>

nvidia-smi will return 100% if there is any work in the compute engine (kernel with single thread).  
Nsight Compute (NCU) has counters per Streaming Multiprocessor (SM) that can count cycles that SM has any work `sm __cycles_active.avg.pct_of_peak_sustained_elapsed` or warp occupancy `sm__ warps_active.avg.pct_of_peak_sustained_elapsed`.

Nsight Systems can capture both of these metrics at very high sampling rates.

On data center GPUs the stats can also be collected via DCGM (Data Center GPU Monitoring). The default sampling rate is 1 Hz.