# How to monitor SM utilization and SM occupancy?

**URL:** <https://forums.developer.nvidia.com/t/how-to-monitor-sm-utilization-and-sm-occupancy/177735>\
**Category:** System Management and Monitoring (NVML)\
**Created:** [May 12, 2021, 3:05am UTC](https://forums.developer.nvidia.com/t/how-to-monitor-sm-utilization-and-sm-occupancy/177735 "2021-05-12T03:05:48Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Noname.no](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Noname.no](https://forums.developer.nvidia.com/u/Noname.no)\
**Post date:** [May 12, 2021, 3:05am UTC](https://forums.developer.nvidia.com/t/how-to-monitor-sm-utilization-and-sm-occupancy/177735/1 "2021-05-12T03:05:48Z")

</div>

`nvidia-smi`gives volatile GPU util. which is useful if you want to know if the GPU is being used or not. It gives the amount of time a kernel was running on the GPU during a sampling interval.  
`nvidia-smi dmon` gives an `sm%`. I do not understand what it means exactly.

```auto
# gpu pwr temp sm mem enc dec mclk pclk
# Idx W C % % % % MHz MHz
    0 43 48 0 1 0 0 3505 936
    0 43 48 0 1 0 0 3505 936

```

I was hoping to know how this sm% number was calculated.

NVML, on the other hand, gives the current and max sm clock frequency. Is there a way to get the SM utilization percentage similar to CPU utilization% to measure how well a program uses the SM cores.

Also, how does one go about monitoring the SM occupancy?

From my understanding of Nvidia GPUs, it looks like to fully maximize utilization, one needs to occupy all the SMs and run the maximum number of instructions possible.  
Can the NVML API be used to get this information?

---

<div class="post-metadata">

**Author:** ![wychen2018](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@wychen2018](https://forums.developer.nvidia.com/u/wychen2018)\
**Post date:** [October 7, 2021, 6:52am UTC](https://forums.developer.nvidia.com/t/how-to-monitor-sm-utilization-and-sm-occupancy/177735/2 "2021-10-07T06:52:07Z")

</div>

Hi, have you solved this problem? I also want to understand this sm(%) meaning but it seems that there is no document on it.

---

<div class="post-metadata">

**Author:** ![BrentS](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@BrentS](https://forums.developer.nvidia.com/u/BrentS)\
**Post date:** [November 5, 2021, 2:34pm UTC](https://forums.developer.nvidia.com/t/how-to-monitor-sm-utilization-and-sm-occupancy/177735/3 "2021-11-05T14:34:03Z")

</div>

You can do this with Data Center GPU Manager (DCGM)

[https://docs.nvidia.com/datacenter/dcgm/latest/dcgm-user-guide/feature-overview.html#profiling](https://docs.nvidia.com/datacenter/dcgm/latest/dcgm-user-guide/feature-overview.html#profiling)

Download from here:

> **[NVIDIA DCGM](https://developer.nvidia.com/dcgm)**
>
> Manage and Monitor GPUs in Cluster Environments

---

<div class="post-metadata">

**Author:** ![skps23](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@skps23](https://forums.developer.nvidia.com/u/skps23)\
**Post date:** [April 3, 2023, 2:17pm UTC](https://forums.developer.nvidia.com/t/how-to-monitor-sm-utilization-and-sm-occupancy/177735/4 "2023-04-03T14:17:16Z")

</div>

Kindly answer what `sm` means as others asked. Thanks.

---

<div class="post-metadata">

**Author:** ![samyak.k.gupta](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@samyak.k.gupta](https://forums.developer.nvidia.com/u/samyak.k.gupta)\
**Post date:** [April 10, 2023, 3:53am UTC](https://forums.developer.nvidia.com/t/how-to-monitor-sm-utilization-and-sm-occupancy/177735/5 "2023-04-10T03:53:28Z")

</div>

Hi,

The python bindings for DCHM appear to be broken. Is there anywhere I can report bugs?

---

<div class="post-metadata">

**Author:** ![subramaniyan.4](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@subramaniyan.4](https://forums.developer.nvidia.com/u/subramaniyan.4)\
**Post date:** [April 27, 2023, 3:52am UTC](https://forums.developer.nvidia.com/t/how-to-monitor-sm-utilization-and-sm-occupancy/177735/6 "2023-04-27T03:52:23Z")

</div>

Hi, I am facing the same issue. I am not able to get

 ![untitled](https://global.discourse-cdn.com/nvidia/original/3X/c/b/cb43160474e526b51aea70f1372cdcb7c7153243.jpeg)  
utilization for my individual process. Any ideas on how to tackle this?

---

<div class="post-metadata">

**Author:** ![1033234617](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@1033234617](https://forums.developer.nvidia.com/u/1033234617)\
**Post date:** [January 11, 2024, 1:41pm UTC](https://forums.developer.nvidia.com/t/how-to-monitor-sm-utilization-and-sm-occupancy/177735/7 "2024-01-11T13:41:08Z")

</div>

Hi,  
Through NVML API has already support SM active and SM occupy in Hopper architecture. I have same problem with you in Ampere architecture. Anyone who provides help is very grateful！

nvml GPM API: [NVML API Reference Guide :: GPU Deployment and Management Documentation](https://docs.nvidia.com/deploy/nvml-api/group __nvmlGpmFunctions.html#group__ nvmlGpmFunctions)

thanks.

---

<div class="post-metadata">

**Author:** ![chelo.canino](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@chelo.canino](https://forums.developer.nvidia.com/u/chelo.canino)\
**Post date:** [January 12, 2024, 7:52pm UTC](https://forums.developer.nvidia.com/t/how-to-monitor-sm-utilization-and-sm-occupancy/177735/8 "2024-01-12T19:52:24Z")

</div>

testing on where this is located
