# GPU Utility Problem

**URL:** <https://forums.developer.nvidia.com/t/gpu-utility-problem/35448>\
**Category:** CUDA Programming and Performance\
**Created:** [November 4, 2014, 5:59am UTC](https://forums.developer.nvidia.com/t/gpu-utility-problem/35448 "2014-11-04T05:59:14Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![udaysankar](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@udaysankar](https://forums.developer.nvidia.com/u/udaysankar)\
**Post date:** [November 4, 2014, 5:59am UTC](https://forums.developer.nvidia.com/t/gpu-utility-problem/35448/1 "2014-11-04T05:59:14Z")

</div>

Hi All,  
I encounter a dramatic slow down of the performance in a gpu node. When I submit job in the node through a queue, the job initially run well. Output of nvidia-smi shows high GPU utility.  
±-----------------------------------------------------+  
| NVIDIA-SMI 5.319.37 Driver Version: 319.37 |  
|-------------------------------±---------------------±---------------------+  
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |  
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |  
|===============================+======================+======================|  
| 0 Tesla M2090 Off | 0000:0A:00.0 Off | 0 |  
| N/A N/A P0 81W / N/A | 159MB / 5375MB | 65% Default |  
±------------------------------±---------------------±---------------------+  
| 1 Tesla M2090 Off | 0000:0D:00.0 Off | 0 |  
| N/A N/A P0 106W / N/A | 159MB / 5375MB | 70% Default |  
±------------------------------±---------------------±---------------------+  
| 2 Tesla M2090 Off | 0000:2B:00.0 Off | 0 |  
| N/A N/A P0 90W / N/A | 158MB / 5375MB | 65% Default |  
±------------------------------±---------------------±---------------------+  
| 3 Tesla M2090 Off | 0000:30:00.0 Off | 0 |  
| N/A N/A P0 112W / N/A | 158MB / 5375MB | 67% Default |  
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+  
| Compute processes: GPU Memory |  
| GPU PID Process name Usage |  
|=============================================================================|  
| 0 5531 …/NAMD\_2.8\_Source/Linux-x86\_64-g++.cudanet/namd2 71MB |  
| 0 5527 …/NAMD\_2.8\_Source/Linux-x86\_64-g++.cudanet/namd2 71MB |  
| 1 5530 …/NAMD\_2.8\_Source/Linux-x86\_64-g++.cudanet/namd2 72MB |  
| 1 5526 …/NAMD\_2.8\_Source/Linux-x86\_64-g++.cudanet/namd2 71MB |  
| 2 5532 …/NAMD\_2.8\_Source/Linux-x86\_64-g++.cudanet/namd2 71MB |  
| 2 5528 …/NAMD\_2.8\_Source/Linux-x86\_64-g++.cudanet/namd2 71MB |  
| 3 5533 …/NAMD\_2.8\_Source/Linux-x86\_64-g++.cudanet/namd2 72MB |  
| 3 5529 …/NAMD\_2.8\_Source/Linux-x86\_64-g++.cudanet/namd2 71MB |  
±----------------------------------------------------------------------------+

But after 10-15 minutes the job slows down dramatically and the nvidia-smi shows 0 GPU utility.

±-----------------------------------------------------+  
| NVIDIA-SMI 5.319.37 Driver Version: 319.37 |  
|-------------------------------±---------------------±---------------------+  
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |  
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |  
|===============================+======================+======================|  
| 0 Tesla M2090 Off | 0000:0A:00.0 Off | 0 |  
| N/A N/A P0 78W / N/A | 159MB / 5375MB | 0% Default |  
±------------------------------±---------------------±---------------------+  
| 1 Tesla M2090 Off | 0000:0D:00.0 Off | 0 |  
| N/A N/A P0 78W / N/A | 159MB / 5375MB | 0% Default |  
±------------------------------±---------------------±---------------------+  
| 2 Tesla M2090 Off | 0000:2B:00.0 Off | 0 |  
| N/A N/A P0 81W / N/A | 158MB / 5375MB | 0% Default |  
±------------------------------±---------------------±---------------------+  
| 3 Tesla M2090 Off | 0000:30:00.0 Off | 0 |  
| N/A N/A P0 81W / N/A | 158MB / 5375MB | 0% Default |  
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+  
| Compute processes: GPU Memory |  
| GPU PID Process name Usage |  
|=============================================================================|  
| 0 5531 …/NAMD\_2.8\_Source/Linux-x86\_64-g++.cudanet/namd2 71MB |  
| 0 5527 …/NAMD\_2.8\_Source/Linux-x86\_64-g++.cudanet/namd2 71MB |  
| 1 5530 …/NAMD\_2.8\_Source/Linux-x86\_64-g++.cudanet/namd2 72MB |  
| 1 5526 …/NAMD\_2.8\_Source/Linux-x86\_64-g++.cudanet/namd2 71MB |  
| 2 5532 …/NAMD\_2.8\_Source/Linux-x86\_64-g++.cudanet/namd2 71MB |  
| 2 5528 …/NAMD\_2.8\_Source/Linux-x86\_64-g++.cudanet/namd2 71MB |  
| 3 5533 …/NAMD\_2.8\_Source/Linux-x86\_64-g++.cudanet/namd2 72MB |  
| 3 5529 …/NAMD\_2.8\_Source/Linux-x86\_64-g++.cudanet/namd2 71MB |  
±----------------------------------------------------------------------------+

Any suggestions.

---

<div class="post-metadata">

**Author:** ![Robert\_Crovella](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/robert_crovella/32/14043_2.png) [@Robert\_Crovella](https://forums.developer.nvidia.com/u/Robert_Crovella)\
**Post date:** [November 5, 2014, 2:45am UTC](https://forums.developer.nvidia.com/t/gpu-utility-problem/35448/2 "2014-11-05T02:45:01Z")

</div>

Are you running 2 MPI ranks per GPU?  
Does this job ever recover?  
What happens if you only run 1 MPI rank per GPU?

---

<div class="post-metadata">

**Author:** ![udaysankar](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@udaysankar](https://forums.developer.nvidia.com/u/udaysankar)\
**Post date:** [November 5, 2014, 7:08am UTC](https://forums.developer.nvidia.com/t/gpu-utility-problem/35448/3 "2014-11-05T07:08:20Z")

</div>

Same things happens.