nvidia-smi Volatile GPU-Util 100%, always, reboot operating system can not fix

Dear all,

I have a problem. GPU-Util 100%, I have try following commands, but these are not working.

  1. sudo nvidia-smi -pm 1
  2. sudo nvidia-smi -r
  3. reboot Operating System

How can i fix this problem, here is nvidia-smi output

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.66       Driver Version: 410.66       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  On   | 00000000:40:00.0 Off |                  N/A |
| 31%   44C    P0    72W / 250W |      0MiB / 11178MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

nvidia-smi -a

Fan Speed                       : 18 %                                                                                                                                                                          
    Performance State               : P0                                                                                                                                                                            
    Clocks Throttle Reasons                                                                                                                                                                                         
        Idle                        : Not Active                                                                                                                                                                    
        Applications Clocks Setting : Not Active                                                                                                                                                                    
        SW Power Cap                : Not Active                                                                                                                                                                    
        HW Slowdown                 : Not Active                                                                                                                                                                    
            HW Thermal Slowdown     : Not Active                                                                                                                                                                    
            HW Power Brake Slowdown : Not Active                                                                                                                                                                    
        Sync Boost                  : Not Active                                                                                                                                                                    
        SW Thermal Slowdown         : Not Active                                                                                                                                                                    
        Display Clock Setting       : Not Active                                                                                                                                                                    
    FB Memory Usage                                                                                                                                                                                                 
        Total                       : 11178 MiB                                                                                                                                                                     
        Used                        : 0 MiB                                                                                                                                                                         
        Free                        : 11178 MiB                                                                                                                                                                     
    BAR1 Memory Usage                                                                                                                                                                                               
        Total                       : 256 MiB                                                                                                                                                                       
        Used                        : 2 MiB                                                                                                                                                                         
        Free                        : 254 MiB                                                                                                                                                                       
    Compute Mode                    : Default                                                                                                                                                                       
    Utilization                                                                                                                                                                                                     
        Gpu                         : 100 %                                                                                                                                                                         
        Memory                      : 0 %                                                                                                                                                                           
        Encoder                     : 0 %                                                                                                                                                                           
        Decoder                     : 0 %                                                                                                                                                                           
    Encoder Stats                                                                                                                                                                                                   
        Active Sessions             : 0                                                                                                                                                                             
        Average FPS                 : 0                                                                                                                                                                             
        Average Latency             : 0                                                                                                                                                                             
    FBC Stats                                                                                                                                                                                                       
        Active Sessions             : 0                                                                                                                                                                             
        Average FPS                 : 0                                                                                                                                                                             
        Average Latency             : 0                                                                                                                                                                             
    Ecc Mode                                                                                                                                                                                                        
        Current                     : N/A                                                                                                                                                                           
        Pending                     : N/A

Thank you for your time.

Hi,
I have the same problem : GPU seems to be at 100% and no process are running.
Could you please give a piece of advice to find a solution ? because it slows the general GPU activity.

Best regards.
Benoit

It’s possible that the act of running nvidia-smi can report 100% utilization. This will usually only affect 1 GPU in a multi-GPU setup.

Hi, were you able to fix the issue?
I got stuck with the same problem, and trying to find a solution but still not luck :(

HI there!
Did find a solution to the problem?
I got stuck with the same problem :(

anyone resolve the problem? The GPU cannot use anymore. I always need to restart machine when this happen

It might be that a process has hung on the GPU - try this

sudo fuser -k -9 /dev/nvidia0 <-replace 0 with the gpu index