nvidia-smi is slow on Ubuntu 16.04

Accelerated Computing CUDA CUDA Setup and Installation
1

I have 2 P-100s. I installed 375-66 drivers and CUDA 8.0 on Ubuntu 14.04 and 16.04. Running nvidia-smi command on 14.04 instantaneously printed below GPU information out but nvidia-smi command always has 1~2 seconds of pause before printing out the information on 16.04. Why is nvidia-smi command slow on Ubuntu 16.04? My driver installation was successful and there were no Xid errors when I running the command.

±----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66 Driver Version: 375.66 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla P100-PCIE… On | 0000:02:00.0 Off | 0 |
| N/A 29C P0 25W / 250W | 0MiB / 16276MiB | 0% Default |
±------------------------------±---------------------±---------------------+
| 1 Tesla P100-PCIE… On | 0000:81:00.0 Off | Off |
| N/A 32C P0 25W / 250W | 0MiB / 16276MiB | 0% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
±----------------------------------------------------------------------------+

2

If GPUs are in persistence mode, or they are hosting a display, then the nvidia-smi command will complete quickly.

If these things are not the case, then the GPUs have to be “woken up” from idle, and that may take some time.

If you put one or both of the GPUs in persistence mode on the “slow” machine, you should see it speed up.

You can do this with the nvidia-smi command, use

nvidia-smi --help

to get command-line help.

Your printout shows that persistence mode is on, so I am assuming that is from the “fast” machine.

1 Like
3

Thanks for you comment txbob, this is very helpful. I run nvidia-persistenced --persistence-mode --user but it did not get faster. I think my Ub 16.04 server machine’s X-server or display was on because Ub 14.04 always turns its display off. I will check it out.

4

This worked for me under Ubuntu 16.04.2! I ran the command under sudo:
sudo nvidia-persistenced --persistence-mode

I had to run it as root in order for persistence-mode to activate, then nvidia-smi ran instantly. (It used to take about 5 seconds to check the status of 8 GPUs on one server).

5

Hi, tinkerthinker

I think you are right. I needed to active persistence-mode under sudo.

Thanks!