Can't get power usage on Titan V with driver 410.93

Graphics / Linux Linux
1

Hi!
I recently updated one server with a Titan V GPU, running CentOS 7.5.1804, to CUDA 10. Following the requirements, I updated the nvidia driver to version 410.93. (I previously had CUDA 9.2 and driver 396.37).
However, now when I try to get the power consumption of the GPU I get N/A. It is worth noting that with the previous CUDA and driver versions I had no issue getting the power usage of the same GPU.

Example:

$ nvidia-smi
Mon Jan 28 17:31:27 2019
±----------------------------------------------------------------------------+
| NVIDIA-SMI 410.93 Driver Version: 410.93 CUDA Version: 10.0 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 TITAN V Off | 00000000:01:00.0 Off | N/A |
| 40% 52C P8 N/A / N/A | 0MiB / 12036MiB | 0% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
±----------------------------------------------------------------------------+

For curiosity, I also tried with the driver version 415.27, and ended up with the same results:

$ nvidia-smi
Mon Jan 28 17:43:21 2019
±----------------------------------------------------------------------------+
| NVIDIA-SMI 415.27 Driver Version: 415.27 CUDA Version: 10.0 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 TITAN V Off | 00000000:01:00.0 Off | N/A |
| 40% 52C P8 N/A / N/A | 0MiB / 12036MiB | 0% Default |
±------------------------------±---------------------±---------------------+

Is this a known issue of these driver versions + the GPU? Am I doing something wrong?
nvidia-bug-report.log.gz (1.03 MB)

2

Please start the nvidia-persistenced to keep the gpu initialized.

3

That seems to only change the persistence mode, with no effect on how nvidia-smi reports the power usage:

$ nvidia-smi
Tue Jan 29 01:14:42 2019
±----------------------------------------------------------------------------+
| NVIDIA-SMI 415.27 Driver Version: 415.27 CUDA Version: 10.0 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 TITAN V On | 00000000:01:00.0 Off | N/A |
| 37% 51C P8 N/A / N/A | 0MiB / 12036MiB | 0% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
±----------------------------------------------------------------------------+

4

Further updates:
I have tried to put a different GPU (Titan Xp) on this machine and the above-presented problem is not observed. The power usage can be obtained, as expected, using nvidia-smi.

I have also tried to put the Titan V GPU on a different machine, where the exact same behaviour is observed:

  • No power usage on Titan V for drivers >= 410.93 (tested with 410.93 and 415.27)
  • otherwise power usage can be obtained as expected (tested with drivers 396.37 and 396.54)