Hi,
I have 3 GTX 1080Ti I use for Deep Learning training. All 3 were working fine. Now one GPU seems to have some issues.
My PC (Ubuntu 16) starts fine, although it takes more time than it usually does.
When I run nvidia-smi tool the PC freeze till the tool returns and all the information is shown. Normally the nvidia-smi command takes no time to show the GPU stats (under 1 sec). It now takes over 15 seconds to show all the information. And I see that one GPU has an error (GPU usage and power usage). Here what it looks like (second line):
±----------------------------------------------------------------------------+
| NVIDIA-SMI 396.44 Driver Version: 396.44 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108… Off | 00000000:02:00.0 On | N/A |
| 0% 39C P8 15W / 250W | 192MiB / 11175MiB | 12% Default |
±------------------------------±---------------------±---------------------+
| 1 GeForce GTX 108… Off | 00000000:03:00.0 Off | N/A |
|ERR! 55C P0 ERR! / 250W | 2MiB / 11178MiB | 0% Default |
±------------------------------±---------------------±---------------------+
| 2 GeForce GTX 108… Off | 00000000:04:00.0 Off | N/A |
| 0% 42C P8 9W / 250W | 2MiB / 11178MiB | 0% Default |
±------------------------------±---------------------±---------------------+
When I remove the faulty GPU from the PC and leave only the 2 good ones, everything works as expected and nvidia-smi does not freeze neither my PC does.
I also noticed that when I start my PC, only the potentially faulty GPU LED flashes (the 2 others stay on).
What do you guys think?