I have a setup I just build where I have 4 2080ti and a 1600w psu and running ubuntu 20.04. When I start up the machine and run nvidia-smi I get the response below. Basically, gpu 1 has almost 3x the watts in usage vs the cap. What happens is if I try to run any model training then all the usage rises until one of the GPUs will crash as it has drawn too much power. I’ve tried swapping around the card location, verifying the power all with the same result. No matter which of my 4 cards is installed its always gpu number 1 that has this over power issue. The behavior persists if there are 3 gpus but goes away if there are 2.
Mon Oct 26 14:34:11 2020
±----------------------------------------------------------------------------+
| NVIDIA-SMI 450.66 Driver Version: 450.66 CUDA Version: 11.0 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 208… On | 00000000:19:00.0 Off | N/A |
| 30% 30C P8 16W / 250W | 6MiB / 11019MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 1 GeForce RTX 208… On | 00000000:1A:00.0 Off | N/A |
| 30% 32C P8 693W / 250W | 6MiB / 11019MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 2 GeForce RTX 208… Off | 00000000:67:00.0 Off | N/A |
| 22% 35C P8 4W / 250W | 6MiB / 11019MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 3 GeForce RTX 208… On | 00000000:68:00.0 On | N/A |
| 0% 57C P8 40W / 250W | 342MiB / 11016MiB | 27% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 2044 G /usr/lib/xorg/Xorg 4MiB |
| 1 N/A N/A 2044 G /usr/lib/xorg/Xorg 4MiB |
| 2 N/A N/A 2044 G /usr/lib/xorg/Xorg 4MiB |
| 3 N/A N/A 2044 G /usr/lib/xorg/Xorg 122MiB |
| 3 N/A N/A 2359 G /usr/bin/gnome-shell 91MiB |
| 3 N/A N/A 3082 G …AAAAAAAAA= --shared-files 126MiB |
±----------------------------------------------------------------------------+
