GPU SLOW | LINUX | UBUNTU 16.04 | Is this drive issue caused?

slow per usage:

nvidia-smi

Thu Dec 28 11:10:43 2017
±----------------------------------------------------------------------------+
| NVIDIA-SMI 387.26 Driver Version: 387.26 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K80 On | 00009EB9:00:00.0 Off | 0 |
| N/A 52C P0 143W / 149W | 10073MiB / 11439MiB | 6% Default |
±------------------------------±---------------------±---------------------+
| 1 Tesla K80 On | 0000B5B3:00:00.0 Off | 0 |
| N/A 69C P0 124W / 149W | 10073MiB / 11439MiB | 0% Default |
±------------------------------±---------------------±---------------------+
| 2 Tesla K80 On | 0000CBFB:00:00.0 Off | 0 |
| N/A 67C P0 115W / 149W | 10073MiB / 11439MiB | 49% Default |
±------------------------------±---------------------±---------------------+
| 3 Tesla K80 On | 0000E81C:00:00.0 Off | 0 |
| N/A 52C P0 94W / 149W | 10064MiB / 11439MiB | 0% Default |
±------------------------------±---------------------±---------------------+

# nvidia-smi -q 

PCI
        Bus                         : 0x00
        D[b]evice                      : 0x00[/b]
        Domain                      : 0xB5B3
        Device Id                   : 0x102D10DE
        Bus Id                      : 0000B5B3:00:00.0
        Sub System Id               : 0x106C10DE

PCI Device 0x00 ( looks like nothing attached on PCI)


# dmesg | grep nvidia

[   28.162333] nvidia: loading out-of-tree module taints kernel.
[   28.162337] nvidia: module license 'NVIDIA' taints kernel.
[   28.167500] nvidia: module verification failed: signature and/or required key missing - tainting kernel
[   28.173718] nvidia-nvlink: Nvlink Core is being initialized, major device number 242
[   28.176071] nvidia 9eb9:00:00.0: can't derive routing for PCI INT A
[   28.176072] nvidia 9eb9:00:00.0: PCI INT A: no GSI
[   28.192488] nvidia b5b3:00:00.0: can't derive routing for PCI INT A
[   28.192490] nvidia b5b3:00:00.0: PCI INT A: no GSI
[   28.204105] nvidia cbfb:00:00.0: can't derive routing for PCI INT A
[   28.204106] nvidia cbfb:00:00.0: PCI INT A: no GSI
[   28.215051] nvidia e81c:00:00.0: can't derive routing for PCI INT A
[   28.215052] nvidia e81c:00:00.0: PCI INT A: no GSI
[   28.235071] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  387.26  Thu Nov  2 20:22:07 PDT 2017
[   28.240994] [drm] [nvidia-drm] [GPU ID 0x9eb90000] Loading driver
[   28.240996] [drm] Initialized nvidia-drm 0.0.0 20160202 for 9eb9:00:00.0 on minor 0
[   28.241298] [drm] [nvidia-drm] [GPU ID 0xb5b30000] Loading driver
[   28.241299] [drm] Initialized nvidia-drm 0.0.0 20160202 for b5b3:00:00.0 on minor 1
[   28.241636] [drm] [nvidia-drm] [GPU ID 0xcbfb0000] Loading driver
[   28.241636] [drm] Initialized nvidia-drm 0.0.0 20160202 for cbfb:00:00.0 on minor 2
[   28.241852] [drm] [nvidia-drm] [GPU ID 0xe81c0000] Loading driver
[   28.241853] [drm] Initialized nvidia-drm 0.0.0 20160202 for e81c:00:00.0 on minor 3
[   28.297913] nvidia-uvm: Loaded the UVM driver in 8 mode, major device number 241
[ 3173.928591] nvidia-modeset: Allocated GPU:0 (GPU-9e750155-112c-7c87-e35b-0c9f97d7b45e) @ PCI:9eb9:00:00.0
[ 3173.928833] nvidia-modeset: WARNING: GPU:0: Correcting number of heads for current head configuration (0x00)
[ 3173.928942] nvidia-modeset: Freed GPU:0 (GPU-9e750155-112c-7c87-e35b-0c9f97d7b45e) @ PCI:9eb9:00:00.0
[ 3173.929225] nvidia-modeset: Allocated GPU:0 (GPU-9e750155-112c-7c87-e35b-0c9f97d7b45e) @ PCI:9eb9:00:00.0
[ 3173.929461] nvidia-modeset: WARNING: GPU:0: Correcting number of heads for current head configuration (0x00)
[ 3173.929554] nvidia-modeset: Freed GPU:0 (GPU-9e750155-112c-7c87-e35b-0c9f97d7b45e) @ PCI:9eb9:00:00.0
[ 3173.936748] nvidia-modeset: Allocated GPU:0 (GPU-eb532b16-1499-0bdb-f075-58c949bbd9ba) @ PCI:b5b3:00:00.0
[ 3173.936980] nvidia-modeset: WARNING: GPU:0: Correcting number of heads for current head configuration (0x00)
[ 3173.937076] nvidia-modeset: Freed GPU:0 (GPU-eb532b16-1499-0bdb-f075-58c949bbd9ba) @ PCI:b5b3:00:00.0
[ 3173.937355] nvidia-modeset: Allocated GPU:0 (GPU-eb532b16-1499-0bdb-f075-58c949bbd9ba) @ PCI:b5b3:00:00.0
[ 3173.937582] nvidia-modeset: WARNING: GPU:0: Correcting number of heads for current head configuration (0x00)
[ 3173.937676] nvidia-modeset: Freed GPU:0 (GPU-eb532b16-1499-0bdb-f075-58c949bbd9ba) @ PCI:b5b3:00:00.0
[ 3173.945096] nvidia-modeset: Allocated GPU:0 (GPU-8c85039d-caaf-9efe-763b-302f00314522) @ PCI:cbfb:00:00.0
[ 3173.945347] nvidia-modeset: WARNING: GPU:0: Correcting number of heads for current head configuration (0x00)
[ 3173.945447] nvidia-modeset: Freed GPU:0 (GPU-8c85039d-caaf-9efe-763b-302f00314522) @ PCI:cbfb:00:00.0
[ 3173.945743] nvidia-modeset: Allocated GPU:0 (GPU-8c85039d-caaf-9efe-763b-302f00314522) @ PCI:cbfb:00:00.0
[ 3173.945985] nvidia-modeset: WARNING: GPU:0: Correcting number of heads for current head configuration (0x00)
[ 3173.946083] nvidia-modeset: Freed GPU:0 (GPU-8c85039d-caaf-9efe-763b-302f00314522) @ PCI:cbfb:00:00.0
[ 3173.953263] nvidia-modeset: Allocated GPU:0 (GPU-d20aa673-b44e-62ef-abee-803ed9d4f79f) @ PCI:e81c:00:00.0
[ 3173.953502] nvidia-modeset: WARNING: GPU:0: Correcting number of heads for current head configuration (0x00)
[ 3173.953600] nvidia-modeset: Freed GPU:0 (GPU-d20aa673-b44e-62ef-abee-803ed9d4f79f) @ PCI:e81c:00:00.0
[ 3173.953887] nvidia-modeset: Allocated GPU:0 (GPU-d20aa673-b44e-62ef-abee-803ed9d4f79f) @ PCI:e81c:00:00.0
[ 3173.954113] nvidia-modeset: WARNING: GPU:0: Correcting number of heads for current head configuration (0x00)
[ 3173.954203] nvidia-modeset: Freed GPU:0 (GPU-d20aa673-b44e-62ef-abee-803ed9d4f79f) @ PCI:e81c:00:00.0
~~~~~~~~~~
can't derive routing for PCI INT A        |  is this means drive not installed well?

Check for a bios update.
Are you using some special kernel parameters?

This is on Azure VM. How to check Bios update?

Azure is a different thing, see this for hints:
[url]System uses only 1 out of 4 GPUs at a time on Azure NC instance - CUDA Setup and Installation - NVIDIA Developer Forums