NVIDIA Kernel Driver in use on only 4 of 6 GPUs

I have 6 GPUs attached via PCI slots on a machine. They are all recognized being recognized:

$ lspci -v | grep 'VGA'
01:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])
02:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])
04:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])
05:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])
07:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])
08:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])

However, the NVIDIA kernel driver is only being used on 4 of them:

$ lspci -v | grep -A 10 'VGA' | grep 'Kernel driver in use:'
Kernel driver in use: nvidia
Kernel driver in use: nvidia
Kernel driver in use: nvidia
Kernel driver in use: nvidia

On the other 2, no kernel driver is even listed:

$ lspci -v | ...
07:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation GP104 [GeForce GTX 1070]
Flags: fast devsel, IRQ 18
Memory at d6000000 (32-bit, non-prefetchable) 
Memory at <ignored> (64-bit, prefetchable)
Memory at <ignored> (64-bit, prefetchable)
I/O ports at a000 
Expansion ROM at d7000000 [disabled] 
Capabilities: <access denied>
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

A full listing for a GPU that has a kernel driver listed is presented for comparison:

05:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation GP104 [GeForce GTX 1070]
Flags: bus master, fast devsel, latency 0, IRQ 325
Memory at d8000000 (32-bit, non-prefetchable) 
Memory at c0000000 (64-bit, prefetchable) 
Memory at b8000000 (64-bit, prefetchable) 
I/O ports at b000 
[virtual] Expansion ROM at d9000000 [disabled] 
Capabilities: <access denied>
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

The 5th and 6th gpus are not recognized in nvidia-settings, either.

How can I get the 5th and 6th GPUs to start using the NVIDIA kernel driver?

Using: Ubuntu 17.04, NVIDIA driver version 375.66.

You need to enable “4G decoding” in bios.

We are having a similiar problem. Have you tried the 384.47 beta drivers?

https://devtalk.nvidia.com/default/topic/1016125/unix-graphics-announcements-and-news/-linux-solaris-and-freebsd-driver-384-47-beta-/#reply