384.47 Drivers. 8 GPUs detected by OS, 7 GPUs detected by Nvidia Drivers

I installed the 384.47 beta drivers as per the announcement here https://devtalk.nvidia.com/default/topic/1016125/unix-graphics-announcements-and-news/-linux-solaris-and-freebsd-driver-384-47-beta-/.

I am trying to run 8 P106-100 GPUs on a CentOS 7.3 system. Running lspci I can clearly see I have 8 PCI devices labeled’3D controller: NVIDIA Corporation Device 1c07 (rev a1)’. However when I run nvidia-smi or try to interact with the GPUs in anyway I can only detect and use 7.

[root@localhost ~]# lspci
00:00.0 Host bridge: Intel Corporation Broadwell-U Host Bridge -OPI (rev 09)
00:02.0 VGA compatible controller: Intel Corporation HD Graphics (rev 09)
00:03.0 Audio device: Intel Corporation Broadwell-U Audio Controller (rev 09)
00:14.0 USB controller: Intel Corporation Wildcat Point-LP USB xHCI Controller (rev 03)
00:16.0 Communication controller: Intel Corporation Wildcat Point-LP MEI Controller #1 (rev 03)
00:1c.0 PCI bridge: Intel Corporation Wildcat Point-LP PCI Express Root Port #1 (rev e3)
00:1c.1 PCI bridge: Intel Corporation Wildcat Point-LP PCI Express Root Port #2 (rev e3)
00:1c.2 PCI bridge: Intel Corporation Wildcat Point-LP PCI Express Root Port #3 (rev e3)
00:1c.3 PCI bridge: Intel Corporation Wildcat Point-LP PCI Express Root Port #4 (rev e3)
00:1c.4 PCI bridge: Intel Corporation Wildcat Point-LP PCI Express Root Port #5 (rev e3)
00:1c.5 PCI bridge: Intel Corporation Wildcat Point-LP PCI Express Root Port #6 (rev e3)
00:1d.0 USB controller: Intel Corporation Wildcat Point-LP USB EHCI Controller (rev 03)
00:1f.0 ISA bridge: Intel Corporation Wildcat Point-LP LPC Controller (rev 03)
00:1f.2 SATA controller: Intel Corporation Wildcat Point-LP SATA Controller [AHCI Mode] (rev 03)
00:1f.3 SMBus: Intel Corporation Wildcat Point-LP SMBus Controller (rev 03)
01:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
02:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
03:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
04:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
05:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
06:00.0 PCI bridge: Pericom Semiconductor Device 2608
07:01.0 PCI bridge: Pericom Semiconductor Device 2608
07:02.0 PCI bridge: Pericom Semiconductor Device 2608
07:03.0 PCI bridge: Pericom Semiconductor Device 2608
07:04.0 PCI bridge: Pericom Semiconductor Device 2608
08:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
09:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
0a:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
0b:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
[root@localhost ~]# nvidia-smi -L
GPU 0: P106-100 (UUID: GPU-480e8473-b5c2-1621-2126-8a06050c00c3)
GPU 1: P106-100 (UUID: GPU-b4361a96-84fe-0862-674a-ea0da03444ba)
GPU 2: P106-100 (UUID: GPU-53107551-b2fa-520f-96fd-98904a2268c3)
GPU 3: P106-100 (UUID: GPU-98050cda-ba7b-0cc4-c3bb-269be3600381)
GPU 4: P106-100 (UUID: GPU-6798b77c-f1aa-2794-d2dd-e25a2c8600e5)
GPU 5: P106-100 (UUID: GPU-eb558b78-dc6e-007b-6982-17a470d1785e)
GPU 6: P106-100 (UUID: GPU-8c5a5f16-9f2a-f7db-5399-2573bbe78f61)

I have tried reinstalling the drivers, reinstalling all the hardware, swapping the GPUs around to different PCI slots and reinstalling the OS and hardware to no avail. As I can run 8 AMD GPUs on the same system and lspci shows 8 3d Controllers I think it is mostly likely an issue with the drivers.

Also I am unable to set memory and core clock speeds with nvidia-smi -ac

nvidia-smi -ac 9468,1578
Setting applications clocks is not supported for GPU 00000000:01:00.0.
Treating as warning and moving on.
Setting applications clocks is not supported for GPU 00000000:02:00.0.
Treating as warning and moving on.
Setting applications clocks is not supported for GPU 00000000:03:00.0.
Treating as warning and moving on.
Setting applications clocks is not supported for GPU 00000000:05:00.0.
Treating as warning and moving on.
Setting applications clocks is not supported for GPU 00000000:08:00.0.
Treating as warning and moving on.
Setting applications clocks is not supported for GPU 00000000:09:00.0.
Treating as warning and moving on.
Setting applications clocks is not supported for GPU 00000000:0B:00.0.
Treating as warning and moving on.
All done.
nvidia-smi -q -d SUPPORTED_CLOCKS

==============NVSMI LOG==============

Timestamp                           : Mon Jul 17 15:45:48 2017
Driver Version                      : 384.47

Attached GPUs                       : 7
GPU 00000000:02:00.0
    Supported Clocks                : N/A

GPU 00000000:03:00.0
    Supported Clocks                : N/A

GPU 00000000:05:00.0
    Supported Clocks                : N/A

GPU 00000000:08:00.0
    Supported Clocks                : N/A

GPU 00000000:09:00.0
    Supported Clocks                : N/A

GPU 00000000:0A:00.0
    Supported Clocks                : N/A

GPU 00000000:0B:00.0
    Supported Clocks                : N/A
nvidia-smi -q -d CLOCK
GPU 00000000:0B:00.0
    Clocks
        Graphics                    : 1506 MHz
        SM                          : 1506 MHz
        Memory                      : 4006 MHz
        Video                       : 1354 MHz
    Applications Clocks
        Graphics                    : N/A
        Memory                      : N/A
    Default Applications Clocks
        Graphics                    : N/A
        Memory                      : N/A
    Max Clocks
        Graphics                    : 1911 MHz
        SM                          : 1911 MHz
        Memory                      : 4004 MHz
        Video                       : 1708 MHz
    Max Customer Boost Clocks
        Graphics                    : N/A
    SM Clock Samples
        Duration                    : 0.20 sec
        Number of Samples           : 14
        Max                         : 1506 MHz
        Min                         : 139 MHz
        Avg                         : 1461 MHz
    Memory Clock Samples
        Duration                    : 0.20 sec
        Number of Samples           : 14
        Max                         : 4006 MHz
        Min                         : 405 MHz
        Avg                         : 3878 MHz
    Clock Policy
        Auto Boost                  : N/A
        Auto Boost Default          : N/A

Is there a solution to this? Or is there a new driver out that I am unaware of?

I also have the same problem.
Has anyone solved it?

Some more people are complaining about this but none hat yet run nvidia-bug-report.sh and attached the output file.