I installed the 384.47 beta drivers as per the announcement here https://devtalk.nvidia.com/default/topic/1016125/unix-graphics-announcements-and-news/-linux-solaris-and-freebsd-driver-384-47-beta-/.
I am trying to run 8 P106-100 GPUs on a CentOS 7.3 system. Running lspci I can clearly see I have 8 PCI devices labeled’3D controller: NVIDIA Corporation Device 1c07 (rev a1)'. However when I run nvidia-smi or try to interact with the GPUs in anyway I can only detect and use 7.
[root@localhost ~]# lspci
00:00.0 Host bridge: Intel Corporation Broadwell-U Host Bridge -OPI (rev 09)
00:02.0 VGA compatible controller: Intel Corporation HD Graphics (rev 09)
00:03.0 Audio device: Intel Corporation Broadwell-U Audio Controller (rev 09)
00:14.0 USB controller: Intel Corporation Wildcat Point-LP USB xHCI Controller (rev 03)
00:16.0 Communication controller: Intel Corporation Wildcat Point-LP MEI Controller #1 (rev 03)
00:1c.0 PCI bridge: Intel Corporation Wildcat Point-LP PCI Express Root Port #1 (rev e3)
00:1c.1 PCI bridge: Intel Corporation Wildcat Point-LP PCI Express Root Port #2 (rev e3)
00:1c.2 PCI bridge: Intel Corporation Wildcat Point-LP PCI Express Root Port #3 (rev e3)
00:1c.3 PCI bridge: Intel Corporation Wildcat Point-LP PCI Express Root Port #4 (rev e3)
00:1c.4 PCI bridge: Intel Corporation Wildcat Point-LP PCI Express Root Port #5 (rev e3)
00:1c.5 PCI bridge: Intel Corporation Wildcat Point-LP PCI Express Root Port #6 (rev e3)
00:1d.0 USB controller: Intel Corporation Wildcat Point-LP USB EHCI Controller (rev 03)
00:1f.0 ISA bridge: Intel Corporation Wildcat Point-LP LPC Controller (rev 03)
00:1f.2 SATA controller: Intel Corporation Wildcat Point-LP SATA Controller [AHCI Mode] (rev 03)
00:1f.3 SMBus: Intel Corporation Wildcat Point-LP SMBus Controller (rev 03)
01:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
02:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
03:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
04:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
05:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
06:00.0 PCI bridge: Pericom Semiconductor Device 2608
07:01.0 PCI bridge: Pericom Semiconductor Device 2608
07:02.0 PCI bridge: Pericom Semiconductor Device 2608
07:03.0 PCI bridge: Pericom Semiconductor Device 2608
07:04.0 PCI bridge: Pericom Semiconductor Device 2608
08:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
09:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
0a:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
0b:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
[root@localhost ~]# nvidia-smi -L
GPU 0: P106-100 (UUID: GPU-480e8473-b5c2-1621-2126-8a06050c00c3)
GPU 1: P106-100 (UUID: GPU-b4361a96-84fe-0862-674a-ea0da03444ba)
GPU 2: P106-100 (UUID: GPU-53107551-b2fa-520f-96fd-98904a2268c3)
GPU 3: P106-100 (UUID: GPU-98050cda-ba7b-0cc4-c3bb-269be3600381)
GPU 4: P106-100 (UUID: GPU-6798b77c-f1aa-2794-d2dd-e25a2c8600e5)
GPU 5: P106-100 (UUID: GPU-eb558b78-dc6e-007b-6982-17a470d1785e)
GPU 6: P106-100 (UUID: GPU-8c5a5f16-9f2a-f7db-5399-2573bbe78f61)
I have tried reinstalling the drivers, reinstalling all the hardware, swapping the GPUs around to different PCI slots and reinstalling the OS and hardware to no avail. As I can run 8 AMD GPUs on the same system and lspci shows 8 3d Controllers I think it is mostly likely an issue with the drivers.
Also I am unable to set memory and core clock speeds with nvidia-smi -ac
nvidia-smi -ac 9468,1578
Setting applications clocks is not supported for GPU 00000000:01:00.0.
Treating as warning and moving on.
Setting applications clocks is not supported for GPU 00000000:02:00.0.
Treating as warning and moving on.
Setting applications clocks is not supported for GPU 00000000:03:00.0.
Treating as warning and moving on.
Setting applications clocks is not supported for GPU 00000000:05:00.0.
Treating as warning and moving on.
Setting applications clocks is not supported for GPU 00000000:08:00.0.
Treating as warning and moving on.
Setting applications clocks is not supported for GPU 00000000:09:00.0.
Treating as warning and moving on.
Setting applications clocks is not supported for GPU 00000000:0B:00.0.
Treating as warning and moving on.
All done.
nvidia-smi -q -d SUPPORTED_CLOCKS
==============NVSMI LOG==============
Timestamp : Mon Jul 17 15:45:48 2017
Driver Version : 384.47
Attached GPUs : 7
GPU 00000000:02:00.0
Supported Clocks : N/A
GPU 00000000:03:00.0
Supported Clocks : N/A
GPU 00000000:05:00.0
Supported Clocks : N/A
GPU 00000000:08:00.0
Supported Clocks : N/A
GPU 00000000:09:00.0
Supported Clocks : N/A
GPU 00000000:0A:00.0
Supported Clocks : N/A
GPU 00000000:0B:00.0
Supported Clocks : N/A
nvidia-smi -q -d CLOCK
GPU 00000000:0B:00.0
Clocks
Graphics : 1506 MHz
SM : 1506 MHz
Memory : 4006 MHz
Video : 1354 MHz
Applications Clocks
Graphics : N/A
Memory : N/A
Default Applications Clocks
Graphics : N/A
Memory : N/A
Max Clocks
Graphics : 1911 MHz
SM : 1911 MHz
Memory : 4004 MHz
Video : 1708 MHz
Max Customer Boost Clocks
Graphics : N/A
SM Clock Samples
Duration : 0.20 sec
Number of Samples : 14
Max : 1506 MHz
Min : 139 MHz
Avg : 1461 MHz
Memory Clock Samples
Duration : 0.20 sec
Number of Samples : 14
Max : 4006 MHz
Min : 405 MHz
Avg : 3878 MHz
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Is there a solution to this? Or is there a new driver out that I am unaware of?