CentOS 6.9 Machine is missing one of it's 8 GTX1080s when running nvidia-smi

Hello,

I am running a CentOS 6.9 GPU server that has 8xGTX1080s. However, after a recent restart, when I run nvidia-smi, only seven of the eight GPUs show up. Upon running nvidia-smi -q, it shows Attached GPUS: 7

I am on nvidia driver version: 375.26. Using Linux kernel 2.6.32-642.15.1.el6.centos.plus.x86_64.

The nvidia drivers were obtained through elrepo by installing the kmod-nvidia.x86_64 package.

Here, lspci shows nvidia PCI devices attached:

$ lspci | grep NVIDIA | grep VGA
04:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1080] (rev a1)
05:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1080] (rev a1)
08:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1080] (rev a1)
09:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1080] (rev a1)
85:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1080] (rev a1)
86:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1080] (rev a1)
89:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1080] (rev a1)
8a:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1080] (rev a1)

nvidia-bug-report.log (1.21 MB)