Hello,
Machine is Dell C4130 with CentOS 7. It has two GPU cart Tesla K80, each cart has two GPU, so in total 4 GPU. Before the four GPU were detected correctly by nvida-smi command. Then a few days a ago, machine automatically rebooted, then after the boot, only three GPU are detected. and there is error
dmesg | grep NVRM
NVRM: GPU 0000:05:00.0: RmInitAdapter failed! (0x25:0xffff:1214)
NVRM: GPU 0000:05:00.0: rm_init_adapter failed, device minor number 1
nvidia-smi -L
GPU 0: NVIDIA Tesla K80 (UUID: GPU-0e409d2e-1f52-ae83-61c6-8d8747a2fdbb)
GPU 1: NVIDIA Tesla K80 (UUID: GPU-69a5d11b-93d8-1369-9a7d-825051b3ea2a)
GPU 2: NVIDIA Tesla K80 (UUID: GPU-7b09c8eb-7604-2420-cf90-52532fde1408)
The kernel package is updated after the reboot. Installed kernel and nvidia package are.
kernel-devel-3.10.0-1160.66.1.el7.x86_64
kernel-headers-3.10.0-1160.71.1.el7.x86_64
kmod-nvidia-latest-dkms-465.19.01-1.el7.x86_64
nvidia-driver-latest-dkms-465.19.01-1.el7.x86_64
nvidia-driver-latest-dkms-cuda-465.19.01-1.el7.x86_64
nvidia-driver-latest-dkms-cuda-libs-465.19.01-1.el7.x86_64
nvidia-driver-latest-dkms-devel-465.19.01-1.el7.x86_64
nvidia-driver-latest-dkms-libs-465.19.01-1.el7.x86_64
nvidia-driver-latest-dkms-NvFBCOpenGL-465.19.01-1.el7.x86_64
nvidia-driver-latest-dkms-NVML-465.19.01-1.el7.x86_64
nvidia-modprobe-latest-dkms-465.19.01-1.el7.x86_64
nvidia-persistenced-latest-dkms-465.19.01-1.el7.x86_64
nvidia-xconfig-latest-dkms-465.19.01-1.el7.x86_64
I suspected it is a hardware problem. But not totally sure?
Anything I can try to solve the problem?