NVIDIA-SMI failure on RHEL 8

Hello,

I’m trying to install CUDA 10.2 on Azure VM with 2xV100 GPU’s. I installed the driver

sudo dkms status
nvidia, 450.51.06, 4.18.0-193.14.3.el8_2.x86_64, x86_64: installed

But CUDA doesn’t work. When trying to run the code I get:

cupy.cuda.runtime.CUDARuntimeError: cudaErrorNoDevice: no CUDA-capable device is detected

After some investigation I found, that nvidia-smi throws this error:

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Can anyone help to figure out what is going on and how it can be fixed?

nvidia-bug-report.log.gz (1.9 MB)