I’m trying set up a VMware virtual environment for my developers to work in. I have ten ESXi hosts clustered together with A10 graphics cards. I’ve been following the steps in Virtual GPU Software User Guide :: NVIDIA Virtual GPU Software Documentation and am currently on part 4.2.
After I install the NVIDIA graphics driver on Linux and try to run the nvidia-smi command to confirm it’s working I get the following message: NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
When I check the status of the nvidia-grid service I get this:
nvidia-gridd.service - NVIDIA Grid Daemon
Loaded: loaded (/usr/lib/systemd/syste/nvidia-gridd.service; enabled; vendor present: disabled)
Active: failed (Result: exit-code) since Mon 2021-11-29 08:08:46 EST; 28min ago
Process: 1342 ExecStopPost=/bin/rm -rf /var/run/nvidia-gridd (code=exited, status=0/SUCCESS)
Process: 1107 ExecStart=/usr/bin/nvidia-gridd (code=exited, status=0/SUCCESS)
Main PID: 1125 (code=exited, status=1/FAILURE)
Nov 29 08:08:45 rhel8vdi systemd[1]: Starting NVIDIA Grid Daemon…
Nov 29 08:08:45 rhel8vdi nvidia-gridd[1125]: Started (1125)
Nov 29 08:08:45 rhel8vdi systemd[1]: Started NVIDIA Grid Daemon.
Nov 29 08:08:46 rhel8vdi nvidia-gridd[1125]: Failed to initialize RM client
Nov 29 08:08:46 rhel8vdi nvidia-gridd[1125]: Failed to initialize RM client
Nov 29 08:08:46 rhel8vdi nvidia-gridd[1125]: Failed to unlock PID file: Bad file descriptor
Nov 29 08:08:46 rhel8vdi nvidia-gridd[1125]: Failed to close PID file: Bad file descriptor
Nov 29 08:08:46 rhel8vdi nvidia-gridd[1125]: Shutdown (1125)
Nov 29 08:08:46 rhel8vdi systemd[1]: nvidia-gridd.service: Main process exited, code=exited, status=1/FAILURE
Nov 29 08:08:46 rhel8vdi systemd[1]: nvidia-gridd.service: Failed with result ‘exit-code’.
I have tried uninstalling and reinstalling the driver, purging the install, installing an earlier version of the driver, using different VM PCI profiles, and installing on a clean install of RHEL 8. I know the VIB is installed correctly on the ESXi hosts because I’ve successfully created a Windows 10 Desktop Pool and Licensing Server with GRID vGPUs in the same environment.
nvidia-bug-report.sh returns an empty file except for the opening statement, nvidia-settings returns ERROR: Unable to find display on any available system.
I am using:
Hypervisor: VMware vSphere 7.0.2
OS: RedHat Enterprise Linux 8.4
Driver: NVIDIA-Linux-x86_64-470-82-01-grid.run
Graphics Card: A10 Tensor Core
GPU Profile: NVIDIA GRID vGPU nvidia_a10-24q