NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
Mar 7 09:36:53 sny001 kernel: NVRM: The NVIDIA Tesla K40m GPU installed in this system is#012NVRM: supported through the NVIDIA 470.xx Legacy drivers. Please#012NVRM: visit Unix Drivers | NVIDIA for more#012NVRM: information. The 525.60.13 NVIDIA driver will ignore#012NVRM: this GPU. Continuing probe…
Mar 7 09:36:53 sny001 kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 510
Hello @sajanraj.t.d and welcome to the NVIDIA developer forums.
I am not sure I am completely clear on what you need help with. But I can try re-phrasing the log output for you.
The way I interpret it is that you have driver version 525 installed with a Kepler generation GPU. But since Kepler is deprecated in newer drivers, you cannot use that driver. That means you need to roll back to the latest driver version that still supports Kepler.
I also assume you are using Linux? Then go to Official Drivers | NVIDIA and put details about your operating system and GPU into the search fields. If I use K40m and “Linux 64 bit” wirh the latest CUDA toolkit I get v460.106.00.
I hope that helps.
Thanku you for the response. The solution is to replace the provisioning image with the specific drivier, in our case it is 470.xx. we fixed it.