Getting docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: nvml error: driver/library version mismatch: unknown. this error while running docker for Cuopt .
Please guide me through this
This can happen when the kernel is using a different version of the nvidia driver than the client program. You can try this, there may be an error message there
dmesg | grep NVRM
Here are a couple of things to try:
- do you get the same error from “nvidia-smi” (assuming it is installed)
- have you tried simply rebooting? If the driver version has been updated, a reboot is necessary.
Let’s start there.
I didn’t what exactly are you trying to say. Can you please elaborate?
These steps may help diagnose the issue. The error can happen when there is a mismatch between a client program or packages on the system and the version of the nvidia driver that is being used by the kernel.
The following command on a Linux system might give us extra information from the system logs:
$ dmesg | grep NVRM
If you have the nvidia-smi executable installed on your system, that might give us a clue too (if nvidia-smi returns a result, but docker has errors, then we know it’s something specific to the docker setup). Run it like this:
Lastly, if there has been an nvidia driver update, but the system has not been rebooted since the update, rebooting the machine may clear the issue.
After following the steps you mentioned I got the following error
please guide me through the next steps.
okay that gives some clarity. The driver version is older than the installed cuda version, most likely from mixed install methods.
Your best course of action is follow this page from NVIDIA
specifically this section:
Incidentally, if you can create a fresh Ubuntu 22.04 machine, you can use this script to install everything you need for cuOpt. It is super-simple and it works well. Are you able to create a new Ubuntu 22.04 machine?