Nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory

I have seen a lot of chatter about this error, but no solution.

I am running Ubuntu 22.04.4 LTS
dpkg -l | grep -i docker
ii docker-buildx-plugin 0.13.1-1~ubuntu.22.04~jammy amd64 Docker Buildx cli plugin.
ii docker-ce 5:26.0.0-1~ubuntu.22.04~jammy amd64 Docker: the open-source application container engine
ii docker-ce-cli 5:26.0.0-1~ubuntu.22.04~jammy amd64 Docker CLI: the open-source application container engine
ii docker-ce-rootless-extras 5:26.0.0-1~ubuntu.22.04~jammy amd64 Rootless support for Docker.
ii docker-compose-plugin 2.25.0-1~ubuntu.22.04~jammy amd64 Docker Compose (V2) plugin for the Docker CLI.
rc docker.io 24.0.5-0ubuntu1~22.04.1 amd64 Linux container runtime
ii nvidia-docker2 2.13.0-1 all nvidia-docker CLI wrapper
ii python3-docker 5.0.3-1 all Python 3 wrapper to access docker.io’s control socket

nvidia-container-cli info
NVRM version: 550.54.15
CUDA version: 12.4

Device Index: 0
Device Minor: 0
Model: NVIDIA GeForce RTX 4090
Brand: GeForce
GPU UUID: GPU-85dddb97-58f5-8c23-f4c9-eda2104c3b5a
Bus Location: 00000000:01:00.0
Architecture: 8.9

Device Index: 1
Device Minor: 1
Model: NVIDIA GeForce RTX 4090
Brand: GeForce
GPU UUID: GPU-8f940736-ea6e-d06d-ae6c-3b7dcb4cc0ab
Bus Location: 00000000:03:00.0
Architecture: 8.9

/code/container/oobabooga$ docker start 01148fb55628
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as ‘legacy’
nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown
Error: failed to start containers: 01148fb55628

/code/container/oobabooga$ sudo docker start 01148fb55628
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as ‘legacy’
nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown
Error: failed to start containers: 01148fb55628

/code/container/oobabooga$ docker version
Client: Docker Engine - Community
Version: 26.0.0
API version: 1.43 (downgraded from 1.45)
Go version: go1.21.8
Git commit: 2ae903e
Built: Wed Mar 20 15:17:48 2024
OS/Arch: linux/amd64
Context: default

Server:
Engine:
Version: 24.0.5
API version: 1.43 (minimum version 1.12)
Go version: go1.20.8
Git commit: a61e2b4
Built: Sat Oct 7 00:14:30 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.6.21
GitCommit: 3dce8eb055cbb6872793272b4f20ed16117344f8
runc:
Version: 1.1.7
GitCommit:
docker-init:
Version: 0.19.0
GitCommit: de40ad0

nvidia-container-cli -V
cli-version: 1.14.6
lib-version: 1.14.6
build date: 2024-02-27T20:51+00:00
build revision: d2eb0afe86f0b643e33624ee64f065dd60e952d4
build compiler: x86_64-linux-gnu-gcc-7 7.5.0
build platform: x86_64
build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fplan9-extensions -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,–gc-sections

I also have Kubernetes on this server. The missing library is in /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1

Any help is appreciated!!

Thanks.

I have relocated your question to a more relevant forum.