Error response from daemon: failed to create task for container

2025-04-09 16:49:33 UTC: docker start C.19335572 2>&1
2025-04-09 16:49:34 UTC: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdo2025-04-09 16:49:34 UTC: ut: , stderr: Auto-detected mode as ‘legacy’
2025-04-09 16:49:34 UTC: nvidia-container-cli: device error: 1: unknown device: unknown
2025-04-09 16:49:34 UTC: Error: failed to start containers: C.19335572
2025-04-09 16:49:34 UTC: Error: failed to start containers: C.19335572

Client: Docker Engine - Community
Version: 27.5.1
Context: default
Debug Mode: false
Plugins:
compose: Docker Compose (Docker Inc.)
Version: v2.34.0
Path: /usr/libexec/docker/cli-plugins/docker-compose

Server:
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 3
Server Version: 27.5.1
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 nvidia runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
runc version: v1.2.5-0-g59923ef
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
Kernel Version: 6.1.0
Operating System: Ubuntu 20.04.6 LTS
OSType: linux
Architecture: x86_64
CPUs: 96
Total Memory: 251.5GiB
Name: S409076422
ID: dd6d4f1d-6596-4309-bec5
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
• DeepStream Version
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only) 550.144.03
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Are you creating the container on dGPU or Jetson? What is your command line?

This seems to be a problem caused by nvidia-container-toolkit/gpu driver not being installed correctly. Please check this problem first.