Cannot run docker containers, incorrect nvidia-container-toolkit dependencies?

[tmirzq]

Hello,

I am using a Jetson Orin AGX machine. After completing the setup, I installed JetPack, Docker, and the NVIDIA container toolkit. Docker was installed using the instructions from the official Docker website:

Ubuntu

The nvidia-container was installed following the instructions here:
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

After the installation, I downloaded a Docker image from the NVIDIA website. Unfortunately, when trying to run a Docker container, I encounter the following error

sudo docker run -it --rm --net=host --runtime nvidia -e DISPLAY=$DISPLAY -v /tmp/.X11-unix/:/tmp/.X11-unix nvcr.io/nvidia/l4t-base:r35.1.0
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running createContainer hook #1: exit status 1, stdout: , stderr: time="2025-01-15T07:57:21+01:00" level=info msg="Symlinking /var/lib/docker/overlay2/5ce1d3567960332e5fae7bb8428fad75c3448afd4fac461892c45696d30f1e83/merged/etc/vulkan/icd.d/nvidia_icd.json to /usr/lib/aarch64-linux-gnu/tegra/nvidia_icd.json"
time="2025-01-15T07:57:21+01:00" level=error msg="failed to create link [/usr/lib/aarch64-linux-gnu/tegra/nvidia_icd.json /etc/vulkan/icd.d/nvidia_icd.json]: failed to create symlink: failed to remove existing file: remove /var/lib/docker/overlay2/5ce1d3567960332e5fae7bb8428fad75c3448afd4fac461892c45696d30f1e83/merged/etc/vulkan/icd.d/nvidia_icd.json: device or resource busy": unknown.

Docker info:

dpkg -l | grep docker.io
rc  docker.io                                  24.0.7-0ubuntu2~20.04.1               arm64        Linux container runtime

NVIDIA container toolkit details:

apt show nvidia-container-toolkit
Package: nvidia-container-toolkit
Version: 1.17.3-1
Priority: optional
Section: utils
Maintainer: NVIDIA CORPORATION <cudatools@nvidia.com>
Installed-Size: 4046 kB
Depends: nvidia-container-toolkit-base (= 1.17.3-1), libnvidia-container-tools (>= 1.17.3-1), libnvidia-container-tools (<< 2.0.0)
Breaks: nvidia-container-runtime (<= 3.5.0-1), nvidia-container-runtime-hook
Replaces: nvidia-container-runtime (<= 3.5.0-1), nvidia-container-runtime-hook
Homepage: https://github.com/NVIDIA/nvidia-container-toolkit
Download-Size: 1025 kB
APT-Manual-Installed: yes
APT-Sources: https://nvidia.github.io/libnvidia-container/stable/deb/arm64  Packages
Description: NVIDIA Container toolkit
 Provides tools and utilities to enable GPU support in containers.

NVIDIA container details (Why do I have nvidia-container-toolkit version 1.17.3-1 installed if the nvidia-container package depends on nvidia-container-toolkit (= 1.14.2-1)):

apt show nvidia-container
Package: nvidia-container
Version: 6.1+b123
Priority: standard
Section: metapackages
Source: nvidia-container (6.1)
Maintainer: NVIDIA Corporation
Installed-Size: 200 kB
Depends: nvidia-container-toolkit-base (= 1.14.2-1), libnvidia-container-tools (= 1.14.2-1), nvidia-container-toolkit (= 1.14.2-1), libnvidia-container1 (= 1.14.2-1)
Homepage: http://developer.nvidia.com/jetson
Download-Size: 29,6 kB
APT-Sources: https://repo.download.nvidia.com/jetson/common r36.4/main arm64 Packages
Description: NVIDIA Container Meta Package

Jetpack details:

apt show nvidia-jetpack
Package: nvidia-jetpack
Version: 6.1+b123
Priority: standard
Section: metapackages
Source: nvidia-jetpack (6.1)
Maintainer: NVIDIA Corporation
Installed-Size: 199 kB
Depends: nvidia-jetpack-runtime (= 6.1+b123), nvidia-jetpack-dev (= 6.1+b123)
Homepage: http://developer.nvidia.com/jetson
Download-Size: 29,3 kB
APT-Sources: https://repo.download.nvidia.com/jetson/common r36.4/main arm64 Packages
Description: NVIDIA Jetpack Meta Package

Can you assist in resolving this issue?
Thank you.

[AastaLLL]

Hi,

nvidia-container-toolkit should be installed by default when installing JetPack.
Please follow the command below to see if it can work:

Issues related with docker (or jetpack) when trying NanoOWL

Hi, Please set up the docker tool first. Here are the steps for your reference: Step 1,2 in Install Docker Engine on Ubuntu | Docker Docs # Add Docker's official GPG key: $ sudo apt-get update $ sudo apt-get install ca-certificates curl $ sudo install -m 0755 -d /etc/apt/keyrings $ sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc $ sudo chmod a+r /etc/apt/keyrings/docker.asc # Add the repository to Apt sources: $ echo \ "deb [arch=$(dpkg --print…

Thanks.

[tmirzq]

Thanks for the reply. In my opinion, Docker isn’t the issue since I installed it using the provided instructions. Could this issue be caused by broken packages or incorrect versions? The following output shows that I have JetPack version 6.1 installed, but inside jtop INFO, I see JetPack version 5.1.2.

sudo apt show nvidia-jetpack -a
Package: nvidia-jetpack
Version: 6.1+b123
Priority: standard
Section: metapackages
Source: nvidia-jetpack (6.1)
Maintainer: NVIDIA Corporation
Installed-Size: 199 kB
Depends: nvidia-jetpack-runtime (= 6.1+b123), nvidia-jetpack-dev (= 6.1+b123)
Homepage: http://developer.nvidia.com/jetson
Download-Size: 29,3 kB
APT-Sources: https://repo.download.nvidia.com/jetson/common r36.4/main arm64 Packages
Description: NVIDIA Jetpack Meta Package

[AastaLLL]

Hi

How do you set up your environment?
Do you flash from the SDK manager or upgrade from an older BSP?

Thanks.