Error run docker image nvcr.io/nvidia/l4t-base:r32.6.1 on jetson AGX

datlt · October 12, 2021, 8:35am

I have a Jetson AGX, and I want to run docker image nvcr.io/nvidia/l4t-base:r32.6.1. I referenced many tutorials they told me that I need to install nvidia-container-runtime.

All commands I used to install nvidia-container-runtime listed below:

curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update
sudo apt-get install nvidia-docker2 -y

In addition, I modified the /etc/docker/daemon.json with content:

{
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
         } 
    },
    "default-runtime": "nvidia" 
}

After all, I restart docker daemon:

sudo systemctl restart docker

All above steps I did on other Jetson NX and docker work fine, but when i run docker image l4t-base:r32.6.1 on my Jetson AGX with interactive mode

sudo docker run -it --rm --net=host --runtime nvidia -e DISPLAY=$DISPLAY -v /tmp/.X11-unix/:/tmp/.X11-unix nvcr.io/nvidia/l4t-base:r32.6.1

I received error:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: driver error: failed to process request: unknown.

This image depict my jetson’s infomation
.
And other docker’s infomation:

vizgard@vizgard:~$ sudo dpkg --get-selections | grep nvidia
libnvidia-container-tools                       install
libnvidia-container1:arm64                      install
nvidia-container-runtime                        install
nvidia-container-toolkit                        install
nvidia-docker2                                  install
nvidia-l4t-3d-core                              install
nvidia-l4t-apt-source                           install
nvidia-l4t-bootloader                           install
nvidia-l4t-camera                               install
nvidia-l4t-configs                              install
nvidia-l4t-core                                 install
nvidia-l4t-cuda                                 install
nvidia-l4t-firmware                             install
nvidia-l4t-graphics-demos                       install
nvidia-l4t-gstreamer                            install
nvidia-l4t-init                                 install
nvidia-l4t-initrd                               install
nvidia-l4t-jetson-io                            install
nvidia-l4t-kernel                               install
nvidia-l4t-kernel-dtbs                          install
nvidia-l4t-kernel-headers                       install
nvidia-l4t-multimedia                           install
nvidia-l4t-multimedia-utils                     install
nvidia-l4t-oem-config                           install
nvidia-l4t-tools                                install
nvidia-l4t-wayland                              install
nvidia-l4t-weston                               install
nvidia-l4t-x11                                  install
nvidia-l4t-xusb-firmware                        install

vizgard@vizgard:~$ sudo docker info | grep nvidia
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux nvidia runc
 Default Runtime: nvidia

AastaLLL · October 12, 2021, 8:56am

Hi,

The container requires r32.6.1 OS but you are using r32.5.0 which is incompatible.
Please upgrade your Xavier with JetPack4.6 or use nvcr.io/nvidia/l4t-base:r32.5.0 instead.

Thanks.

datlt · October 12, 2021, 9:10am

I downloaded nvcr.io/nvidia/l4t-base:r32.5.0 and run, but the same error with r32.6.1

vizgard@vizgard:~$ docker run -it --rm --net=host --runtime nvidia -e DISPLAY=$DISPLAY -v /tmp/.X11-unix/:/tmp/.X11-unix nvcr.io/nvidia/l4t-base:r32.5.0 
Unable to find image 'nvcr.io/nvidia/l4t-base:r32.5.0' locally
r32.5.0: Pulling from nvidia/l4t-base
5000a6c32c5a: Pull complete 
8e855b69096a: Pull complete 
8db8dbbd4bb9: Pull complete 
833dc3235950: Pull complete 
f79d264135a3: Pull complete 
1c40f77bb35b: Pull complete 
1990ecf0bfb7: Pull complete 
c8ffbfd7f0aa: Pull complete 
ba785779122a: Pull complete 
024ce79b6790: Pull complete 
9b09da3b5483: Pull complete 
17f974a43cf9: Pull complete 
211b56b73ff1: Pull complete 
78aca4be1f3b: Pull complete 
95f34310bbda: Pull complete 
678c9d1557e9: Pull complete 
ec17ad7cab01: Pull complete 
08fb1eee5328: Pull complete 
Digest: sha256:c7207a13da6054c738b530aad93e8c620182db85eb7fbb40b103757145e8b5f4
Status: Downloaded newer image for nvcr.io/nvidia/l4t-base:r32.5.0
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: driver error: failed to process request: unknown.

alex15 · October 13, 2021, 5:00am

Hi @AastaLLL thank you for your assistance so far. Can you urgently let us know your thoughts on why it still doesn’t work after changing to a compatible version?

datlt · October 15, 2021, 7:50am

@AastaLLL , I reinstall AGX with r32.6.1 OS, and no more error driver notification. But I ran sudo apt-get install nvidia-docker2 -y again, AGX error driver again. Do you have a way restore the original state of jetpack, because I cannot reinstall OS again?

alex15 · October 16, 2021, 11:45am

@alanz do you have any thoughts on datlt’s last comment?

alanz · October 16, 2021, 12:23pm

You don’t need to install the docker runtime, It’s already there when you flashed the Xavier with JP4.6.1. Here is the default components lists.

xavier@xavier-desktop:~$ sudo dpkg --get-selections | grep nvidia
libnvidia-container-tools			install
libnvidia-container0:arm64			install
nvidia-container-csv-cuda			install
nvidia-container-csv-cudnn			install
nvidia-container-csv-tensorrt			install
nvidia-container-runtime			install
nvidia-container-toolkit			install
nvidia-docker2					install
nvidia-l4t-3d-core				install
nvidia-l4t-apt-source				install
nvidia-l4t-bootloader				install
nvidia-l4t-camera				install
nvidia-l4t-configs				install
nvidia-l4t-core					install
nvidia-l4t-cuda					install
nvidia-l4t-firmware				install
nvidia-l4t-gputools				install
nvidia-l4t-graphics-demos			install
nvidia-l4t-gstreamer				install
nvidia-l4t-init					install
nvidia-l4t-initrd				install
nvidia-l4t-jetson-io				install
nvidia-l4t-jetson-multimedia-api		install
nvidia-l4t-kernel				install
nvidia-l4t-kernel-dtbs				install
nvidia-l4t-kernel-headers			install
nvidia-l4t-libvulkan				install
nvidia-l4t-multimedia				install
nvidia-l4t-multimedia-utils			install
nvidia-l4t-oem-config				install
nvidia-l4t-tools				install
nvidia-l4t-wayland				install
nvidia-l4t-weston				install
nvidia-l4t-x11					install
nvidia-l4t-xusb-firmware			install

I don’t see any issue.

xavier@xavier-desktop:~$ sudo docker pull nvcr.io/nvidia/l4t-base:r32.6.1
r32.6.1: Pulling from nvidia/l4t-base
e47a8a86d66c: Pull complete 
bdce3430dad6: Pull complete 
c26a6b81c746: Pull complete 
a70635e646de: Pull complete 
24cbe60e3161: Pull complete 
c7f64cc97a39: Pull complete 
8777adb92eda: Pull complete 
542a24b3572f: Pull complete 
3d5ade2b8849: Pull complete 
3e865584f789: Pull complete 
4811af6cacf1: Pull complete 
db645757aac7: Pull complete 
0105dc23dc93: Pull complete 
c8aff3baf097: Pull complete 
5f50b3e38480: Pull complete 
b5418d85b074: Pull complete 
Digest: sha256:66e8d2e16d4e0745d9d24e40271511dfb287bd4718b4d8a68c177a8981decb16
Status: Downloaded newer image for nvcr.io/nvidia/l4t-base:r32.6.1
nvcr.io/nvidia/l4t-base:r32.6.1
xavier@xavier-desktop:~$ sudo dpkg --get-selections | grep nvidia^C
xavier@xavier-desktop:~$ sudo docker run -it --rm --net=host --runtime nvidia -e DISPLAY=$DISPLAY -v /tmp/.X11-unix/:/tmp/.X11-unix nvcr.io/nvidia/l4t-base:r32.6.1
root@xavier-desktop:/# 
root@xavier-desktop:/# 
root@xavier-desktop:/# 
root@xavier-desktop:/# ls
bin  boot  dev  dst  etc  home  lib  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
root@xavier-desktop:/#

alex15 · November 1, 2021, 3:10pm

Hi @alanz , we got this fully up and running in the end so thank you for your help. There was one issue, however. In order to use docker, we had to install the entire SDK Toolkit so that the container components were installed. This used up valuable space which was probably unnecessary as our docker image contained all the libraries we would need. When datlt tried to install just nvidia container components it didn’t appear to install.

Next time, I would like to just flash the jetson, install whatever is required to allow docker images, and then install the docker image. How do we install what is required after the basic flashing has completed?

alanz · November 2, 2021, 12:45am

Pls file another topic to tracking this.

Thanks.

Topic		Replies	Views
Unable to run nvidia docker Jetson Xavier NX docker	4	3575	December 8, 2021
Creating Containers Using nvidia-docker with AGX Xavier Jetson AGX Xavier docker	8	2744	October 18, 2021
Jetson AGX Xavier cannot start a basic docker Jetson AGX Xavier docker	8	1290	June 23, 2021
Cannot run docker containers, incorrect nvidia-container-toolkit dependencies? Jetson AGX Orin docker , containers , jetson	3	54	January 16, 2025
Unable to run a nvidia docker on AGX Xavier Jetson AGX Xavier docker	6	3553	October 18, 2021
Cannot bring up docker container using runtime nvidia with nvidia package verion 32.7.1 Jetson Xavier NX	4	1765	May 13, 2022
Cannot run docker with --runtime nvidia Jetson Xavier NX docker , containers	8	7415	December 22, 2021
Docker container runtime issue Jetson Nano docker	10	4398	October 15, 2021
Docker Container: nvidia:l4t-ml wont run Jetson Xavier NX docker	13	4399	October 18, 2021
Getting GPU docker passthrough working Jetson AGX Xavier docker	8	8957	July 12, 2022

Error run docker image nvcr.io/nvidia/l4t-base:r32.6.1 on jetson AGX

Related topics