Cannot run docker with --runtime nvidia

Autonomous Machines Jetson & Embedded Systems Jetson Xavier NX
,
1

I am trying to use docker in xavier. In the beginning I just install normal docker and use it, it works normally. But with other problem I have here: RTSP stream with openCV: Missing installation? - #8 by jobpasin

I noticed that my docker does not contain /etc/nvidia-container-runtime/host-files-for-container.d/l4t.csv file inside docker and apparently I need to use --runtime nvidia option but it gives error


docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: driver error: failed to process request: unknown.

I tried doing what other said in other post but none works for me and I think this problem worth creating a new post, What should I do here?

Some Info

$ dpkg -l | grep docker.io
ii  docker.io                                  20.10.7-0ubuntu5~18.04.3                   arm64        Linux container runtime

$ sudo apt-cache show nvidia-jetpack
Package: nvidia-jetpack
Version: 4.6-b199
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-cuda (= 4.6-b199), nvidia-opencv (= 4.6-b199), nvidia-cudnn8 (= 4.6-b199), nvidia-tensorrt (= 4.6-b199), nvidia-visionworks (= 4.6-b199), nvidia-container (= 4.6-b199), nvidia-vpi (= 4.6-b199), nvidia-l4t-jetson-multimedia-api (>> 32.6-0), nvidia-l4t-jetson-multimedia-api (<< 32.7-0)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_4.6-b199_arm64.deb
Size: 29376
SHA256: d67b85293cade45d81dcafebd46c70a97a0b0d1379ca48aaa79d70d8ba99ddf8
SHA1: 74d9cbdfe9af52baa667e321749b9771101fc6de
MD5sum: cd1b3a0b651cd214b15fa76f6b5af2cd
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

Package: nvidia-jetpack
Version: 4.6-b197
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-cuda (= 4.6-b197), nvidia-opencv (= 4.6-b197), nvidia-cudnn8 (= 4.6-b197), nvidia-tensorrt (= 4.6-b197), nvidia-visionworks (= 4.6-b197), nvidia-container (= 4.6-b197), nvidia-vpi (= 4.6-b197), nvidia-l4t-jetson-multimedia-api (>> 32.6-0), nvidia-l4t-jetson-multimedia-api (<< 32.7-0)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_4.6-b197_arm64.deb
Size: 29372
SHA256: acec83ad0c1ef05caf9b8ccc6a975c4fb2a7f7830cbe63bbcf7b196a6c1f350e
SHA1: 3e11456cf0ec6b3a40d81b80ca1e14cebafa65ff
MD5sum: 72b2b7b280793bd4abdabe0d38b08535
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

(Not sure why they show two jetpack version here]

Also, I moved my docker image location to SD card because the memory of xavier is already full, hopefully this is not the source of the problem.

2

Hi @jobpasin, can you try installing these packages with apt?

apt-cache search nvidia-container
libnvidia-container-tools - NVIDIA container runtime library (command-line tools)
libnvidia-container0 - NVIDIA container runtime library
nvidia-container-csv-cuda - Jetpack CUDA CSV file
nvidia-container-csv-cudnn - Jetpack CUDNN CSV file
nvidia-container-csv-tensorrt - Jetpack TensorRT CSV file
nvidia-container-csv-visionworks - Jetpack VisionWorks CSV file
nvidia-container-runtime - NVIDIA container runtime
nvidia-container-toolkit - NVIDIA container runtime hook
nvidia-docker2 - nvidia-docker CLI wrapper
nvidia-container - NVIDIA Container Meta Package
nvidia-container-csv-opencv - Jetpack OpenCV CSV file

If you are using the Xavier NX SD card image, or flashed with SDK Manager and had it install nvidia-docker, then Docker with NVIDIA Container Runtime should have already been installed on your Jetson.

3 
$ apt-cache search nvidia-container
libnvidia-container-dev - NVIDIA container runtime library (development files)
libnvidia-container-tools - NVIDIA container runtime library (command-line tools)
libnvidia-container1-dbg - NVIDIA container runtime library (debugging symbols)
libnvidia-container1 - NVIDIA container runtime library
nvidia-container-runtime - NVIDIA container runtime
nvidia-container-toolkit - NVIDIA container runtime hook
libnvidia-container0 - NVIDIA container runtime library
nvidia-container-csv-cuda - Jetpack CUDA CSV file
nvidia-container-csv-cudnn - Jetpack CUDNN CSV file
nvidia-container-csv-tensorrt - Jetpack TensorRT CSV file
nvidia-container-csv-visionworks - Jetpack VisionWorks CSV file
nvidia-container - NVIDIA Container Meta Package

Seems like it misses couple of these. (nvidia-docker2, nvidia-container-runtime, nvidia-container-toolkit, nvidia-container-csv-opencv]

4

Hmm, maybe it’s a difference between our respective versions of JetPack-L4T that we are running. I think if you do a sudo apt-get install nvidia-container that should install all of them, because that’s the meta-package (check the dependencies below)

apt-cache show nvidia-container
Package: nvidia-container
Version: 4.6-b199
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: libnvidia-container0 (= 0.10.0+jetpack), libnvidia-container-tools (= 0.10.0+jetpack), nvidia-container-csv-cuda (= 10.2.460-1), nvidia-container-csv-cudnn (= 8.2.1.32-1+cuda10.2), nvidia-container-csv-tensorrt (= 8.0.1.6-1+cuda10.2), nvidia-container-csv-visionworks (= 1.6.0.501), nvidia-container-runtime (= 3.1.0-1), nvidia-container-toolkit (= 1.0.1-1), nvidia-docker2 (= 2.2.0-1)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-container/nvidia-container_4.6-b199_arm64.deb
Size: 29428
SHA256: a8505abde1a5d65e997cd8e4c8f004d2dcc97dacdda964efe2807e5e0772e5e4
SHA1: 1ede3b0efedfebfc11981b10e40cc0d54f7f7dbb
MD5sum: dd20534b574181b6fc21d8e85315d556
Description: NVIDIA Container Meta Package
Description-md5: 12843014cb77898a6a19d4b272a070a5
6

Here is what I got

$ sudo apt-get install nvidia-container
[sudo] password for ndrs:
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 nvidia-container : Depends: libnvidia-container-tools (= 0.10.0+jetpack) but 1.6.0-1 is to be installed
                    Depends: nvidia-container-runtime (= 3.1.0-1) but 3.6.0-1 is to be installed
                    Depends: nvidia-container-toolkit (= 1.0.1-1) but 1.6.0-1 is to be installed
                    Depends: nvidia-docker2 (= 2.2.0-1) but 2.7.0-1 is to be installed
E: Unable to correct problems, you have held broken packages.

I’ll look into if I can somehow fix this. Or maybe is it better to install JetPack again to fix all the problem?

7

If you’re able to backup your work and reflash your SD card with JetPack, then I would probably recommend doing that as opposed to spending more time trying to fix the Docker install. Docker with NVIDIA Container Runtime should be working out-of-the-box after you flash the SD card. NVIDIA Docker comes pre-installed on the SD card image (or if you are flashing a production module with SDK Manager, SDK Manager will install NVIDIA Docker for you in the post-flashing install steps)

8

Hi, I have reinstalled Jetpack again and in the beginning, it looks very promising

:~$ apt-cache search nvidia-container

libnvidia-container-dev - NVIDIA container runtime library (development files)
libnvidia-container-tools - NVIDIA container runtime library (command-line tools)
libnvidia-container1-dbg - NVIDIA container runtime library (debugging symbols)
libnvidia-container1 - NVIDIA container runtime library
nvidia-container-runtime - NVIDIA container runtime
nvidia-container-toolkit - NVIDIA container runtime hook
libnvidia-container0 -NVIDIA container runtime library
nvidia-container-csv-cuda - Jetpack CUDA CSV file
nvidia-container-csv-cudnn - Jetpack CUDNN CSV file
nvidia-container-csv-tensorrt - Jetpack TensorRT CSV file
nvidia-container-csv-visionworks - Jetpack VisionWorks CSV file
nvidia-container - NVIDIA Container Meta Package

After I tried to

And I checked again, the package got changed

$ apt-cache search nvidia-container
libnvidia-container-tools - NVIDIA container runtime library (command-line tools)
libnvidia-container0 - NVIDIA container runtime library
nvidia-container-csv-cuda - Jetpack CUDA CSV file
nvidia-container-csv-cudnn - Jetpack CUDNN CSV file
nvidia-container-csv-tensorrt - Jetpack TensorRT CSV file
nvidia-container-csv-visionworks - Jetpack VisionWorks CSV file
nvidia-container-runtime - NVIDIA container runtime
nvidia-container-toolkit - NVIDIA container runtime hook
nvidia-docker2 - nvidia-docker CLI wrapper
nvidia-container - NVIDIA Container Meta Package

Seems like a couple of them is still missing, not sure if this is fatal or not though.

However when I tried to run

$ sudo docker run -it --rm --net=host --runtime nvidia -e DISPLAY=$DISPLAY -v /tmp/.X11-unix/:/tmp/.X11-unix nvcr.io/nvidia/l4t-base:r32.4.3

docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall clone3: permission denied: unknown.

It failed again with same error as before. Not sure what went wrong here (Does it required display to be connected?)

9

Unfortunately there was an issue with recent Ubuntu package updates to docker.io/containerd packages which renders them incompatible with the NVIDIA Container Runtime. It’s recommended to downgrade/pin these packages until the issue is resolved. Please see this post for more info:

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.