Cannot bring up docker container using runtime nvidia with nvidia package verion 32.7.1

Hi,
We followed the link Getting Started With Jetson Xavier NX Developer Kit | NVIDIA Developer to install the devkit in our jetson. However, docker container failed for runtime nvidia:

docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #1:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: mount error: file creation failed: /var/lib/docker/overlay2/58e0a82789241861c88742638db1d504f417ecdb20069cae81f77a2f7e7f9470/merged/usr/lib/aarch64-linux-gnu/tegra/libnvidia-fatbinaryloader.so.440.18: file exists: unknown.

A similar issue in the forum Docker run error with stderr: nvidia-container-cli - Jetson & Embedded Systems / Jetson Xavier NX - NVIDIA Developer Forums suggests to downgrade nvidia packge version. My nvidia package ( sudo dpkg-query -l ) is:

nvidia-l4t-3d-core 32.7.1-20220219090344

Please advise how to downgrade the nvidia package version.

Thanks!

Hi @fayw, here are the steps for fixing the docker versions: https://github.com/dusty-nv/jetson-containers/issues/108#issuecomment-995090398

However the error message you would have recieved for that problem different:

docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall

so I think that’s another issue.

What is the container you are running? Is it an r32.7.1 container? Can you try the following:

sudo docker run -it --rm --net=host --runtime nvidia nvcr.io/nvidia/l4t-base:r32.7.1

The command works fine:
nvaoi@nvaoi-jetson:~$ sudo docker run -it --rm --net=host --runtime nvidia nvcr.io/nvidia/l4t-base:r32.7.1
root@nvaoi-jetson:/#

However, the command below to bring up container using my docker image failed:
nvaoi@nvaoi-jetson:~$ sudo docker run -it --rm --net=host --runtime nvidia ads/tri:latest
docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: mount error: file creation failed: /var/lib/docker/overlay2/f5dcd945a10fe51402595835b9085ffb8f74c7d3ecb1a740752eecbd1eea8249/merged/usr/lib/aarch64-linux-gnu/tegra/libnvidia-fatbinaryloader.so.440.18: file exists: unknown.

This is my Dockerfile:
FROM nvcr.io/nvidia/l4t-pytorch:r32.4.3-pth1.6-py3

How to build the image compatible with r32.7.1?

From NVIDIA L4T PyTorch | NVIDIA NGC, the newest JetPack version is l4t-pytorch:r32.6.1-pth1.9-py3. Is there JetPack version for r32.7.1? If not, how to downgrade the nvidia package version to work with, say, r32.4.3?

The latest l4t-pytorch tags can be found here: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/l4t-pytorch/tags

You can use either of these for r32.7.1:

  • nvcr.io/nvidia/l4t-pytorch:r32.7.1-pth1.10-py3
  • nvcr.io/nvidia/l4t-pytorch:r32.7.1-pth1.9-py3