Error running privileged L4T containers with the NVidia Runtime on JetPack 5.0 Developer Preview

[rollc_at]

Hello,

I’ve just grabbed the JetPack 5.0 Developer Preview for Xavier NX (T194), and tried running a couple Docker containers on it.

I’ve managed to run the following Docker containers without any trouble. Alpine:

$ docker run -ti --rm alpine true
$ docker run -ti --rm --runtime nvidia alpine true
$ docker run -ti --rm --privileged alpine true
$ docker run -ti --rm --privileged --runtime nvidia alpine true

The super freshly just-dropped r34.1 release of L4T:

$ docker run -ti --rm nvcr.io/nvidia/l4t-base:r34.1 true        
$ docker run -ti --rm --privileged nvcr.io/nvidia/l4t-base:r34.1 true
$ docker run -ti --rm --runtime nvidia nvcr.io/nvidia/l4t-base:r34.1 true

All of the above 7 variants work fine. However, combining the privileged flag with the NVidia Docker runtime, on this specific image, causes a lengthy wall of errors:

$ docker run -ti --rm --privileged --runtime nvidia nvcr.io/nvidia/l4t-base:r34.1 true

docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #1:: error running hook: exit status 1, stdout: src: /etc/vulkan/icd.d/nvidia_icd.json, src_lnk: /usr/lib/aarch64-linux-gnu/tegra/nvidia_icd.json, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/etc/vulkan/icd.d/nvidia_icd.json, dst_lnk: /usr/lib/aarch64-linux-gnu/tegra/nvidia_icd.json
src: /usr/lib/aarch64-linux-gnu/libcuda.so, src_lnk: tegra/libcuda.so, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/libcuda.so, dst_lnk: tegra/libcuda.so
src: /usr/lib/aarch64-linux-gnu/libnvcucompat.so, src_lnk: tegra/libnvcucompat.so, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/libnvcucompat.so, dst_lnk: tegra/libnvcucompat.so
src: /usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so, src_lnk: tegra/libnvidia-nvvm.so, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so, dst_lnk: tegra/libnvidia-nvvm.so
src: /usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.999999, src_lnk: tegra/libnvv4l2.so, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.999999, dst_lnk: tegra/libnvv4l2.so
src: /usr/lib/aarch64-linux-gnu/libv4lconvert.so.0.0.999999, src_lnk: tegra/libnvv4lconvert.so, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/libv4lconvert.so.0.0.999999, dst_lnk: tegra/libnvv4lconvert.so
src: /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvargus.so, src_lnk: ../../../tegra/libv4l2_nvargus.so, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvargus.so, dst_lnk: ../../../tegra/libv4l2_nvargus.so
src: /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvcuvidvideocodec.so, src_lnk: ../../../tegra/libv4l2_nvcuvidvideocodec.so, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvcuvidvideocodec.so, dst_lnk: ../../../tegra/libv4l2_nvcuvidvideocodec.so
src: /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvideocodec.so, src_lnk: ../../../tegra/libv4l2_nvvideocodec.so, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvideocodec.so, dst_lnk: ../../../tegra/libv4l2_nvvideocodec.so
src: /usr/lib/aarch64-linux-gnu/libvulkan.so.1.3.203, src_lnk: tegra/libvulkan.so.1.3.203, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/libvulkan.so.1.3.203, dst_lnk: tegra/libvulkan.so.1.3.203
src: /usr/lib/aarch64-linux-gnu/tegra/libcuda.so, src_lnk: libcuda.so.1.1, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/tegra/libcuda.so, dst_lnk: libcuda.so.1.1
src: /usr/lib/aarch64-linux-gnu/tegra/libgstnvdsseimeta.so, src_lnk: libgstnvdsseimeta.so.1.0.0, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/tegra/libgstnvdsseimeta.so, dst_lnk: libgstnvdsseimeta.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurface.so, src_lnk: libnvbufsurface.so.1.0.0, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/tegra/libnvbufsurface.so, dst_lnk: libnvbufsurface.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurftransform.so, src_lnk: libnvbufsurftransform.so.1.0.0, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/tegra/libnvbufsurftransform.so, dst_lnk: libnvbufsurftransform.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so, src_lnk: libnvbuf_utils.so.1.0.0, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so, dst_lnk: libnvbuf_utils.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvdsbufferpool.so, src_lnk: libnvdsbufferpool.so.1.0.0, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/tegra/libnvdsbufferpool.so, dst_lnk: libnvdsbufferpool.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvidia-nvvm.so, src_lnk: libnvidia-nvvm.so.4, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/tegra/libnvidia-nvvm.so, dst_lnk: libnvidia-nvvm.so.4
src: /usr/lib/aarch64-linux-gnu/tegra/libnvidia-nvvm.so.4, src_lnk: libnvidia-nvvm.so.4.0.0, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/tegra/libnvidia-nvvm.so.4, dst_lnk: libnvidia-nvvm.so.4.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvid_mapper.so, src_lnk: libnvid_mapper.so.1.0.0, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/tegra/libnvid_mapper.so, dst_lnk: libnvid_mapper.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvscibuf.so, src_lnk: libnvscibuf.so.1, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/tegra/libnvscibuf.so, dst_lnk: libnvscibuf.so.1
src: /usr/lib/aarch64-linux-gnu/tegra/libnvscicommon.so, src_lnk: libnvscicommon.so.1, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/tegra/libnvscicommon.so, dst_lnk: libnvscicommon.so.1
src: /usr/lib/aarch64-linux-gnu/tegra/libnvscistream.so, src_lnk: libnvscistream.so.1, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/tegra/libnvscistream.so, dst_lnk: libnvscistream.so.1
src: /usr/lib/aarch64-linux-gnu/tegra/libnvscisync.so, src_lnk: libnvscisync.so.1, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/lib/aarch64-linux-gnu/tegra/libnvscisync.so, dst_lnk: libnvscisync.so.1
src: /usr/share/glvnd/egl_vendor.d/10_nvidia.json, src_lnk: ../../../lib/aarch64-linux-gnu/tegra-egl/nvidia.json, dst: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/usr/share/glvnd/egl_vendor.d/10_nvidia.json, dst_lnk: ../../../lib/aarch64-linux-gnu/tegra-egl/nvidia.json
, stderr: nvidia-container-cli: mount error: file creation failed: /var/lib/docker/overlay2/7fa5fde86f94e78b85f15f95e051bded2eb32e2e192a6dc776a8b275ac16e245/merged/dev/nvhost-as-gpu: invalid argument: unknown.

I did remove the following packages from the host OS to get a slimmed down, headless environment:

    sudo apt-get remove --autoremove -qy \
        lxde 'lxde-*' lxsession unity 'unity-*' 'gnome-*' xorg 'xserver-*' 'fonts-*' \
        lightdm gdm3 'update-*' 'vlc-*' 'rhythmbox-*' 'libreoffice-*' orca 'cuda-*' \
        'chromium-*' 'python2.*' 'python3.*' 'gstreamer1.0-*' 'xdg-*' 'x11-*' \
        'printer-*' ghostscript

An identical test on the stable release of JetPack, with the same L4T base image on the corresponding tag (r32.6.1), with the same flags, on the same hardware, has been working fine.

I can re-flash the SD card and try again if removing the above packages is the suspected root cause, although I would love to keep the base system slimmed down.

[AastaLLL]

Hi,

This issue can be reproduced in our environment as well.

We are checking this with our internal team.
Will get back to you later.

Thanks.

[AastaLLL]

Hi,

Could you check if the following suggestion helps in your use case?

Docker isn't working after apt upgrade

I followed the instructions from: https://github.com/dusty-nv/jetson-containers/issues/108 https://forums.developer.nvidia.com/t/docker-containers-wont-run-after-recent-apt-get-upgrade/194369/11 I combined the instructions, with a revision to the /etc/apt/preferences section and here they are: wget https://launchpad.net/ubuntu/+source/docker.io/20.10.2-0ubuntu1~18.04.2/+build/21335731/+files/docker.io_20.10.2-0ubuntu1~18.04.2_arm64.deb sudo dpkg -i docker.io_20.10.2-0ubuntu1~18.04.2_arm64.de…

Thanks.

[rollc_at]

Hi @AastaLLL,

I’ve tested the following:

docker run -ti --rm --privileged --runtime nvidia --security-opt seccomp=unconfined nvcr.io/nvidia/l4t-base:r34.1 true

Getting the same symptoms.

I’ve tried downgrading Docker. These are the only versions I have available in the repositories:

$ apt-cache madison docker.io containerd
 docker.io | 20.10.7-0ubuntu5~20.04.2 | http://ports.ubuntu.com/ubuntu-ports focal-updates/universe arm64 Packages
 docker.io | 20.10.7-0ubuntu5~20.04.2 | http://ports.ubuntu.com/ubuntu-ports focal-security/universe arm64 Packages
 docker.io | 19.03.8-0ubuntu1 | http://ports.ubuntu.com/ubuntu-ports focal/universe arm64 Packages
containerd | 1.5.5-0ubuntu3~20.04.2 | http://ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 Packages
containerd | 1.5.5-0ubuntu3~20.04.2 | http://ports.ubuntu.com/ubuntu-ports focal-security/main arm64 Packages
containerd | 1.3.3-0ubuntu2 | http://ports.ubuntu.com/ubuntu-ports focal/main arm64 Packages

I stop the Docker daemon, wipe the working dataset, downgrade the packages, and start the daemon again:

service docker stop
rm -rf /var/lib/docker
apt-get install --no-install-recommends docker.io=19.03.8-0ubuntu1 containerd=1.3.3-0ubuntu2
service docker start

Still same problem.

I speculate that this is specifically related to the forwarding of /dev/nvhost-as-gpu. I’ve dropped the --privileged flag, replaced it with --device /dev/nvhost-as-gpu, and getting the same error. I’ve tried forwarding a couple other devices more selectively (nvhost-ctrl, nvhost-nvjpg, nvhost-nvdec) and the container starts fine. In the meantime I will verify whether we actually need nvhost-as-gpuin the container for our application, perhaps I can continue testing the JetPack upgrade without it.

[AastaLLL]

Hi,

Thanks for the update.

We are checking this internally.
Will share more information with you later.

[alexs512]

I am seeing the same issue trying to run container built using GitHub - dusty-nv/jetson-containers: Machine Learning Containers for NVIDIA Jetson and JetPack-L4T on jetpack 5.0.
the container builds ok.
but when running using the following

nvidia-docker run -it -d --rm
–privileged=True
–volume=“/dev:/dev”
–net=host
–env DISPLAY=${DISPLAY}
–env=TERM=“xterm-color”
–env NVIDIA_DISABLE_REQUIRE=1
–env NVIDIA_VISIBLE_DEVICES=0
–env NVIDIA_DRIVER_CAPABILITIES=all
–volume=“/tmp/.X11-unix:/tmp/.X11-unix:rw”
…

I get lengthy error log:

docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: src: /etc/vulkan/icd.d/nvidia_icd.json, src_lnk: /usr/lib/aarch64-linux-gnu/tegra/nvidia_icd.json, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/etc/vulkan/icd.d/nvidia_icd.json, dst_lnk: /usr/lib/aarch64-linux-gnu/tegra/nvidia_icd.json
src: /usr/lib/aarch64-linux-gnu/libcuda.so, src_lnk: tegra/libcuda.so, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/libcuda.so, dst_lnk: tegra/libcuda.so
src: /usr/lib/aarch64-linux-gnu/libnvcucompat.so, src_lnk: tegra/libnvcucompat.so, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/libnvcucompat.so, dst_lnk: tegra/libnvcucompat.so
src: /usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so, src_lnk: tegra/libnvidia-nvvm.so, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so, dst_lnk: tegra/libnvidia-nvvm.so
src: /usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.999999, src_lnk: tegra/libnvv4l2.so, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.999999, dst_lnk: tegra/libnvv4l2.so
src: /usr/lib/aarch64-linux-gnu/libv4lconvert.so.0.0.999999, src_lnk: tegra/libnvv4lconvert.so, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/libv4lconvert.so.0.0.999999, dst_lnk: tegra/libnvv4lconvert.so
src: /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvargus.so, src_lnk: …/…/…/tegra/libv4l2_nvargus.so, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvargus.so, dst_lnk: …/…/…/tegra/libv4l2_nvargus.so
src: /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvcuvidvideocodec.so, src_lnk: …/…/…/tegra/libv4l2_nvcuvidvideocodec.so, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvcuvidvideocodec.so, dst_lnk: …/…/…/tegra/libv4l2_nvcuvidvideocodec.so
src: /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvideocodec.so, src_lnk: …/…/…/tegra/libv4l2_nvvideocodec.so, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvideocodec.so, dst_lnk: …/…/…/tegra/libv4l2_nvvideocodec.so
src: /usr/lib/aarch64-linux-gnu/libvulkan.so.1.3.203, src_lnk: tegra/libvulkan.so.1.3.203, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/libvulkan.so.1.3.203, dst_lnk: tegra/libvulkan.so.1.3.203
src: /usr/lib/aarch64-linux-gnu/tegra/libcuda.so, src_lnk: libcuda.so.1.1, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/tegra/libcuda.so, dst_lnk: libcuda.so.1.1
src: /usr/lib/aarch64-linux-gnu/tegra/libgstnvdsseimeta.so, src_lnk: libgstnvdsseimeta.so.1.0.0, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/tegra/libgstnvdsseimeta.so, dst_lnk: libgstnvdsseimeta.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurface.so, src_lnk: libnvbufsurface.so.1.0.0, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/tegra/libnvbufsurface.so, dst_lnk: libnvbufsurface.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurftransform.so, src_lnk: libnvbufsurftransform.so.1.0.0, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/tegra/libnvbufsurftransform.so, dst_lnk: libnvbufsurftransform.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so, src_lnk: libnvbuf_utils.so.1.0.0, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so, dst_lnk: libnvbuf_utils.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvdsbufferpool.so, src_lnk: libnvdsbufferpool.so.1.0.0, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/tegra/libnvdsbufferpool.so, dst_lnk: libnvdsbufferpool.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvidia-nvvm.so, src_lnk: libnvidia-nvvm.so.4, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/tegra/libnvidia-nvvm.so, dst_lnk: libnvidia-nvvm.so.4
src: /usr/lib/aarch64-linux-gnu/tegra/libnvidia-nvvm.so.4, src_lnk: libnvidia-nvvm.so.4.0.0, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/tegra/libnvidia-nvvm.so.4, dst_lnk: libnvidia-nvvm.so.4.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvid_mapper.so, src_lnk: libnvid_mapper.so.1.0.0, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/tegra/libnvid_mapper.so, dst_lnk: libnvid_mapper.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvscibuf.so, src_lnk: libnvscibuf.so.1, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/tegra/libnvscibuf.so, dst_lnk: libnvscibuf.so.1
src: /usr/lib/aarch64-linux-gnu/tegra/libnvscicommon.so, src_lnk: libnvscicommon.so.1, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/tegra/libnvscicommon.so, dst_lnk: libnvscicommon.so.1
src: /usr/lib/aarch64-linux-gnu/tegra/libnvscistream.so, src_lnk: libnvscistream.so.1, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/tegra/libnvscistream.so, dst_lnk: libnvscistream.so.1
src: /usr/lib/aarch64-linux-gnu/tegra/libnvscisync.so, src_lnk: libnvscisync.so.1, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/lib/aarch64-linux-gnu/tegra/libnvscisync.so, dst_lnk: libnvscisync.so.1
src: /usr/share/glvnd/egl_vendor.d/10_nvidia.json, src_lnk: …/…/…/lib/aarch64-linux-gnu/tegra-egl/nvidia.json, dst: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/usr/share/glvnd/egl_vendor.d/10_nvidia.json, dst_lnk: …/…/…/lib/aarch64-linux-gnu/tegra-egl/nvidia.json
, stderr: nvidia-container-cli: mount error: file creation failed: /var/lib/docker/overlay2/be7392c101197ef7cbbc0ab2a4f14a3513d1a8d01e93cc920e444c4b0c4068ef/merged/dev/nvhost-as-gpu: invalid argument: unknown.

[AastaLLL]

Hi, alexs512

Thanks for reporting this.
This issue can be reproduced with either --privileged or mounting /dev.
Our internal is actively working on this. Will share more information with you later.

Thanks.

[AastaLLL]

Hi, all

We found a way to run --privileged and --runtime nvidia together.
Please edit the following file and comment out the /dev/nvhost-as-gpu.

/etc/nvidia-container-runtime/host-files-for-container.d/l4t.csv

dev, /dev/fb0
dev, /dev/fb1
#dev, /dev/nvhost-as-gpu
dev, /dev/nvhost-ctrl
...

Although the node is commented out, you can still access it within the docker.
We tested a CUDA sample and it can run normally.

Please note that if you run the container without --privileged, the full l4t.csv is required.
If any issue occurs after applying this workaround, please let us know.

Thanks.

[AastaLLL]

Hi, all

We solve this issue in our latest NVIDIA Container Toolkit v1.10.0.
Please try the following to get the fix.

$ sudo apt install curl
$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
      && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
      && curl -s -L https://nvidia.github.io/libnvidia-container/experimental/$distribution/libnvidia-container.list | \
         sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
         sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
$ sudo apt update
$ sudo apt-get install nvidia-container-toolkit=1.10.0~rc.3-1

With the latest nvidia-container-toolkit, you don’t need the workaround shared above anymore (edit l4t.csv file).
The library is also included in our future releases.

Thanks.