NVIDIA docker issue with privileged

We have been using docker in Xavier for a long time, to encapsulate our programs but have access to the GPU, NVENC, NVDEC, DLA, VI, etc.
We normally run the container with --privileged and mounting all devices (NVIDIA GPU, but also CAN, USB, etc).

With r34.1 and Orin, this command produces errors:

verdant@orin:~/verdant/docker/aarch64/verdant (master)$ docker run --rm -it --net=host --runtime nvidia --privileged nvcr.io/nvidia/l4t-base:r34.1
docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: src: /etc/vulkan/icd.d/nvidia_icd.json, src_lnk: /usr/lib/aarch64-linux-gnu/tegra/nvidia_icd.json, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/etc/vulkan/icd.d/nvidia_icd.json, dst_lnk: /usr/lib/aarch64-linux-gnu/tegra/nvidia_icd.json
src: /usr/lib/aarch64-linux-gnu/libcuda.so, src_lnk: tegra/libcuda.so, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/libcuda.so, dst_lnk: tegra/libcuda.so
src: /usr/lib/aarch64-linux-gnu/libnvcucompat.so, src_lnk: tegra/libnvcucompat.so, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/libnvcucompat.so, dst_lnk: tegra/libnvcucompat.so
src: /usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so, src_lnk: tegra/libnvidia-nvvm.so, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so, dst_lnk: tegra/libnvidia-nvvm.so
src: /usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.999999, src_lnk: tegra/libnvv4l2.so, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.999999, dst_lnk: tegra/libnvv4l2.so
src: /usr/lib/aarch64-linux-gnu/libv4lconvert.so.0.0.999999, src_lnk: tegra/libnvv4lconvert.so, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/libv4lconvert.so.0.0.999999, dst_lnk: tegra/libnvv4lconvert.so
src: /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvargus.so, src_lnk: …/…/…/tegra/libv4l2_nvargus.so, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvargus.so, dst_lnk: …/…/…/tegra/libv4l2_nvargus.so
src: /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvcuvidvideocodec.so, src_lnk: …/…/…/tegra/libv4l2_nvcuvidvideocodec.so, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvcuvidvideocodec.so, dst_lnk: …/…/…/tegra/libv4l2_nvcuvidvideocodec.so
src: /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvideocodec.so, src_lnk: …/…/…/tegra/libv4l2_nvvideocodec.so, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvideocodec.so, dst_lnk: …/…/…/tegra/libv4l2_nvvideocodec.so
src: /usr/lib/aarch64-linux-gnu/libvulkan.so.1.3.203, src_lnk: tegra/libvulkan.so.1.3.203, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/libvulkan.so.1.3.203, dst_lnk: tegra/libvulkan.so.1.3.203
src: /usr/lib/aarch64-linux-gnu/tegra/libcuda.so, src_lnk: libcuda.so.1.1, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/tegra/libcuda.so, dst_lnk: libcuda.so.1.1
src: /usr/lib/aarch64-linux-gnu/tegra/libgstnvdsseimeta.so, src_lnk: libgstnvdsseimeta.so.1.0.0, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/tegra/libgstnvdsseimeta.so, dst_lnk: libgstnvdsseimeta.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurface.so, src_lnk: libnvbufsurface.so.1.0.0, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/tegra/libnvbufsurface.so, dst_lnk: libnvbufsurface.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurftransform.so, src_lnk: libnvbufsurftransform.so.1.0.0, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/tegra/libnvbufsurftransform.so, dst_lnk: libnvbufsurftransform.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so, src_lnk: libnvbuf_utils.so.1.0.0, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so, dst_lnk: libnvbuf_utils.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvdsbufferpool.so, src_lnk: libnvdsbufferpool.so.1.0.0, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/tegra/libnvdsbufferpool.so, dst_lnk: libnvdsbufferpool.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvidia-nvvm.so, src_lnk: libnvidia-nvvm.so.4, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/tegra/libnvidia-nvvm.so, dst_lnk: libnvidia-nvvm.so.4
src: /usr/lib/aarch64-linux-gnu/tegra/libnvidia-nvvm.so.4, src_lnk: libnvidia-nvvm.so.4.0.0, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/tegra/libnvidia-nvvm.so.4, dst_lnk: libnvidia-nvvm.so.4.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvid_mapper.so, src_lnk: libnvid_mapper.so.1.0.0, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/tegra/libnvid_mapper.so, dst_lnk: libnvid_mapper.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvscibuf.so, src_lnk: libnvscibuf.so.1, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/tegra/libnvscibuf.so, dst_lnk: libnvscibuf.so.1
src: /usr/lib/aarch64-linux-gnu/tegra/libnvscicommon.so, src_lnk: libnvscicommon.so.1, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/tegra/libnvscicommon.so, dst_lnk: libnvscicommon.so.1
src: /usr/lib/aarch64-linux-gnu/tegra/libnvscistream.so, src_lnk: libnvscistream.so.1, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/tegra/libnvscistream.so, dst_lnk: libnvscistream.so.1
src: /usr/lib/aarch64-linux-gnu/tegra/libnvscisync.so, src_lnk: libnvscisync.so.1, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/lib/aarch64-linux-gnu/tegra/libnvscisync.so, dst_lnk: libnvscisync.so.1
src: /usr/share/glvnd/egl_vendor.d/10_nvidia.json, src_lnk: …/…/…/lib/aarch64-linux-gnu/tegra-egl/nvidia.json, dst: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/usr/share/glvnd/egl_vendor.d/10_nvidia.json, dst_lnk: …/…/…/lib/aarch64-linux-gnu/tegra-egl/nvidia.json
, stderr: nvidia-container-cli: mount error: file creation failed: /var/lib/docker/overlay2/ef5cd95ef8a533c5cd01fa226c86b75ff406e123485aba59f6bbdfea4e737bed/merged/dev/nvhost-as-gpu: invalid argument: unknown.

Dear @lawrence8pqxb ,
It looks like Jetson platform, moving to relavant forum

Hi,

It seems this duplicate to the below topic:

We are checking this internally.
Will share more information with you later.
Thanks.

Hi,

Please check the below comment for the workaround:

Thanks.

Thank you so much!

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.