Jetson Xavier AGX CUDNN Shared Object Files Mystery

I am working on creating a Docker image for my Xavier similar to this one here:

I am running Jetpack 4.6 and will be using Pytorch and Tensorflow in the container.

The image is being built using a gitlab cicd runner with 64 bit arm based GPUs nodes in AWS. Node details:
“Amazon EC2 G5g instances are powered by AWS Graviton2 processors and feature NVIDIA T4G Tensor Core GPUs to provide the best price performance in Amazon EC2 for graphics workloads such as Android game streaming. They are the first Arm-based instances in a major cloud to feature GPU acceleration. Customers can also use G5g instances for cost-effective ML inference.”

The problem I am running into is that my build succeeds, then I go to run the container on our Xavier devkit. I can run the original l4t Pytorch image and see that torch is using the GPU. I can see all the shared object files are in the requisite place when the container is run with --gpus=all

SO path when base pytorch image is run w/o --gpus flag

dso-dat-dev3@dso-dat-dev3:~$ docker run -it --rm nvcr.io/nvidia/l4t-pytorch:r32.6.1-pth1.9-py3 /bin/bash
root@aee59d62865b:/# ls /usr/local/cuda-10.2/targets/aarch64-linux/lib
libcudadevrt.a libcudart_static.a stubs

SO path when base pytorch image is run w/ --gpus flag

dso-dat-dev3@dso-dat-dev3:~$ docker run -it --rm --gpus=all nvcr.io/nvidia/l4t-pytorch:r32.6.1-pth1.9-py3 /bin/bash
root@4d96784d3f18:/# ls /usr/local/cuda-10.2/targets/aarch64-linux/lib
libcublas.so libcudnn_static.a libnppc.so.10.2.1.300 libnppisu.so
libcublas.so.10 libcudnn_static_v8.a libnppc_static.a libnppisu.so.10
libcublas.so.10.2.3.300 libcufft.so libnppial.so libnppisu.so.10.2.1.300
libcublasLt.so libcufft.so.10 libnppial.so.10 libnppisu_static.a
libcublasLt.so.10 libcufft.so.10.1.2.300 libnppial.so.10.2.1.300 libnppitc.so
libcublasLt.so.10.2.3.300 libcufft_static.a libnppial_static.a libnppitc.so.10
libcublasLt_static.a libcufft_static_nocallback.a libnppicc.so libnppitc.so.10.2.1.300
libcublas_static.a libcufftw.so libnppicc.so.10 libnppitc_static.a
libcudadevrt.a libcufftw.so.10 libnppicc.so.10.2.1.300 libnpps.so
libcudart.so libcufftw.so.10.1.2.300 libnppicc_static.a libnpps.so.10
libcudart.so.10.2 libcufftw_static.a libnppicom.so libnpps.so.10.2.1.300
libcudart.so.10.2.300 libcuinj64.so libnppicom.so.10 libnpps_static.a
libcudart_static.a libcuinj64.so.10.2 libnppicom.so.10.2.1.300 libnvToolsExt.so
libcudnn.so libcuinj64.so.10.2.300 libnppicom_static.a libnvToolsExt.so.1
libcudnn.so.8 libculibos.a libnppidei.so libnvToolsExt.so.1.0.0
libcudnn.so.8.2.1 libcupti.so libnppidei.so.10 libnvblas.so
libcudnn_adv_infer.so libcupti.so.10.2 libnppidei.so.10.2.1.300 libnvblas.so.10
libcudnn_adv_infer.so.8 libcupti.so.10.2.175 libnppidei_static.a libnvblas.so.10.2.3.300
libcudnn_adv_infer.so.8.2.1 libcurand.so libnppif.so libnvgraph.so
libcudnn_adv_train.so libcurand.so.10 libnppif.so.10 libnvgraph.so.10
libcudnn_adv_train.so.8 libcurand.so.10.1.2.300 libnppif.so.10.2.1.300 libnvgraph.so.10.2.300
libcudnn_adv_train.so.8.2.1 libcurand_static.a libnppif_static.a libnvgraph_static.a
libcudnn_cnn_infer.so libcusolver.so libnppig.so libnvperf_host.so
libcudnn_cnn_infer.so.8 libcusolver.so.10 libnppig.so.10 libnvperf_target.so
libcudnn_cnn_infer.so.8.2.1 libcusolver.so.10.3.0.300 libnppig.so.10.2.1.300 libnvrtc-builtins.so
libcudnn_cnn_train.so libcusolver_static.a libnppig_static.a libnvrtc-builtins.so.10.2
libcudnn_cnn_train.so.8 libcusparse.so libnppim.so libnvrtc-builtins.so.10.2.300
libcudnn_cnn_train.so.8.2.1 libcusparse.so.10 libnppim.so.10 libnvrtc.so
libcudnn_ops_infer.so libcusparse.so.10.3.1.300 libnppim.so.10.2.1.300 libnvrtc.so.10.2
libcudnn_ops_infer.so.8 libcusparse_static.a libnppim_static.a libnvrtc.so.10.2.300
libcudnn_ops_infer.so.8.2.1 liblapack_static.a libnppist.so stubs
libcudnn_ops_train.so libmetis_static.a libnppist.so.10
libcudnn_ops_train.so.8 libnppc.so libnppist.so.10.2.1.300
libcudnn_ops_train.so.8.2.1 libnppc.so.10 libnppist_static.a

SO path in my custom image w/ --gpus flag

dso-dat-dev3@dso-dat-dev3:~$ docker run -it --rm --gpus=all <our ecr uri>/ai-platform-nvidia-conda-ml:5e13d2e0-arm64-gpu bash
(base) root@c4e0876d6d2c:/# ls /usr/local/cuda-10.2/targets/aarch64-linux/lib
libcublas.so.10 libcufftw.so.10.1.2.300 libnppicc.so.10 libnppim.so.10.2.1.300 libnvToolsExt.so.1.0.0
libcublas.so.10.2.3.300 libcurand.so.10 libnppicc.so.10.2.1.300 libnppist.so.10 libnvblas.so.10
libcublasLt.so.10 libcurand.so.10.1.2.300 libnppicom.so.10 libnppist.so.10.2.1.300 libnvblas.so.10.2.3.300
libcublasLt.so.10.2.3.300 libcusolver.so.10 libnppicom.so.10.2.1.300 libnppisu.so.10 libnvgraph.so.10
libcudadevrt.a libcusolver.so.10.3.0.300 libnppidei.so.10 libnppisu.so.10.2.1.300 libnvgraph.so.10.2.300
libcudart.so.10.2 libcusparse.so.10 libnppidei.so.10.2.1.300 libnppitc.so.10 libnvrtc-builtins.so
libcudart.so.10.2.300 libcusparse.so.10.3.1.300 libnppif.so.10 libnppitc.so.10.2.1.300 libnvrtc-builtins.so.10.2
libcudart_static.a libnppc.so.10 libnppif.so.10.2.1.300 libnpps.so.10 libnvrtc-builtins.so.10.2.300
libcufft.so.10 libnppc.so.10.2.1.300 libnppig.so.10 libnpps.so.10.2.1.300 libnvrtc.so.10.2
libcufft.so.10.1.2.300 libnppial.so.10 libnppig.so.10.2.1.300 libnvToolsExt.so libnvrtc.so.10.2.300
libcufftw.so.10 libnppial.so.10.2.1.300 libnppim.so.10 libnvToolsExt.so.1 stubs

The subset of shared objects here is causing pytorch fail at import with an OSerror related to libcudnn.so.8 .

Perhaps I am not doing something correctly, but the docker image I am building is based closely on the one linked above. The base image I am using is: nvcr. io/nvidia/l4t-cuda:10.2.460-runtime whereas the one in the build above is nvcr. io/nvidia/l4t-base:r32.4.4. But this should not make a different right?

Since your docker host is running JP 4.6, the cuda/cudnn/etc libraries are actually supposed to be mounted from the host. Check /etc/nvidia-container-runtime/host-files-for-container.d directory.
You should see those files: cuda.csv cudnn.csv l4t.csv tensorrt.csv They define what files/dir are volume mounted into the container. In your case, libcudnn.so.8 is from cudnn.csv (came from package nvidia-container-csv-cudnn),

Hi @user100090 thank you for the reply. I have looked at this files in /etc/nvidia-container-runtime/host-files-for-container.d and unfortunately can’t figure out why I’m not seeing the correct shared objects files in my custom container. The correct ones are shown above when I ls the directory /usr/local/cuda-10.2/targets/aarch64-linux/lib for the Pytorch container run with the gpus flag. It seems like they aren’t being mounted correctly when I run my custom container with the GPUs flag.

Hi,

Would you mind replacing '–gpus=all with '--runtime nvidia to see if it works?

$ sudo docker run -it --rm --runtime nvidia --network host nvcr.io/nvidia/l4t-pytorch:r32.6.1-pth1.9-py3

More, maybe you just check the incorrect repository.
For example, I can find them in the /usr/lib/aarch64-linux-gnu/.

# ls /usr/lib/aarch64-linux-gnu/libcudnn*
/usr/lib/aarch64-linux-gnu/libcudnn.so                  /usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so.8.2.1  /usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so.8.2.1
/usr/lib/aarch64-linux-gnu/libcudnn.so.8                /usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so        /usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so
/usr/lib/aarch64-linux-gnu/libcudnn.so.8.2.1            /usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8      /usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so.8
/usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so        /usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8.2.1  /usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so.8.2.1
/usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so.8      /usr/lib/aarch64-linux-gnu/libcudnn_cnn_train.so.8      /usr/lib/aarch64-linux-gnu/libcudnn_static.a
/usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so.8.2.1  /usr/lib/aarch64-linux-gnu/libcudnn_cnn_train.so.8.2.1  /usr/lib/aarch64-linux-gnu/libcudnn_static_v8.a
/usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so        /usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so
/usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so.8      /usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so.8

Thanks.

I see those files on the device root filesystem. They are definitely there.

but when I run the container I don’t see them.

dso-dat-dev2@ubuntu:~$ docker run -it --runtime=nvidia <our ecr uri>/ai-platform-nvidia-conda-ml:7e6ee134-arm64-gpu /bin/bash
(base) root@166d0c1b769d:/# conda activate venv
(venv) root@166d0c1b769d:/# python
Python 3.6.15 | packaged by conda-forge | (default, Dec  3 2021, 19:12:04)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/envs/venv/lib/python3.6/site-packages/torch/__init__.py", line 196, in <module>
    _load_global_deps()
  File "/opt/conda/envs/venv/lib/python3.6/site-packages/torch/__init__.py", line 149, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/opt/conda/envs/venv/lib/python3.6/ctypes/__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libcudnn.so.8: cannot open shared object file: No such file or directory
>>>
(venv) root@166d0c1b769d:/# ls /usr/lib/aarch64-linux-gnu/
Mcrt1.o                                 libhdf5_serialhl_fortran.so.100
Scrt1.o                                 libhdf5_serialhl_fortran.so.100.0.0
audit                                   libheimbase.so.1
avahi                                   libheimbase.so.1.0.0
blas                                    libheimntlm.so.0
caca                                    libheimntlm.so.0.1.0
coreutils                               libhogweed.so.4
crt1.o                                  libhogweed.so.4.5
crti.o                                  libhwloc.so.0
crtn.o                                  libhwloc.so.1
dri                                     libhwloc.so.2
engines-1.1                             libhwloc.so.3
gconv                                   libhwloc.so.4
gcrt1.o                                 libhwloc.so.5
gdk-pixbuf-2.0                          libhwloc.so.5.7.6
gio                                     libhx509.so.5
glib-2.0                                libhx509.so.5.0.0
gstreamer-1.0                           libi2c.so.0
gstreamer1.0                            libi2c.so.0.1.0
gtk-3.0                                 libibverbs.so.1
hdf5                                    libibverbs.so.1.1.17.1
hwloc                                   libicudata.so.60
krb5                                    libicudata.so.60.2
lapack                                  libicui18n.so.60
ldscripts                               libicui18n.so.60.2
libBrokenLocale.a                       libicuio.so.60
libBrokenLocale.so                      libicuio.so.60.2
libEGL.so                               libicutest.so.60
libEGL.so.1                             libicutest.so.60.2
libEGL.so.1.0.0                         libicutu.so.60
libEGL_mesa.so.0                        libicutu.so.60.2
libEGL_mesa.so.0.0.0                    libicuuc.so.60
libFLAC.so.8                            libicuuc.so.60.2
libFLAC.so.8.3.0                        libidn2.so.0
libGL.so                                libidn2.so.0.3.3
libGL.so.1                              libiec61883.so.0
libGL.so.1.0.0                          libiec61883.so.0.1.1
libGLESv1_CM.so                         libisl.so.19
libGLESv1_CM.so.1                       libisl.so.19.0.0
libGLESv1_CM.so.1.0.0                   libitm.so.1
libGLESv2.so                            libitm.so.1.0.0
libGLESv2.so.2                          libjack.so.0
libGLESv2.so.2.0.0                      libjack.so.0.1.0
libGLU.a                                libjacknet.so.0
libGLU.so                               libjacknet.so.0.1.0
libGLU.so.1                             libjackserver.so.0
libGLU.so.1.3.1                         libjackserver.so.0.1.0
libGLX.so                               libjbig.so.0
libGLX.so.0                             libjpeg.a
libGLX.so.0.0.0                         libjpeg.so
libGLX_indirect.so.0                    libjpeg.so.8
libGLX_mesa.so.0                        libjpeg.so.8.1.2
libGLX_mesa.so.0.0.0                    libjson-glib-1.0.so.0
libGLdispatch.so                        libjson-glib-1.0.so.0.400.2
libGLdispatch.so.0                      libjsoncpp.so.1
libGLdispatch.so.0.0.0                  libjsoncpp.so.1.7.4
libHalf.so.12                           libk5crypto.so.3
libHalf.so.12.0.0                       libk5crypto.so.3.1
libICE.a                                libkate.so.1
libICE.so                               libkate.so.1.3.0
libICE.so.6                             libkrb5.so.26
libICE.so.6.3.0                         libkrb5.so.26.0.0
libIex-2_2.so.12                        libkrb5.so.3
libIex-2_2.so.12.0.0                    libkrb5.so.3.3
libIexMath-2_2.so.12                    libkrb5support.so.0
libIexMath-2_2.so.12.0.0                libkrb5support.so.0.1
libIlmImf-2_2.so.22                     libksba.so.8
libIlmImf-2_2.so.22.0.0                 libksba.so.8.11.6
libIlmImfUtil-2_2.so.22                 liblapack.a
libIlmImfUtil-2_2.so.22.0.0             liblapack.so
libIlmThread-2_2.so.12                  liblapack.so.3
libIlmThread-2_2.so.12.0.0              liblber-2.4.so.2
libImath-2_2.so.12                      liblber-2.4.so.2.10.8
libImath-2_2.so.12.0.0                  liblcms2.so.2
libLLVM-10.so                           liblcms2.so.2.0.8
libLLVM-10.so.1                         libldap-2.4.so.2
libLLVM-9.0.0.so.1                      libldap_r-2.4.so.2
libLLVM-9.so                            libldap_r-2.4.so.2.10.8
libLLVM-9.so.1                          liblsan.so.0
libOpenCL.so.1                          liblsan.so.0.0.0
libOpenCL.so.1.0.0                      libltdl.so.7
libOpenGL.so                            libltdl.so.7.3.1
libOpenGL.so.0                          liblz4.so.1
libOpenGL.so.0.0.0                      liblz4.so.1.7.1
libSM.a                                 libm.a
libSM.so                                libm.so
libSM.so.6                              libmagic.so.1
libSM.so.6.0.1                          libmagic.so.1.0.0
libSoundTouch.so.1                      libmca_common_sm.so.20
libSoundTouch.so.1.0.0                  libmca_common_sm.so.20.10.1
libWildMidi.so.2                        libmca_common_verbs.so.20
libWildMidi.so.2.0.1                    libmca_common_verbs.so.20.10.0
libX11-xcb.a                            libmcheck.a
libX11-xcb.so                           libmenu.so.5
libX11-xcb.so.1                         libmenu.so.5.9
libX11-xcb.so.1.0.0                     libmenuw.so.5
libX11.a                                libmenuw.so.5.9
libX11.so                               libmjpegutils-2.1.so.0
libX11.so.6                             libmjpegutils-2.1.so.0.0.0
libX11.so.6.3.0                         libmms.so.0
libXau.a                                libmms.so.0.0.2
libXau.so                               libmodplug.so.1
libXau.so.6                             libmodplug.so.1.0.0
libXau.so.6.0.0                         libmp3lame.so.0
libXcomposite.so.1                      libmp3lame.so.0.0.0
libXcomposite.so.1.0.0                  libmpc.so.3
libXcursor.so.1                         libmpc.so.3.1.0
libXcursor.so.1.0.2                     libmpcdec.so.6
libXdamage.a                            libmpcdec.so.6.1.0
libXdamage.so                           libmpdec.so.2
libXdamage.so.1                         libmpdec.so.2.4.2
libXdamage.so.1.1.0                     libmpeg2.so.0
libXdmcp.a                              libmpeg2.so.0.1.0
libXdmcp.so                             libmpeg2convert.so.0
libXdmcp.so.6                           libmpeg2convert.so.0.0.0
libXdmcp.so.6.0.0                       libmpeg2encpp-2.1.so.0
libXext.a                               libmpeg2encpp-2.1.so.0.0.0
libXext.so                              libmpfr.so.6
libXext.so.6                            libmpfr.so.6.0.1
libXext.so.6.4.0                        libmpg123.so.0
libXfixes.a                             libmpg123.so.0.44.8
libXfixes.so                            libmpi.so.20
libXfixes.so.3                          libmpi.so.20.10.1
libXfixes.so.3.1.0                      libmpi_cxx.so.20
libXi.so.6                              libmpi_cxx.so.20.10.0
libXi.so.6.1.0                          libmpi_java.so.20
libXinerama.so.1                        libmpi_java.so.20.10.0
libXinerama.so.1.0.0                    libmpi_mpifh.so.20
libXrandr.so.2                          libmpi_mpifh.so.20.11.0
libXrandr.so.2.2.0                      libmpi_usempi_ignore_tkr.so.20
libXrender.so.1                         libmpi_usempi_ignore_tkr.so.20.10.0
libXrender.so.1.3.0                     libmpi_usempif08.so.20
libXt.a                                 libmpi_usempif08.so.20.10.0
libXt.so                                libmplex2-2.1.so.0
libXt.so.6                              libmplex2-2.1.so.0.0.0
libXt.so.6.0.0                          libmysofa.so.0
libXxf86vm.a                            libmysofa.so.0.5.1
libXxf86vm.so                           libnettle.so.6
libXxf86vm.so.1                         libnettle.so.6.5
libXxf86vm.so.1.0.0                     libnghttp2.so.14
liba52-0.7.4.so                         libnghttp2.so.14.15.2
libaa.so.1                              libnice.so.10
libaa.so.1.0.4                          libnice.so.10.7.0
libaec.a                                libnl-route-3.so.200
libaec.so                               libnl-route-3.so.200.24.0
libaec.so.0                             libnorm.so.1
libaec.so.0.0.3                         libnorm.so.1.0.0
libanl.a                                libnpth.so.0
libanl.so                               libnpth.so.0.1.1
libapt-inst.so.2.0                      libnsl.a
libapt-inst.so.2.0.0                    libnsl.so
libapt-pkg.so.5.0                       libnss_compat.so
libapt-pkg.so.5.0.2                     libnss_dns.so
libapt-private.so.0.0                   libnss_files.so
libapt-private.so.0.0.0                 libnss_hesiod.so
libarchive.so.13                        libnss_nis.so
libarchive.so.13.2.2                    libnss_nisplus.so
libasan.so.4                            libnuma.so.1
libasan.so.4.0.0                        libnuma.so.1.0.0
libasn1.so.8                            libnvsample_cudaprocess.so
libasn1.so.8.0.0                        libofa.so.0
libasound.so.2                          libofa.so.0.0.0
libasound.so.2.0.0                      libogg.so.0
libass.so.9                             libogg.so.0.8.2
libass.so.9.0.2                         libompitrace.so.20
libassuan.so.0                          libompitrace.so.20.10.0
libassuan.so.0.8.1                      libopcodes-2.30-system.so
libasyncns.so.0                         libopen-pal.so.20
libasyncns.so.0.3.1                     libopen-pal.so.20.10.1
libatk-1.0.so.0                         libopen-rte.so.20
libatk-1.0.so.0.22810.1                 libopen-rte.so.20.10.1
libatk-bridge-2.0.so.0                  libopenal.so.1
libatk-bridge-2.0.so.0.0.0              libopenal.so.1.18.2
libatomic.so.1                          libopenblas.a
libatomic.so.1.2.0                      libopenblas.so
libatspi.so.0                           libopenblas.so.0
libatspi.so.0.0.1                       libopenblas_armv8p-r0.2.20.a
libavahi-client.so.3                    libopenblas_armv8p-r0.2.20.so
libavahi-client.so.3.2.9                libopencore-amrnb.so.0
libavahi-common.so.3                    libopencore-amrnb.so.0.0.3
libavahi-common.so.3.5.3                libopencore-amrwb.so.0
libavc1394.so.0                         libopencore-amrwb.so.0.0.3
libavc1394.so.0.3.0                     libopenjp2.so.2.3.0
libavcodec.so.57                        libopenjp2.so.7
libavcodec.so.57.107.100                libopenmpt.so.0
libavfilter.so.6                        libopenmpt.so.0.1.1
libavfilter.so.6.107.100                libopus.so.0
libavformat.so.57                       libopus.so.0.5.2
libavformat.so.57.83.100                liborc-0.4.so.0
libavresample.so.3                      liborc-0.4.so.0.28.0
libavresample.so.3.7.0                  liborc-test-0.4.so.0
libavutil.so.55                         liborc-test-0.4.so.0.28.0
libavutil.so.55.78.100                  liboshmem.so.20
libbfd-2.30-system.so                   libp11-kit.so.0
libblas.a                               libp11-kit.so.0.3.0
libblas.so                              libpanel.so.5
libblas.so.3                            libpanel.so.5.9
libbluray.so.2                          libpanelw.so.5
libbluray.so.2.0.2                      libpanelw.so.5.9
libbs2b.so.0                            libpango-1.0.so.0
libbs2b.so.0.0.0                        libpango-1.0.so.0.4000.14
libc.a                                  libpangocairo-1.0.so.0
libc.so                                 libpangocairo-1.0.so.0.4000.14
libc_nonshared.a                        libpangoft2-1.0.so.0
libcaca++.so.0                          libpangoft2-1.0.so.0.4000.14
libcaca++.so.0.99.19                    libpciaccess.so.0
libcaca.so.0                            libpciaccess.so.0.11.1
libcaca.so.0.99.19                      libpcreposix.so.3
libcairo-gobject.so.2                   libpcreposix.so.3.13.3
libcairo-gobject.so.2.11510.0           libpcsclite.so.1
libcairo.so.2                           libpcsclite.so.1.0.0
libcairo.so.2.11510.0                   libperl.so.5.26
libcanberra-0.30                        libperl.so.5.26.1
libcanberra-gtk3.so.0                   libpfm.so.4
libcanberra-gtk3.so.0.1.9               libpfm.so.4.9.0
libcanberra.so.0                        libpgm-5.2.so.0
libcanberra.so.0.2.5                    libpgm-5.2.so.0.0.122
libcc1.so.0                             libpipeline.so.1
libcc1.so.0.0.0                         libpipeline.so.1.5.0
libcdda_interface.so.0                  libpixman-1.so.0
libcdda_interface.so.0.10.2             libpixman-1.so.0.34.0
libcdda_paranoia.so.0                   libpng16.so.16
libcdda_paranoia.so.0.10.2              libpng16.so.16.34.0
libcdio.so.17                           libpostproc.so.54
libcdio.so.17.0.0                       libpostproc.so.54.7.100
libchromaprint.so.1                     libprotobuf-lite.a
libchromaprint.so.1.4.3                 libprotobuf-lite.so
libcidn.so                              libprotobuf-lite.so.10
libcolord.so.2                          libprotobuf-lite.so.10.0.0
libcolord.so.2.0.5                      libprotobuf.a
libcolordprivate.so.2                   libprotobuf.so
libcolordprivate.so.2.0.5               libprotobuf.so.10
libcroco-0.6.so.3                       libprotobuf.so.10.0.0
libcroco-0.6.so.3.0.1                   libprotoc.a
libcrypt.a                              libprotoc.so
libcrypt.so                             libprotoc.so.10
libcrypto.so.1.1                        libprotoc.so.10.0.0
libcuda.so                              libproxy.so.1
libcups.so.2                            libproxy.so.1.0.0
libcurl-gnutls.so.3                     libpsl.so.5
libcurl-gnutls.so.4                     libpsl.so.5.2.0
libcurl-gnutls.so.4.5.0                 libpthread.a
libcurl.so.4                            libpthread.so
libcurl.so.4.5.0                        libpthread_nonshared.a
libdatrie.so.1                          libpulse-simple.so.0
libdatrie.so.1.3.3                      libpulse-simple.so.0.1.1
libdb-5.3.so                            libpulse.so.0
libdc1394.so.22                         libpulse.so.0.20.2
libdc1394.so.22.2.1                     libpython2.7.a
libdca.so.0                             libpython2.7.so
libdca.so.0.0.0                         libpython2.7.so.1
libdconf.so.1                           libpython2.7.so.1.0
libdconf.so.1.0.0                       libraw1394.so.11
libde265.so.0                           libraw1394.so.11.1.0
libde265.so.0.0.10                      libresolv.a
libdebconfclient.so.0                   libresolv.so
libdebconfclient.so.0.0.0               librest-0.7.so.0
libdl.a                                 librest-0.7.so.0.0.0
libdl.so                                librhash.so.0
libdrm.so                               libroken.so.18
libdrm.so.2                             libroken.so.18.1.0
libdrm.so.2.4.0                         librom1394.so.0
libdrm_amdgpu.so                        librom1394.so.0.3.0
libdrm_amdgpu.so.1                      librpcsvc.a
libdrm_amdgpu.so.1.0.0                  librsvg-2.so.2
libdrm_etnaviv.so                       librsvg-2.so.2.40.20
libdrm_etnaviv.so.1                     librt.a
libdrm_etnaviv.so.1.0.0                 librt.so
libdrm_freedreno.so                     librtmp.so.1
libdrm_freedreno.so.1                   librubberband.so.2
libdrm_freedreno.so.1.0.0               librubberband.so.2.1.0
libdrm_nouveau.so                       libsamplerate.so.0
libdrm_nouveau.so.2                     libsamplerate.so.0.1.8
libdrm_nouveau.so.2.0.0                 libsasl2.so.2
libdrm_nvdc.so                          libsasl2.so.2.0.25
libdrm_radeon.so                        libsbc.so.1
libdrm_radeon.so.1                      libsbc.so.1.2.1
libdrm_radeon.so.1.0.1                  libsemanage.so.1
libdrm_tegra.so                         libsensors.so.4
libdrm_tegra.so.0                       libsensors.so.4.4.0
libdrm_tegra.so.0.0.0                   libserd-0.so.0
libdv.so.4                              libserd-0.so.0.28.0
libdv.so.4.0.3                          libshine.so.3
libdvdnav.so.4                          libshine.so.3.0.1
libdvdnav.so.4.2.0                      libshout.so.3
libdvdread.so.4                         libshout.so.3.2.0
libdvdread.so.4.2.0                     libsidplay.so.1
libedit.so.2                            libsidplay.so.1.0.3
libedit.so.2.0.56                       libsnappy.so.1
libelf-0.170.so                         libsnappy.so.1.1.7
libelf.so.1                             libsndfile.so.1
libepoxy.so.0                           libsndfile.so.1.0.28
libepoxy.so.0.0.0                       libsndio.so.6.1
libexpat.a                              libsodium.so.23
libexpat.so                             libsodium.so.23.1.0
libexpatw.a                             libsord-0.so.0
libexpatw.so                            libsord-0.so.0.16.0
libexpatw.so.1                          libsoup-2.4.so.1
libexpatw.so.1.6.7                      libsoup-2.4.so.1.8.0
libfaad.so.2                            libsoup-gnome-2.4.so.1
libfaad.so.2.0.0                        libsoup-gnome-2.4.so.1.8.0
libfaad_drm.so.2                        libsox.so.3
libfaad_drm.so.2.0.0                    libsox.so.3.0.0
libffi.a                                libsoxr.so.0
libffi.so                               libsoxr.so.0.1.1
libffi.so.6                             libspandsp.so.2
libffi.so.6.0.4                         libspandsp.so.2.0.0
libffi_pic.a                            libspeex.so.1
libfftw3.so.3                           libspeex.so.1.5.0
libfftw3.so.3.5.7                       libsqlite3.so.0
libfftw3_omp.so.3                       libsqlite3.so.0.8.6
libfftw3_omp.so.3.5.7                   libsratom-0.so.0
libfftw3_threads.so.3                   libsratom-0.so.0.6.0
libfftw3_threads.so.3.5.7               libsrtp2.so.1
libflite.so.1                           libssh-gcrypt.so.4
libflite.so.2.1                         libssh-gcrypt.so.4.5.0
libflite_cmu_grapheme_lang.so.1         libssh-gcrypt_threads.so.4
libflite_cmu_grapheme_lang.so.2.1       libssh-gcrypt_threads.so.4.5.0
libflite_cmu_grapheme_lex.so.1          libssl.so.1.1
libflite_cmu_grapheme_lex.so.2.1        libstdc++.so.6
libflite_cmu_indic_lang.so.1            libstdc++.so.6.0.25
libflite_cmu_indic_lang.so.2.1          libswresample.so.2
libflite_cmu_indic_lex.so.1             libswresample.so.2.9.100
libflite_cmu_indic_lex.so.2.1           libswscale.so.4
libflite_cmu_time_awb.so.1              libswscale.so.4.8.100
libflite_cmu_time_awb.so.2.1            libsz.a
libflite_cmu_us_awb.so.1                libsz.so
libflite_cmu_us_awb.so.2.1              libsz.so.2
libflite_cmu_us_kal.so.1                libsz.so.2.0.1
libflite_cmu_us_kal.so.2.1              libtag.so.1
libflite_cmu_us_kal16.so.1              libtag.so.1.17.0
libflite_cmu_us_kal16.so.2.1            libtasn1.so.6
libflite_cmu_us_rms.so.1                libtasn1.so.6.5.5
libflite_cmu_us_rms.so.2.1              libtdb.so.1
libflite_cmu_us_slt.so.1                libtdb.so.1.3.15
libflite_cmu_us_slt.so.2.1              libtermcap.a
libflite_cmulex.so.1                    libtermcap.so
libflite_cmulex.so.2.1                  libthai.so.0
libflite_usenglish.so.1                 libthai.so.0.3.0
libflite_usenglish.so.2.1               libtheora.so.0
libfluidsynth.so.1                      libtheora.so.0.3.10
libfluidsynth.so.1.7.0                  libtheoradec.so.1
libfontconfig.so.1                      libtheoradec.so.1.1.4
libfontconfig.so.1.10.1                 libtheoraenc.so.1
libform.so.5                            libtheoraenc.so.1.1.2
libform.so.5.9                          libthread_db.so
libformw.so.5                           libtic.a
libformw.so.5.9                         libtic.so
libfreetype.so.6                        libtic.so.5
libfreetype.so.6.15.0                   libtic.so.5.9
libfribidi.so.0                         libtiff.so.5
libfribidi.so.0.3.6                     libtiff.so.5.3.0
libg.a                                  libtinfo.a
libgbm.so.1                             libtinfo.so
libgbm.so.1.0.0                         libtsan.so.0
libgdbm.so.5                            libtsan.so.0.0.0
libgdbm.so.5.0.0                        libtsan_preinit.o
libgdbm_compat.so.4                     libtwolame.so.0
libgdbm_compat.so.4.0.0                 libtwolame.so.0.0.0
libgdk-3.so.0                           libubsan.so.0
libgdk-3.so.0.2200.30                   libubsan.so.0.0.0
libgdk_pixbuf-2.0.so.0                  libunistring.so.2
libgdk_pixbuf-2.0.so.0.3611.0           libunistring.so.2.1.0
libgdk_pixbuf_xlib-2.0.so.0             libutil.a
libgdk_pixbuf_xlib-2.0.so.0.3611.0      libutil.so
libgfortran.so.4                        libuv.so.1
libgfortran.so.4.0.0                    libuv.so.1.0.0
libgio-2.0.so.0                         libv4l
libgio-2.0.so.0.5600.4                  libv4l1.so.0
libglapi.so.0                           libv4l1.so.0.0.0
libglapi.so.0.0.0                       libv4l2.so.0
libglib-2.0.so.0                        libv4l2.so.0.0.0
libglib-2.0.so.0.5600.4                 libv4l2.so.0.0.999999
libglut.a                               libv4lconvert.so.0
libglut.so                              libv4lconvert.so.0.0.0
libglut.so.3                            libv4lconvert.so.0.0.999999
libglut.so.3.9.0                        libv4lconvert0
libgme.so.0                             libva-drm.so.2
libgme.so.0.6.2                         libva-drm.so.2.100.0
libgmodule-2.0.so.0                     libva-x11.so.2
libgmodule-2.0.so.0.5600.4              libva-x11.so.2.100.0
libgmp.so.10                            libva.so.2
libgmp.so.10.3.2                        libva.so.2.100.0
libgnutls.so.30                         libvdpau.so.1
libgnutls.so.30.14.10                   libvdpau.so.1.0.0
libgobject-2.0.so.0                     libvisual-0.4.so.0
libgobject-2.0.so.0.5600.4              libvisual-0.4.so.0.0.0
libgomp.so.1                            libvo-aacenc.so.0
libgomp.so.1.0.0                        libvo-aacenc.so.0.0.4
libgpm.so.2                             libvo-amrwbenc.so.0
libgraphite2.so.2.0.0                   libvo-amrwbenc.so.0.0.4
libgraphite2.so.3                       libvorbis.so.0
libgraphite2.so.3.0.1                   libvorbis.so.0.4.8
libgsm.so.1                             libvorbisenc.so.2
libgsm.so.1.0.12                        libvorbisenc.so.2.0.11
libgssapi.so.3                          libvorbisfile.so.3
libgssapi.so.3.0.0                      libvorbisfile.so.3.3.7
libgssapi_krb5.so.2                     libvpx.so.5
libgssapi_krb5.so.2.2                   libvpx.so.5.0
libgssdp-1.0.so.3                       libvpx.so.5.0.0
libgssdp-1.0.so.3.0.1                   libvulkan.so.1
libgstadaptivedemux-1.0.so.0            libvulkan.so.1.1.70
libgstadaptivedemux-1.0.so.0.1405.0     libvulkan.so.1.2.141
libgstallocators-1.0.so.0               libwavpack.so.1
libgstallocators-1.0.so.0.1405.0        libwavpack.so.1.2.0
libgstapp-1.0.so.0                      libwayland-client.so.0
libgstapp-1.0.so.0.1405.0               libwayland-client.so.0.3.0
libgstaudio-1.0.so.0                    libwayland-cursor.so.0
libgstaudio-1.0.so.0.1405.0             libwayland-cursor.so.0.0.0
libgstbadaudio-1.0.so.0                 libwayland-egl.so.1
libgstbadaudio-1.0.so.0.1405.0          libwayland-egl.so.1.0.0
libgstbadvideo-1.0.so.0                 libwayland-server.so.0
libgstbadvideo-1.0.so.0.1405.0          libwayland-server.so.0.1.0
libgstbase-1.0.so.0                     libwebp.so.6
libgstbase-1.0.so.0.1405.0              libwebp.so.6.0.2
libgstbasecamerabinsrc-1.0.so.0         libwebpmux.so.3
libgstbasecamerabinsrc-1.0.so.0.1405.0  libwebpmux.so.3.0.1
libgstcheck-1.0.so.0                    libwebrtc_audio_processing.so.1
libgstcheck-1.0.so.0.1405.0             libwebrtc_audio_processing.so.1.0.0
libgstcodecparsers-1.0.so.0             libwind.so.0
libgstcodecparsers-1.0.so.0.1405.0      libwind.so.0.0.0
libgstcontroller-1.0.so.0               libx264.so.152
libgstcontroller-1.0.so.0.1405.0        libx265.so.146
libgstfft-1.0.so.0                      libxcb-dri2.a
libgstfft-1.0.so.0.1405.0               libxcb-dri2.so
libgstgl-1.0.so.0                       libxcb-dri2.so.0
libgstgl-1.0.so.0.1405.0                libxcb-dri2.so.0.0.0
libgstinsertbin-1.0.so.0                libxcb-dri3.a
libgstinsertbin-1.0.so.0.1405.0         libxcb-dri3.so
libgstisoff-1.0.so.0                    libxcb-dri3.so.0
libgstisoff-1.0.so.0.1405.0             libxcb-dri3.so.0.0.0
libgstmpegts-1.0.so.0                   libxcb-glx.a
libgstmpegts-1.0.so.0.1405.0            libxcb-glx.so
libgstnet-1.0.so.0                      libxcb-glx.so.0
libgstnet-1.0.so.0.1405.0               libxcb-glx.so.0.0.0
libgstnvegl-1.0.so.0                    libxcb-present.a
libgstnvexifmeta.so                     libxcb-present.so
libgstnvivameta.so                      libxcb-present.so.0
libgstpbutils-1.0.so.0                  libxcb-present.so.0.0.0
libgstpbutils-1.0.so.0.1405.0           libxcb-randr.a
libgstphotography-1.0.so.0              libxcb-randr.so
libgstphotography-1.0.so.0.1405.0       libxcb-randr.so.0
libgstplayer-1.0.so.0                   libxcb-randr.so.0.1.0
libgstplayer-1.0.so.0.1405.0            libxcb-render.a
libgstreamer-1.0.so.0                   libxcb-render.so
libgstreamer-1.0.so.0.1405.0            libxcb-render.so.0
libgstriff-1.0.so.0                     libxcb-render.so.0.0.0
libgstriff-1.0.so.0.1405.0              libxcb-shape.a
libgstrtp-1.0.so.0                      libxcb-shape.so
libgstrtp-1.0.so.0.1405.0               libxcb-shape.so.0
libgstrtsp-1.0.so.0                     libxcb-shape.so.0.0.0
libgstrtsp-1.0.so.0.1405.0              libxcb-shm.so.0
libgstsdp-1.0.so.0                      libxcb-shm.so.0.0.0
libgstsdp-1.0.so.0.1405.0               libxcb-sync.a
libgsttag-1.0.so.0                      libxcb-sync.so
libgsttag-1.0.so.0.1405.0               libxcb-sync.so.1
libgsturidownloader-1.0.so.0            libxcb-sync.so.1.0.0
libgsturidownloader-1.0.so.0.1405.0     libxcb-xfixes.a
libgstvideo-1.0.so.0                    libxcb-xfixes.so
libgstvideo-1.0.so.0.1405.0             libxcb-xfixes.so.0
libgstwayland-1.0.so.0                  libxcb-xfixes.so.0.0.0
libgstwayland-1.0.so.0.1405.0           libxcb.a
libgstwebrtc-1.0.so.0                   libxcb.so
libgstwebrtc-1.0.so.0.1405.0            libxcb.so.1
libgthread-2.0.so.0                     libxcb.so.1.1.0
libgthread-2.0.so.0.5600.4              libxkbcommon.so.0
libgtk-3-0                              libxkbcommon.so.0.0.0
libgtk-3.so.0                           libxml2.so.2
libgtk-3.so.0.2200.30                   libxml2.so.2.9.4
libgudev-1.0.so.0                       libxshmfence.a
libgudev-1.0.so.0.2.0                   libxshmfence.so
libgupnp-1.0.so.4                       libxshmfence.so.1
libgupnp-1.0.so.4.0.1                   libxshmfence.so.1.0.0
libgupnp-igd-1.0.so.4                   libxvidcore.so.4
libgupnp-igd-1.0.so.4.2.0               libxvidcore.so.4.3
libharfbuzz.so.0                        libyaml-0.so.2
libharfbuzz.so.0.10702.0                libyaml-0.so.2.0.5
libhcrypto.so.4                         libz.a
libhcrypto.so.4.1.0                     libz.so
libhdf5_cpp.a                           libzbar.so.0
libhdf5_cpp.so                          libzbar.so.0.2.0
libhdf5_cpp.so.100                      libzmq.so.5
libhdf5_cpp.so.100.0.0                  libzmq.so.5.1.5
libhdf5_hl_cpp.a                        libzstd.so.1
libhdf5_hl_cpp.so                       libzstd.so.1.3.3
libhdf5_hl_cpp.so.100                   libzvbi-chains.so.0
libhdf5_hl_cpp.so.100.0.0               libzvbi-chains.so.0.0.0
libhdf5_serial.a                        libzvbi.so.0
libhdf5_serial.settings                 libzvbi.so.0.13.2
libhdf5_serial.so                       openblas
libhdf5_serial.so.100                   openmpi
libhdf5_serial.so.100.0.1               perl
libhdf5_serial_fortran.a                perl-base
libhdf5_serial_fortran.so               perl5
libhdf5_serial_fortran.so.100           pkgconfig
libhdf5_serial_fortran.so.100.0.1       pulseaudio
libhdf5_serial_hl.a                     sasl2
libhdf5_serial_hl.so                    sox
libhdf5_serial_hl.so.100                tegra
libhdf5_serial_hl.so.100.0.0            tegra-egl
libhdf5_serialhl_fortran.a              vdpau
libhdf5_serialhl_fortran.so             x264-10bit
(venv) root@166d0c1b769d:/# 

Can you paste the output of cat /etc/nvidia-container-runtime/host-files-for-container.d/cudnn.csv from your device. It should list those cudnn libraries from your device that are supposed to be mounted into the container.

I think I found the fix but not the issue. The fix is to use nvcr.io/nvidia/l4t-base:r32.4.4 as a base image instead of nvcr.io/nvidia/l4t-cuda:10.2.460-runtime

I’m not as familiar with the cuda runtime image and I thought using it would be good but perhaps that is more for cuda development? Anyways the GPU works with pytorch/tensorflow w/ the flags --runtime=nvidia or --gpus=all when I use the l4t-base image.