I am working on creating a Docker image for my Xavier similar to this one here:
# Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.
This file has been truncated. show original
I am running Jetpack 4.6 and will be using Pytorch and Tensorflow in the container.
The image is being built using a gitlab cicd runner with 64 bit arm based GPUs nodes in AWS. Node details:
“Amazon EC2 G5g instances are powered by AWS Graviton2 processors and feature NVIDIA T4G Tensor Core GPUs to provide the best price performance in Amazon EC2 for graphics workloads such as Android game streaming. They are the first Arm-based instances in a major cloud to feature GPU acceleration. Customers can also use G5g instances for cost-effective ML inference.”
The problem I am running into is that my build succeeds, then I go to run the container on our Xavier devkit. I can run the original l4t Pytorch image and see that torch is using the GPU. I can see all the shared object files are in the requisite place when the container is run with --gpus=all
SO path when base pytorch image is run w/o --gpus flag
dso-dat-dev3@dso-dat-dev3:~$ docker run -it --rm nvcr.io/nvidia/l4t-pytorch:r32.6.1-pth1.9-py3 /bin/bash
root@aee59d62865b:/# ls /usr/local/cuda-10.2/targets/aarch64-linux/lib
libcudadevrt.a libcudart_static.a stubs
SO path when base pytorch image is run w/ --gpus flag
dso-dat-dev3@dso-dat-dev3:~$ docker run -it --rm --gpus=all nvcr.io/nvidia/l4t-pytorch:r32.6.1-pth1.9-py3 /bin/bash
root@4d96784d3f18:/# ls /usr/local/cuda-10.2/targets/aarch64-linux/lib
libcublas.so libcudnn_static.a libnppc.so.10.2.1.300 libnppisu.so
libcublas.so.10 libcudnn_static_v8.a libnppc_static.a libnppisu.so.10
libcublas.so.10.2.3.300 libcufft.so libnppial.so libnppisu.so.10.2.1.300
libcublasLt.so libcufft.so.10 libnppial.so.10 libnppisu_static.a
libcublasLt.so.10 libcufft.so.10.1.2.300 libnppial.so.10.2.1.300 libnppitc.so
libcublasLt.so.10.2.3.300 libcufft_static.a libnppial_static.a libnppitc.so.10
libcublasLt_static.a libcufft_static_nocallback.a libnppicc.so libnppitc.so.10.2.1.300
libcublas_static.a libcufftw.so libnppicc.so.10 libnppitc_static.a
libcudadevrt.a libcufftw.so.10 libnppicc.so.10.2.1.300 libnpps.so
libcudart.so libcufftw.so.10.1.2.300 libnppicc_static.a libnpps.so.10
libcudart.so.10.2 libcufftw_static.a libnppicom.so libnpps.so.10.2.1.300
libcudart.so.10.2.300 libcuinj64.so libnppicom.so.10 libnpps_static.a
libcudart_static.a libcuinj64.so.10.2 libnppicom.so.10.2.1.300 libnvToolsExt.so
libcudnn.so libcuinj64.so.10.2.300 libnppicom_static.a libnvToolsExt.so.1
libcudnn.so.8 libculibos.a libnppidei.so libnvToolsExt.so.1.0.0
libcudnn.so.8.2.1 libcupti.so libnppidei.so.10 libnvblas.so
libcudnn_adv_infer.so libcupti.so.10.2 libnppidei.so.10.2.1.300 libnvblas.so.10
libcudnn_adv_infer.so.8 libcupti.so.10.2.175 libnppidei_static.a libnvblas.so.10.2.3.300
libcudnn_adv_infer.so.8.2.1 libcurand.so libnppif.so libnvgraph.so
libcudnn_adv_train.so libcurand.so.10 libnppif.so.10 libnvgraph.so.10
libcudnn_adv_train.so.8 libcurand.so.10.1.2.300 libnppif.so.10.2.1.300 libnvgraph.so.10.2.300
libcudnn_adv_train.so.8.2.1 libcurand_static.a libnppif_static.a libnvgraph_static.a
libcudnn_cnn_infer.so libcusolver.so libnppig.so libnvperf_host.so
libcudnn_cnn_infer.so.8 libcusolver.so.10 libnppig.so.10 libnvperf_target.so
libcudnn_cnn_infer.so.8.2.1 libcusolver.so.10.3.0.300 libnppig.so.10.2.1.300 libnvrtc-builtins.so
libcudnn_cnn_train.so libcusolver_static.a libnppig_static.a libnvrtc-builtins.so.10.2
libcudnn_cnn_train.so.8 libcusparse.so libnppim.so libnvrtc-builtins.so.10.2.300
libcudnn_cnn_train.so.8.2.1 libcusparse.so.10 libnppim.so.10 libnvrtc.so
libcudnn_ops_infer.so libcusparse.so.10.3.1.300 libnppim.so.10.2.1.300 libnvrtc.so.10.2
libcudnn_ops_infer.so.8 libcusparse_static.a libnppim_static.a libnvrtc.so.10.2.300
libcudnn_ops_infer.so.8.2.1 liblapack_static.a libnppist.so stubs
libcudnn_ops_train.so libmetis_static.a libnppist.so.10
libcudnn_ops_train.so.8 libnppc.so libnppist.so.10.2.1.300
libcudnn_ops_train.so.8.2.1 libnppc.so.10 libnppist_static.a
SO path in my custom image w/ --gpus flag
dso-dat-dev3@dso-dat-dev3:~$ docker run -it --rm --gpus=all <our ecr uri>/ai-platform-nvidia-conda-ml:5e13d2e0-arm64-gpu bash
(base) root@c4e0876d6d2c:/# ls /usr/local/cuda-10.2/targets/aarch64-linux/lib
libcublas.so.10 libcufftw.so.10.1.2.300 libnppicc.so.10 libnppim.so.10.2.1.300 libnvToolsExt.so.1.0.0
libcublas.so.10.2.3.300 libcurand.so.10 libnppicc.so.10.2.1.300 libnppist.so.10 libnvblas.so.10
libcublasLt.so.10 libcurand.so.10.1.2.300 libnppicom.so.10 libnppist.so.10.2.1.300 libnvblas.so.10.2.3.300
libcublasLt.so.10.2.3.300 libcusolver.so.10 libnppicom.so.10.2.1.300 libnppisu.so.10 libnvgraph.so.10
libcudadevrt.a libcusolver.so.10.3.0.300 libnppidei.so.10 libnppisu.so.10.2.1.300 libnvgraph.so.10.2.300
libcudart.so.10.2 libcusparse.so.10 libnppidei.so.10.2.1.300 libnppitc.so.10 libnvrtc-builtins.so
libcudart.so.10.2.300 libcusparse.so.10.3.1.300 libnppif.so.10 libnppitc.so.10.2.1.300 libnvrtc-builtins.so.10.2
libcudart_static.a libnppc.so.10 libnppif.so.10.2.1.300 libnpps.so.10 libnvrtc-builtins.so.10.2.300
libcufft.so.10 libnppc.so.10.2.1.300 libnppig.so.10 libnpps.so.10.2.1.300 libnvrtc.so.10.2
libcufft.so.10.1.2.300 libnppial.so.10 libnppig.so.10.2.1.300 libnvToolsExt.so libnvrtc.so.10.2.300
libcufftw.so.10 libnppial.so.10.2.1.300 libnppim.so.10 libnvToolsExt.so.1 stubs
The subset of shared objects here is causing pytorch fail at import with an OSerror related to libcudnn.so.8 .
Perhaps I am not doing something correctly, but the docker image I am building is based closely on the one linked above. The base image I am using is: nvcr. io/nvidia/l4t-cuda:10.2.460-runtime whereas the one in the build above is nvcr. io/nvidia/l4t-base:r32.4.4 . But this should not make a different right?
Since your docker host is running JP 4.6, the cuda/cudnn/etc libraries are actually supposed to be mounted from the host. Check /etc/nvidia-container-runtime/host-files-for-container.d directory.
You should see those files: cuda.csv cudnn.csv l4t.csv tensorrt.csv They define what files/dir are volume mounted into the container. In your case, libcudnn.so.8 is from cudnn.csv (came from package nvidia-container-csv-cudnn),
Hi @user100090 thank you for the reply. I have looked at this files in /etc/nvidia-container-runtime/host-files-for-container.d and unfortunately can’t figure out why I’m not seeing the correct shared objects files in my custom container. The correct ones are shown above when I ls the directory /usr/local/cuda-10.2/targets/aarch64-linux/lib for the Pytorch container run with the gpus flag. It seems like they aren’t being mounted correctly when I run my custom container with the GPUs flag.
Hi,
Would you mind replacing '–gpus=all with '--runtime nvidia
to see if it works?
$ sudo docker run -it --rm --runtime nvidia --network host nvcr.io/nvidia/l4t-pytorch:r32.6.1-pth1.9-py3
More, maybe you just check the incorrect repository.
For example, I can find them in the /usr/lib/aarch64-linux-gnu/
.
# ls /usr/lib/aarch64-linux-gnu/libcudnn*
/usr/lib/aarch64-linux-gnu/libcudnn.so /usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so.8.2.1 /usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so.8.2.1
/usr/lib/aarch64-linux-gnu/libcudnn.so.8 /usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so /usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so
/usr/lib/aarch64-linux-gnu/libcudnn.so.8.2.1 /usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8 /usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so.8
/usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so /usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8.2.1 /usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so.8.2.1
/usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so.8 /usr/lib/aarch64-linux-gnu/libcudnn_cnn_train.so.8 /usr/lib/aarch64-linux-gnu/libcudnn_static.a
/usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so.8.2.1 /usr/lib/aarch64-linux-gnu/libcudnn_cnn_train.so.8.2.1 /usr/lib/aarch64-linux-gnu/libcudnn_static_v8.a
/usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so /usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so
/usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so.8 /usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so.8
Thanks.
I see those files on the device root filesystem. They are definitely there.
but when I run the container I don’t see them.
dso-dat-dev2@ubuntu:~$ docker run -it --runtime=nvidia <our ecr uri>/ai-platform-nvidia-conda-ml:7e6ee134-arm64-gpu /bin/bash
(base) root@166d0c1b769d:/# conda activate venv
(venv) root@166d0c1b769d:/# python
Python 3.6.15 | packaged by conda-forge | (default, Dec 3 2021, 19:12:04)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/conda/envs/venv/lib/python3.6/site-packages/torch/__init__.py", line 196, in <module>
_load_global_deps()
File "/opt/conda/envs/venv/lib/python3.6/site-packages/torch/__init__.py", line 149, in _load_global_deps
ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
File "/opt/conda/envs/venv/lib/python3.6/ctypes/__init__.py", line 348, in __init__
self._handle = _dlopen(self._name, mode)
OSError: libcudnn.so.8: cannot open shared object file: No such file or directory
>>>
(venv) root@166d0c1b769d:/# ls /usr/lib/aarch64-linux-gnu/
Mcrt1.o libhdf5_serialhl_fortran.so.100
Scrt1.o libhdf5_serialhl_fortran.so.100.0.0
audit libheimbase.so.1
avahi libheimbase.so.1.0.0
blas libheimntlm.so.0
caca libheimntlm.so.0.1.0
coreutils libhogweed.so.4
crt1.o libhogweed.so.4.5
crti.o libhwloc.so.0
crtn.o libhwloc.so.1
dri libhwloc.so.2
engines-1.1 libhwloc.so.3
gconv libhwloc.so.4
gcrt1.o libhwloc.so.5
gdk-pixbuf-2.0 libhwloc.so.5.7.6
gio libhx509.so.5
glib-2.0 libhx509.so.5.0.0
gstreamer-1.0 libi2c.so.0
gstreamer1.0 libi2c.so.0.1.0
gtk-3.0 libibverbs.so.1
hdf5 libibverbs.so.1.1.17.1
hwloc libicudata.so.60
krb5 libicudata.so.60.2
lapack libicui18n.so.60
ldscripts libicui18n.so.60.2
libBrokenLocale.a libicuio.so.60
libBrokenLocale.so libicuio.so.60.2
libEGL.so libicutest.so.60
libEGL.so.1 libicutest.so.60.2
libEGL.so.1.0.0 libicutu.so.60
libEGL_mesa.so.0 libicutu.so.60.2
libEGL_mesa.so.0.0.0 libicuuc.so.60
libFLAC.so.8 libicuuc.so.60.2
libFLAC.so.8.3.0 libidn2.so.0
libGL.so libidn2.so.0.3.3
libGL.so.1 libiec61883.so.0
libGL.so.1.0.0 libiec61883.so.0.1.1
libGLESv1_CM.so libisl.so.19
libGLESv1_CM.so.1 libisl.so.19.0.0
libGLESv1_CM.so.1.0.0 libitm.so.1
libGLESv2.so libitm.so.1.0.0
libGLESv2.so.2 libjack.so.0
libGLESv2.so.2.0.0 libjack.so.0.1.0
libGLU.a libjacknet.so.0
libGLU.so libjacknet.so.0.1.0
libGLU.so.1 libjackserver.so.0
libGLU.so.1.3.1 libjackserver.so.0.1.0
libGLX.so libjbig.so.0
libGLX.so.0 libjpeg.a
libGLX.so.0.0.0 libjpeg.so
libGLX_indirect.so.0 libjpeg.so.8
libGLX_mesa.so.0 libjpeg.so.8.1.2
libGLX_mesa.so.0.0.0 libjson-glib-1.0.so.0
libGLdispatch.so libjson-glib-1.0.so.0.400.2
libGLdispatch.so.0 libjsoncpp.so.1
libGLdispatch.so.0.0.0 libjsoncpp.so.1.7.4
libHalf.so.12 libk5crypto.so.3
libHalf.so.12.0.0 libk5crypto.so.3.1
libICE.a libkate.so.1
libICE.so libkate.so.1.3.0
libICE.so.6 libkrb5.so.26
libICE.so.6.3.0 libkrb5.so.26.0.0
libIex-2_2.so.12 libkrb5.so.3
libIex-2_2.so.12.0.0 libkrb5.so.3.3
libIexMath-2_2.so.12 libkrb5support.so.0
libIexMath-2_2.so.12.0.0 libkrb5support.so.0.1
libIlmImf-2_2.so.22 libksba.so.8
libIlmImf-2_2.so.22.0.0 libksba.so.8.11.6
libIlmImfUtil-2_2.so.22 liblapack.a
libIlmImfUtil-2_2.so.22.0.0 liblapack.so
libIlmThread-2_2.so.12 liblapack.so.3
libIlmThread-2_2.so.12.0.0 liblber-2.4.so.2
libImath-2_2.so.12 liblber-2.4.so.2.10.8
libImath-2_2.so.12.0.0 liblcms2.so.2
libLLVM-10.so liblcms2.so.2.0.8
libLLVM-10.so.1 libldap-2.4.so.2
libLLVM-9.0.0.so.1 libldap_r-2.4.so.2
libLLVM-9.so libldap_r-2.4.so.2.10.8
libLLVM-9.so.1 liblsan.so.0
libOpenCL.so.1 liblsan.so.0.0.0
libOpenCL.so.1.0.0 libltdl.so.7
libOpenGL.so libltdl.so.7.3.1
libOpenGL.so.0 liblz4.so.1
libOpenGL.so.0.0.0 liblz4.so.1.7.1
libSM.a libm.a
libSM.so libm.so
libSM.so.6 libmagic.so.1
libSM.so.6.0.1 libmagic.so.1.0.0
libSoundTouch.so.1 libmca_common_sm.so.20
libSoundTouch.so.1.0.0 libmca_common_sm.so.20.10.1
libWildMidi.so.2 libmca_common_verbs.so.20
libWildMidi.so.2.0.1 libmca_common_verbs.so.20.10.0
libX11-xcb.a libmcheck.a
libX11-xcb.so libmenu.so.5
libX11-xcb.so.1 libmenu.so.5.9
libX11-xcb.so.1.0.0 libmenuw.so.5
libX11.a libmenuw.so.5.9
libX11.so libmjpegutils-2.1.so.0
libX11.so.6 libmjpegutils-2.1.so.0.0.0
libX11.so.6.3.0 libmms.so.0
libXau.a libmms.so.0.0.2
libXau.so libmodplug.so.1
libXau.so.6 libmodplug.so.1.0.0
libXau.so.6.0.0 libmp3lame.so.0
libXcomposite.so.1 libmp3lame.so.0.0.0
libXcomposite.so.1.0.0 libmpc.so.3
libXcursor.so.1 libmpc.so.3.1.0
libXcursor.so.1.0.2 libmpcdec.so.6
libXdamage.a libmpcdec.so.6.1.0
libXdamage.so libmpdec.so.2
libXdamage.so.1 libmpdec.so.2.4.2
libXdamage.so.1.1.0 libmpeg2.so.0
libXdmcp.a libmpeg2.so.0.1.0
libXdmcp.so libmpeg2convert.so.0
libXdmcp.so.6 libmpeg2convert.so.0.0.0
libXdmcp.so.6.0.0 libmpeg2encpp-2.1.so.0
libXext.a libmpeg2encpp-2.1.so.0.0.0
libXext.so libmpfr.so.6
libXext.so.6 libmpfr.so.6.0.1
libXext.so.6.4.0 libmpg123.so.0
libXfixes.a libmpg123.so.0.44.8
libXfixes.so libmpi.so.20
libXfixes.so.3 libmpi.so.20.10.1
libXfixes.so.3.1.0 libmpi_cxx.so.20
libXi.so.6 libmpi_cxx.so.20.10.0
libXi.so.6.1.0 libmpi_java.so.20
libXinerama.so.1 libmpi_java.so.20.10.0
libXinerama.so.1.0.0 libmpi_mpifh.so.20
libXrandr.so.2 libmpi_mpifh.so.20.11.0
libXrandr.so.2.2.0 libmpi_usempi_ignore_tkr.so.20
libXrender.so.1 libmpi_usempi_ignore_tkr.so.20.10.0
libXrender.so.1.3.0 libmpi_usempif08.so.20
libXt.a libmpi_usempif08.so.20.10.0
libXt.so libmplex2-2.1.so.0
libXt.so.6 libmplex2-2.1.so.0.0.0
libXt.so.6.0.0 libmysofa.so.0
libXxf86vm.a libmysofa.so.0.5.1
libXxf86vm.so libnettle.so.6
libXxf86vm.so.1 libnettle.so.6.5
libXxf86vm.so.1.0.0 libnghttp2.so.14
liba52-0.7.4.so libnghttp2.so.14.15.2
libaa.so.1 libnice.so.10
libaa.so.1.0.4 libnice.so.10.7.0
libaec.a libnl-route-3.so.200
libaec.so libnl-route-3.so.200.24.0
libaec.so.0 libnorm.so.1
libaec.so.0.0.3 libnorm.so.1.0.0
libanl.a libnpth.so.0
libanl.so libnpth.so.0.1.1
libapt-inst.so.2.0 libnsl.a
libapt-inst.so.2.0.0 libnsl.so
libapt-pkg.so.5.0 libnss_compat.so
libapt-pkg.so.5.0.2 libnss_dns.so
libapt-private.so.0.0 libnss_files.so
libapt-private.so.0.0.0 libnss_hesiod.so
libarchive.so.13 libnss_nis.so
libarchive.so.13.2.2 libnss_nisplus.so
libasan.so.4 libnuma.so.1
libasan.so.4.0.0 libnuma.so.1.0.0
libasn1.so.8 libnvsample_cudaprocess.so
libasn1.so.8.0.0 libofa.so.0
libasound.so.2 libofa.so.0.0.0
libasound.so.2.0.0 libogg.so.0
libass.so.9 libogg.so.0.8.2
libass.so.9.0.2 libompitrace.so.20
libassuan.so.0 libompitrace.so.20.10.0
libassuan.so.0.8.1 libopcodes-2.30-system.so
libasyncns.so.0 libopen-pal.so.20
libasyncns.so.0.3.1 libopen-pal.so.20.10.1
libatk-1.0.so.0 libopen-rte.so.20
libatk-1.0.so.0.22810.1 libopen-rte.so.20.10.1
libatk-bridge-2.0.so.0 libopenal.so.1
libatk-bridge-2.0.so.0.0.0 libopenal.so.1.18.2
libatomic.so.1 libopenblas.a
libatomic.so.1.2.0 libopenblas.so
libatspi.so.0 libopenblas.so.0
libatspi.so.0.0.1 libopenblas_armv8p-r0.2.20.a
libavahi-client.so.3 libopenblas_armv8p-r0.2.20.so
libavahi-client.so.3.2.9 libopencore-amrnb.so.0
libavahi-common.so.3 libopencore-amrnb.so.0.0.3
libavahi-common.so.3.5.3 libopencore-amrwb.so.0
libavc1394.so.0 libopencore-amrwb.so.0.0.3
libavc1394.so.0.3.0 libopenjp2.so.2.3.0
libavcodec.so.57 libopenjp2.so.7
libavcodec.so.57.107.100 libopenmpt.so.0
libavfilter.so.6 libopenmpt.so.0.1.1
libavfilter.so.6.107.100 libopus.so.0
libavformat.so.57 libopus.so.0.5.2
libavformat.so.57.83.100 liborc-0.4.so.0
libavresample.so.3 liborc-0.4.so.0.28.0
libavresample.so.3.7.0 liborc-test-0.4.so.0
libavutil.so.55 liborc-test-0.4.so.0.28.0
libavutil.so.55.78.100 liboshmem.so.20
libbfd-2.30-system.so libp11-kit.so.0
libblas.a libp11-kit.so.0.3.0
libblas.so libpanel.so.5
libblas.so.3 libpanel.so.5.9
libbluray.so.2 libpanelw.so.5
libbluray.so.2.0.2 libpanelw.so.5.9
libbs2b.so.0 libpango-1.0.so.0
libbs2b.so.0.0.0 libpango-1.0.so.0.4000.14
libc.a libpangocairo-1.0.so.0
libc.so libpangocairo-1.0.so.0.4000.14
libc_nonshared.a libpangoft2-1.0.so.0
libcaca++.so.0 libpangoft2-1.0.so.0.4000.14
libcaca++.so.0.99.19 libpciaccess.so.0
libcaca.so.0 libpciaccess.so.0.11.1
libcaca.so.0.99.19 libpcreposix.so.3
libcairo-gobject.so.2 libpcreposix.so.3.13.3
libcairo-gobject.so.2.11510.0 libpcsclite.so.1
libcairo.so.2 libpcsclite.so.1.0.0
libcairo.so.2.11510.0 libperl.so.5.26
libcanberra-0.30 libperl.so.5.26.1
libcanberra-gtk3.so.0 libpfm.so.4
libcanberra-gtk3.so.0.1.9 libpfm.so.4.9.0
libcanberra.so.0 libpgm-5.2.so.0
libcanberra.so.0.2.5 libpgm-5.2.so.0.0.122
libcc1.so.0 libpipeline.so.1
libcc1.so.0.0.0 libpipeline.so.1.5.0
libcdda_interface.so.0 libpixman-1.so.0
libcdda_interface.so.0.10.2 libpixman-1.so.0.34.0
libcdda_paranoia.so.0 libpng16.so.16
libcdda_paranoia.so.0.10.2 libpng16.so.16.34.0
libcdio.so.17 libpostproc.so.54
libcdio.so.17.0.0 libpostproc.so.54.7.100
libchromaprint.so.1 libprotobuf-lite.a
libchromaprint.so.1.4.3 libprotobuf-lite.so
libcidn.so libprotobuf-lite.so.10
libcolord.so.2 libprotobuf-lite.so.10.0.0
libcolord.so.2.0.5 libprotobuf.a
libcolordprivate.so.2 libprotobuf.so
libcolordprivate.so.2.0.5 libprotobuf.so.10
libcroco-0.6.so.3 libprotobuf.so.10.0.0
libcroco-0.6.so.3.0.1 libprotoc.a
libcrypt.a libprotoc.so
libcrypt.so libprotoc.so.10
libcrypto.so.1.1 libprotoc.so.10.0.0
libcuda.so libproxy.so.1
libcups.so.2 libproxy.so.1.0.0
libcurl-gnutls.so.3 libpsl.so.5
libcurl-gnutls.so.4 libpsl.so.5.2.0
libcurl-gnutls.so.4.5.0 libpthread.a
libcurl.so.4 libpthread.so
libcurl.so.4.5.0 libpthread_nonshared.a
libdatrie.so.1 libpulse-simple.so.0
libdatrie.so.1.3.3 libpulse-simple.so.0.1.1
libdb-5.3.so libpulse.so.0
libdc1394.so.22 libpulse.so.0.20.2
libdc1394.so.22.2.1 libpython2.7.a
libdca.so.0 libpython2.7.so
libdca.so.0.0.0 libpython2.7.so.1
libdconf.so.1 libpython2.7.so.1.0
libdconf.so.1.0.0 libraw1394.so.11
libde265.so.0 libraw1394.so.11.1.0
libde265.so.0.0.10 libresolv.a
libdebconfclient.so.0 libresolv.so
libdebconfclient.so.0.0.0 librest-0.7.so.0
libdl.a librest-0.7.so.0.0.0
libdl.so librhash.so.0
libdrm.so libroken.so.18
libdrm.so.2 libroken.so.18.1.0
libdrm.so.2.4.0 librom1394.so.0
libdrm_amdgpu.so librom1394.so.0.3.0
libdrm_amdgpu.so.1 librpcsvc.a
libdrm_amdgpu.so.1.0.0 librsvg-2.so.2
libdrm_etnaviv.so librsvg-2.so.2.40.20
libdrm_etnaviv.so.1 librt.a
libdrm_etnaviv.so.1.0.0 librt.so
libdrm_freedreno.so librtmp.so.1
libdrm_freedreno.so.1 librubberband.so.2
libdrm_freedreno.so.1.0.0 librubberband.so.2.1.0
libdrm_nouveau.so libsamplerate.so.0
libdrm_nouveau.so.2 libsamplerate.so.0.1.8
libdrm_nouveau.so.2.0.0 libsasl2.so.2
libdrm_nvdc.so libsasl2.so.2.0.25
libdrm_radeon.so libsbc.so.1
libdrm_radeon.so.1 libsbc.so.1.2.1
libdrm_radeon.so.1.0.1 libsemanage.so.1
libdrm_tegra.so libsensors.so.4
libdrm_tegra.so.0 libsensors.so.4.4.0
libdrm_tegra.so.0.0.0 libserd-0.so.0
libdv.so.4 libserd-0.so.0.28.0
libdv.so.4.0.3 libshine.so.3
libdvdnav.so.4 libshine.so.3.0.1
libdvdnav.so.4.2.0 libshout.so.3
libdvdread.so.4 libshout.so.3.2.0
libdvdread.so.4.2.0 libsidplay.so.1
libedit.so.2 libsidplay.so.1.0.3
libedit.so.2.0.56 libsnappy.so.1
libelf-0.170.so libsnappy.so.1.1.7
libelf.so.1 libsndfile.so.1
libepoxy.so.0 libsndfile.so.1.0.28
libepoxy.so.0.0.0 libsndio.so.6.1
libexpat.a libsodium.so.23
libexpat.so libsodium.so.23.1.0
libexpatw.a libsord-0.so.0
libexpatw.so libsord-0.so.0.16.0
libexpatw.so.1 libsoup-2.4.so.1
libexpatw.so.1.6.7 libsoup-2.4.so.1.8.0
libfaad.so.2 libsoup-gnome-2.4.so.1
libfaad.so.2.0.0 libsoup-gnome-2.4.so.1.8.0
libfaad_drm.so.2 libsox.so.3
libfaad_drm.so.2.0.0 libsox.so.3.0.0
libffi.a libsoxr.so.0
libffi.so libsoxr.so.0.1.1
libffi.so.6 libspandsp.so.2
libffi.so.6.0.4 libspandsp.so.2.0.0
libffi_pic.a libspeex.so.1
libfftw3.so.3 libspeex.so.1.5.0
libfftw3.so.3.5.7 libsqlite3.so.0
libfftw3_omp.so.3 libsqlite3.so.0.8.6
libfftw3_omp.so.3.5.7 libsratom-0.so.0
libfftw3_threads.so.3 libsratom-0.so.0.6.0
libfftw3_threads.so.3.5.7 libsrtp2.so.1
libflite.so.1 libssh-gcrypt.so.4
libflite.so.2.1 libssh-gcrypt.so.4.5.0
libflite_cmu_grapheme_lang.so.1 libssh-gcrypt_threads.so.4
libflite_cmu_grapheme_lang.so.2.1 libssh-gcrypt_threads.so.4.5.0
libflite_cmu_grapheme_lex.so.1 libssl.so.1.1
libflite_cmu_grapheme_lex.so.2.1 libstdc++.so.6
libflite_cmu_indic_lang.so.1 libstdc++.so.6.0.25
libflite_cmu_indic_lang.so.2.1 libswresample.so.2
libflite_cmu_indic_lex.so.1 libswresample.so.2.9.100
libflite_cmu_indic_lex.so.2.1 libswscale.so.4
libflite_cmu_time_awb.so.1 libswscale.so.4.8.100
libflite_cmu_time_awb.so.2.1 libsz.a
libflite_cmu_us_awb.so.1 libsz.so
libflite_cmu_us_awb.so.2.1 libsz.so.2
libflite_cmu_us_kal.so.1 libsz.so.2.0.1
libflite_cmu_us_kal.so.2.1 libtag.so.1
libflite_cmu_us_kal16.so.1 libtag.so.1.17.0
libflite_cmu_us_kal16.so.2.1 libtasn1.so.6
libflite_cmu_us_rms.so.1 libtasn1.so.6.5.5
libflite_cmu_us_rms.so.2.1 libtdb.so.1
libflite_cmu_us_slt.so.1 libtdb.so.1.3.15
libflite_cmu_us_slt.so.2.1 libtermcap.a
libflite_cmulex.so.1 libtermcap.so
libflite_cmulex.so.2.1 libthai.so.0
libflite_usenglish.so.1 libthai.so.0.3.0
libflite_usenglish.so.2.1 libtheora.so.0
libfluidsynth.so.1 libtheora.so.0.3.10
libfluidsynth.so.1.7.0 libtheoradec.so.1
libfontconfig.so.1 libtheoradec.so.1.1.4
libfontconfig.so.1.10.1 libtheoraenc.so.1
libform.so.5 libtheoraenc.so.1.1.2
libform.so.5.9 libthread_db.so
libformw.so.5 libtic.a
libformw.so.5.9 libtic.so
libfreetype.so.6 libtic.so.5
libfreetype.so.6.15.0 libtic.so.5.9
libfribidi.so.0 libtiff.so.5
libfribidi.so.0.3.6 libtiff.so.5.3.0
libg.a libtinfo.a
libgbm.so.1 libtinfo.so
libgbm.so.1.0.0 libtsan.so.0
libgdbm.so.5 libtsan.so.0.0.0
libgdbm.so.5.0.0 libtsan_preinit.o
libgdbm_compat.so.4 libtwolame.so.0
libgdbm_compat.so.4.0.0 libtwolame.so.0.0.0
libgdk-3.so.0 libubsan.so.0
libgdk-3.so.0.2200.30 libubsan.so.0.0.0
libgdk_pixbuf-2.0.so.0 libunistring.so.2
libgdk_pixbuf-2.0.so.0.3611.0 libunistring.so.2.1.0
libgdk_pixbuf_xlib-2.0.so.0 libutil.a
libgdk_pixbuf_xlib-2.0.so.0.3611.0 libutil.so
libgfortran.so.4 libuv.so.1
libgfortran.so.4.0.0 libuv.so.1.0.0
libgio-2.0.so.0 libv4l
libgio-2.0.so.0.5600.4 libv4l1.so.0
libglapi.so.0 libv4l1.so.0.0.0
libglapi.so.0.0.0 libv4l2.so.0
libglib-2.0.so.0 libv4l2.so.0.0.0
libglib-2.0.so.0.5600.4 libv4l2.so.0.0.999999
libglut.a libv4lconvert.so.0
libglut.so libv4lconvert.so.0.0.0
libglut.so.3 libv4lconvert.so.0.0.999999
libglut.so.3.9.0 libv4lconvert0
libgme.so.0 libva-drm.so.2
libgme.so.0.6.2 libva-drm.so.2.100.0
libgmodule-2.0.so.0 libva-x11.so.2
libgmodule-2.0.so.0.5600.4 libva-x11.so.2.100.0
libgmp.so.10 libva.so.2
libgmp.so.10.3.2 libva.so.2.100.0
libgnutls.so.30 libvdpau.so.1
libgnutls.so.30.14.10 libvdpau.so.1.0.0
libgobject-2.0.so.0 libvisual-0.4.so.0
libgobject-2.0.so.0.5600.4 libvisual-0.4.so.0.0.0
libgomp.so.1 libvo-aacenc.so.0
libgomp.so.1.0.0 libvo-aacenc.so.0.0.4
libgpm.so.2 libvo-amrwbenc.so.0
libgraphite2.so.2.0.0 libvo-amrwbenc.so.0.0.4
libgraphite2.so.3 libvorbis.so.0
libgraphite2.so.3.0.1 libvorbis.so.0.4.8
libgsm.so.1 libvorbisenc.so.2
libgsm.so.1.0.12 libvorbisenc.so.2.0.11
libgssapi.so.3 libvorbisfile.so.3
libgssapi.so.3.0.0 libvorbisfile.so.3.3.7
libgssapi_krb5.so.2 libvpx.so.5
libgssapi_krb5.so.2.2 libvpx.so.5.0
libgssdp-1.0.so.3 libvpx.so.5.0.0
libgssdp-1.0.so.3.0.1 libvulkan.so.1
libgstadaptivedemux-1.0.so.0 libvulkan.so.1.1.70
libgstadaptivedemux-1.0.so.0.1405.0 libvulkan.so.1.2.141
libgstallocators-1.0.so.0 libwavpack.so.1
libgstallocators-1.0.so.0.1405.0 libwavpack.so.1.2.0
libgstapp-1.0.so.0 libwayland-client.so.0
libgstapp-1.0.so.0.1405.0 libwayland-client.so.0.3.0
libgstaudio-1.0.so.0 libwayland-cursor.so.0
libgstaudio-1.0.so.0.1405.0 libwayland-cursor.so.0.0.0
libgstbadaudio-1.0.so.0 libwayland-egl.so.1
libgstbadaudio-1.0.so.0.1405.0 libwayland-egl.so.1.0.0
libgstbadvideo-1.0.so.0 libwayland-server.so.0
libgstbadvideo-1.0.so.0.1405.0 libwayland-server.so.0.1.0
libgstbase-1.0.so.0 libwebp.so.6
libgstbase-1.0.so.0.1405.0 libwebp.so.6.0.2
libgstbasecamerabinsrc-1.0.so.0 libwebpmux.so.3
libgstbasecamerabinsrc-1.0.so.0.1405.0 libwebpmux.so.3.0.1
libgstcheck-1.0.so.0 libwebrtc_audio_processing.so.1
libgstcheck-1.0.so.0.1405.0 libwebrtc_audio_processing.so.1.0.0
libgstcodecparsers-1.0.so.0 libwind.so.0
libgstcodecparsers-1.0.so.0.1405.0 libwind.so.0.0.0
libgstcontroller-1.0.so.0 libx264.so.152
libgstcontroller-1.0.so.0.1405.0 libx265.so.146
libgstfft-1.0.so.0 libxcb-dri2.a
libgstfft-1.0.so.0.1405.0 libxcb-dri2.so
libgstgl-1.0.so.0 libxcb-dri2.so.0
libgstgl-1.0.so.0.1405.0 libxcb-dri2.so.0.0.0
libgstinsertbin-1.0.so.0 libxcb-dri3.a
libgstinsertbin-1.0.so.0.1405.0 libxcb-dri3.so
libgstisoff-1.0.so.0 libxcb-dri3.so.0
libgstisoff-1.0.so.0.1405.0 libxcb-dri3.so.0.0.0
libgstmpegts-1.0.so.0 libxcb-glx.a
libgstmpegts-1.0.so.0.1405.0 libxcb-glx.so
libgstnet-1.0.so.0 libxcb-glx.so.0
libgstnet-1.0.so.0.1405.0 libxcb-glx.so.0.0.0
libgstnvegl-1.0.so.0 libxcb-present.a
libgstnvexifmeta.so libxcb-present.so
libgstnvivameta.so libxcb-present.so.0
libgstpbutils-1.0.so.0 libxcb-present.so.0.0.0
libgstpbutils-1.0.so.0.1405.0 libxcb-randr.a
libgstphotography-1.0.so.0 libxcb-randr.so
libgstphotography-1.0.so.0.1405.0 libxcb-randr.so.0
libgstplayer-1.0.so.0 libxcb-randr.so.0.1.0
libgstplayer-1.0.so.0.1405.0 libxcb-render.a
libgstreamer-1.0.so.0 libxcb-render.so
libgstreamer-1.0.so.0.1405.0 libxcb-render.so.0
libgstriff-1.0.so.0 libxcb-render.so.0.0.0
libgstriff-1.0.so.0.1405.0 libxcb-shape.a
libgstrtp-1.0.so.0 libxcb-shape.so
libgstrtp-1.0.so.0.1405.0 libxcb-shape.so.0
libgstrtsp-1.0.so.0 libxcb-shape.so.0.0.0
libgstrtsp-1.0.so.0.1405.0 libxcb-shm.so.0
libgstsdp-1.0.so.0 libxcb-shm.so.0.0.0
libgstsdp-1.0.so.0.1405.0 libxcb-sync.a
libgsttag-1.0.so.0 libxcb-sync.so
libgsttag-1.0.so.0.1405.0 libxcb-sync.so.1
libgsturidownloader-1.0.so.0 libxcb-sync.so.1.0.0
libgsturidownloader-1.0.so.0.1405.0 libxcb-xfixes.a
libgstvideo-1.0.so.0 libxcb-xfixes.so
libgstvideo-1.0.so.0.1405.0 libxcb-xfixes.so.0
libgstwayland-1.0.so.0 libxcb-xfixes.so.0.0.0
libgstwayland-1.0.so.0.1405.0 libxcb.a
libgstwebrtc-1.0.so.0 libxcb.so
libgstwebrtc-1.0.so.0.1405.0 libxcb.so.1
libgthread-2.0.so.0 libxcb.so.1.1.0
libgthread-2.0.so.0.5600.4 libxkbcommon.so.0
libgtk-3-0 libxkbcommon.so.0.0.0
libgtk-3.so.0 libxml2.so.2
libgtk-3.so.0.2200.30 libxml2.so.2.9.4
libgudev-1.0.so.0 libxshmfence.a
libgudev-1.0.so.0.2.0 libxshmfence.so
libgupnp-1.0.so.4 libxshmfence.so.1
libgupnp-1.0.so.4.0.1 libxshmfence.so.1.0.0
libgupnp-igd-1.0.so.4 libxvidcore.so.4
libgupnp-igd-1.0.so.4.2.0 libxvidcore.so.4.3
libharfbuzz.so.0 libyaml-0.so.2
libharfbuzz.so.0.10702.0 libyaml-0.so.2.0.5
libhcrypto.so.4 libz.a
libhcrypto.so.4.1.0 libz.so
libhdf5_cpp.a libzbar.so.0
libhdf5_cpp.so libzbar.so.0.2.0
libhdf5_cpp.so.100 libzmq.so.5
libhdf5_cpp.so.100.0.0 libzmq.so.5.1.5
libhdf5_hl_cpp.a libzstd.so.1
libhdf5_hl_cpp.so libzstd.so.1.3.3
libhdf5_hl_cpp.so.100 libzvbi-chains.so.0
libhdf5_hl_cpp.so.100.0.0 libzvbi-chains.so.0.0.0
libhdf5_serial.a libzvbi.so.0
libhdf5_serial.settings libzvbi.so.0.13.2
libhdf5_serial.so openblas
libhdf5_serial.so.100 openmpi
libhdf5_serial.so.100.0.1 perl
libhdf5_serial_fortran.a perl-base
libhdf5_serial_fortran.so perl5
libhdf5_serial_fortran.so.100 pkgconfig
libhdf5_serial_fortran.so.100.0.1 pulseaudio
libhdf5_serial_hl.a sasl2
libhdf5_serial_hl.so sox
libhdf5_serial_hl.so.100 tegra
libhdf5_serial_hl.so.100.0.0 tegra-egl
libhdf5_serialhl_fortran.a vdpau
libhdf5_serialhl_fortran.so x264-10bit
(venv) root@166d0c1b769d:/#
Can you paste the output of cat /etc/nvidia-container-runtime/host-files-for-container.d/cudnn.csv
from your device. It should list those cudnn libraries from your device that are supposed to be mounted into the container.
I think I found the fix but not the issue. The fix is to use nvcr.io/nvidia/l4t-base:r32.4.4
as a base image instead of nvcr.io/nvidia/l4t-cuda:10.2.460-runtime
I’m not as familiar with the cuda runtime image and I thought using it would be good but perhaps that is more for cuda development? Anyways the GPU works with pytorch/tensorflow w/ the flags --runtime=nvidia
or --gpus=all
when I use the l4t-base image.
system
Closed
May 25, 2022, 3:09am
10
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.