There are a TON of these “OSError: libcurand.so.10: cannot open shared object file: No such file or directory” reported on the forums. Seems all have different solutions. None seem to work for me.
Cloned the jetson_voice repo. Just ran ./docker/run.sh and then ./tests/run_tests.sh. All fail and throw exceptions with this as the message. Running fine on a Nano running the same JetPack. Not so fine on this NX.
The obvious cause is this:
root@myhostname:/jetson-voice# ll /usr/local/cuda/lib64/
total 1544
drwxr-xr-x 1 root root 4096 Jul 23 14:19 ./
drwxr-xr-x 1 root root 4096 Jul 23 14:19 ../
-rw-r--r-- 1 root root 679636 Jul 23 14:11 libcudadevrt.a
-rw-r--r-- 1 root root 888074 Jul 23 14:11 libcudart_static.a
drwxr-xr-x 2 root root 4096 Jul 23 14:19 stubs/
All files are there on the host, but this is all I see in the container. Any clue what could be causing THAT? I did verify that the --runtime nvidia is being appended to the docker run command. I have nvidia as the default runtime in daemon.json. nvidia-container-runtime is installed at 3.1.0-1. nvidia-docker2 is 2.2.0-1.
I’m at a loss.
1 Like
Hi @pault45z6 , here is what my /usr/local/cuda/lib64/
from outside the container (this is on JetPack 4.6):
ls -ll /usr/local/cuda/lib64/
total 2259940
lrwxrwxrwx 1 root root 17 Mar 1 2021 libcublasLt.so -> libcublasLt.so.10
lrwxrwxrwx 1 root root 25 Mar 1 2021 libcublasLt.so.10 -> libcublasLt.so.10.2.3.300
-rw-r--r-- 1 root root 33562824 Mar 1 2021 libcublasLt.so.10.2.3.300
-rw-r--r-- 1 root root 36011742 Mar 1 2021 libcublasLt_static.a
lrwxrwxrwx 1 root root 15 Mar 1 2021 libcublas.so -> libcublas.so.10
lrwxrwxrwx 1 root root 23 Mar 1 2021 libcublas.so.10 -> libcublas.so.10.2.3.300
-rw-r--r-- 1 root root 81096256 Mar 1 2021 libcublas.so.10.2.3.300
-rw-r--r-- 1 root root 96903266 Mar 1 2021 libcublas_static.a
-rw-r--r-- 1 root root 679636 Mar 1 2021 libcudadevrt.a
lrwxrwxrwx 1 root root 17 Mar 1 2021 libcudart.so -> libcudart.so.10.2
lrwxrwxrwx 1 root root 21 Mar 1 2021 libcudart.so.10.2 -> libcudart.so.10.2.300
-rw-r--r-- 1 root root 490664 Mar 1 2021 libcudart.so.10.2.300
-rw-r--r-- 1 root root 888074 Mar 1 2021 libcudart_static.a
lrwxrwxrwx 1 root root 14 Mar 1 2021 libcufft.so -> libcufft.so.10
lrwxrwxrwx 1 root root 22 Mar 1 2021 libcufft.so.10 -> libcufft.so.10.1.2.300
-rw-r--r-- 1 root root 201494704 Mar 1 2021 libcufft.so.10.1.2.300
-rw-r--r-- 1 root root 192531512 Mar 1 2021 libcufft_static.a
-rw-r--r-- 1 root root 210524874 Mar 1 2021 libcufft_static_nocallback.a
lrwxrwxrwx 1 root root 15 Mar 1 2021 libcufftw.so -> libcufftw.so.10
lrwxrwxrwx 1 root root 23 Mar 1 2021 libcufftw.so.10 -> libcufftw.so.10.1.2.300
-rw-r--r-- 1 root root 503192 Mar 1 2021 libcufftw.so.10.1.2.300
-rw-r--r-- 1 root root 31970 Mar 1 2021 libcufftw_static.a
lrwxrwxrwx 1 root root 18 Mar 1 2021 libcuinj64.so -> libcuinj64.so.10.2
lrwxrwxrwx 1 root root 22 Mar 1 2021 libcuinj64.so.10.2 -> libcuinj64.so.10.2.300
-rw-r--r-- 1 root root 1535464 Mar 1 2021 libcuinj64.so.10.2.300
-rw-r--r-- 1 root root 33242 Mar 1 2021 libculibos.a
lrwxrwxrwx 1 root root 16 Mar 1 2021 libcupti.so -> libcupti.so.10.2
lrwxrwxrwx 1 root root 20 Mar 1 2021 libcupti.so.10.2 -> libcupti.so.10.2.175
-rw-r--r-- 1 root root 4526616 Mar 1 2021 libcupti.so.10.2.175
lrwxrwxrwx 1 root root 15 Mar 1 2021 libcurand.so -> libcurand.so.10
lrwxrwxrwx 1 root root 23 Mar 1 2021 libcurand.so.10 -> libcurand.so.10.1.2.300
-rw-r--r-- 1 root root 62698584 Mar 1 2021 libcurand.so.10.1.2.300
-rw-r--r-- 1 root root 62767380 Mar 1 2021 libcurand_static.a
lrwxrwxrwx 1 root root 17 Mar 1 2021 libcusolver.so -> libcusolver.so.10
lrwxrwxrwx 1 root root 25 Mar 1 2021 libcusolver.so.10 -> libcusolver.so.10.3.0.300
-rw-r--r-- 1 root root 218927328 Mar 1 2021 libcusolver.so.10.3.0.300
-rw-r--r-- 1 root root 123895098 Mar 1 2021 libcusolver_static.a
lrwxrwxrwx 1 root root 17 Mar 1 2021 libcusparse.so -> libcusparse.so.10
lrwxrwxrwx 1 root root 25 Mar 1 2021 libcusparse.so.10 -> libcusparse.so.10.3.1.300
-rw-r--r-- 1 root root 141252584 Mar 1 2021 libcusparse.so.10.3.1.300
-rw-r--r-- 1 root root 149512102 Mar 1 2021 libcusparse_static.a
-rw-r--r-- 1 root root 8319056 Mar 1 2021 liblapack_static.a
-rw-r--r-- 1 root root 909274 Mar 1 2021 libmetis_static.a
lrwxrwxrwx 1 root root 13 Mar 1 2021 libnppc.so -> libnppc.so.10
lrwxrwxrwx 1 root root 21 Mar 1 2021 libnppc.so.10 -> libnppc.so.10.2.1.300
-rw-r--r-- 1 root root 503184 Mar 1 2021 libnppc.so.10.2.1.300
-rw-r--r-- 1 root root 26846 Mar 1 2021 libnppc_static.a
lrwxrwxrwx 1 root root 15 Mar 1 2021 libnppial.so -> libnppial.so.10
lrwxrwxrwx 1 root root 23 Mar 1 2021 libnppial.so.10 -> libnppial.so.10.2.1.300
-rw-r--r-- 1 root root 11509472 Mar 1 2021 libnppial.so.10.2.1.300
-rw-r--r-- 1 root root 14410930 Mar 1 2021 libnppial_static.a
lrwxrwxrwx 1 root root 15 Mar 1 2021 libnppicc.so -> libnppicc.so.10
lrwxrwxrwx 1 root root 23 Mar 1 2021 libnppicc.so.10 -> libnppicc.so.10.2.1.300
-rw-r--r-- 1 root root 4914920 Mar 1 2021 libnppicc.so.10.2.1.300
-rw-r--r-- 1 root root 5722536 Mar 1 2021 libnppicc_static.a
lrwxrwxrwx 1 root root 16 Mar 1 2021 libnppicom.so -> libnppicom.so.10
lrwxrwxrwx 1 root root 24 Mar 1 2021 libnppicom.so.10 -> libnppicom.so.10.2.1.300
-rw-r--r-- 1 root root 1453728 Mar 1 2021 libnppicom.so.10.2.1.300
-rw-r--r-- 1 root root 1093680 Mar 1 2021 libnppicom_static.a
lrwxrwxrwx 1 root root 16 Mar 1 2021 libnppidei.so -> libnppidei.so.10
lrwxrwxrwx 1 root root 24 Mar 1 2021 libnppidei.so.10 -> libnppidei.so.10.2.1.300
-rw-r--r-- 1 root root 8175688 Mar 1 2021 libnppidei.so.10.2.1.300
-rw-r--r-- 1 root root 10762478 Mar 1 2021 libnppidei_static.a
lrwxrwxrwx 1 root root 14 Mar 1 2021 libnppif.so -> libnppif.so.10
lrwxrwxrwx 1 root root 22 Mar 1 2021 libnppif.so.10 -> libnppif.so.10.2.1.300
-rw-r--r-- 1 root root 54362944 Mar 1 2021 libnppif.so.10.2.1.300
-rw-r--r-- 1 root root 58471042 Mar 1 2021 libnppif_static.a
lrwxrwxrwx 1 root root 14 Mar 1 2021 libnppig.so -> libnppig.so.10
lrwxrwxrwx 1 root root 22 Mar 1 2021 libnppig.so.10 -> libnppig.so.10.2.1.300
-rw-r--r-- 1 root root 28761920 Mar 1 2021 libnppig.so.10.2.1.300
-rw-r--r-- 1 root root 31432462 Mar 1 2021 libnppig_static.a
lrwxrwxrwx 1 root root 14 Mar 1 2021 libnppim.so -> libnppim.so.10
lrwxrwxrwx 1 root root 22 Mar 1 2021 libnppim.so.10 -> libnppim.so.10.2.1.300
-rw-r--r-- 1 root root 7163640 Mar 1 2021 libnppim.so.10.2.1.300
-rw-r--r-- 1 root root 7396476 Mar 1 2021 libnppim_static.a
lrwxrwxrwx 1 root root 15 Mar 1 2021 libnppist.so -> libnppist.so.10
lrwxrwxrwx 1 root root 23 Mar 1 2021 libnppist.so.10 -> libnppist.so.10.2.1.300
-rw-r--r-- 1 root root 20877336 Mar 1 2021 libnppist.so.10.2.1.300
-rw-r--r-- 1 root root 23399160 Mar 1 2021 libnppist_static.a
lrwxrwxrwx 1 root root 15 Mar 1 2021 libnppisu.so -> libnppisu.so.10
lrwxrwxrwx 1 root root 23 Mar 1 2021 libnppisu.so.10 -> libnppisu.so.10.2.1.300
-rw-r--r-- 1 root root 486576 Mar 1 2021 libnppisu.so.10.2.1.300
-rw-r--r-- 1 root root 11458 Mar 1 2021 libnppisu_static.a
lrwxrwxrwx 1 root root 15 Mar 1 2021 libnppitc.so -> libnppitc.so.10
lrwxrwxrwx 1 root root 23 Mar 1 2021 libnppitc.so.10 -> libnppitc.so.10.2.1.300
-rw-r--r-- 1 root root 3112480 Mar 1 2021 libnppitc.so.10.2.1.300
-rw-r--r-- 1 root root 3205362 Mar 1 2021 libnppitc_static.a
lrwxrwxrwx 1 root root 13 Mar 1 2021 libnpps.so -> libnpps.so.10
lrwxrwxrwx 1 root root 21 Mar 1 2021 libnpps.so.10 -> libnpps.so.10.2.1.300
-rw-r--r-- 1 root root 9539760 Mar 1 2021 libnpps.so.10.2.1.300
-rw-r--r-- 1 root root 10690508 Mar 1 2021 libnpps_static.a
lrwxrwxrwx 1 root root 15 Mar 1 2021 libnvblas.so -> libnvblas.so.10
lrwxrwxrwx 1 root root 23 Mar 1 2021 libnvblas.so.10 -> libnvblas.so.10.2.3.300
-rw-r--r-- 1 root root 540232 Mar 1 2021 libnvblas.so.10.2.3.300
lrwxrwxrwx 1 root root 16 Mar 1 2021 libnvgraph.so -> libnvgraph.so.10
lrwxrwxrwx 1 root root 22 Mar 1 2021 libnvgraph.so.10 -> libnvgraph.so.10.2.300
-rw-r--r-- 1 root root 165012616 Mar 1 2021 libnvgraph.so.10.2.300
-rw-r--r-- 1 root root 168141386 Mar 1 2021 libnvgraph_static.a
-rw-r--r-- 1 root root 7430712 Mar 1 2021 libnvperf_host.so
-rw-r--r-- 1 root root 1096016 Mar 1 2021 libnvperf_target.so
lrwxrwxrwx 1 root root 25 Mar 1 2021 libnvrtc-builtins.so -> libnvrtc-builtins.so.10.2
lrwxrwxrwx 1 root root 29 Mar 1 2021 libnvrtc-builtins.so.10.2 -> libnvrtc-builtins.so.10.2.300
-rw-r--r-- 1 root root 4794168 Mar 1 2021 libnvrtc-builtins.so.10.2.300
lrwxrwxrwx 1 root root 16 Mar 1 2021 libnvrtc.so -> libnvrtc.so.10.2
lrwxrwxrwx 1 root root 20 Mar 1 2021 libnvrtc.so.10.2 -> libnvrtc.so.10.2.300
-rw-r--r-- 1 root root 20432800 Mar 1 2021 libnvrtc.so.10.2.300
lrwxrwxrwx 1 root root 18 Mar 1 2021 libnvToolsExt.so -> libnvToolsExt.so.1
lrwxrwxrwx 1 root root 22 Mar 1 2021 libnvToolsExt.so.1 -> libnvToolsExt.so.1.0.0
-rw-r--r-- 1 root root 44088 Mar 1 2021 libnvToolsExt.so.1.0.0
drwxr-xr-x 2 root root 4096 Jul 26 22:31 stubs
And here is what it looks like from inside the container:
ls -ll /usr/local/cuda/lib64/
total 2259940
lrwxrwxrwx 1 root root 17 Mar 1 2021 libcublasLt.so -> libcublasLt.so.10
lrwxrwxrwx 1 root root 25 Mar 1 2021 libcublasLt.so.10 -> libcublasLt.so.10.2.3.300
-rw-r--r-- 1 root root 33562824 Mar 1 2021 libcublasLt.so.10.2.3.300
-rw-r--r-- 1 root root 36011742 Mar 1 2021 libcublasLt_static.a
lrwxrwxrwx 1 root root 15 Mar 1 2021 libcublas.so -> libcublas.so.10
lrwxrwxrwx 1 root root 23 Mar 1 2021 libcublas.so.10 -> libcublas.so.10.2.3.300
-rw-r--r-- 1 root root 81096256 Mar 1 2021 libcublas.so.10.2.3.300
-rw-r--r-- 1 root root 96903266 Mar 1 2021 libcublas_static.a
-rw-r--r-- 1 root root 679636 Mar 1 2021 libcudadevrt.a
lrwxrwxrwx 1 root root 17 Mar 1 2021 libcudart.so -> libcudart.so.10.2
lrwxrwxrwx 1 root root 21 Mar 1 2021 libcudart.so.10.2 -> libcudart.so.10.2.300
-rw-r--r-- 1 root root 490664 Mar 1 2021 libcudart.so.10.2.300
-rw-r--r-- 1 root root 888074 Mar 1 2021 libcudart_static.a
lrwxrwxrwx 1 root root 14 Mar 1 2021 libcufft.so -> libcufft.so.10
lrwxrwxrwx 1 root root 22 Mar 1 2021 libcufft.so.10 -> libcufft.so.10.1.2.300
-rw-r--r-- 1 root root 201494704 Mar 1 2021 libcufft.so.10.1.2.300
-rw-r--r-- 1 root root 192531512 Mar 1 2021 libcufft_static.a
-rw-r--r-- 1 root root 210524874 Mar 1 2021 libcufft_static_nocallback.a
lrwxrwxrwx 1 root root 15 Mar 1 2021 libcufftw.so -> libcufftw.so.10
lrwxrwxrwx 1 root root 23 Mar 1 2021 libcufftw.so.10 -> libcufftw.so.10.1.2.300
-rw-r--r-- 1 root root 503192 Mar 1 2021 libcufftw.so.10.1.2.300
-rw-r--r-- 1 root root 31970 Mar 1 2021 libcufftw_static.a
lrwxrwxrwx 1 root root 18 Mar 1 2021 libcuinj64.so -> libcuinj64.so.10.2
lrwxrwxrwx 1 root root 22 Mar 1 2021 libcuinj64.so.10.2 -> libcuinj64.so.10.2.300
-rw-r--r-- 1 root root 1535464 Mar 1 2021 libcuinj64.so.10.2.300
-rw-r--r-- 1 root root 33242 Mar 1 2021 libculibos.a
lrwxrwxrwx 1 root root 16 Mar 1 2021 libcupti.so -> libcupti.so.10.2
lrwxrwxrwx 1 root root 20 Mar 1 2021 libcupti.so.10.2 -> libcupti.so.10.2.175
-rw-r--r-- 1 root root 4526616 Mar 1 2021 libcupti.so.10.2.175
lrwxrwxrwx 1 root root 15 Mar 1 2021 libcurand.so -> libcurand.so.10
lrwxrwxrwx 1 root root 23 Mar 1 2021 libcurand.so.10 -> libcurand.so.10.1.2.300
-rw-r--r-- 1 root root 62698584 Mar 1 2021 libcurand.so.10.1.2.300
-rw-r--r-- 1 root root 62767380 Mar 1 2021 libcurand_static.a
lrwxrwxrwx 1 root root 17 Mar 1 2021 libcusolver.so -> libcusolver.so.10
lrwxrwxrwx 1 root root 25 Mar 1 2021 libcusolver.so.10 -> libcusolver.so.10.3.0.300
-rw-r--r-- 1 root root 218927328 Mar 1 2021 libcusolver.so.10.3.0.300
-rw-r--r-- 1 root root 123895098 Mar 1 2021 libcusolver_static.a
lrwxrwxrwx 1 root root 17 Mar 1 2021 libcusparse.so -> libcusparse.so.10
lrwxrwxrwx 1 root root 25 Mar 1 2021 libcusparse.so.10 -> libcusparse.so.10.3.1.300
-rw-r--r-- 1 root root 141252584 Mar 1 2021 libcusparse.so.10.3.1.300
-rw-r--r-- 1 root root 149512102 Mar 1 2021 libcusparse_static.a
-rw-r--r-- 1 root root 8319056 Mar 1 2021 liblapack_static.a
-rw-r--r-- 1 root root 909274 Mar 1 2021 libmetis_static.a
lrwxrwxrwx 1 root root 13 Mar 1 2021 libnppc.so -> libnppc.so.10
lrwxrwxrwx 1 root root 21 Mar 1 2021 libnppc.so.10 -> libnppc.so.10.2.1.300
-rw-r--r-- 1 root root 503184 Mar 1 2021 libnppc.so.10.2.1.300
-rw-r--r-- 1 root root 26846 Mar 1 2021 libnppc_static.a
lrwxrwxrwx 1 root root 15 Mar 1 2021 libnppial.so -> libnppial.so.10
lrwxrwxrwx 1 root root 23 Mar 1 2021 libnppial.so.10 -> libnppial.so.10.2.1.300
-rw-r--r-- 1 root root 11509472 Mar 1 2021 libnppial.so.10.2.1.300
-rw-r--r-- 1 root root 14410930 Mar 1 2021 libnppial_static.a
lrwxrwxrwx 1 root root 15 Mar 1 2021 libnppicc.so -> libnppicc.so.10
lrwxrwxrwx 1 root root 23 Mar 1 2021 libnppicc.so.10 -> libnppicc.so.10.2.1.300
-rw-r--r-- 1 root root 4914920 Mar 1 2021 libnppicc.so.10.2.1.300
-rw-r--r-- 1 root root 5722536 Mar 1 2021 libnppicc_static.a
lrwxrwxrwx 1 root root 16 Mar 1 2021 libnppicom.so -> libnppicom.so.10
lrwxrwxrwx 1 root root 24 Mar 1 2021 libnppicom.so.10 -> libnppicom.so.10.2.1.300
-rw-r--r-- 1 root root 1453728 Mar 1 2021 libnppicom.so.10.2.1.300
-rw-r--r-- 1 root root 1093680 Mar 1 2021 libnppicom_static.a
lrwxrwxrwx 1 root root 16 Mar 1 2021 libnppidei.so -> libnppidei.so.10
lrwxrwxrwx 1 root root 24 Mar 1 2021 libnppidei.so.10 -> libnppidei.so.10.2.1.300
-rw-r--r-- 1 root root 8175688 Mar 1 2021 libnppidei.so.10.2.1.300
-rw-r--r-- 1 root root 10762478 Mar 1 2021 libnppidei_static.a
lrwxrwxrwx 1 root root 14 Mar 1 2021 libnppif.so -> libnppif.so.10
lrwxrwxrwx 1 root root 22 Mar 1 2021 libnppif.so.10 -> libnppif.so.10.2.1.300
-rw-r--r-- 1 root root 54362944 Mar 1 2021 libnppif.so.10.2.1.300
-rw-r--r-- 1 root root 58471042 Mar 1 2021 libnppif_static.a
lrwxrwxrwx 1 root root 14 Mar 1 2021 libnppig.so -> libnppig.so.10
lrwxrwxrwx 1 root root 22 Mar 1 2021 libnppig.so.10 -> libnppig.so.10.2.1.300
-rw-r--r-- 1 root root 28761920 Mar 1 2021 libnppig.so.10.2.1.300
-rw-r--r-- 1 root root 31432462 Mar 1 2021 libnppig_static.a
lrwxrwxrwx 1 root root 14 Mar 1 2021 libnppim.so -> libnppim.so.10
lrwxrwxrwx 1 root root 22 Mar 1 2021 libnppim.so.10 -> libnppim.so.10.2.1.300
-rw-r--r-- 1 root root 7163640 Mar 1 2021 libnppim.so.10.2.1.300
-rw-r--r-- 1 root root 7396476 Mar 1 2021 libnppim_static.a
lrwxrwxrwx 1 root root 15 Mar 1 2021 libnppist.so -> libnppist.so.10
lrwxrwxrwx 1 root root 23 Mar 1 2021 libnppist.so.10 -> libnppist.so.10.2.1.300
-rw-r--r-- 1 root root 20877336 Mar 1 2021 libnppist.so.10.2.1.300
-rw-r--r-- 1 root root 23399160 Mar 1 2021 libnppist_static.a
lrwxrwxrwx 1 root root 15 Mar 1 2021 libnppisu.so -> libnppisu.so.10
lrwxrwxrwx 1 root root 23 Mar 1 2021 libnppisu.so.10 -> libnppisu.so.10.2.1.300
-rw-r--r-- 1 root root 486576 Mar 1 2021 libnppisu.so.10.2.1.300
-rw-r--r-- 1 root root 11458 Mar 1 2021 libnppisu_static.a
lrwxrwxrwx 1 root root 15 Mar 1 2021 libnppitc.so -> libnppitc.so.10
lrwxrwxrwx 1 root root 23 Mar 1 2021 libnppitc.so.10 -> libnppitc.so.10.2.1.300
-rw-r--r-- 1 root root 3112480 Mar 1 2021 libnppitc.so.10.2.1.300
-rw-r--r-- 1 root root 3205362 Mar 1 2021 libnppitc_static.a
lrwxrwxrwx 1 root root 13 Mar 1 2021 libnpps.so -> libnpps.so.10
lrwxrwxrwx 1 root root 21 Mar 1 2021 libnpps.so.10 -> libnpps.so.10.2.1.300
-rw-r--r-- 1 root root 9539760 Mar 1 2021 libnpps.so.10.2.1.300
-rw-r--r-- 1 root root 10690508 Mar 1 2021 libnpps_static.a
lrwxrwxrwx 1 root root 15 Mar 1 2021 libnvblas.so -> libnvblas.so.10
lrwxrwxrwx 1 root root 23 Mar 1 2021 libnvblas.so.10 -> libnvblas.so.10.2.3.300
-rw-r--r-- 1 root root 540232 Mar 1 2021 libnvblas.so.10.2.3.300
lrwxrwxrwx 1 root root 16 Mar 1 2021 libnvgraph.so -> libnvgraph.so.10
lrwxrwxrwx 1 root root 22 Mar 1 2021 libnvgraph.so.10 -> libnvgraph.so.10.2.300
-rw-r--r-- 1 root root 165012616 Mar 1 2021 libnvgraph.so.10.2.300
-rw-r--r-- 1 root root 168141386 Mar 1 2021 libnvgraph_static.a
-rw-r--r-- 1 root root 7430712 Mar 1 2021 libnvperf_host.so
-rw-r--r-- 1 root root 1096016 Mar 1 2021 libnvperf_target.so
lrwxrwxrwx 1 root root 25 Mar 1 2021 libnvrtc-builtins.so -> libnvrtc-builtins.so.10.2
lrwxrwxrwx 1 root root 29 Mar 1 2021 libnvrtc-builtins.so.10.2 -> libnvrtc-builtins.so.10.2.300
-rw-r--r-- 1 root root 4794168 Mar 1 2021 libnvrtc-builtins.so.10.2.300
lrwxrwxrwx 1 root root 16 Mar 1 2021 libnvrtc.so -> libnvrtc.so.10.2
lrwxrwxrwx 1 root root 20 Mar 1 2021 libnvrtc.so.10.2 -> libnvrtc.so.10.2.300
-rw-r--r-- 1 root root 20432800 Mar 1 2021 libnvrtc.so.10.2.300
lrwxrwxrwx 1 root root 18 Mar 1 2021 libnvToolsExt.so -> libnvToolsExt.so.1
lrwxrwxrwx 1 root root 22 Mar 1 2021 libnvToolsExt.so.1 -> libnvToolsExt.so.1.0.0
-rw-r--r-- 1 root root 44088 Mar 1 2021 libnvToolsExt.so.1.0.0
drwxr-xr-x 2 root root 4096 Jul 26 22:31 stubs
If you are missing libraries, you may want to re-install JetPack or reflash your device / SD card.
I see the same as you outside on the host. I’m just missing everything inside the container. It seems not to be mapping from the container to the host (which I though the nvidia-container-host was supposed to do).
Hoping to avoid reflashing if possible.
One other bit of info. I ran into the issue where I apt upgrade’d containerd and docker.io . It borked. Had to manually downgrade. Could this potentially be a root cause?
If you run other containers (such as l4t-base or l4t-pytorch or jetson-inference) with --runtime nvidia
, does it mount the files correctly, or is this issue specific to the jetson-voice container?
Ah - I’ve not done the upgrade/downgrade myself, but it could be. Can you try adding --security-opt seccomp=unconfined
as a flag in the docker run
command in this line of the docker/run.sh script?
https://github.com/dusty-nv/jetson-voice/blob/1ec13b77f493d399f31c31f0af0650bcdbca8bc0/docker/run.sh#L176
If you’re unable to get the Docker runtime working again where it mounts the files correctly, you may save yourself more time by just re-flashing - sorry about that.
You SHOULD! It’s a blast. ;)
Unfortunately, yes, the behavior is apparently the same in all containers. Setting the security-opt flag as such:
sudo nvidia-docker run --runtime nvidia -it --rm \
--security-opt seccomp=unconfined \
--name=$CONTAINER_NAME \
--network host \
$MOUNTS $CONTAINER_IMAGE $USER_COMMAND
Didn’t work.
One other variable. I’ve configured this to boot from NVMe using ./nvsdkmanager_flash.sh –storage nvme0n1p1
. Before reflashing only to find out the problem didn’t go away, is there anything about using this flash method that might impact this? Maybe a hard-coded location in nvidia-container-runtime that presumes SD storage?
pault45z6:
I’ve configured this to boot from NVMe using ./nvsdkmanager_flash.sh –storage nvme0n1p1
. Before reflashing only to find out the problem didn’t go away, is there anything about using this flash method that might impact this? Maybe a hard-coded location in nvidia-container-runtime that presumes SD storage?
Offhand I don’t believe so, although I haven’t done boot from NVME myself either (sorry) - typically I just put my docker root directory (where all the containers themselves live) onto NVME and call it a day.
The CSV files that contain the files to mount are located under /etc/nvidia-container-runtime/host-files-for-container.d/
, so you might want to check that you still have these:
ls /etc/nvidia-container-runtime/host-files-for-container.d/
cuda.csv cudnn.csv l4t.csv opencv.csv tensorrt.csv visionworks.csv
BINGO! I was missing cuda.csv. Copied it from one of my Nanos and that fixed that problem. Presumably sudo apt install nvidia-container-csv-cuda
would have worked, too.
Of course, then I found out that during my “reinstall everything” efforts, I must’ve nuked pycuda and with it the ability to use TensorRT from Python. Which I can’t figure out how to properly install. Arrhhgghah! Smelling a reflash in my future. :(
Thankfully @kangalow saved the day without even knowing it. Just had to run the mammoth apt install from bootFromExternalStorage/install_jetson_default_packages.sh (github.com) . At least it got me to the point that I could run everything. Doubtless would’ve fixed it from the start. Need to remember this next time.
Still some issues (e.g., failing a few of the tests) that I’m sure I’ll ask about in a separate thread. Regardless, thanks for all of your help @dusty_nv !!
1 Like
OK great, glad to hear that you got it working!
Regarding the tests, I think I know what you mean - IIRC the ASR classification models (Matchboxnet / Marblenet) fail on JetPack 4.6. Been meaning to look into that - I recall it being a pre-processing error due to using a newer version of PyTorch. There may also have been a failure in one of the NLP tests, but I think it was just because of a minorly different output than expected, but was still functional (I do a simple text comparison to see if the outputs match, and sometimes they can be slightly different).
system
Closed
December 8, 2021, 3:34am
12
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.