Jetson_voice docker image run gives "OSError: libcurand.so.10: cannot open shared object file: No such file or directory"

There are a TON of these “OSError: libcurand.so.10: cannot open shared object file: No such file or directory” reported on the forums. Seems all have different solutions. None seem to work for me.

Cloned the jetson_voice repo. Just ran ./docker/run.sh and then ./tests/run_tests.sh. All fail and throw exceptions with this as the message. Running fine on a Nano running the same JetPack. Not so fine on this NX.

The obvious cause is this:

root@myhostname:/jetson-voice# ll /usr/local/cuda/lib64/
total 1544
drwxr-xr-x 1 root root   4096 Jul 23 14:19 ./
drwxr-xr-x 1 root root   4096 Jul 23 14:19 ../
-rw-r--r-- 1 root root 679636 Jul 23 14:11 libcudadevrt.a
-rw-r--r-- 1 root root 888074 Jul 23 14:11 libcudart_static.a
drwxr-xr-x 2 root root   4096 Jul 23 14:19 stubs/

All files are there on the host, but this is all I see in the container. Any clue what could be causing THAT? I did verify that the --runtime nvidia is being appended to the docker run command. I have nvidia as the default runtime in daemon.json. nvidia-container-runtime is installed at 3.1.0-1. nvidia-docker2 is 2.2.0-1.

I’m at a loss.

Hi @pault45z6, here is what my /usr/local/cuda/lib64/ from outside the container (this is on JetPack 4.6):

ls -ll /usr/local/cuda/lib64/
total 2259940
lrwxrwxrwx 1 root root        17 Mar  1  2021 libcublasLt.so -> libcublasLt.so.10
lrwxrwxrwx 1 root root        25 Mar  1  2021 libcublasLt.so.10 -> libcublasLt.so.10.2.3.300
-rw-r--r-- 1 root root  33562824 Mar  1  2021 libcublasLt.so.10.2.3.300
-rw-r--r-- 1 root root  36011742 Mar  1  2021 libcublasLt_static.a
lrwxrwxrwx 1 root root        15 Mar  1  2021 libcublas.so -> libcublas.so.10
lrwxrwxrwx 1 root root        23 Mar  1  2021 libcublas.so.10 -> libcublas.so.10.2.3.300
-rw-r--r-- 1 root root  81096256 Mar  1  2021 libcublas.so.10.2.3.300
-rw-r--r-- 1 root root  96903266 Mar  1  2021 libcublas_static.a
-rw-r--r-- 1 root root    679636 Mar  1  2021 libcudadevrt.a
lrwxrwxrwx 1 root root        17 Mar  1  2021 libcudart.so -> libcudart.so.10.2
lrwxrwxrwx 1 root root        21 Mar  1  2021 libcudart.so.10.2 -> libcudart.so.10.2.300
-rw-r--r-- 1 root root    490664 Mar  1  2021 libcudart.so.10.2.300
-rw-r--r-- 1 root root    888074 Mar  1  2021 libcudart_static.a
lrwxrwxrwx 1 root root        14 Mar  1  2021 libcufft.so -> libcufft.so.10
lrwxrwxrwx 1 root root        22 Mar  1  2021 libcufft.so.10 -> libcufft.so.10.1.2.300
-rw-r--r-- 1 root root 201494704 Mar  1  2021 libcufft.so.10.1.2.300
-rw-r--r-- 1 root root 192531512 Mar  1  2021 libcufft_static.a
-rw-r--r-- 1 root root 210524874 Mar  1  2021 libcufft_static_nocallback.a
lrwxrwxrwx 1 root root        15 Mar  1  2021 libcufftw.so -> libcufftw.so.10
lrwxrwxrwx 1 root root        23 Mar  1  2021 libcufftw.so.10 -> libcufftw.so.10.1.2.300
-rw-r--r-- 1 root root    503192 Mar  1  2021 libcufftw.so.10.1.2.300
-rw-r--r-- 1 root root     31970 Mar  1  2021 libcufftw_static.a
lrwxrwxrwx 1 root root        18 Mar  1  2021 libcuinj64.so -> libcuinj64.so.10.2
lrwxrwxrwx 1 root root        22 Mar  1  2021 libcuinj64.so.10.2 -> libcuinj64.so.10.2.300
-rw-r--r-- 1 root root   1535464 Mar  1  2021 libcuinj64.so.10.2.300
-rw-r--r-- 1 root root     33242 Mar  1  2021 libculibos.a
lrwxrwxrwx 1 root root        16 Mar  1  2021 libcupti.so -> libcupti.so.10.2
lrwxrwxrwx 1 root root        20 Mar  1  2021 libcupti.so.10.2 -> libcupti.so.10.2.175
-rw-r--r-- 1 root root   4526616 Mar  1  2021 libcupti.so.10.2.175
lrwxrwxrwx 1 root root        15 Mar  1  2021 libcurand.so -> libcurand.so.10
lrwxrwxrwx 1 root root        23 Mar  1  2021 libcurand.so.10 -> libcurand.so.10.1.2.300
-rw-r--r-- 1 root root  62698584 Mar  1  2021 libcurand.so.10.1.2.300
-rw-r--r-- 1 root root  62767380 Mar  1  2021 libcurand_static.a
lrwxrwxrwx 1 root root        17 Mar  1  2021 libcusolver.so -> libcusolver.so.10
lrwxrwxrwx 1 root root        25 Mar  1  2021 libcusolver.so.10 -> libcusolver.so.10.3.0.300
-rw-r--r-- 1 root root 218927328 Mar  1  2021 libcusolver.so.10.3.0.300
-rw-r--r-- 1 root root 123895098 Mar  1  2021 libcusolver_static.a
lrwxrwxrwx 1 root root        17 Mar  1  2021 libcusparse.so -> libcusparse.so.10
lrwxrwxrwx 1 root root        25 Mar  1  2021 libcusparse.so.10 -> libcusparse.so.10.3.1.300
-rw-r--r-- 1 root root 141252584 Mar  1  2021 libcusparse.so.10.3.1.300
-rw-r--r-- 1 root root 149512102 Mar  1  2021 libcusparse_static.a
-rw-r--r-- 1 root root   8319056 Mar  1  2021 liblapack_static.a
-rw-r--r-- 1 root root    909274 Mar  1  2021 libmetis_static.a
lrwxrwxrwx 1 root root        13 Mar  1  2021 libnppc.so -> libnppc.so.10
lrwxrwxrwx 1 root root        21 Mar  1  2021 libnppc.so.10 -> libnppc.so.10.2.1.300
-rw-r--r-- 1 root root    503184 Mar  1  2021 libnppc.so.10.2.1.300
-rw-r--r-- 1 root root     26846 Mar  1  2021 libnppc_static.a
lrwxrwxrwx 1 root root        15 Mar  1  2021 libnppial.so -> libnppial.so.10
lrwxrwxrwx 1 root root        23 Mar  1  2021 libnppial.so.10 -> libnppial.so.10.2.1.300
-rw-r--r-- 1 root root  11509472 Mar  1  2021 libnppial.so.10.2.1.300
-rw-r--r-- 1 root root  14410930 Mar  1  2021 libnppial_static.a
lrwxrwxrwx 1 root root        15 Mar  1  2021 libnppicc.so -> libnppicc.so.10
lrwxrwxrwx 1 root root        23 Mar  1  2021 libnppicc.so.10 -> libnppicc.so.10.2.1.300
-rw-r--r-- 1 root root   4914920 Mar  1  2021 libnppicc.so.10.2.1.300
-rw-r--r-- 1 root root   5722536 Mar  1  2021 libnppicc_static.a
lrwxrwxrwx 1 root root        16 Mar  1  2021 libnppicom.so -> libnppicom.so.10
lrwxrwxrwx 1 root root        24 Mar  1  2021 libnppicom.so.10 -> libnppicom.so.10.2.1.300
-rw-r--r-- 1 root root   1453728 Mar  1  2021 libnppicom.so.10.2.1.300
-rw-r--r-- 1 root root   1093680 Mar  1  2021 libnppicom_static.a
lrwxrwxrwx 1 root root        16 Mar  1  2021 libnppidei.so -> libnppidei.so.10
lrwxrwxrwx 1 root root        24 Mar  1  2021 libnppidei.so.10 -> libnppidei.so.10.2.1.300
-rw-r--r-- 1 root root   8175688 Mar  1  2021 libnppidei.so.10.2.1.300
-rw-r--r-- 1 root root  10762478 Mar  1  2021 libnppidei_static.a
lrwxrwxrwx 1 root root        14 Mar  1  2021 libnppif.so -> libnppif.so.10
lrwxrwxrwx 1 root root        22 Mar  1  2021 libnppif.so.10 -> libnppif.so.10.2.1.300
-rw-r--r-- 1 root root  54362944 Mar  1  2021 libnppif.so.10.2.1.300
-rw-r--r-- 1 root root  58471042 Mar  1  2021 libnppif_static.a
lrwxrwxrwx 1 root root        14 Mar  1  2021 libnppig.so -> libnppig.so.10
lrwxrwxrwx 1 root root        22 Mar  1  2021 libnppig.so.10 -> libnppig.so.10.2.1.300
-rw-r--r-- 1 root root  28761920 Mar  1  2021 libnppig.so.10.2.1.300
-rw-r--r-- 1 root root  31432462 Mar  1  2021 libnppig_static.a
lrwxrwxrwx 1 root root        14 Mar  1  2021 libnppim.so -> libnppim.so.10
lrwxrwxrwx 1 root root        22 Mar  1  2021 libnppim.so.10 -> libnppim.so.10.2.1.300
-rw-r--r-- 1 root root   7163640 Mar  1  2021 libnppim.so.10.2.1.300
-rw-r--r-- 1 root root   7396476 Mar  1  2021 libnppim_static.a
lrwxrwxrwx 1 root root        15 Mar  1  2021 libnppist.so -> libnppist.so.10
lrwxrwxrwx 1 root root        23 Mar  1  2021 libnppist.so.10 -> libnppist.so.10.2.1.300
-rw-r--r-- 1 root root  20877336 Mar  1  2021 libnppist.so.10.2.1.300
-rw-r--r-- 1 root root  23399160 Mar  1  2021 libnppist_static.a
lrwxrwxrwx 1 root root        15 Mar  1  2021 libnppisu.so -> libnppisu.so.10
lrwxrwxrwx 1 root root        23 Mar  1  2021 libnppisu.so.10 -> libnppisu.so.10.2.1.300
-rw-r--r-- 1 root root    486576 Mar  1  2021 libnppisu.so.10.2.1.300
-rw-r--r-- 1 root root     11458 Mar  1  2021 libnppisu_static.a
lrwxrwxrwx 1 root root        15 Mar  1  2021 libnppitc.so -> libnppitc.so.10
lrwxrwxrwx 1 root root        23 Mar  1  2021 libnppitc.so.10 -> libnppitc.so.10.2.1.300
-rw-r--r-- 1 root root   3112480 Mar  1  2021 libnppitc.so.10.2.1.300
-rw-r--r-- 1 root root   3205362 Mar  1  2021 libnppitc_static.a
lrwxrwxrwx 1 root root        13 Mar  1  2021 libnpps.so -> libnpps.so.10
lrwxrwxrwx 1 root root        21 Mar  1  2021 libnpps.so.10 -> libnpps.so.10.2.1.300
-rw-r--r-- 1 root root   9539760 Mar  1  2021 libnpps.so.10.2.1.300
-rw-r--r-- 1 root root  10690508 Mar  1  2021 libnpps_static.a
lrwxrwxrwx 1 root root        15 Mar  1  2021 libnvblas.so -> libnvblas.so.10
lrwxrwxrwx 1 root root        23 Mar  1  2021 libnvblas.so.10 -> libnvblas.so.10.2.3.300
-rw-r--r-- 1 root root    540232 Mar  1  2021 libnvblas.so.10.2.3.300
lrwxrwxrwx 1 root root        16 Mar  1  2021 libnvgraph.so -> libnvgraph.so.10
lrwxrwxrwx 1 root root        22 Mar  1  2021 libnvgraph.so.10 -> libnvgraph.so.10.2.300
-rw-r--r-- 1 root root 165012616 Mar  1  2021 libnvgraph.so.10.2.300
-rw-r--r-- 1 root root 168141386 Mar  1  2021 libnvgraph_static.a
-rw-r--r-- 1 root root   7430712 Mar  1  2021 libnvperf_host.so
-rw-r--r-- 1 root root   1096016 Mar  1  2021 libnvperf_target.so
lrwxrwxrwx 1 root root        25 Mar  1  2021 libnvrtc-builtins.so -> libnvrtc-builtins.so.10.2
lrwxrwxrwx 1 root root        29 Mar  1  2021 libnvrtc-builtins.so.10.2 -> libnvrtc-builtins.so.10.2.300
-rw-r--r-- 1 root root   4794168 Mar  1  2021 libnvrtc-builtins.so.10.2.300
lrwxrwxrwx 1 root root        16 Mar  1  2021 libnvrtc.so -> libnvrtc.so.10.2
lrwxrwxrwx 1 root root        20 Mar  1  2021 libnvrtc.so.10.2 -> libnvrtc.so.10.2.300
-rw-r--r-- 1 root root  20432800 Mar  1  2021 libnvrtc.so.10.2.300
lrwxrwxrwx 1 root root        18 Mar  1  2021 libnvToolsExt.so -> libnvToolsExt.so.1
lrwxrwxrwx 1 root root        22 Mar  1  2021 libnvToolsExt.so.1 -> libnvToolsExt.so.1.0.0
-rw-r--r-- 1 root root     44088 Mar  1  2021 libnvToolsExt.so.1.0.0
drwxr-xr-x 2 root root      4096 Jul 26 22:31 stubs

And here is what it looks like from inside the container:

ls -ll /usr/local/cuda/lib64/
total 2259940
lrwxrwxrwx 1 root root        17 Mar  1  2021 libcublasLt.so -> libcublasLt.so.10
lrwxrwxrwx 1 root root        25 Mar  1  2021 libcublasLt.so.10 -> libcublasLt.so.10.2.3.300
-rw-r--r-- 1 root root  33562824 Mar  1  2021 libcublasLt.so.10.2.3.300
-rw-r--r-- 1 root root  36011742 Mar  1  2021 libcublasLt_static.a
lrwxrwxrwx 1 root root        15 Mar  1  2021 libcublas.so -> libcublas.so.10
lrwxrwxrwx 1 root root        23 Mar  1  2021 libcublas.so.10 -> libcublas.so.10.2.3.300
-rw-r--r-- 1 root root  81096256 Mar  1  2021 libcublas.so.10.2.3.300
-rw-r--r-- 1 root root  96903266 Mar  1  2021 libcublas_static.a
-rw-r--r-- 1 root root    679636 Mar  1  2021 libcudadevrt.a
lrwxrwxrwx 1 root root        17 Mar  1  2021 libcudart.so -> libcudart.so.10.2
lrwxrwxrwx 1 root root        21 Mar  1  2021 libcudart.so.10.2 -> libcudart.so.10.2.300
-rw-r--r-- 1 root root    490664 Mar  1  2021 libcudart.so.10.2.300
-rw-r--r-- 1 root root    888074 Mar  1  2021 libcudart_static.a
lrwxrwxrwx 1 root root        14 Mar  1  2021 libcufft.so -> libcufft.so.10
lrwxrwxrwx 1 root root        22 Mar  1  2021 libcufft.so.10 -> libcufft.so.10.1.2.300
-rw-r--r-- 1 root root 201494704 Mar  1  2021 libcufft.so.10.1.2.300
-rw-r--r-- 1 root root 192531512 Mar  1  2021 libcufft_static.a
-rw-r--r-- 1 root root 210524874 Mar  1  2021 libcufft_static_nocallback.a
lrwxrwxrwx 1 root root        15 Mar  1  2021 libcufftw.so -> libcufftw.so.10
lrwxrwxrwx 1 root root        23 Mar  1  2021 libcufftw.so.10 -> libcufftw.so.10.1.2.300
-rw-r--r-- 1 root root    503192 Mar  1  2021 libcufftw.so.10.1.2.300
-rw-r--r-- 1 root root     31970 Mar  1  2021 libcufftw_static.a
lrwxrwxrwx 1 root root        18 Mar  1  2021 libcuinj64.so -> libcuinj64.so.10.2
lrwxrwxrwx 1 root root        22 Mar  1  2021 libcuinj64.so.10.2 -> libcuinj64.so.10.2.300
-rw-r--r-- 1 root root   1535464 Mar  1  2021 libcuinj64.so.10.2.300
-rw-r--r-- 1 root root     33242 Mar  1  2021 libculibos.a
lrwxrwxrwx 1 root root        16 Mar  1  2021 libcupti.so -> libcupti.so.10.2
lrwxrwxrwx 1 root root        20 Mar  1  2021 libcupti.so.10.2 -> libcupti.so.10.2.175
-rw-r--r-- 1 root root   4526616 Mar  1  2021 libcupti.so.10.2.175
lrwxrwxrwx 1 root root        15 Mar  1  2021 libcurand.so -> libcurand.so.10
lrwxrwxrwx 1 root root        23 Mar  1  2021 libcurand.so.10 -> libcurand.so.10.1.2.300
-rw-r--r-- 1 root root  62698584 Mar  1  2021 libcurand.so.10.1.2.300
-rw-r--r-- 1 root root  62767380 Mar  1  2021 libcurand_static.a
lrwxrwxrwx 1 root root        17 Mar  1  2021 libcusolver.so -> libcusolver.so.10
lrwxrwxrwx 1 root root        25 Mar  1  2021 libcusolver.so.10 -> libcusolver.so.10.3.0.300
-rw-r--r-- 1 root root 218927328 Mar  1  2021 libcusolver.so.10.3.0.300
-rw-r--r-- 1 root root 123895098 Mar  1  2021 libcusolver_static.a
lrwxrwxrwx 1 root root        17 Mar  1  2021 libcusparse.so -> libcusparse.so.10
lrwxrwxrwx 1 root root        25 Mar  1  2021 libcusparse.so.10 -> libcusparse.so.10.3.1.300
-rw-r--r-- 1 root root 141252584 Mar  1  2021 libcusparse.so.10.3.1.300
-rw-r--r-- 1 root root 149512102 Mar  1  2021 libcusparse_static.a
-rw-r--r-- 1 root root   8319056 Mar  1  2021 liblapack_static.a
-rw-r--r-- 1 root root    909274 Mar  1  2021 libmetis_static.a
lrwxrwxrwx 1 root root        13 Mar  1  2021 libnppc.so -> libnppc.so.10
lrwxrwxrwx 1 root root        21 Mar  1  2021 libnppc.so.10 -> libnppc.so.10.2.1.300
-rw-r--r-- 1 root root    503184 Mar  1  2021 libnppc.so.10.2.1.300
-rw-r--r-- 1 root root     26846 Mar  1  2021 libnppc_static.a
lrwxrwxrwx 1 root root        15 Mar  1  2021 libnppial.so -> libnppial.so.10
lrwxrwxrwx 1 root root        23 Mar  1  2021 libnppial.so.10 -> libnppial.so.10.2.1.300
-rw-r--r-- 1 root root  11509472 Mar  1  2021 libnppial.so.10.2.1.300
-rw-r--r-- 1 root root  14410930 Mar  1  2021 libnppial_static.a
lrwxrwxrwx 1 root root        15 Mar  1  2021 libnppicc.so -> libnppicc.so.10
lrwxrwxrwx 1 root root        23 Mar  1  2021 libnppicc.so.10 -> libnppicc.so.10.2.1.300
-rw-r--r-- 1 root root   4914920 Mar  1  2021 libnppicc.so.10.2.1.300
-rw-r--r-- 1 root root   5722536 Mar  1  2021 libnppicc_static.a
lrwxrwxrwx 1 root root        16 Mar  1  2021 libnppicom.so -> libnppicom.so.10
lrwxrwxrwx 1 root root        24 Mar  1  2021 libnppicom.so.10 -> libnppicom.so.10.2.1.300
-rw-r--r-- 1 root root   1453728 Mar  1  2021 libnppicom.so.10.2.1.300
-rw-r--r-- 1 root root   1093680 Mar  1  2021 libnppicom_static.a
lrwxrwxrwx 1 root root        16 Mar  1  2021 libnppidei.so -> libnppidei.so.10
lrwxrwxrwx 1 root root        24 Mar  1  2021 libnppidei.so.10 -> libnppidei.so.10.2.1.300
-rw-r--r-- 1 root root   8175688 Mar  1  2021 libnppidei.so.10.2.1.300
-rw-r--r-- 1 root root  10762478 Mar  1  2021 libnppidei_static.a
lrwxrwxrwx 1 root root        14 Mar  1  2021 libnppif.so -> libnppif.so.10
lrwxrwxrwx 1 root root        22 Mar  1  2021 libnppif.so.10 -> libnppif.so.10.2.1.300
-rw-r--r-- 1 root root  54362944 Mar  1  2021 libnppif.so.10.2.1.300
-rw-r--r-- 1 root root  58471042 Mar  1  2021 libnppif_static.a
lrwxrwxrwx 1 root root        14 Mar  1  2021 libnppig.so -> libnppig.so.10
lrwxrwxrwx 1 root root        22 Mar  1  2021 libnppig.so.10 -> libnppig.so.10.2.1.300
-rw-r--r-- 1 root root  28761920 Mar  1  2021 libnppig.so.10.2.1.300
-rw-r--r-- 1 root root  31432462 Mar  1  2021 libnppig_static.a
lrwxrwxrwx 1 root root        14 Mar  1  2021 libnppim.so -> libnppim.so.10
lrwxrwxrwx 1 root root        22 Mar  1  2021 libnppim.so.10 -> libnppim.so.10.2.1.300
-rw-r--r-- 1 root root   7163640 Mar  1  2021 libnppim.so.10.2.1.300
-rw-r--r-- 1 root root   7396476 Mar  1  2021 libnppim_static.a
lrwxrwxrwx 1 root root        15 Mar  1  2021 libnppist.so -> libnppist.so.10
lrwxrwxrwx 1 root root        23 Mar  1  2021 libnppist.so.10 -> libnppist.so.10.2.1.300
-rw-r--r-- 1 root root  20877336 Mar  1  2021 libnppist.so.10.2.1.300
-rw-r--r-- 1 root root  23399160 Mar  1  2021 libnppist_static.a
lrwxrwxrwx 1 root root        15 Mar  1  2021 libnppisu.so -> libnppisu.so.10
lrwxrwxrwx 1 root root        23 Mar  1  2021 libnppisu.so.10 -> libnppisu.so.10.2.1.300
-rw-r--r-- 1 root root    486576 Mar  1  2021 libnppisu.so.10.2.1.300
-rw-r--r-- 1 root root     11458 Mar  1  2021 libnppisu_static.a
lrwxrwxrwx 1 root root        15 Mar  1  2021 libnppitc.so -> libnppitc.so.10
lrwxrwxrwx 1 root root        23 Mar  1  2021 libnppitc.so.10 -> libnppitc.so.10.2.1.300
-rw-r--r-- 1 root root   3112480 Mar  1  2021 libnppitc.so.10.2.1.300
-rw-r--r-- 1 root root   3205362 Mar  1  2021 libnppitc_static.a
lrwxrwxrwx 1 root root        13 Mar  1  2021 libnpps.so -> libnpps.so.10
lrwxrwxrwx 1 root root        21 Mar  1  2021 libnpps.so.10 -> libnpps.so.10.2.1.300
-rw-r--r-- 1 root root   9539760 Mar  1  2021 libnpps.so.10.2.1.300
-rw-r--r-- 1 root root  10690508 Mar  1  2021 libnpps_static.a
lrwxrwxrwx 1 root root        15 Mar  1  2021 libnvblas.so -> libnvblas.so.10
lrwxrwxrwx 1 root root        23 Mar  1  2021 libnvblas.so.10 -> libnvblas.so.10.2.3.300
-rw-r--r-- 1 root root    540232 Mar  1  2021 libnvblas.so.10.2.3.300
lrwxrwxrwx 1 root root        16 Mar  1  2021 libnvgraph.so -> libnvgraph.so.10
lrwxrwxrwx 1 root root        22 Mar  1  2021 libnvgraph.so.10 -> libnvgraph.so.10.2.300
-rw-r--r-- 1 root root 165012616 Mar  1  2021 libnvgraph.so.10.2.300
-rw-r--r-- 1 root root 168141386 Mar  1  2021 libnvgraph_static.a
-rw-r--r-- 1 root root   7430712 Mar  1  2021 libnvperf_host.so
-rw-r--r-- 1 root root   1096016 Mar  1  2021 libnvperf_target.so
lrwxrwxrwx 1 root root        25 Mar  1  2021 libnvrtc-builtins.so -> libnvrtc-builtins.so.10.2
lrwxrwxrwx 1 root root        29 Mar  1  2021 libnvrtc-builtins.so.10.2 -> libnvrtc-builtins.so.10.2.300
-rw-r--r-- 1 root root   4794168 Mar  1  2021 libnvrtc-builtins.so.10.2.300
lrwxrwxrwx 1 root root        16 Mar  1  2021 libnvrtc.so -> libnvrtc.so.10.2
lrwxrwxrwx 1 root root        20 Mar  1  2021 libnvrtc.so.10.2 -> libnvrtc.so.10.2.300
-rw-r--r-- 1 root root  20432800 Mar  1  2021 libnvrtc.so.10.2.300
lrwxrwxrwx 1 root root        18 Mar  1  2021 libnvToolsExt.so -> libnvToolsExt.so.1
lrwxrwxrwx 1 root root        22 Mar  1  2021 libnvToolsExt.so.1 -> libnvToolsExt.so.1.0.0
-rw-r--r-- 1 root root     44088 Mar  1  2021 libnvToolsExt.so.1.0.0
drwxr-xr-x 2 root root      4096 Jul 26 22:31 stubs

If you are missing libraries, you may want to re-install JetPack or reflash your device / SD card.

I see the same as you outside on the host. I’m just missing everything inside the container. It seems not to be mapping from the container to the host (which I though the nvidia-container-host was supposed to do).

Hoping to avoid reflashing if possible.

One other bit of info. I ran into the issue where I apt upgrade’d containerd and docker.io. It borked. Had to manually downgrade. Could this potentially be a root cause?

If you run other containers (such as l4t-base or l4t-pytorch or jetson-inference) with --runtime nvidia, does it mount the files correctly, or is this issue specific to the jetson-voice container?

Ah - I’ve not done the upgrade/downgrade myself, but it could be. Can you try adding --security-opt seccomp=unconfined as a flag in the docker run command in this line of the docker/run.sh script?

https://github.com/dusty-nv/jetson-voice/blob/1ec13b77f493d399f31c31f0af0650bcdbca8bc0/docker/run.sh#L176

If you’re unable to get the Docker runtime working again where it mounts the files correctly, you may save yourself more time by just re-flashing - sorry about that.

You SHOULD! It’s a blast. ;)

Unfortunately, yes, the behavior is apparently the same in all containers. Setting the security-opt flag as such:

        sudo nvidia-docker run --runtime nvidia -it --rm \
                --security-opt seccomp=unconfined \
                --name=$CONTAINER_NAME \
                --network host \
                $MOUNTS $CONTAINER_IMAGE $USER_COMMAND

Didn’t work.

One other variable. I’ve configured this to boot from NVMe using ./nvsdkmanager_flash.sh –storage nvme0n1p1. Before reflashing only to find out the problem didn’t go away, is there anything about using this flash method that might impact this? Maybe a hard-coded location in nvidia-container-runtime that presumes SD storage?

Offhand I don’t believe so, although I haven’t done boot from NVME myself either (sorry) - typically I just put my docker root directory (where all the containers themselves live) onto NVME and call it a day.

The CSV files that contain the files to mount are located under /etc/nvidia-container-runtime/host-files-for-container.d/, so you might want to check that you still have these:

ls /etc/nvidia-container-runtime/host-files-for-container.d/
cuda.csv  cudnn.csv  l4t.csv  opencv.csv  tensorrt.csv  visionworks.csv

BINGO! I was missing cuda.csv. Copied it from one of my Nanos and that fixed that problem. Presumably sudo apt install nvidia-container-csv-cuda would have worked, too.

Of course, then I found out that during my “reinstall everything” efforts, I must’ve nuked pycuda and with it the ability to use TensorRT from Python. Which I can’t figure out how to properly install. Arrhhgghah! Smelling a reflash in my future. :(

Thankfully @kangalow saved the day without even knowing it. Just had to run the mammoth apt install from bootFromExternalStorage/install_jetson_default_packages.sh (github.com). At least it got me to the point that I could run everything. Doubtless would’ve fixed it from the start. Need to remember this next time.

Still some issues (e.g., failing a few of the tests) that I’m sure I’ll ask about in a separate thread. Regardless, thanks for all of your help @dusty_nv!!

1 Like

OK great, glad to hear that you got it working!

Regarding the tests, I think I know what you mean - IIRC the ASR classification models (Matchboxnet / Marblenet) fail on JetPack 4.6. Been meaning to look into that - I recall it being a pre-processing error due to using a newer version of PyTorch. There may also have been a failure in one of the NLP tests, but I think it was just because of a minorly different output than expected, but was still functional (I do a simple text comparison to see if the outputs match, and sometimes they can be slightly different).

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.