I am trying to use nvidia containers in docker with the Xavier. I have it successfully deploying to two jetson nanos, but the master (Xavier) does not find CUDA. I installed it via the SDK manager and all deployed well. I added the path to LD_LIBRARY_PATH as shown in output but it says it isn’t there.
tz@tz-ai:~/tz-nn-api$ sudo docker run -it --rm --network host 192.168.1.137:5000/tz-ai
2020-07-29 16:50:24.337633: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library ‘libcudart.so.10.2’; dlerror: libcudart.so.10.2: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.2/targets/aarch64-linux/lib/:/usr/local/lib/:/usr/local/cuda-10.2/lib64:/usr/lib/aarch64-linux-gnu:/usr/lib/aarch64-linux-gnu/tegra:/lib/aarch64-linux-gnu:/usr/local/cuda/lib64:/usr/local/cuda-10.2/targets/aarch64-linux/lib:
2020-07-29 16:50:24.337706: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2020-07-29 16:50:24.337936: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library ‘libcudart.so.10.2’; dlerror: libcudart.so.10.2: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.2/targets/aarch64-linux/lib/:/usr/local/lib/:/usr/local/cuda-10.2/lib64:/usr/lib/aarch64-linux-gnu:/usr/lib/aarch64-linux-gnu/tegra:/lib/aarch64-linux-gnu:/usr/local/cuda/lib64:/usr/local/cuda-10.2/targets/aarch64-linux/lib:
2020-07-29 16:50:24.337977: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
tz@tz-ai:~/tz-nn-api$ cd /usr/local/cuda-10.2/targets/aarch64-linux/lib
tz@tz-ai:/usr/local/cuda-10.2/targets/aarch64-linux/lib$ ls
libcudadevrt.a libcuinj64.so.10.2 libcusparse.so.10 libnppicc.so.10.2.1.89 libnppig.so.10 libnppitc.so libnvrtc-builtins.so.10.2
libcudart.so libcuinj64.so.10.2.89 libcusparse.so.10.3.1.89 libnppicc_static.a libnppig.so.10.2.1.89 libnppitc.so.10 libnvrtc-builtins.so.10.2.89
libcudart.so.10.2 libculibos.a libcusparse_static.a libnppicom.so libnppig_static.a libnppitc.so.10.2.1.89 libnvrtc.so
libcudart.so.10.2.89 libcupti.so liblapack_static.a libnppicom.so.10 libnppim.so libnppitc_static.a libnvrtc.so.10.2
libcudart_static.a libcupti.so.10.2 libmetis_static.a libnppicom.so.10.2.1.89 libnppim.so.10 libnpps.so libnvrtc.so.10.2.89
libcufft.so libcupti.so.10.2.75 libnppc.so libnppicom_static.a libnppim.so.10.2.1.89 libnpps.so.10 libnvToolsExt.so
libcufft.so.10 libcurand.so libnppc.so.10 libnppidei.so libnppim_static.a libnpps.so.10.2.1.89 libnvToolsExt.so.1
libcufft.so.10.1.2.89 libcurand.so.10 libnppc.so.10.2.1.89 libnppidei.so.10 libnppist.so libnpps_static.a libnvToolsExt.so.1.0.0
libcufft_static.a libcurand.so.10.1.2.89 libnppc_static.a libnppidei.so.10.2.1.89 libnppist.so.10 libnvgraph.so stubs
libcufft_static_nocallback.a libcurand_static.a libnppial.so libnppidei_static.a libnppist.so.10.2.1.89 libnvgraph.so.10
libcufftw.so libcusolver.so libnppial.so.10 libnppif.so libnppist_static.a libnvgraph.so.10.2.89
libcufftw.so.10 libcusolver.so.10 libnppial.so.10.2.1.89 libnppif.so.10 libnppisu.so libnvgraph_static.a
libcufftw.so.10.1.2.89 libcusolver.so.10.3.0.89 libnppial_static.a libnppif.so.10.2.1.89 libnppisu.so.10 libnvperf_host.so
libcufftw_static.a libcusolver_static.a libnppicc.so libnppif_static.a libnppisu.so.10.2.1.89 libnvperf_target.so
libcuinj64.so libcusparse.so libnppicc.so.10 libnppig.so libnppisu_static.a libnvrtc-builtins.so
tz@tz-ai:/usr/local/cuda-10.2/targets/aarch64-linux/lib$