Isaac Sim 2019.2 "Warehouse Scene Navigation": Isaac application terminated unexpectedly

bsamadi · August 5, 2019, 2:24pm

When I run the Warehouse Scene Navigation, the simulation shows up but the app fails with this error:

2019-08-05 09:51:04.355 INFO    engine/alice/backend/modules.cpp@283: Loaded module 'packages/viewers/libviewers_module.so': Now has 120 components total
2019-08-05 09:51:04.356 ERROR   engine/alice/backend/modules.cpp@307: ml: /home/nubot/.cache/bazel/_bazel_nubot/24cb462690d68313a57e2ec216711a4d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/carter/navsim/navsim_navigate.runfiles/com_nvidia_isaac//external/com_nvidia_isaac/packages/ml/libml_module.so: cannot open shared object file: No such file or directory
2019-08-05 09:51:04.356 PANIC   engine/alice/backend/modules.cpp@309: Could not load all required modules for application
====================================================================================================
|                            Isaac application terminated unexpectedly                             |
====================================================================================================

The same error happens for the “Object Detection Training Data” example.

qianl · August 5, 2019, 10:40pm

Please install Isaac SDK dependencies first
bob@desktop:~/isaac$ engine/build/scripts/install_dependencies.sh

(https://docs.nvidia.com/isaac/isaac/doc/setup.html)

bsamadi · August 5, 2019, 10:58pm

Thank you qianl. I had done it. I did it again but still I got the same error.

qianl · August 6, 2019, 6:13pm

It’s most likely due to missing depedencies of libml_module.so (or their dependencies).
Try follow the “NVIDIA GPU Driver” session here:
[url]https://docs.nvidia.com/isaac/isaac/doc/setup.html#machine-learning-optional[/url]

behzad.samadi · August 7, 2019, 8:46pm

I have already installed the NVIDIA driver. I think libml_module has something to do with machine learning. Can anyone help me trace the issue?

qianl · August 7, 2019, 11:23pm

libml_module is part of Isaac SDK. navsim.subgraph.json uses a codelet in that module.

goto isaac/bazel-bin/packages/ml, run
sudo updatedb
ldd libml_module.so

and post the result here.

bsamadi · August 8, 2019, 2:53am

Thanks Qianl. I have CUDA 10.1. Could that be the problem?

This is the output:

linux-vdso.so.1 (0x00007ffe6c9f8000)
	libml_module.so => /home/nubot/.cache/bazel/_bazel_nubot/24cb462690d68313a57e2ec216711a4d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/ml/./../../_solib_k8/_U_S_Spackages_Sml_Clibml_Umodule.so___Upackages_Ux86_U64_Sml/libml_module.so (0x00007f5006ffd000)
	libnvinfer.so.5 => /home/nubot/.cache/bazel/_bazel_nubot/24cb462690d68313a57e2ec216711a4d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/ml/./../../_solib_k8/_U@tensorrt_Ux86_U64_S_S_Ctensorrt_Ux86_U64___Uexternal_Stensorrt_Ux86_U64_Slib_Sx86_U64-linux-gnu/libnvinfer.so.5 (0x00007f4fffb9e000)
	libnvinfer_plugin.so.5 => /home/nubot/.cache/bazel/_bazel_nubot/24cb462690d68313a57e2ec216711a4d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/ml/./../../_solib_k8/_U@tensorrt_Ux86_U64_S_S_Ctensorrt_Ux86_U64___Uexternal_Stensorrt_Ux86_U64_Slib_Sx86_U64-linux-gnu/libnvinfer_plugin.so.5 (0x00007f4fff66a000)
	libnvonnxparser.so.0 => /home/nubot/.cache/bazel/_bazel_nubot/24cb462690d68313a57e2ec216711a4d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/ml/./../../_solib_k8/_U@tensorrt_Ux86_U64_S_S_Ctensorrt_Ux86_U64___Uexternal_Stensorrt_Ux86_U64_Slib_Sx86_U64-linux-gnu/libnvonnxparser.so.0 (0x00007f4ffeead000)
	libnvonnxparser_runtime.so.0 => /home/nubot/.cache/bazel/_bazel_nubot/24cb462690d68313a57e2ec216711a4d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/ml/./../../_solib_k8/_U@tensorrt_Ux86_U64_S_S_Ctensorrt_Ux86_U64___Uexternal_Stensorrt_Ux86_U64_Slib_Sx86_U64-linux-gnu/libnvonnxparser_runtime.so.0 (0x00007f4ffe8ff000)
	libnvparsers.so.5 => /home/nubot/.cache/bazel/_bazel_nubot/24cb462690d68313a57e2ec216711a4d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/ml/./../../_solib_k8/_U@tensorrt_Ux86_U64_S_S_Ctensorrt_Ux86_U64___Uexternal_Stensorrt_Ux86_U64_Slib_Sx86_U64-linux-gnu/libnvparsers.so.5 (0x00007f4ffe3e1000)
	libcudnn.so.7 => /home/nubot/.cache/bazel/_bazel_nubot/24cb462690d68313a57e2ec216711a4d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/ml/./../../_solib_k8/_U@cuda_Ux86_U64_S_S_Ccudnn___Uexternal_Scuda_Ux86_U64_Susr_Slib_Sx86_U64-linux-gnu/libcudnn.so.7 (0x00007f4fe95ba000)
	libcudart.so.10.0 => /home/nubot/.cache/bazel/_bazel_nubot/24cb462690d68313a57e2ec216711a4d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/ml/./../../_solib_k8/_U@cuda_Ux86_U64_S_S_Ccudart___Uexternal_Scuda_Ux86_U64_Susr_Slocal_Scuda-10.0_Slib64/libcudart.so.10.0 (0x00007f4fe9340000)
	libcublas.so.10.0 => /home/nubot/.cache/bazel/_bazel_nubot/24cb462690d68313a57e2ec216711a4d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/ml/./../../_solib_k8/_U@cuda_Ux86_U64_S_S_Ccublas___Uexternal_Scuda_Ux86_U64_Susr_Slocal_Scuda-10.0_Slib64/libcublas.so.10.0 (0x00007f4fe4daa000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f4fe49ca000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f4fe462c000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4fe423b000)
	libnppial.so.10.0 => /home/nubot/.cache/bazel/_bazel_nubot/24cb462690d68313a57e2ec216711a4d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/ml/./../../_solib_k8/_U_S_Spackages_Sml_Clibml_Umodule.so___Upackages_Ux86_U64_Sml/../../_solib_k8/_U@cuda_Ux86_U64_S_S_Cnppial___Uexternal_Scuda_Ux86_U64_Susr_Slocal_Scuda-10.0_Slib64/libnppial.so.10.0 (0x00007f4fe3625000)
	libnppidei.so.10.0 => /home/nubot/.cache/bazel/_bazel_nubot/24cb462690d68313a57e2ec216711a4d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/ml/./../../_solib_k8/_U_S_Spackages_Sml_Clibml_Umodule.so___Upackages_Ux86_U64_Sml/../../_solib_k8/_U@cuda_Ux86_U64_S_S_Cnppidei___Uexternal_Scuda_Ux86_U64_Susr_Slocal_Scuda-10.0_Slib64/libnppidei.so.10.0 (0x00007f4fe2d1a000)
	libnppif.so.10.0 => /home/nubot/.cache/bazel/_bazel_nubot/24cb462690d68313a57e2ec216711a4d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/ml/./../../_solib_k8/_U_S_Spackages_Sml_Clibml_Umodule.so___Upackages_Ux86_U64_Sml/../../_solib_k8/_U@cuda_Ux86_U64_S_S_Cnppif___Uexternal_Scuda_Ux86_U64_Susr_Slocal_Scuda-10.0_Slib64/libnppif.so.10.0 (0x00007f4fdfe0a000)
	libnppig.so.10.0 => /home/nubot/.cache/bazel/_bazel_nubot/24cb462690d68313a57e2ec216711a4d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/ml/./../../_solib_k8/_U_S_Spackages_Sml_Clibml_Umodule.so___Upackages_Ux86_U64_Sml/../../_solib_k8/_U@cuda_Ux86_U64_S_S_Cnppig___Uexternal_Scuda_Ux86_U64_Susr_Slocal_Scuda-10.0_Slib64/libnppig.so.10.0 (0x00007f4fde423000)
	libnpps.so.10.0 => /home/nubot/.cache/bazel/_bazel_nubot/24cb462690d68313a57e2ec216711a4d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/ml/./../../_solib_k8/_U_S_Spackages_Sml_Clibml_Umodule.so___Upackages_Ux86_U64_Sml/../../_solib_k8/_U@cuda_Ux86_U64_S_S_Cnpps___Uexternal_Scuda_Ux86_U64_Susr_Slocal_Scuda-10.0_Slib64/libnpps.so.10.0 (0x00007f4fdd9f1000)
	libnppc.so.10.0 => /home/nubot/.cache/bazel/_bazel_nubot/24cb462690d68313a57e2ec216711a4d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/ml/./../../_solib_k8/_U_S_Spackages_Sml_Clibml_Umodule.so___Upackages_Ux86_U64_Sml/../../_solib_k8/_U@cuda_Ux86_U64_S_S_Cnppc___Uexternal_Scuda_Ux86_U64_Susr_Slocal_Scuda-10.0_Slib64/libnppc.so.10.0 (0x00007f4fdd769000)
	libcurand.so.10.0 => not found
	libcusolver.so.10.0 => not found
	libcuda.so.1 => /usr/lib/x86_64-linux-gnu/libcuda.so.1 (0x00007f4fdc5f9000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f4fdc3f5000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f4fdc1dd000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4fdbfbe000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f500746d000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f4fdbdb6000)
	libnvidia-fatbinaryloader.so.430.40 => /usr/lib/x86_64-linux-gnu/libnvidia-fatbinaryloader.so.430.40 (0x00007f4fdbb68000)

qianl · August 8, 2019, 5:07pm

these are the missing pieces, so yes you’ll need CUDA 10.0 (this is the CUDA version used in Jetpack 4.2.1, so we keep it for consistency)

libcurand.so.10.0 => /usr/local/cuda-10.0/targets/x86_64-linux/lib/libcurand.so.10.0 (0x00007faa4f059000)
libcusolver.so.10.0 => /usr/local/cuda-10.0/targets/x86_64-linux/lib/libcusolver.so.10.0 (0x00007faa46971000)

you can just install CUDA 10.0 alongside CUDA 10.1

bsamadi · August 10, 2019, 12:39am

I tried to install CUDA 10.0 from the NVIDIA website (the current default is 10.1) but I didn’t have much success until I found the solution:

sudo apt-get install cuda-10-0

Source: [url]https://devtalk.nvidia.com/default/topic/1050914/cuda-setup-and-installation/cuda-remove-10-1-and-install-10-0-ubuntu-18-04/[/url]

randaz81 · November 26, 2019, 12:32pm

I have the same issue.

Here is another topic describing the same problem:
https://devtalk.nvidia.com/default/topic/1066886/sdk/libml_module-so-panic-error/post/5403415/?offset=2#5405717

shrinv · February 18, 2020, 10:59pm

Please check https://devtalk.nvidia.com/default/topic/1066886/isaac-sdk-isaac-sim-for-navigation/libml_module-so-panic-error/post/5405873/#5405873

infox22oo · February 25, 2020, 9:54pm

Just a quick note that a fresh install of Isaac 2019.3 on Ubuntu 18.04 LTS does allow this example to work