Isaac SDK 2019.3: Isaac application terminated unexpectedly

When I run the Warehouse Scene Navigation, or any kind of navigation command, I get an error. The full message from the terminal is shown below:

soeren@soeren-MS-7B98:~/isaac$ bazel run //apps/navsim:navsim_navigate -- --more packages/navsim/maps/medium_warehouse.json,packages/navsim/robots/carter.json
WARNING: /home/soeren/isaac/packages/viewers/BUILD:14:12: in srcs attribute of cc_binary rule //packages/viewers:libviewers_module.so: please do not import '//packages_x86_64/viewers:libviewers_module.so' directly. You should either move the file to this package or depend on an appropriate rule there
WARNING: /home/soeren/isaac/packages/perception/BUILD:34:12: in srcs attribute of cc_binary rule //packages/perception:libperception_module.so: please do not import '//packages_x86_64/perception:libperception_module.so' directly. You should either move the file to this package or depend on an appropriate rule there
WARNING: /home/soeren/isaac/packages/ml/BUILD:15:12: in srcs attribute of cc_binary rule //packages/ml:libml_module.so: please do not import '//packages_x86_64/ml:libml_module.so' directly. You should either move the file to this package or depend on an appropriate rule there
INFO: Analysed target //apps/navsim:navsim_navigate (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //apps/navsim:navsim_navigate up-to-date:
  bazel-bin/apps/navsim/navsim_navigate
INFO: Elapsed time: 0.229s, Critical Path: 0.00s, Remote (0.00% of the time): [queue: 0.00%, setup: 0.00%, process: 0.00%]
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/apps/navsim/navsim_navigate --more packageINFO: Build completed successfully, 1 total action
2020-02-17 12:22:55.872 INFO  engine/alice/tools/websight.cpp@166: Loading websight...
2020-02-17 12:22:55.873 WARN  engine/alice/backend/application_json_loader.cpp@171: This application does not have an explicit scheduler configuration. One will be autogenerated to the best of the system's abilities if possible.
2020-02-17 12:22:55.873 INFO  engine/alice/backend/application_json_loader.cpp@340: Modules requested by subgraph 'simulation': navsim, ml, perception, utils, 
2020-02-17 12:22:55.873 INFO  engine/alice/backend/application_json_loader.cpp@340: Modules requested by subgraph 'noiser': (navsim), 
2020-02-17 12:22:55.873 INFO  engine/alice/backend/application_json_loader.cpp@340: Modules requested by subgraph 'navigation': map, navigation, planner, (sight), 
2020-02-17 12:22:55.873 INFO  engine/alice/backend/application_json_loader.cpp@340: Modules requested by subgraph 'local_map': (navigation), 
2020-02-17 12:22:55.873 INFO  engine/alice/backend/application_json_loader.cpp@340: Modules requested by subgraph 'go_to': (navigation), (viewers), 
2020-02-17 12:22:55.873 INFO  engine/alice/backend/application_json_loader.cpp@340: Modules requested by subgraph 'planner': (navigation), (planner), 
2020-02-17 12:22:55.873 INFO  engine/alice/backend/application_json_loader.cpp@340: Modules requested by subgraph 'control': (navigation), (planner), 
2020-02-17 12:22:55.874 INFO  engine/alice/backend/application_json_loader.cpp@340: Modules requested by subgraph 'localization': (navigation), (utils), (viewers), 
2020-02-17 12:22:55.874 INFO  engine/alice/backend/application_json_loader.cpp@340: Modules requested by subgraph 'imu_odometry': imu, (navigation), 
2020-02-17 12:22:55.874 INFO  engine/alice/backend/application_json_loader.cpp@340: Modules requested by subgraph 'goals': (navigation), 
2020-02-17 12:22:55.877 WARN  engine/alice/backend/backend.cpp@183: This application does not have an execution group configuration. One will be autogenerated to the best of the systems abilities if possible.
2020-02-17 12:22:55.877 WARN  engine/gems/scheduler/scheduler.cpp@337: No default execution groups specified. Attempting to create scheduler configuration for 8 remaining cores. This may be non optimal for the system and application.
2020-02-17 12:22:55.877 INFO  engine/gems/scheduler/scheduler.cpp@290: Scheduler execution groups are:
2020-02-17 12:22:55.877 INFO  engine/gems/scheduler/scheduler.cpp@299: __BlockerGroup__: Cores = [6, 7], Workers = No
2020-02-17 12:22:55.877 INFO  engine/gems/scheduler/scheduler.cpp@299: __WorkerGroup__: Cores = [0, 1, 2, 3, 4, 5], Workers = Yes
2020-02-17 12:22:55.881 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/imu/libimu_module.so': Now has 50 components total
2020-02-17 12:22:55.885 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/map/libmap_module.so': Now has 57 components total
2020-02-17 12:22:55.893 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/navigation/libnavigation_module.so': Now has 94 components total
2020-02-17 12:22:55.899 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/navsim/libnavsim_module.so': Now has 97 components total
2020-02-17 12:22:55.906 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/perception/libperception_module.so': Now has 108 components total
2020-02-17 12:22:55.914 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/planner/libplanner_module.so': Now has 117 components total
2020-02-17 12:22:55.921 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/sight/libsight_module.so': Now has 119 components total
2020-02-17 12:22:55.928 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/utils/libutils_module.so': Now has 130 components total
2020-02-17 12:22:55.937 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/viewers/libviewers_module.so': Now has 149 components total
2020-02-17 12:22:55.937 ERROR engine/alice/backend/modules.cpp@250: packages/ml/libml_module.so: libcurand.so.10.0: cannot open shared object file: No such file or directory
2020-02-17 12:22:55.937 PANIC engine/alice/backend/modules.cpp@252: Could not load all required modules for application
====================================================================================================
|                            Isaac application terminated unexpectedly                             |
====================================================================================================
#01 engine/alice/tools/main(+0x880fa) [0x5631d596c0fa]
#02 engine/alice/tools/main(+0xf97d0) [0x5631d59dd7d0]
#03 engine/alice/tools/main(+0xf9b40) [0x5631d59ddb40]
#04 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f59f39c6890]
#05 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7f59f3060e97]
#06 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7f59f3062801]
#07 engine/alice/tools/main(+0xa33c7) [0x5631d59873c7]
#08 engine/alice/tools/main(+0x3b91c) [0x5631d591f91c]
#09 engine/alice/tools/main(+0x3cc39) [0x5631d5920c39]
#10 engine/alice/tools/main(+0x3cfdf) [0x5631d5920fdf]
#11 engine/alice/tools/main(+0x17433) [0x5631d58fb433]
#12 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f59f3043b97]
#13 engine/alice/tools/main(+0x1d4ca) [0x5631d59014ca]
====================================================================================================
Minidump written to: /tmp/c360f386-2b40-43c6-3c5deaa2-30aa6c77.dmp
/home/soeren/.cache/bazel/_bazel_soeren/095f5ffa1280b1937123489b32371c1b/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/navsim/navsim_navigate: line 38: 21963 Aborted                 (core dumped) engine/alice/tools/main --app apps/navsim/navsim_navigate.app.json --more "packages/navsim/robots/$ROBOT_ID.json,packages/navsim/maps/$MAP.json" $@

The setup which is been used, is a clean system with Ubuntu 18.04 LTS installed. To install the programs the guide described in your documentation was followed. Each step I took is explained below, so you can see if I made an error somewhere.
First two folders, isaac and isaac_sim_unity3d, was created. The Isaac SDK (2019.3) and Isaac Sim files was extracted to their folders using the command

tar -xJvf isaac_sdk-2019.1-17919.tar.xz -C ~/isaac/

Afterwards the dependencies was installed using

engine/build/scripts/install_dependencies.sh

Afterwards Unity Hub was installed from the following link https://forum.unity.com/threads/unity-hub-v-1-3-2-is-now-available.594139/

And then I ran the two commands

chmod +x UnityHub.AppImage

and

/UnityHub.AppImage

When the last command is run it opens Unity as it’s supposed to, but give an small error message:

Gtk-Message: 14:20:31.879: Failed to load module "canberra-gtk-module"

The newest version of Unity was installed (2019.3.1f1), and no extra modules was installed.

The simulator was started up using the command:

./isaac_sim_unity3d.x86_64 --scene small_warehouse

Which seems to work perfectly, but when I run the command

bazel run //apps/navsim:navsim_navigate

I get the error message seen below, which is almost identical, to the one all the way at the top:

soeren@soeren-MS-7B98:~/isaac$ bazel run //apps/navsim:navsim_navigate
WARNING: /home/soeren/isaac/packages/viewers/BUILD:14:12: in srcs attribute of cc_binary rule //packages/viewers:libviewers_module.so: please do not import '//packages_x86_64/viewers:libviewers_module.so' directly. You should either move the file to this package or depend on an appropriate rule there
WARNING: /home/soeren/isaac/packages/perception/BUILD:34:12: in srcs attribute of cc_binary rule //packages/perception:libperception_module.so: please do not import '//packages_x86_64/perception:libperception_module.so' directly. You should either move the file to this package or depend on an appropriate rule there
WARNING: /home/soeren/isaac/packages/ml/BUILD:15:12: in srcs attribute of cc_binary rule //packages/ml:libml_module.so: please do not import '//packages_x86_64/ml:libml_module.so' directly. You should either move the file to this package or depend on an appropriate rule there
INFO: Analysed target //apps/navsim:navsim_navigate (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //apps/navsim:navsim_navigate up-to-date:
  bazel-bin/apps/navsim/navsim_navigate
INFO: Elapsed time: 0.151s, Critical Path: 0.00s, Remote (0.00% of the time): [queue: 0.00%, setup: 0.00%, process: 0.00%]
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
2020-02-17 14:05:14.357 INFO  engine/alice/tools/websight.cpp@166: Loading websight...
2020-02-17 14:05:14.357 WARN  engine/alice/backend/application_json_loader.cpp@171: This application does not have an explicit scheduler configuration. One will be autogenerated to the best of the system's abilities if possible.
2020-02-17 14:05:14.358 INFO  engine/alice/backend/application_json_loader.cpp@340: Modules requested by subgraph 'simulation': navsim, ml, perception, utils, 
2020-02-17 14:05:14.358 INFO  engine/alice/backend/application_json_loader.cpp@340: Modules requested by subgraph 'noiser': (navsim), 
2020-02-17 14:05:14.358 INFO  engine/alice/backend/application_json_loader.cpp@340: Modules requested by subgraph 'navigation': map, navigation, planner, (sight), 
2020-02-17 14:05:14.358 INFO  engine/alice/backend/application_json_loader.cpp@340: Modules requested by subgraph 'local_map': (navigation), 
2020-02-17 14:05:14.358 INFO  engine/alice/backend/application_json_loader.cpp@340: Modules requested by subgraph 'go_to': (navigation), (viewers), 
2020-02-17 14:05:14.358 INFO  engine/alice/backend/application_json_loader.cpp@340: Modules requested by subgraph 'planner': (navigation), (planner), 
2020-02-17 14:05:14.358 INFO  engine/alice/backend/application_json_loader.cpp@340: Modules requested by subgraph 'control': (navigation), (planner), 
2020-02-17 14:05:14.358 INFO  engine/alice/backend/application_json_loader.cpp@340: Modules requested by subgraph 'localization': (navigation), (utils), (viewers), 
2020-02-17 14:05:14.359 INFO  engine/alice/backend/application_json_loader.cpp@340: Modules requested by subgraph 'imu_odometry': imu, (navigation), 
2020-02-17 14:05:14.359 INFO  engine/alice/backend/application_json_loader.cpp@340: Modules requested by subgraph 'goals': (navigation), 
2020-02-17 14:05:14.359 WARN  engine/alice/backend/backend.cpp@183: This application does not have an execution group configuration. One will be autogenerated to the best of the systems abilities if possible.
2020-02-17 14:05:14.359 WARN  engine/gems/scheduler/scheduler.cpp@337: No default execution groups specified. Attempting to create scheduler configuration for 8 remaining cores. This may be non optimal for the system and application.
2020-02-17 14:05:14.359 INFO  engine/gems/scheduler/scheduler.cpp@290: Scheduler execution groups are:
2020-02-17 14:05:14.359 INFO  engine/gems/scheduler/scheduler.cpp@299: __BlockerGroup__: Cores = [6, 7], Workers = No
2020-02-17 14:05:14.359 INFO  engine/gems/scheduler/scheduler.cpp@299: __WorkerGroup__: Cores = [0, 1, 2, 3, 4, 5], Workers = Yes
2020-02-17 14:05:14.363 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/imu/libimu_module.so': Now has 50 components total
2020-02-17 14:05:14.367 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/map/libmap_module.so': Now has 57 components total
2020-02-17 14:05:14.374 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/navigation/libnavigation_module.so': Now has 94 components total
2020-02-17 14:05:14.380 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/navsim/libnavsim_module.so': Now has 97 components total
2020-02-17 14:05:14.388 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/perception/libperception_module.so': Now has 108 components total
2020-02-17 14:05:14.395 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/planner/libplanner_module.so': Now has 117 components total
2020-02-17 14:05:14.402 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/sight/libsight_module.so': Now has 119 components total
2020-02-17 14:05:14.410 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/utils/libutils_module.so': Now has 130 components total
2020-02-17 14:05:14.419 INFO  engine/alice/backend/modules.cpp@226: Loaded module 'packages/viewers/libviewers_module.so': Now has 149 components total
2020-02-17 14:05:14.419 ERROR engine/alice/backend/modules.cpp@250: packages/ml/libml_module.so: libcurand.so.10.0: cannot open shared object file: No such file or directory
2020-02-17 14:05:14.419 PANIC engine/alice/backend/modules.cpp@252: Could not load all required modules for application
====================================================================================================
|                            Isaac application terminated unexpectedly                             |
====================================================================================================
#01 engine/alice/tools/main(+0x880fa) [0x55725ad810fa]
#02 engine/alice/tools/main(+0xf97d0) [0x55725adf27d0]
#03 engine/alice/tools/main(+0xf9b40) [0x55725adf2b40]
#04 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f04774d9890]
#05 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7f0476b73e97]
#06 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7f0476b75801]
#07 engine/alice/tools/main(+0xa33c7) [0x55725ad9c3c7]
#08 engine/alice/tools/main(+0x3b91c) [0x55725ad3491c]
#09 engine/alice/tools/main(+0x3cc39) [0x55725ad35c39]
#10 engine/alice/tools/main(+0x3cfdf) [0x55725ad35fdf]
#11 engine/alice/tools/main(+0x17433) [0x55725ad10433]
#12 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f0476b56b97]
#13 engine/alice/tools/main(+0x1d4ca) [0x55725ad164ca]
====================================================================================================
Minidump written to: /tmp/77737d0e-c172-47d3-55ab0198-f26d9de9.dmp
/home/soeren/.cache/bazel/_bazel_soeren/095f5ffa1280b1937123489b32371c1b/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/navsim/navsim_navigate: line 38: 28197 Aborted                 (core dumped) engine/alice/tools/main --app apps/navsim/navsim_navigate.app.json --more "packages/navsim/robots/$ROBOT_ID.json,packages/navsim/maps/$MAP.json" $@

Also tried to run the editor mode of Isaac Sim. Creating a project and opening medium_warehouse scene worked fine, but again the navigation app was a problem. Ran the following command:

bazel run //apps/navsim:navsim_navigate -- --more packages/navsim/maps/medium_warehouse.json,packages/navsim/robots/carter.json

Which gives the error message shown in the top.

Tried to solve it by following some other forum posts, and from them it seems there might be a problem with CUDA. Was told I needed to install CUDA 10.0, so tried to do that manually. As shown below, it seems like it got installed correctly, but I still get the same error message.

soeren@soeren-MS-7B98:/usr/local$ ls
bin  cuda  cuda-10.0  etc  games  include  lib  man  sbin  share  src
soeren@soeren-MS-7B98:/usr/local$ cat /usr/local/cuda/version.txt
CUDA Version 10.0.130

Also read somewhere it could be a driver problem. I’m using GeForce RTX 2070 with the driver 435.21. I don’t think that should mean anything, but just in case you need the information.

Hope somebody can help me with this problem.

Hi jeive17,
Can you do a couple of checks?

  • Is libcudart.so.10.0 in /usr/local/cuda/lib64?
  • Is cuda 10.0 symlinked in /usr/local/cuda?
  • Is /usr/local/cuda/lib64 included in the LD_LIBRARY_PATH environment variable?
  • Which version of cuDNN? ``` dpkg -l | grep cudnn ```
  • Which version of TensorRT? ``` dpkg -l | grep nvinfer ```
  • Yes, the file is located in the directory
    soeren@soeren-MS-7B98:/usr/local/cuda/lib64$ find libcudart.so.10.0
    libcudart.so.10.0
    
  • cuda 10.0 is not symlinked in /usr/local/cuda, but is symlinked to /usr/local.
    soeren@soeren-MS-7B98:~$ ls /usr/local/cuda
    bin     include  libnsight          nsightee_plugins  pkgconfig  src
    doc     jre      libnvvp            nvml              samples    tools
    extras  lib64    NsightCompute-1.0  nvvm              share      version.txt
    soeren@soeren-MS-7B98:/usr/local$ ls
    bin  cuda  cuda-10.0  etc  games  include  lib  man  sbin  share  src
    
  • There is nothing included in the LD_LIBRARY_PATH, should I include the path?
  • I don't think I have any version of cuDNN and TensorRT since there is no output when I run the two commands you gave
  • I included /usr/local/cuda/lib64 in the LD_LIBRARY_PATH and everything seems to work now.
    Thanks for the help