Playing the "Will it work" lottery this evening - Documentation Example Fails

Greetings,

I attempted to run the example app located here. And it fails to load.

Here’s the Console Output:

coach@tesseract-ventures:~/Documents/Isaac_SDK/sdk$ bazel run packages/cart_delivery/apps:cart_delivery -- --map_json apps/assets/maps/virtual_test_warehouse_1.json --pose2_planner ""
WARNING: /home/coach/Documents/Isaac_SDK/sdk/packages/map/BUILD:24:1: in linkstatic attribute of cc_library rule //packages/map:public_headers: setting 'linkstatic=1' is recommended if there are no object files
WARNING: /home/coach/.cache/bazel/_bazel_coach/c000136ed42492210f86c3ae1f4e90d6/external/opencv_x86_64/BUILD.bazel:11:1: in linkstatic attribute of cc_library rule @opencv_x86_64//:opencv_x86_64: setting 'linkstatic=1' is recommended if there are no object files
INFO: Analyzed target //packages/cart_delivery/apps:cart_delivery (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //packages/cart_delivery/apps:cart_delivery up-to-date:
  bazel-bin/packages/cart_delivery/apps/cart_delivery
INFO: Elapsed time: 0.110s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
/home/coach/.local/lib/python3.6/site-packages/numba/core/errors.py:154: UserWarning: Insufficiently recent colorama version found. Numba requires colorama >= 0.3.9
  warnings.warn(msg)
2021-12-02 19:59:35.540 WARN  external/com_nvidia_isaac_engine/engine/alice/backend/application_json_loader.cpp@183: This application does not have an explicit scheduler configuration. One will be autogenerated to the best of the system's abilities if possible.
2021-12-02 19:59:35.541 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/application_json_loader.cpp@375: Modules requested by subgraph 'delivery': atlas, (behavior_tree), message_generators, navigation, otg5, planner_cost, utils, 
2021-12-02 19:59:35.542 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/application_json_loader.cpp@375: Modules requested by subgraph 'navigation': (map), (navigation), planner, sight, 
2021-12-02 19:59:35.543 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/application_json_loader.cpp@375: Modules requested by subgraph 'local_map': (message_generators), (navigation), viewers, //packages/egm_fusion:evidence_grid_map, 
2021-12-02 19:59:35.543 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/application_json_loader.cpp@375: Modules requested by subgraph 'go_to': (behavior_tree), (navigation), (viewers), 
2021-12-02 19:59:35.543 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/application_json_loader.cpp@375: Modules requested by subgraph 'planner': (behavior_tree), (navigation), path_planner, (planner), (viewers), (//packages/egm_fusion:evidence_grid_map), 
2021-12-02 19:59:35.544 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/application_json_loader.cpp@375: Modules requested by subgraph 'control': (behavior_tree), controller, lqr, (message_generators), (navigation), (planner_cost), trajectory_validation, 
2021-12-02 19:59:35.544 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/application_json_loader.cpp@375: Modules requested by subgraph 'localization': (atlas), (behavior_tree), flatscan_localization, monitor, (navigation), (utils), (viewers), 
2021-12-02 19:59:35.545 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/application_json_loader.cpp@375: Modules requested by subgraph 'imu_odometry': imu, (navigation), 
2021-12-02 19:59:35.545 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/application_json_loader.cpp@375: Modules requested by subgraph 'otg5': (behavior_tree), (otg5), (utils), 
2021-12-02 19:59:35.545 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/application_json_loader.cpp@375: Modules requested by subgraph 'detection_pose_estimation': (sight), (utils), (viewers), 
2021-12-02 19:59:35.545 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/application_json_loader.cpp@375: Modules requested by subgraph 'object_detection': detect_net, ml, perception, (sight), (viewers), 
2021-12-02 19:59:35.546 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/application_json_loader.cpp@375: Modules requested by subgraph 'object_pose_estimation': (ml), object_pose_estimation, (utils), 
2021-12-02 19:59:35.546 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/application_json_loader.cpp@375: Modules requested by subgraph 'simulation': (atlas), engine_tcp_udp, (imu), navsim, (utils), 
2021-12-02 19:59:35.547 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/application_json_loader.cpp@375: Modules requested by subgraph 'noisy_flatscan_1': (perception), flatsim, 
2021-12-02 19:59:35.547 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/application_json_loader.cpp@375: Modules requested by subgraph 'noisy_flatscan_2': (perception), (flatsim), 
2021-12-02 19:59:35.547 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/application_json_loader.cpp@375: Modules requested by subgraph 'noisy_flatscan_3': (perception), (flatsim), 
2021-12-02 19:59:35.547 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/application_json_loader.cpp@375: Modules requested by subgraph 'noisy_flatscan_4': (perception), (flatsim), 
2021-12-02 19:59:35.548 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/redis_backend.cpp@40: Successfully connected to Redis server.

2021-12-02 19:59:35.548 WARN  external/com_nvidia_isaac_engine/engine/alice/backend/backend.cpp@200: This application does not have an execution group configuration. One will be autogenerated to the best of the systems abilities if possible.
2021-12-02 19:59:35.548 WARN  external/com_nvidia_isaac_engine/engine/gems/scheduler/scheduler.cpp@341: No default execution groups specified. Attempting to create scheduler configuration for 16 remaining cores. This may be non optimal for the system and application.
2021-12-02 19:59:35.548 INFO  external/com_nvidia_isaac_engine/engine/gems/scheduler/scheduler.cpp@291: Scheduler execution groups are:
2021-12-02 19:59:35.548 INFO  external/com_nvidia_isaac_engine/engine/gems/scheduler/scheduler.cpp@300: __BlockerGroup__: Cores = [2, 3, 4, 5], Workers = No
2021-12-02 19:59:35.548 INFO  external/com_nvidia_isaac_engine/engine/gems/scheduler/scheduler.cpp@300: __WorkerGroup__: Cores = [15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 1, 0], Workers = Yes
2021-12-02 19:59:35.550 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/atlas/libatlas_module.so': Now has 42 components total
2021-12-02 19:59:35.553 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/behavior_tree/libbehavior_tree_module.so': Now has 56 components total
2021-12-02 19:59:35.556 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/controller/libcontroller_module.so': Now has 61 components total
2021-12-02 19:59:35.558 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/detect_net/libdetect_net_module.so': Now has 62 components total
2021-12-02 19:59:35.562 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/egm_fusion/libevidence_grid_map_module.so': Now has 69 components total
2021-12-02 19:59:35.566 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/engine_tcp_udp/libengine_tcp_udp_module.so': Now has 74 components total
2021-12-02 19:59:35.571 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/flatscan_localization/libflatscan_localization_module.so': Now has 83 components total
2021-12-02 19:59:35.574 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/flatsim/libflatsim_module.so': Now has 90 components total
2021-12-02 19:59:35.577 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/imu/libimu_module.so': Now has 94 components total
2021-12-02 19:59:35.581 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/lqr/liblqr_module.so': Now has 100 components total
2021-12-02 19:59:35.585 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/map/libmap_module.so': Now has 112 components total
2021-12-02 19:59:35.590 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/message_generators/libmessage_generators_module.so': Now has 133 components total
2021-12-02 19:59:35.595 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/monitor/libmonitor_module.so': Now has 135 components total
2021-12-02 19:59:35.601 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/navigation/libnavigation_module.so': Now has 165 components total
2021-12-02 19:59:35.607 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/navsim/libnavsim_module.so': Now has 167 components total
2021-12-02 19:59:35.615 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/object_pose_estimation/libobject_pose_estimation_module.so': Now has 178 components total
2021-12-02 19:59:35.622 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/otg5/libotg5_module.so': Now has 182 components total
2021-12-02 19:59:35.629 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/path_planner/libpath_planner_module.so': Now has 191 components total
2021-12-02 19:59:35.636 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/planner/libplanner_module.so': Now has 194 components total
2021-12-02 19:59:35.643 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/planner_cost/libplanner_cost_module.so': Now has 204 components total
2021-12-02 19:59:35.651 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/sight/libsight_module.so': Now has 209 components total
2021-12-02 19:59:35.658 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/trajectory_validation/libtrajectory_validation_module.so': Now has 217 components total
2021-12-02 19:59:35.667 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/utils/libutils_module.so': Now has 236 components total
2021-12-02 19:59:35.677 INFO  external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@271: Loaded module 'packages/viewers/libviewers_module.so': Now has 260 components total
2021-12-02 19:59:35.677 ERROR external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@295: packages/ml/libml_module.so: libnvrtc.so.10.2: cannot open shared object file: No such file or directory
2021-12-02 19:59:35.677 ERROR external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@295: packages/perception/libperception_module.so: /home/coach/.cache/bazel/_bazel_coach/c000136ed42492210f86c3ae1f4e90d6/execroot/com_nvidia_isaac_sdk/bazel-out/k8-opt/bin/packages/cart_delivery/apps/cart_delivery.runfiles/com_nvidia_isaac_sdk//packages/perception/libperception_module.so: undefined symbol: IsaacGatherComponentInfo
2021-12-02 19:59:35.677 PANIC external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@297: Could not load all required modules for application
====================================================================================================
|                            Isaac application terminated unexpectedly                             |
====================================================================================================
#01 /home/coach/.cache/bazel/_bazel_coach/c000136ed42492210f86c3ae1f4e90d6/execroot/com_nvidia_isaac_sdk/bazel-out/k8-opt/bin/packages/cart_delivery/apps/cart_delivery.runfiles/com_nvidia_isaac_sdk/packages/pyalice/bindings.so(+0x16ceea) [0x7f46f14ffeea]
#02 google_breakpad::ExceptionHandler::GenerateDump(google_breakpad::ExceptionHandler::CrashContext*) /home/coach/.cache/bazel/_bazel_coach/c000136ed42492210f86c3ae1f4e90d6/execroot/com_nvidia_isaac_sdk/bazel-out/k8-opt/bin/packages/cart_delivery/apps/cart_delivery.runfiles/com_nvidia_isaac_sdk/packages/pyalice/bindings.so(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3f0) [0x7f46f15a20a0]
#03 google_breakpad::ExceptionHandler::SignalHandler(int, siginfo_t*, void*) /home/coach/.cache/bazel/_bazel_coach/c000136ed42492210f86c3ae1f4e90d6/execroot/com_nvidia_isaac_sdk/bazel-out/k8-opt/bin/packages/cart_delivery/apps/cart_delivery.runfiles/com_nvidia_isaac_sdk/packages/pyalice/bindings.so(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xc0) [0x7f46f15a2410]
#04 /lib/x86_64-linux-gnu/libc.so.6(+0x3f040) [0x7f47201c4040]
#05 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7f47201c3fb7]
#06 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7f47201c5921]
#07 isaac::alice::ModuleManager::loadModules(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) /home/coach/.cache/bazel/_bazel_coach/c000136ed42492210f86c3ae1f4e90d6/execroot/com_nvidia_isaac_sdk/bazel-out/k8-opt/bin/packages/cart_delivery/apps/cart_delivery.runfiles/com_nvidia_isaac_sdk/packages/pyalice/bindings.so(_ZN5isaac5alice13ModuleManager11loadModulesERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS8_EE+0x80e) [0x7f46f150bc3e]
#08 isaac::alice::Application::createMore(isaac::alice::ApplicationJsonLoader const&) /home/coach/.cache/bazel/_bazel_coach/c000136ed42492210f86c3ae1f4e90d6/execroot/com_nvidia_isaac_sdk/bazel-out/k8-opt/bin/packages/cart_delivery/apps/cart_delivery.runfiles/com_nvidia_isaac_sdk/packages/pyalice/bindings.so(_ZN5isaac5alice11Application10createMoreERKNS0_21ApplicationJsonLoaderE+0x67) [0x7f46f14c1f97]
#09 isaac::alice::Application::createApplication(isaac::alice::ApplicationJsonLoader const&) /home/coach/.cache/bazel/_bazel_coach/c000136ed42492210f86c3ae1f4e90d6/execroot/com_nvidia_isaac_sdk/bazel-out/k8-opt/bin/packages/cart_delivery/apps/cart_delivery.runfiles/com_nvidia_isaac_sdk/packages/pyalice/bindings.so(_ZN5isaac5alice11Application17createApplicationERKNS0_21ApplicationJsonLoaderE+0x289) [0x7f46f14c4939]
#10 isaac::alice::Application::Application(isaac::alice::ApplicationJsonLoader const&) /home/coach/.cache/bazel/_bazel_coach/c000136ed42492210f86c3ae1f4e90d6/execroot/com_nvidia_isaac_sdk/bazel-out/k8-opt/bin/packages/cart_delivery/apps/cart_delivery.runfiles/com_nvidia_isaac_sdk/packages/pyalice/bindings.so(_ZN5isaac5alice11ApplicationC1ERKNS0_21ApplicationJsonLoaderE+0x15f) [0x7f46f14c4c8f]
#11 isaac::alice::PybindApplication::PybindApplication(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /home/coach/.cache/bazel/_bazel_coach/c000136ed42492210f86c3ae1f4e90d6/execroot/com_nvidia_isaac_sdk/bazel-out/k8-opt/bin/packages/cart_delivery/apps/cart_delivery.runfiles/com_nvidia_isaac_sdk/packages/pyalice/bindings.so(_ZN5isaac5alice17PybindApplicationC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_S9_+0x21c) [0x7f46f148ea6c]
#12 /home/coach/.cache/bazel/_bazel_coach/c000136ed42492210f86c3ae1f4e90d6/execroot/com_nvidia_isaac_sdk/bazel-out/k8-opt/bin/packages/cart_delivery/apps/cart_delivery.runfiles/com_nvidia_isaac_sdk/packages/pyalice/bindings.so(+0xfbea6) [0x7f46f148eea6]
#13 /home/coach/.cache/bazel/_bazel_coach/c000136ed42492210f86c3ae1f4e90d6/execroot/com_nvidia_isaac_sdk/bazel-out/k8-opt/bin/packages/cart_delivery/apps/cart_delivery.runfiles/com_nvidia_isaac_sdk/packages/pyalice/bindings.so(+0xe9a24) [0x7f46f147ca24]
#14 /usr/bin/python3(_PyCFunction_FastCallDict+0x35c) [0x566bec]
#15 /usr/bin/python3() [0x595291]
#16 /usr/bin/python3() [0x549ce5]
#17 /usr/bin/python3() [0x551391]
#18 /usr/bin/python3(_PyObject_FastCallKeywords+0x19c) [0x5a9d2c]
#19 /usr/bin/python3() [0x50a533]
#20 /usr/bin/python3(_PyEval_EvalFrameDefault+0x444) [0x50bf44]
#21 /usr/bin/python3() [0x507cd4]
#22 /usr/bin/python3(_PyFunction_FastCallDict+0x2e2) [0x508fb2]
#23 /usr/bin/python3() [0x595221]
#24 /usr/bin/python3() [0x549b3f]
#25 /usr/bin/python3() [0x551391]
#26 /usr/bin/python3(_PyObject_FastCallKeywords+0x19c) [0x5a9d2c]
#27 /usr/bin/python3() [0x50a533]
#28 /usr/bin/python3(_PyEval_EvalFrameDefault+0x1215) [0x50cd15]
#29 /usr/bin/python3() [0x507cd4]
#30 /usr/bin/python3(PyEval_EvalCode+0x23) [0x50ae13]
#31 /usr/bin/python3() [0x635262]
#32 /usr/bin/python3(PyRun_FileExFlags+0x97) [0x635317]
#33 /usr/bin/python3(PyRun_SimpleFileExFlags+0x17f) [0x638acf]
#34 /usr/bin/python3(Py_Main+0x591) [0x639671]
#35 /usr/bin/python3(main+0xe0) [0x4b0e40]
#36 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f47201a6bf7]
#37 /usr/bin/python3(_start+0x2a) [0x5b2f0a]
====================================================================================================
Minidump written to: /tmp/d422974d-fc19-4c96-54909c84-90ad9df5.dmp
Segmentation fault (core dumped)

I feel like it’s playing the lottery trying to learn from these docs.

Turns out this issue is most likely related to CUDA. My machine has 11.5. This is a little frustrating because I am would need to purge nvidia everything and try to grab CUDA 10.0 to make things work. When will NVIDIA make things more straight forward?

I literally followed all the steps for setup and find out that I have update to date packages for the world but too far up to date for NVIDIA.

Also turns out the RTX 3060 may not like older drivers and the older cuda versions. Well I guess this game NVIDIA plays that we build very specific computers to dev with outdated kit!

As you determined, the error you’re getting for libnvrtc.so is indicative of a CUDA mismatch or the library not being on LD_LIBRARY_PATH as part of CUDA setup for deep learning inference (see Setup — ISAAC 2021.1 documentation).

RTX3060 is built with the powerful Ampere architecture, and Ampere is only supported by CUDA 11 or higher. Isaac Sim will be able to take full advantage of this GPU. However, Isaac SDK is built against CUDA 10.2 which is the version supported on Jetson and JetPack 4.5.1, so many of its CUDA kernels in the SDK are not precompiled for CUDA 11+, nor are they forwards compatible to the Ampere architecture. You can still develop Isaac SDK-based applications on an RTX3060, but DNN inference and an number of the built-in packages will not work unfortunately. There were fixes for this included in Isaac SDK 2021.1 but your mileage may vary.

When the dependencies are installed, the installer script outputs the following message:

./engine/engine/build/scripts/install_dependencies.sh
This is a helper script for installing dependencies to build the Isaac SDK.

First argument is the optional CUDA version. By default CUDA 10.2 dependent
packages are installed. GPUs with Maxwell, Pascal, Volta and Turing
architectures are supported in this configuration. Experimental support of
newer architecture GPUs is available with the --config=x86_64_cuda_11_1
option.

CUDA 10.2 support was selected (default option).

And then the same --config=x86_64_cuda_11_1 option can be included to bazel. This result in CUDA 11.1 build, compatible with Ampere architecture. But like noted above, your mileage may vary, depending on the packages that you use.