Hi there, hoping to get some help with this. I’m trying to run the ROS2 multi-navigation examples. Whenever I launch the carter_navigation launch file, however, Isaac will crash. Here’s my setup:
I’m running a headless cloud computer with ubuntu 20.04. Everything is running in the Isaac Sim Docker Container (22.04) with ROS2 Humble (plus the nav2 and isaac workspaces) installed in the container (ROS2 is not installed on the host).
Because this is headless, I can’t launch rviz. So I’m playing the hospital example then running the launch file from the container like so:
# ros2 launch carter_navigation multiple_robot_carter_navigation_hospital.launch.py use_rviz:=false
Then I get a crash as soon as this loads up. Not sure why. Here’s the log of all the [Fatal] lines (full log attached):
2024-08-19 16:01:00 [1,230ms] [Fatal] [carb.crashreporter-breakpad.plugin] [crash] Thread 74 backtrace follows:
2024-08-19 16:01:00 [1,249ms] [Fatal] [carb.crashreporter-breakpad.plugin] 000: libc.so.6!__sigaction+0x50 (??:?)
2024-08-19 16:01:00 [1,279ms] [Fatal] [carb.crashreporter-breakpad.plugin] 001: libc.so.6!pthread_kill+0x12c (??:?)
2024-08-19 16:01:00 [1,300ms] [Fatal] [carb.crashreporter-breakpad.plugin] 002: libc.so.6!raise+0x16 (??:?)
2024-08-19 16:01:00 [1,320ms] [Fatal] [carb.crashreporter-breakpad.plugin] 003: libc.so.6!abort+0xd3 (??:?)
2024-08-19 16:01:00 [1,342ms] [Fatal] [carb.crashreporter-breakpad.plugin] 004: libstdc++.so.6!__cxa_throw_bad_array_new_length+0x552 (??:?)
2024-08-19 16:01:00 [1,375ms] [Fatal] [carb.crashreporter-breakpad.plugin] 005: libstdc++.so.6!std::rethrow_exception(std::__exception_ptr::exception_ptr)+0x7c (??:?)
2024-08-19 16:01:00 [1,399ms] [Fatal] [carb.crashreporter-breakpad.plugin] 006: libstdc++.so.6!std::terminate()+0x17 (??:?)
2024-08-19 16:01:00 [1,410ms] [Fatal] [carb.crashreporter-breakpad.plugin] 007: libomni.isaac.sensor.plugin.so!std::string::compare(char const*) const+0x1039 (??:?)
2024-08-19 16:01:00 [1,421ms] [Fatal] [carb.crashreporter-breakpad.plugin] 008: libomni.isaac.sensor.plugin.so!std::string::compare(char const*) const+0x6149 (??:?)
2024-08-19 16:01:00 [1,431ms] [Fatal] [carb.crashreporter-breakpad.plugin] 009: libomni.isaac.sensor.plugin.so!std::string::compare(char const*) const+0x6a38 (??:?)
2024-08-19 16:01:00 [1,443ms] [Fatal] [carb.crashreporter-breakpad.plugin] 010: libomni.graph.action_core.plugin.so!std::char_traits<char>::length(char const*)+0x1c6b (??:?)
2024-08-19 16:01:00 [1,460ms] [Fatal] [carb.crashreporter-breakpad.plugin] 011: libomni.graph.action_core.plugin.so!std::_Function_base::_Base_manager<omni::graph::exec::unstable::SerialScheduler::schedule<omni::graph::exec::unstable::Executor<omni::graph::exec::unstable::INode, omni::graph::exec::unstable::ExecutionVisit, omni::graph::exec::unstable::ExecutionNodeData, omni::graph::exec::unstable::SerialScheduler, omni::graph::exec::unstable::DefaultSchedulingStrategy, omni::graph::exec::unstable::IExecutor>::scheduleInternal<omni::graph::exec::unstable::SerialScheduler>(omni::graph::exec::unstable::ExecutionTask&&, std::enable_if<!omni::graph::exec::unstable::Executor<omni::graph::exec::unstable::INode, omni::graph::exec::unstable::ExecutionVisit, omni::graph::exec::unstable::ExecutionNodeData, omni::graph::exec::unstable::SerialScheduler, omni::graph::exec::unstable::DefaultSchedulingStrategy, omni::graph::exec::unstable::IExecutor>::is_deferred<omni::graph::exec::unstable::SerialScheduler, void>::value, void>::type*)::{lambda()#1}>(omni::graph::exec::unstable::Executor<omni::graph::exec::unstable::INode, omni::graph::exec::unstable::ExecutionVisit, omni::graph::exec::unstable::ExecutionNodeData, omni::graph::exec::unstable::SerialScheduler, omni::graph::exec::unstable::DefaultSchedulingStrategy, omni::graph::exec::unstable::IExecutor>::scheduleInternal<omni::graph::exec::unstable::SerialScheduler>(omni::graph::exec::unstable::ExecutionTask&&, std::enable_if<!omni::graph::exec::unstable::Executor<omni::graph::exec::unstable::INode, omni::graph::exec::unstable::ExecutionVisit, omni::graph::exec::unstable::ExecutionNodeData, omni::graph::exec::unstable::SerialScheduler, omni::graph::exec::unstable::DefaultSchedulingStrategy, omni::graph::exec::unstable::IExecutor>::is_deferred<omni::graph::exec::unstable::SerialScheduler, void>::value, void>::type*)::{lambda()#1}&&, omni::graph::exec::unstable::SchedulingInfo)::{lambda()#1}>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation)+0x193a (??:?)
2024-08-19 16:01:00 [1,475ms] [Fatal] [carb.crashreporter-breakpad.plugin] 012: libomni.graph.exec.plugin.so!_init+0x24d0 (??:0)
2024-08-19 16:01:00 [1,489ms] [Fatal] [carb.crashreporter-breakpad.plugin] 013: libomni.graph.action_core.plugin.so!std::string::append(char const*, unsigned long)+0x5af (??:?)
2024-08-19 16:01:00 [1,499ms] [Fatal] [carb.crashreporter-breakpad.plugin] 014: libomni.graph.action_core.plugin.so!void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<omni::graph::exec::unstable::CompactUniqueIndex::{unnamed type#1}>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, omni::graph::exec::unstable::CompactUniqueIndex::{unnamed type#1}&&)+0x9cc (??:?)
2024-08-19 16:01:01 [1,512ms] [Fatal] [carb.crashreporter-breakpad.plugin] 015: libomni.graph.action_core.plugin.so!void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<omni::graph::exec::unstable::CompactUniqueIndex::{unnamed type#1}>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, omni::graph::exec::unstable::CompactUniqueIndex::{unnamed type#1}&&)+0xdff (??:?)
2024-08-19 16:01:01 [1,525ms] [Fatal] [carb.crashreporter-breakpad.plugin] 016: libomni.graph.action_core.plugin.so!std::deque<std::function<void ()>, std::allocator<std::function<void ()> > >::_M_destroy_data_aux(std::_Deque_iterator<std::function<void ()>, std::function<void ()>&, std::function<void ()>*>, std::_Deque_iterator<std::function<void ()>, std::function<void ()>&, std::function<void ()>*>)+0x169db (??:?)
2024-08-19 16:01:01 [1,539ms] [Fatal] [carb.crashreporter-breakpad.plugin] 017: libomni.graph.exec.plugin.so!_init+0x2efd (??:0)
2024-08-19 16:01:01 [1,555ms] [Fatal] [carb.crashreporter-breakpad.plugin] 018: libomni.kit.exec.core.plugin.so!std::__detail::_Map_base<std::thread::id, std::pair<std::thread::id const, unsigned long>, std::allocator<std::pair<std::thread::id const, unsigned long> >, std::__detail::_Select1st, std::equal_to<std::thread::id>, std::hash<std::thread::id>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator[](std::thread::id&&)+0x8e4 (??:?)
2024-08-19 16:01:01 [1,575ms] [Fatal] [carb.crashreporter-breakpad.plugin] 019: libomni.kit.exec.core.plugin.so!std::__detail::_Map_base<std::thread::id, std::pair<std::thread::id const, unsigned long>, std::allocator<std::pair<std::thread::id const, unsigned long> >, std::__detail::_Select1st, std::equal_to<std::thread::id>, std::hash<std::thread::id>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator[](std::thread::id&&)+0xb02 (??:?)
2024-08-19 16:01:01 [1,591ms] [Fatal] [carb.crashreporter-breakpad.plugin] 020: libomni.kit.exec.core.plugin.so!_init+0x32d0 (??:0)
2024-08-19 16:01:01 [1,610ms] [Fatal] [carb.crashreporter-breakpad.plugin] 021: libomni.kit.exec.core.plugin.so!std::__detail::_Map_base<std::thread::id, std::pair<std::thread::id const, unsigned long>, std::allocator<std::pair<std::thread::id const, unsigned long> >, std::__detail::_Select1st, std::equal_to<std::thread::id>, std::hash<std::thread::id>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator[](std::thread::id&&)+0x21c4 (??:?)
2024-08-19 16:01:01 [1,621ms] [Fatal] [carb.crashreporter-breakpad.plugin] 022: libomni.kit.exec.core.plugin.so!std::__detail::_Map_base<std::thread::id, std::pair<std::thread::id const, unsigned long>, std::allocator<std::pair<std::thread::id const, unsigned long> >, std::__detail::_Select1st, std::equal_to<std::thread::id>, std::hash<std::thread::id>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator[](std::thread::id&&)+0x29a1 (??:?)
2024-08-19 16:01:01 [1,629ms] [Fatal] [carb.crashreporter-breakpad.plugin] 023: libomni.kit.exec.core.plugin.so!_init+0x2250 (??:0)
2024-08-19 16:01:01 [1,638ms] [Fatal] [carb.crashreporter-breakpad.plugin] 024: libomni.graph.core.plugin.so!std::string::_M_leak_hard()+0x208a3 (??:?)
2024-08-19 16:01:01 [1,647ms] [Fatal] [carb.crashreporter-breakpad.plugin] 025: _omni_graph_core.cpython-310-x86_64-linux-gnu.so!std::vector<double, std::allocator<double> >::push_back(double const&)+0x10c22 (??:?)
2024-08-19 16:01:01 [1,655ms] [Fatal] [carb.crashreporter-breakpad.plugin] 026: _omni_graph_core.cpython-310-x86_64-linux-gnu.so!std::vector<double, std::allocator<double> >::push_back(double const&)+0x19f93 (??:?)
2024-08-19 16:01:01 [1,663ms] [Fatal] [carb.crashreporter-breakpad.plugin] 027: _omni_graph_core.cpython-310-x86_64-linux-gnu.so!std::__detail::_Map_base<std::string, std::pair<std::string const, void*>, std::allocator<std::pair<std::string const, void*> >, std::__detail::_Select1st, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::operator[](std::string&&)+0x955c (??:?)
2024-08-19 16:01:01 [1,681ms] [Fatal] [carb.crashreporter-breakpad.plugin] 028: libpython3.10.so.1.0!PyBuffer_ToContiguous+0x9a7 (??:?)
2024-08-19 16:01:01 [1,699ms] [Fatal] [carb.crashreporter-breakpad.plugin] 029: libpython3.10.so.1.0!_PyObject_MakeTpCall+0x9a (??:?)
2024-08-19 16:01:01 [1,712ms] [Fatal] [carb.crashreporter-breakpad.plugin] 030: libpython3.10.so.1.0!PyCell_Set+0x8b5 (??:?)
2024-08-19 16:01:01 [1,726ms] [Fatal] [carb.crashreporter-breakpad.plugin] 031: libpython3.10.so.1.0!_init+0xbaed (??:0)
2024-08-19 16:01:01 [1,740ms] [Fatal] [carb.crashreporter-breakpad.plugin] 032: libpython3.10.so.1.0!_PyEval_EvalFrameDefault+0x6e78 (??:?)
2024-08-19 16:01:01 [1,754ms] [Fatal] [carb.crashreporter-breakpad.plugin] 033: libpython3.10.so.1.0!PyEval_EvalFrameEx+0xb24 (??:?)
2024-08-19 16:01:01 [1,769ms] [Fatal] [carb.crashreporter-breakpad.plugin] 034: libpython3.10.so.1.0!PyCell_Set+0x716 (??:?)
2024-08-19 16:01:01 [1,783ms] [Fatal] [carb.crashreporter-breakpad.plugin] 035: libpython3.10.so.1.0!PyVectorcall_Call+0x54 (??:?)
2024-08-19 16:01:01 [1,796ms] [Fatal] [carb.crashreporter-breakpad.plugin] 036: _events.cpython-310-x86_64-linux-gnu.so!std::_Function_base::_Base_manager<pybind11::detail::type_caster<std::function<void (carb::events::IEvent*)>, void>::load(pybind11::handle, bool)::func_wrapper>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation)+0x2ea2 (??:?)
2024-08-19 16:01:01 [1,809ms] [Fatal] [carb.crashreporter-breakpad.plugin] 037: _events.cpython-310-x86_64-linux-gnu.so!std::string __gnu_cxx::__to_xstring<std::string, char>(int (*)(char*, unsigned long, char const*, __va_list_tag*), unsigned long, char const*, ...)+0x4d1b (??:?)
2024-08-19 16:01:01 [1,825ms] [Fatal] [carb.crashreporter-breakpad.plugin] 038: _events.cpython-310-x86_64-linux-gnu.so!_init+0x20f7 (??:0)
2024-08-19 16:01:01 [1,839ms] [Fatal] [carb.crashreporter-breakpad.plugin] 039: libcarb.events.plugin.so!std::string __gnu_cxx::__to_xstring<std::string, char>(int (*)(char*, unsigned long, char const*, __va_list_tag*), unsigned long, char const*, ...)+0x53f8 (??:?)
2024-08-19 16:01:01 [1,853ms] [Fatal] [carb.crashreporter-breakpad.plugin] 040: libcarb.events.plugin.so!std::string __gnu_cxx::__to_xstring<std::string, char>(int (*)(char*, unsigned long, char const*, __va_list_tag*), unsigned long, char const*, ...)+0x574d (??:?)
2024-08-19 16:01:01 [1,868ms] [Fatal] [carb.crashreporter-breakpad.plugin] 041: libcarb.events.plugin.so!std::string __gnu_cxx::__to_xstring<std::string, char>(int (*)(char*, unsigned long, char const*, __va_list_tag*), unsigned long, char const*, ...)+0x5851 (??:?)
2024-08-19 16:01:01 [1,878ms] [Fatal] [carb.crashreporter-breakpad.plugin] 042: libomni.kit.loop-isaac.plugin.so!std::char_traits<char>::length(char const*)+0x5fc (??:?)
2024-08-19 16:01:01 [1,887ms] [Fatal] [carb.crashreporter-breakpad.plugin] 043: libomni.kit.loop-isaac.plugin.so!std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&)+0xf7a4 (??:?)
2024-08-19 16:01:01 [1,901ms] [Fatal] [carb.crashreporter-breakpad.plugin] 044: libomni.kit.app.plugin.so!_init+0x4e42 (??:0)
2024-08-19 16:01:01 [1,913ms] [Fatal] [carb.crashreporter-breakpad.plugin] 045: libomni.kit.app.plugin.so!carbOnPluginPreStartup+0x187d (??:?)
2024-08-19 16:01:01 [1,918ms] [Fatal] [carb.crashreporter-breakpad.plugin] 046: kit!_init+0x6f0 (??:0)
2024-08-19 16:01:01 [1,932ms] [Fatal] [carb.crashreporter-breakpad.plugin] 047: libc.so.6!__libc_init_first+0x90 (??:?)
2024-08-19 16:01:01 [1,945ms] [Fatal] [carb.crashreporter-breakpad.plugin] 048: libc.so.6!__libc_start_main+0x80 (??:?)
2024-08-19 16:01:01 [1,952ms] [Fatal] [carb.crashreporter-breakpad.plugin] 049: kit!_start+0x29 (??:?)
Aborted (core dumped)
Here’s the output of nvidia-smi:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.107.02 Driver Version: 550.107.02 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Quadro RTX 5000 Off | 00000000:00:05.0 Off | Off |
| 42% 67C P2 107W / 230W | 1862MiB / 16384MiB | 53% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 Quadro RTX 5000 Off | 00000000:00:06.0 Off | Off |
| 33% 38C P8 23W / 230W | 446MiB / 16384MiB | 25% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 10913 C+G /isaac-sim/kit/kit 1854MiB |
| 1 N/A N/A 10913 C+G /isaac-sim/kit/kit 439MiB |
+-----------------------------------------------------------------------------------------+
And the host system info:
System: Kernel: 5.4.0-192-generic x86_64 bits: 64 compiler: gcc v: 9.4.0 Console: tty 2 dm: N/A
Distro: Ubuntu 20.04.6 LTS (Focal Fossa)
Machine: Type: Xen System: Xen product: HVM domU v: 4.13 serial: <filter> Chassis: type: 1 serial: <filter>
Mobo: N/A model: N/A serial: N/A BIOS: Xen v: 4.13 date: 04/08/2022
CPU: Topology: 16-Core model: Intel Xeon Silver 4215R bits: 64 type: MCP arch: Cascade Lake rev: 7 L2 cache: 11.0 MiB
flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 bogomips: 102401
Speed: 3200 MHz min/max: N/A Core speeds (MHz): 1: 3200 2: 3200 3: 3200 4: 3200 5: 3200 6: 3200 7: 3200 8: 3200
9: 3200 10: 3200 11: 3200 12: 3200 13: 3200 14: 3200 15: 3200 16: 3200
Graphics: Device-1: vendor: XenSource driver: bochs-drm v: N/A bus ID: 00:02.0 chip ID: 1234:1111
Device-2: NVIDIA TU104GL [Quadro RTX 5000] driver: nvidia v: 550.107.02 bus ID: 00:05.0 chip ID: 10de:1eb0
Device-3: NVIDIA TU104GL [Quadro RTX 5000] driver: nvidia v: 550.107.02 bus ID: 00:06.0 chip ID: 10de:1eb0
Display: server: No display server data found. Headless machine? tty: 240x89
Message: Advanced graphics data unavailable in console. Try -G --display
Audio: Message: No Device data found.
Network: Device-1: Intel 82371AB/EB/MB PIIX4 ACPI vendor: Red Hat Qemu virtual machine type: network bridge driver: N/A
port: c300 bus ID: 00:01.3 chip ID: 8086:7113
IF-ID-1: docker0 state: down mac: <filter>
IF-ID-2: eth0 state: up speed: N/A duplex: N/A mac: <filter>
Drives: Local Storage: total: 500.00 GiB used: 39.74 GiB (7.9%)
Partition: ID-1: / size: 492.02 GiB used: 39.74 GiB (8.1%) fs: ext4 dev: /dev/dm-0
Sensors: Message: No sensors data was found. Is sensors configured?
Info: Processes: 280 Uptime: 27m Memory: 58.84 GiB used: 6.12 GiB (10.4%) Init: systemd v: 245 runlevel: 5 Compilers:
gcc: 9.4.0 alt: 8/9 Shell: bash v: 5.0.17 running in: tty 2 inxi: 3.0.38
Thanks for looking!
isaac_crash_log.txt (45.0 KB)