GPU crash on Hello World example

My system:

  • CPU: x86_64
  • GPU: RTX 4090
  • OS: Ubuntu 20.04
  • Nvidia Driver: 535.104.05 (installed via .run file)

Problem:
First of all, latest containerized Isaac Sim (built via IsaacSim-dockerfiles) has no curl installed, which should probably be fixed, because user can’t run Isaac Sim with --rm flag anymore (you have to install curl and restart the same container).
Then Isaac Sim starts without errors, just a lot of warnings, but ALL Isaac Examples (even “Hello World”) crashes the program with such error:

[Error] [carb.graphics-vulkan.plugin] GPU crash is detected. Crash dump is written into: /root/.nvidia-omniverse/logs/Kit/Isaac-Sim/2022.2/kit_20230908_091716.nv-gpudmp
[Error] [carb.graphics-vulkan.plugin] GPU crash dump is successfully written
[Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_DEVICE_LOST
[Error] [carb.graphics-vulkan.plugin] submitToQueueCommon failed.
[Error] [gpu.foundation.plugin] A GPU crash occurred. Exiting the application...
Reasons for the failure: a device lost, out of memory, or an unexpected bug.
[Error] [carb.crashreporter-breakpad.plugin] crash detected
... a bunch of warnings here ...
[Fatal] [carb.crashreporter-breakpad.plugin] Thread 251 backtrace follows:
[Fatal] [carb.crashreporter-breakpad.plugin] 000: libpthread.so.0!funlockfile+0x60
[Fatal] [carb.crashreporter-breakpad.plugin] 001: libgpu.foundation.plugin.so!carbGetPluginDeps+0x2b3f
[Fatal] [carb.crashreporter-breakpad.plugin] 002: libgpu.foundation.plugin.so!void std::vector<std::tuple<char const*, float, char const*, unsigned int>, std::allocator<std::tuple<char const*, float, char const*, unsigned int> > >::_M_realloc_insert<std::tuple<char const*, float, char const*, unsigned int> >(__gnu_cxx::__normal_iterator<std::tuple<char const*, float, char const*, unsigned int>*, std::vector<std::tuple<char const*, float, char const*, unsigned int>, std::allocator<std::tuple<char const*, float, char const*, unsigned int> > > >, std::tuple<char const*, float, char const*, unsigned int>&&)+0x1168f
[Fatal] [carb.crashreporter-breakpad.plugin] 003: libgpu.foundation.plugin.so!void std::vector<std::tuple<char const*, float, char const*, unsigned int>, std::allocator<std::tuple<char const*, float, char const*, unsigned int> > >::_M_realloc_insert<std::tuple<char const*, float, char const*, unsigned int> >(__gnu_cxx::__normal_iterator<std::tuple<char const*, float, char const*, unsigned int>*, std::vector<std::tuple<char const*, float, char const*, unsigned int>, std::allocator<std::tuple<char const*, float, char const*, unsigned int> > > >, std::tuple<char const*, float, char const*, unsigned int>&&)+0x1b9b9
[Fatal] [carb.crashreporter-breakpad.plugin] 004: libomni.kit.renderer.plugin.so!carbOnPluginPostShutdown+0x2af5
[Fatal] [carb.crashreporter-breakpad.plugin] 005: libomni.kit.renderer.plugin.so!carbOnPluginPostShutdown+0x2ca9
[Fatal] [carb.crashreporter-breakpad.plugin] 006: libomni.kit.renderer.plugin.so!std::__future_base::_State_baseV2::_M_is_deferred_future() const+0x8f
[Fatal] [carb.crashreporter-breakpad.plugin] 007: libcarb.events.plugin.so!std::_Hashtable<unsigned long, unsigned long, std::allocator<unsigned long>, std::__detail::_Identity, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<unsigned long, false>*)+0x188c
[Fatal] [carb.crashreporter-breakpad.plugin] 008: libcarb.events.plugin.so!std::_Hashtable<unsigned long, unsigned long, std::allocator<unsigned long>, std::__detail::_Identity, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<unsigned long, false>*)+0x2026
[Fatal] [carb.crashreporter-breakpad.plugin] 009: libomni.kit.loop-default.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tuple<omni::kit::RunLoopThread::run()::{lambda()#1}> > >::~_State_impl()+0x8ac
[Fatal] [carb.crashreporter-breakpad.plugin] 010: libomni.kit.loop-default.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tuple<omni::kit::RunLoopThread::run()::{lambda()#1}> > >::~_State_impl()+0xe8e3
[Fatal] [carb.crashreporter-breakpad.plugin] 011: libomni.kit.loop-default.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tuple<omni::kit::RunLoopThread::run()::{lambda()#1}> > >::_M_run()+0x1d
[Fatal] [carb.crashreporter-breakpad.plugin] 012: libstdc++.so.6!std::error_code::default_error_condition() const+0x34
[Fatal] [carb.crashreporter-breakpad.plugin] 013: libpthread.so.0!start_thread+0xd9
[Fatal] [carb.crashreporter-breakpad.plugin] 014: libc.so.6!clone+0x43
./runapp.sh: line 4:    14 Segmentation fault      (core dumped) /isaac-sim/isaac-sim.sh --/persistent/isaac/asset_root/default="$OMNI_SERVER" --allow-root $@

My goal:
I try to figure out how to use Isaac ROS, so my first step was to build it and run benchmarks (done), now my second step: run ANY Isaac ROS example with Isaac Sim.

P.S. Isaac Sim itself works, even Isaac Assets work fine in it (I could spawn various robots without any error).

1 Like

@divelix i am just another user passing by. is it possible for you to upload the most recent log in its entirety? it should provide more context to the mods/devs when they get to this thread. you should be able to find it in the Issac Sim subfolder in ~/.nvidia-omniverse/logs/Kit/

does the Hello World example crash when you hit the “Load” button in the World Controls rollout or after a specific action?

Thank you for reply! Crash happens when I click LOAD button of Hello World example.

Log file:
kit_20230919_172308.log (1.3 MB)

P.S. This error happens with both containerized and Workstation installed Isaac Sims on my machine. Log is for workstation one.

Hi. Does it crash when just loading Isaac Sim or running other examples too?
Please try reinstalling the 525.x drivers instead.

Hi. Please try our latest Isaac Sim 2023.1.0 container. Is now has curl included and based on Ubuntu 22.04. The 525.85 drivers is recommended.