Hi,
I was trying to run the Jetbot RL training, when I got the following error:
2021-11-12 08:59:00 [9,189,940ms] [Error] [rtx.scenedb.plugin] allocateFromRanges failed
../../../source/plugins/rtx.scenedb/Context.cpp(923): uint32_t rtx::scenedb::Context::allocateSBTs(uint32_t)(): Assertion (index != ~0u) failed: Ran out of shader blocks!
2021-11-12 08:59:01 [9,190,643ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.crashreporter-breakpad.plugin.so!_init
2021-11-12 08:59:01 [9,190,647ms] [Fatal] [carb.crashreporter-breakpad.plugin] libstdc++.so.6!std::rethrow_exception(std::__exception_ptr::exception_ptr)
2021-11-12 08:59:01 [9,190,651ms] [Fatal] [carb.crashreporter-breakpad.plugin] libstdc++.so.6!std::terminate()
2021-11-12 08:59:01 [9,190,652ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.scenedb.plugin.so!_init
2021-11-12 08:59:01 [9,190,653ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.scenedb.plugin.so!__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > > std::_V2::__rotate<__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > > >(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, __gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, __gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, std::random_access_iterator_tag)
2021-11-12 08:59:01 [9,190,655ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.scenedb.plugin.so!_init
2021-11-12 08:59:01 [9,190,656ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.scenedb.plugin.so!void std::vector<unsigned int, std::allocator<unsigned int> >::emplace_back<unsigned int>(unsigned int&&)
2021-11-12 08:59:01 [9,190,658ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.scenedb.plugin.so!void std::vector<int, std::allocator<int> >::emplace_back<int>(int&&)
2021-11-12 08:59:01 [9,190,659ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.scenerenderer-mux.plugin.so!_init
2021-11-12 08:59:01 [9,190,661ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.hydra.so!std::pair<std::__detail::_Node_iterator<std::string, true, true>, bool> std::_Hashtable<std::string, std::string, std::allocator<std::string>, std::__detail::_Identity, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> >::_M_insert<std::string const&, std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<std::string, true> > > >(std::string const&, std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<std::string, true> > > const&, std::integral_constant<bool, true>)
2021-11-12 08:59:01 [9,190,662ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.hydra.so!std::pair<std::__detail::_Node_iterator<std::string, true, true>, bool> std::_Hashtable<std::string, std::string, std::allocator<std::string>, std::__detail::_Identity, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> >::_M_insert<std::string const&, std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<std::string, true> > > >(std::string const&, std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<std::string, true> > > const&, std::integral_constant<bool, true>)
2021-11-12 08:59:01 [9,190,665ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.tasking.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (carb::tasking::ThreadPool::*)(), carb::tasking::ThreadPool*> > >::~_State_impl()
2021-11-12 08:59:01 [9,190,666ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.hydra.so!std::vector<unsigned short, std::allocator<unsigned short> >::resize(unsigned long)
2021-11-12 08:59:01 [9,190,668ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.hydra.so!std::_Hashtable<unsigned short, std::pair<unsigned short const, std::string>, std::allocator<std::pair<unsigned short const, std::string> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<std::pair<unsigned short const, std::string>, false>*)
2021-11-12 08:59:01 [9,190,670ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.usd.so!std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<omni::usd::UsdContext::Impl::closeAndOpenStage(std::string const&, bool)::{lambda()#1}> >, void> >::_M_invoke(std::_Any_data const&)
2021-11-12 08:59:01 [9,190,671ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.usd.so!std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<omni::usd::UsdContext::Impl::closeAndOpenStage(std::string const&, bool)::{lambda()#1}> >, void> >::_M_invoke(std::_Any_data const&)
2021-11-12 08:59:01 [9,190,673ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext-viewport_window.plugin.so!std::__detail::_Compiler<std::regex_traits<char> >::_M_assertion()
2021-11-12 08:59:01 [9,190,674ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext-viewport_window.plugin.so!std::__future_base::_State_baseV2::_M_is_deferred_future() const
2021-11-12 08:59:01 [9,190,677ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ui.windowmanager.plugin.so!carbOnPluginStartup
2021-11-12 08:59:01 [9,190,679ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.imgui_renderer.plugin.so!void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long const&>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long const&)
2021-11-12 08:59:01 [9,190,681ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.imgui_renderer.plugin.so!carbOnPluginPostShutdown
2021-11-12 08:59:01 [9,190,684ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.events.plugin.so!void std::this_thread::sleep_for<long, std::ratio<1l, 1000000l> >(std::chrono::duration<long, std::ratio<1l, 1000000l> > const&)
2021-11-12 08:59:01 [9,190,686ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long const&>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long const&)
2021-11-12 08:59:01 [9,190,688ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!carbOnPluginRegisterEx
2021-11-12 08:59:01 [9,190,690ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!std::__future_base::_State_baseV2::_M_is_deferred_future() const
2021-11-12 08:59:01 [9,190,693ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.events.plugin.so!void std::this_thread::sleep_for<long, std::ratio<1l, 1000000l> >(std::chrono::duration<long, std::ratio<1l, 1000000l> > const&)
2021-11-12 08:59:01 [9,190,696ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.loop-isaac.plugin.so!std::unique_lock<std::mutex>::unlock()
2021-11-12 08:59:01 [9,190,699ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.app.plugin.so!_init
2021-11-12 08:59:01 [9,190,702ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.app.plugin.so!carbOnPluginPostShutdown
2021-11-12 08:59:01 [9,190,705ms] [Fatal] [carb.crashreporter-breakpad.plugin] _app.cpython-37m-x86_64-linux-gnu.so!std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::~basic_stringbuf()
2021-11-12 08:59:01 [9,190,708ms] [Fatal] [carb.crashreporter-breakpad.plugin] _app.cpython-37m-x86_64-linux-gnu.so!std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::~basic_stringbuf()
2021-11-12 08:59:01 [9,190,712ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyMethodDef_RawFastCallKeywords
2021-11-12 08:59:01 [9,190,716ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyCFunction_FastCallKeywords
2021-11-12 08:59:01 [9,190,720ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,190,724ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalCodeWithName
2021-11-12 08:59:01 [9,190,728ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyFunction_FastCallKeywords
2021-11-12 08:59:01 [9,190,731ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,190,735ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalCodeWithName
2021-11-12 08:59:01 [9,190,739ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyFunction_FastCallKeywords
2021-11-12 08:59:01 [9,190,743ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,190,747ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_init
2021-11-12 08:59:01 [9,190,751ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,190,755ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_init
2021-11-12 08:59:01 [9,190,758ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,190,762ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_init
2021-11-12 08:59:01 [9,190,766ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,190,770ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_init
2021-11-12 08:59:01 [9,190,774ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,190,778ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalCodeWithName
2021-11-12 08:59:01 [9,190,781ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyFunction_FastCallKeywords
2021-11-12 08:59:01 [9,190,785ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,190,789ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalCodeWithName
2021-11-12 08:59:01 [9,190,793ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyFunction_FastCallKeywords
2021-11-12 08:59:01 [9,190,797ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,190,800ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalCodeWithName
2021-11-12 08:59:01 [9,190,804ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyFunction_FastCallKeywords
2021-11-12 08:59:01 [9,190,808ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,190,812ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_init
2021-11-12 08:59:01 [9,190,816ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,190,820ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalCodeWithName
2021-11-12 08:59:01 [9,190,823ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!PyEval_EvalCodeEx
2021-11-12 08:59:01 [9,190,827ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!PyEval_EvalCode
2021-11-12 08:59:01 [9,190,831ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!PyRun_FileExFlags
2021-11-12 08:59:01 [9,190,835ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!PyRun_SimpleFileExFlags
2021-11-12 08:59:01 [9,190,839ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyMainInterpreterConfig_Read
2021-11-12 08:59:01 [9,190,843ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_Py_UnixMain
2021-11-12 08:59:01 [9,190,846ms] [Fatal] [carb.crashreporter-breakpad.plugin] libc.so.6!__libc_start_main
2021-11-12 08:59:01 [9,190,846ms] [Fatal] [carb.crashreporter-breakpad.plugin] python3!_start
terminate called without an active exception
Fatal Python error: Aborted
Thread 0x00007f97f96b6700 (most recent call first):
Thread 0x00007f9781407700 (most recent call first):
File "/home/rambo/.local/share/ov/pkg/isaac_sim-2021.1.0/kit/python/lib/python3.7/threading.py", line 300 in wait
File "/home/rambo/.local/share/ov/pkg/isaac_sim-2021.1.0/kit/python/lib/python3.7/queue.py", line 179 in get
File "/home/rambo/.local/share/ov/pkg/isaac_sim-2021.1.0/kit/python/lib/python3.7/site-packages/tensorboard/summary/writer/event_file_writer.py", line 232 in run
File "/home/rambo/.local/share/ov/pkg/isaac_sim-2021.1.0/kit/python/lib/python3.7/threading.py", line 926 in _bootstrap_inner
File "/home/rambo/.local/share/ov/pkg/isaac_sim-2021.1.0/kit/python/lib/python3.7/threading.py", line 890 in _bootstrap
Current thread 0x00007f9ebfc0c200 (most recent call first):
File "/home/rambo/.local/share/ov/pkg/isaac_sim-2021.1.0/exts/omni.isaac.python_app/omni/isaac/python_app/python_app.py", line 283 in update
File "/home/rambo/.local/share/ov/pkg/isaac_sim-2021.1.0/python_samples/jetbot/jetbot.py", line 81 in teleport
File "/home/rambo/.local/share/ov/pkg/isaac_sim-2021.1.0/python_samples/jetbot/jetbot_env.py", line 239 in reset
File "/home/rambo/.local/share/ov/pkg/isaac_sim-2021.1.0/kit/python/lib/python3.7/site-packages/stable_baselines3/common/vec_env/dummy_vec_env.py", line 49 in step_wait
File "/home/rambo/.local/share/ov/pkg/isaac_sim-2021.1.0/kit/python/lib/python3.7/site-packages/stable_baselines3/common/vec_env/vec_transpose.py", line 48 in step_wait
File "/home/rambo/.local/share/ov/pkg/isaac_sim-2021.1.0/kit/python/lib/python3.7/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 150 in step
File "/home/rambo/.local/share/ov/pkg/isaac_sim-2021.1.0/kit/python/lib/python3.7/site-packages/stable_baselines3/common/on_policy_algorithm.py", line 163 in collect_rollouts
File "/home/rambo/.local/share/ov/pkg/isaac_sim-2021.1.0/kit/python/lib/python3.7/site-packages/stable_baselines3/common/on_policy_algorithm.py", line 222 in learn
File "/home/rambo/.local/share/ov/pkg/isaac_sim-2021.1.0/kit/python/lib/python3.7/site-packages/stable_baselines3/ppo/ppo.py", line 264 in learn
File "python_samples/jetbot/jetbot_train.py", line 67 in train
File "python_samples/jetbot/jetbot_train.py", line 171 in <module>
2021-11-12 08:59:01 [9,191,360ms] [Error] [carb.crashreporter-breakpad.plugin] failed to write .dmp to: '/home/rambo/.local/share/ov/pkg/isaac_sim-2021.1.0/kit/data/Kit/Isaac-Sim/2021.1/8b25951e-9728-4878-303e7cab-92002dd3.dmp'
2021-11-12 08:59:01 [9,191,364ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpthread.so.0!funlockfile
2021-11-12 08:59:01 [9,191,368ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpthread.so.0!raise
2021-11-12 08:59:01 [9,191,372ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpthread.so.0!funlockfile
2021-11-12 08:59:01 [9,191,376ms] [Fatal] [carb.crashreporter-breakpad.plugin] libc.so.6!gsignal
2021-11-12 08:59:01 [9,191,379ms] [Fatal] [carb.crashreporter-breakpad.plugin] libc.so.6!abort
2021-11-12 08:59:01 [9,191,383ms] [Fatal] [carb.crashreporter-breakpad.plugin] libstdc++.so.6!+0x8c957
2021-11-12 08:59:01 [9,191,386ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.crashreporter-breakpad.plugin.so!_init
2021-11-12 08:59:01 [9,191,390ms] [Fatal] [carb.crashreporter-breakpad.plugin] libstdc++.so.6!std::rethrow_exception(std::__exception_ptr::exception_ptr)
2021-11-12 08:59:01 [9,191,393ms] [Fatal] [carb.crashreporter-breakpad.plugin] libstdc++.so.6!std::terminate()
2021-11-12 08:59:01 [9,191,395ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.scenedb.plugin.so!_init
2021-11-12 08:59:01 [9,191,396ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.scenedb.plugin.so!__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > > std::_V2::__rotate<__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > > >(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, __gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, __gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, std::random_access_iterator_tag)
2021-11-12 08:59:01 [9,191,398ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.scenedb.plugin.so!_init
2021-11-12 08:59:01 [9,191,399ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.scenedb.plugin.so!void std::vector<unsigned int, std::allocator<unsigned int> >::emplace_back<unsigned int>(unsigned int&&)
2021-11-12 08:59:01 [9,191,401ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.scenedb.plugin.so!void std::vector<int, std::allocator<int> >::emplace_back<int>(int&&)
2021-11-12 08:59:01 [9,191,402ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.scenerenderer-mux.plugin.so!_init
2021-11-12 08:59:01 [9,191,404ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.hydra.so!std::pair<std::__detail::_Node_iterator<std::string, true, true>, bool> std::_Hashtable<std::string, std::string, std::allocator<std::string>, std::__detail::_Identity, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> >::_M_insert<std::string const&, std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<std::string, true> > > >(std::string const&, std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<std::string, true> > > const&, std::integral_constant<bool, true>)
2021-11-12 08:59:01 [9,191,405ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.hydra.so!std::pair<std::__detail::_Node_iterator<std::string, true, true>, bool> std::_Hashtable<std::string, std::string, std::allocator<std::string>, std::__detail::_Identity, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> >::_M_insert<std::string const&, std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<std::string, true> > > >(std::string const&, std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<std::string, true> > > const&, std::integral_constant<bool, true>)
2021-11-12 08:59:01 [9,191,408ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.tasking.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (carb::tasking::ThreadPool::*)(), carb::tasking::ThreadPool*> > >::~_State_impl()
2021-11-12 08:59:01 [9,191,409ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.hydra.so!std::vector<unsigned short, std::allocator<unsigned short> >::resize(unsigned long)
2021-11-12 08:59:01 [9,191,411ms] [Fatal] [carb.crashreporter-breakpad.plugin] librtx.hydra.so!std::_Hashtable<unsigned short, std::pair<unsigned short const, std::string>, std::allocator<std::pair<unsigned short const, std::string> >, std::__detail::_Select1st, std::equal_to<unsigned short>, std::hash<unsigned short>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<std::pair<unsigned short const, std::string>, false>*)
2021-11-12 08:59:01 [9,191,413ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.usd.so!std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<omni::usd::UsdContext::Impl::closeAndOpenStage(std::string const&, bool)::{lambda()#1}> >, void> >::_M_invoke(std::_Any_data const&)
2021-11-12 08:59:01 [9,191,415ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.usd.so!std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<omni::usd::UsdContext::Impl::closeAndOpenStage(std::string const&, bool)::{lambda()#1}> >, void> >::_M_invoke(std::_Any_data const&)
2021-11-12 08:59:01 [9,191,416ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext-viewport_window.plugin.so!std::__detail::_Compiler<std::regex_traits<char> >::_M_assertion()
2021-11-12 08:59:01 [9,191,418ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext-viewport_window.plugin.so!std::__future_base::_State_baseV2::_M_is_deferred_future() const
2021-11-12 08:59:01 [9,191,420ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ui.windowmanager.plugin.so!carbOnPluginStartup
2021-11-12 08:59:01 [9,191,422ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.imgui_renderer.plugin.so!void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long const&>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long const&)
2021-11-12 08:59:01 [9,191,424ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.imgui_renderer.plugin.so!carbOnPluginPostShutdown
2021-11-12 08:59:01 [9,191,427ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.events.plugin.so!void std::this_thread::sleep_for<long, std::ratio<1l, 1000000l> >(std::chrono::duration<long, std::ratio<1l, 1000000l> > const&)
2021-11-12 08:59:01 [9,191,429ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long const&>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long const&)
2021-11-12 08:59:01 [9,191,432ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!carbOnPluginRegisterEx
2021-11-12 08:59:01 [9,191,434ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.renderer.plugin.so!std::__future_base::_State_baseV2::_M_is_deferred_future() const
2021-11-12 08:59:01 [9,191,437ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.events.plugin.so!void std::this_thread::sleep_for<long, std::ratio<1l, 1000000l> >(std::chrono::duration<long, std::ratio<1l, 1000000l> > const&)
2021-11-12 08:59:01 [9,191,439ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.loop-isaac.plugin.so!std::unique_lock<std::mutex>::unlock()
2021-11-12 08:59:01 [9,191,442ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.app.plugin.so!_init
2021-11-12 08:59:01 [9,191,445ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.app.plugin.so!carbOnPluginPostShutdown
2021-11-12 08:59:01 [9,191,448ms] [Fatal] [carb.crashreporter-breakpad.plugin] _app.cpython-37m-x86_64-linux-gnu.so!std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::~basic_stringbuf()
2021-11-12 08:59:01 [9,191,452ms] [Fatal] [carb.crashreporter-breakpad.plugin] _app.cpython-37m-x86_64-linux-gnu.so!std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::~basic_stringbuf()
2021-11-12 08:59:01 [9,191,456ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyMethodDef_RawFastCallKeywords
2021-11-12 08:59:01 [9,191,460ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyCFunction_FastCallKeywords
2021-11-12 08:59:01 [9,191,463ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,191,467ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalCodeWithName
2021-11-12 08:59:01 [9,191,471ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyFunction_FastCallKeywords
2021-11-12 08:59:01 [9,191,475ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,191,479ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalCodeWithName
2021-11-12 08:59:01 [9,191,483ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyFunction_FastCallKeywords
2021-11-12 08:59:01 [9,191,486ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,191,490ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_init
2021-11-12 08:59:01 [9,191,494ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,191,498ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_init
2021-11-12 08:59:01 [9,191,502ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,191,505ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_init
2021-11-12 08:59:01 [9,191,509ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,191,513ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_init
2021-11-12 08:59:01 [9,191,517ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,191,521ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalCodeWithName
2021-11-12 08:59:01 [9,191,525ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyFunction_FastCallKeywords
2021-11-12 08:59:01 [9,191,528ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,191,532ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalCodeWithName
2021-11-12 08:59:01 [9,191,536ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyFunction_FastCallKeywords
2021-11-12 08:59:01 [9,191,540ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,191,544ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalCodeWithName
2021-11-12 08:59:01 [9,191,548ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyFunction_FastCallKeywords
2021-11-12 08:59:01 [9,191,552ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,191,556ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_init
2021-11-12 08:59:01 [9,191,559ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalFrameDefault
2021-11-12 08:59:01 [9,191,563ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyEval_EvalCodeWithName
2021-11-12 08:59:01 [9,191,567ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!PyEval_EvalCodeEx
2021-11-12 08:59:01 [9,191,571ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!PyEval_EvalCode
2021-11-12 08:59:01 [9,191,575ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!PyRun_FileExFlags
2021-11-12 08:59:01 [9,191,579ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!PyRun_SimpleFileExFlags
2021-11-12 08:59:01 [9,191,582ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_PyMainInterpreterConfig_Read
2021-11-12 08:59:01 [9,191,586ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpython3.7m.so.1.0!_Py_UnixMain
2021-11-12 08:59:01 [9,191,590ms] [Fatal] [carb.crashreporter-breakpad.plugin] libc.so.6!__libc_start_main
2021-11-12 08:59:01 [9,191,590ms] [Fatal] [carb.crashreporter-breakpad.plugin] python3!_start
./python.sh: line 27: 27813 Aborted (core dumped) $python_exe $@
There was an error running python
This was already after more then 10 hours of training. I evaluated the model using the most recently created checkpoint, but it was not satisfying enough. I could restart the training using this latest checkpoint, but it s still frustrating that I have to check it in the middle of the night, if it has an error.
Do you have any idea what the problem might be?
I am running IsaacSim 2021.1.1 (but it gives me the same error with 2021.1.0 . Althought I have successfully run the training in the summer with the 2021.1.0 version)
I have an RTX 3080 Driver Version: 455.38 CUDA Version: 11.1
#define CUDNN_MAJOR 8
#define CUDNN_MINOR 3
#define CUDNN_PATCHLEVEL 0
--
#define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)
#endif /* CUDNN_VERSION_H */
Thanks,
Szilard