Generator.py crashes - 'std::bad_weak_ptr'

terminate called after throwing an instance of ‘std::bad_weak_ptr’
Full run below…

The good news is that I at least get the kitti data out - testing/training. The viewport data seems sketchy - some folders are empty, some only have a limited set of rows, say 0 to 3.

The only ‘yellow’ warning I see that I could’ve (easily) been involved with is the CUDA mismatch - I apparently have 11.0 and Isaac SIM wants 11.1. Not sure how/why - if this is an issue should I just upgrade? (And how to do that for the Isaac SIM install?)

Thanks!

pg@pg-U-18:~/.local/share/ov/pkg/isaac_sim-2021.1.1$ ./python.sh python_samples/syntheticdata/offline_generation/generator.py
[Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn’t available.
[Info] [carb] Logging to file: /home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/logs/Kit/Isaac-Sim/2021.1/kit_20210805_085042.log
[Info] [carb] Initializing plugin: carb.events.plugin (interfaces: [carb::events::IEvents v0.2]) (impl: carb.events.plugin)
[0.008s] Loading user config located at: ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/data/Kit/Isaac-Sim/2021.1/user.config.json’
2021-08-05 12:50:43 [40ms] [Warning] [omni.ext.plugin] Extension with the same id is already registered. ext_id: ‘omni.kit.xr-1.0.0’, ext 1: ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/exts/omni.kit.xr’, ext 2: ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/extscore/omni.kit.xr’
[0.054s] [ext: omni.kit.pipapi-0.0.0] startup
[0.072s] [ext: omni.kit.pip_archive-0.0.0] startup
[0.077s] [ext: omni.usd.config-1.0.0] startup
[0.079s] [ext: omni.usd.libs-1.0.0] startup
[0.242s] [ext: omni.kit.async_engine-0.0.0] startup
[0.244s] [ext: omni.stats-0.0.0] startup
[0.247s] [ext: omni.assets.plugins-0.0.0] startup
[0.248s] [ext: omni.kit.loop-isaac-0.1.0] startup
[0.249s] [ext: omni.client-0.1.0] startup
[0.260s] [ext: omni.appwindow-1.0.0] startup
[0.273s] [ext: omni.kit.renderer.core-0.0.0] startup

|---------------------------------------------------------------------------------------------|
| Driver Version: 460.91 | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
| | | | | | Device-ID | UUID |
|---------------------------------------------------------------------------------------------|
| 0 | GeForce RTX 3070 | Yes | | 8438 MB | 10de | 0 |
| | | | | | 2484 | 1b228445… |
|=============================================================================================|
| OS: Linux pg-U-18, Version: 4.15.0-151-generic
| XServer Vendor: The X.Org Foundation, XServer Version: 11906000 (1.19.6.0)
| Processor: Intel(R) Core™ i7-7700K CPU @ 4.20GHz | Cores: Unknown | Logical: 8
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 32122 | Free Memory: 23576
| Total Page/Swap (MB): 2047 | Free Page/Swap: 2047
|---------------------------------------------------------------------------------------------|
[0.894s] [ext: omni.kit.test-0.0.0] startup
[0.909s] [ext: omni.ui-2.1.8] startup
[0.925s] [ext: omni.kit.mainwindow-0.0.0] startup
[0.928s] [ext: omni.kit.uiapp-0.0.0] startup
[0.929s] [ext: omni.usd.schema.physics-1.0.0] startup
2021-08-05 12:50:43 [932ms] [Warning] [carb.python] There is already a module ‘pxr.UsdPhysics’ in ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/extscore/omni.usd.schema.physics/pxr/UsdPhysics/init.py’. Can’t import another module with the same name in public space from ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/extsPhysics/omni.usd.schema.physics’.
2021-08-05 12:50:43 [935ms] [Warning] [carb.python] There is already a module ‘usd.schema.physics’ in ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/extscore/omni.usd.schema.physics/usd/schema/physics/init.py’. Can’t import another module with the same name in public space from ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/extsPhysics/omni.usd.schema.physics’.
[0.942s] [ext: omni.usd.schema.isaac-0.1.0] startup
[0.983s] [ext: omni.usd.schema.semantics-0.0.0] startup
[0.988s] [ext: omni.usd.schema.audio-0.0.0] startup
[0.994s] [ext: omni.usd.schema.physx-0.0.0] startup
[1.021s] [ext: omni.usd.schema.anim-0.0.0] startup
[1.041s] [ext: omni.services.facilities.base-1.0.0] startup
[1.051s] [ext: omni.services.core-1.1.0] startup
[1.192s] [ext: omni.services.transport.server.base-1.0.0] startup
[1.194s] [ext: omni.services.transport.server.http-1.0.0] startup
[1.430s] [ext: omni.kit.notification_manager-1.0.0] startup
[1.433s] [ext: omni.services.transport.client.base-1.0.0] startup
[1.434s] [ext: omni.kit.agent-0.1.0] startup
[1.435s] [ext: omni.kit.window.popup_dialog-2.0.0] startup
[1.439s] [ext: omni.hydra.scene_delegate-0.1.0] startup
[1.448s] [ext: omni.services.client-0.2.0] startup
[1.501s] [ext: carb.audio-0.1.0] startup
[1.524s] [ext: omni.timeline-1.0.0] startup
[1.527s] [ext: omni.kit.commands-0.0.0] startup
[1.532s] [ext: omni.services.transport.client.http_async-1.0.0] startup
[1.823s] [ext: omni.kit.search_core-1.0.0] startup
[1.825s] [ext: omni.kit.widget.path_field-2.0.2] startup
[1.829s] [ext: omni.usd-1.4.2] startup
[1.859s] [ext: omni.kit.thumbnails.images-0.1.1] startup
[1.863s] [ext: omni.kit.widget.filebrowser-2.2.4] startup
[1.868s] [ext: omni.kit.widget.browser_bar-2.0.1] startup
[1.869s] [ext: omni.kit.stage_templates-1.0.4] startup
[1.872s] [ext: omni.kit.window.filepicker-2.3.1] startup
[1.981s] [ext: omni.kit.window.file-1.2.2] startup
[2.065s] [ext: omni.renderer-rtx-0.0.0] startup
[2.066s] [ext: omni.kvdb-0.0.0] startup
[2.076s] [ext: omni.usdphysics-1.2.0] startup
[2.077s] [ext: omni.convexdecomposition-1.2.0] startup
[2.088s] [ext: omni.localcache-0.0.0] startup
[2.098s] [ext: omni.kit.menu.utils-1.0.1] startup
[2.107s] [ext: omni.physx-1.2.0-5.1] startup
[2.130s] [ext: omni.kit.material.library-1.0.4] startup
[2.143s] [ext: omni.kit.menu.create-1.0.1] startup
[2.144s] [ext: omni.kit.window.drop_support-1.0.0] startup
[2.145s] [ext: omni.kit.context_menu-1.1.0] startup
[2.201s] [ext: omni.hydra.rtx-0.1.0] startup
[2.205s] [ext: omni.kit.window.content_browser-2.3.1] startup
[2.267s] [ext: omni.kit.window.viewport-0.0.0] startup
2021-08-05 12:50:45 [2,264ms] [Warning] [omni.ext.plugin] [ext: omni.kit.window.viewport-0.0.0] ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/plugins/rtx/rtx.reshade.plugin’ in ‘[[native.plugin]]’ was not found.
2021-08-05 12:50:45 [2,479ms] [Warning] [rtx.neuraylib.plugin] [CUDA:RENDER] 0.1 CUDA rend warn : CUDA module initialization failed.
2021-08-05 12:50:45 [2,479ms] [Warning] [rtx.neuraylib.plugin] [CUDA:RENDER] 0.1 CUDA rend warn : The version of your CUDA runtime library is 11.0, but 11.1 is required
2021-08-05 12:50:45 [2,479ms] [Warning] [rtx.neuraylib.plugin] [CUDA:RENDER] 0.1 CUDA rend warn : Please verify that you are using the right shared CUDA libraries.
2021-08-05 12:50:45 [2,565ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER] 1.1 IRAY rend warn : CUDA module initialization failed.
2021-08-05 12:50:45 [2,565ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER] 1.1 IRAY rend warn : The version of your CUDA runtime library is 11.0, but 11.1 is required; iray photoreal can only run in CPU mode.
2021-08-05 12:50:45 [2,565ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER] 1.1 IRAY rend warn : Please verify that you are using the right shared CUDA libraries.
2021-08-05 12:50:45 [2,565ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER] 1.1 IRAY rend warn : There is no CUDA-capable GPU available to the iray photoreal renderer.
2021-08-05 12:50:47 [4,959ms] [Warning] [carb] Acquiring non optional plugin interface which is not listed as dependency: [rtx::raytracing::Subsurface v0.1] (plugin: rtx.raytracing.plugin), by client: carb.scenerenderer-rtx.plugin. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client.
[5.246s] [ext: omni.physx.commands-1.0.0] startup
[5.256s] [ext: omni.kit.widget.stage-2.4.0] startup
[5.264s] [ext: omni.kit.window.property-0.0.0] startup
[5.266s] [ext: omni.kit.widget.settings-1.0.0] startup
[5.267s] [ext: omni.kit.property.usd-3.9.4] startup
[5.286s] [ext: omni.kit.window.preferences-1.1.1] startup
[5.314s] [ext: omni.debugdraw-0.0.0] startup
[5.318s] [ext: omni.kit.property.material-1.6.4] startup
[5.321s] [ext: omni.physx.ui-1.2.0-5.1] startup
[5.344s] [ext: omni.kit.numpy.common-0.1.0] startup
[5.403s] [ext: omni.kit.property.physx-0.1.0] startup
[5.459s] [ext: omni.kit.window.toolbar-1.0.0] startup
[5.466s] [ext: omni.physx.demos-1.0.0] startup
[5.476s] [ext: omni.isaac.dynamic_control-0.1.3] startup
[5.491s] [ext: omni.physx.vehicle-1.2.0-5.1] startup
[5.508s] [ext: omni.physx.cct-1.2.0-5.1] startup
[5.521s] [ext: omni.isaac.utils-0.1.4] startup
[5.532s] [ext: omni.syntheticdata-0.1.5] startup
[5.681s] [ext: omni.physx.bundle-1.2.0-5.1] startup
[5.681s] [ext: omni.isaac.range_sensor-0.1.2] startup
[5.688s] [ext: omni.isaac.robot_engine_bridge_ui-0.1.0] startup
[5.690s] [ext: omni.kit.primitive.mesh-1.0.0] startup
[5.694s] [ext: omni.isaac.robot_engine_bridge-0.1.0] startup
[6.013s] [ext: omni.kit.window.console-0.0.0] startup
[6.016s] [ext: omni.isaac.motion_planning-0.1.4] startup
[6.022s] [ext: omni.isaac.occupancy_map-0.1.0] startup
[6.028s] [ext: omni.kit.widget.prompt-1.0.0] startup
[6.030s] [ext: omni.kit.asset_converter-1.1.22] startup
[6.056s] [ext: omni.kit.window.script_editor-1.5.0] startup
[6.057s] [ext: omni.kit.window.status_bar-0.0.0] startup
[6.059s] [ext: omni.rtx.window.settings-0.6.0] startup
[6.064s] [ext: omni.isaac.synthetic_utils-0.1.0] startup
[6.075s] [ext: omni.kit.widget.graph-1.1.2] startup
[6.080s] [ext: omni.kit.window.profiler-1.0.0] startup
[6.085s] [ext: omni.rtx.settings.core-0.5.4] startup
[6.125s] [ext: omni.isaac.dr-0.1.6] startup
[6.130s] [ext: omni.kit.window.extensions-1.0.2] startup
[6.135s] [ext: omni.isaac.urdf-0.1.4] startup
[6.144s] [ext: omni.isaac.contact_sensor-0.1.0] startup
[6.149s] [ext: omni.isaac.python_app-0.0.0] startup
[6.149s] [ext: omni.isaac.sim.python-2021.1.1] startup
[6.149s] app started
OmniKitHelper Starting up …
OmniKitHelper Startup Complete
Loading materials. Will generate data soon…
2021-08-05 12:51:07 [24,118ms] [Warning] [omni.isaac.dynamic_control.plugin] DcPeekObjectType: Function called while not simulating
Writing data to /home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/output
ID: 1
2021-08-05 12:51:07 [24,301ms] [Warning] [omni.isaac.dynamic_control.plugin] DcPeekObjectType: Function called while not simulating
ID: 2
2021-08-05 12:51:07 [24,383ms] [Warning] [omni.isaac.dynamic_control.plugin] DcPeekObjectType: Function called while not simulating
ID: 3
2021-08-05 12:51:07 [24,463ms] [Warning] [omni.isaac.dynamic_control.plugin] DcPeekObjectType: Function called while not simulating
ID: 4
2021-08-05 12:51:07 [24,546ms] [Warning] [omni.isaac.dynamic_control.plugin] DcPeekObjectType: Function called while not simulating
ID: 5
2021-08-05 12:51:07 [24,627ms] [Warning] [omni.isaac.dynamic_control.plugin] DcPeekObjectType: Function called while not simulating
ID: 6
2021-08-05 12:51:07 [24,713ms] [Warning] [omni.isaac.dynamic_control.plugin] DcPeekObjectType: Function called while not simulating
ID: 7
2021-08-05 12:51:07 [24,804ms] [Warning] [omni.isaac.dynamic_control.plugin] DcPeekObjectType: Function called while not simulating
ID: 8
2021-08-05 12:51:07 [24,880ms] [Warning] [omni.isaac.dynamic_control.plugin] DcPeekObjectType: Function called while not simulating
ID: 9
2021-08-05 12:51:07 [24,958ms] [Warning] [omni.isaac.dynamic_control.plugin] DcPeekObjectType: Function called while not simulating
ID: 10
Shutting Down OmniKitHelper…
[25.044s] [ext: omni.isaac.sim.python-2021.1.1] shutdown
[25.044s] [ext: omni.isaac.contact_sensor-0.1.0] shutdown
[25.086s] [ext: omni.isaac.dr-0.1.6] shutdown
[25.095s] [ext: omni.isaac.motion_planning-0.1.4] shutdown
[25.095s] [ext: omni.isaac.robot_engine_bridge-0.1.0] shutdown
[25.141s] [ext: omni.isaac.occupancy_map-0.1.0] shutdown
[25.181s] [ext: omni.isaac.urdf-0.1.4] shutdown
[25.233s] [ext: omni.isaac.utils-0.1.4] shutdown
[25.430s] [ext: omni.isaac.dynamic_control-0.1.3] shutdown
[25.471s] [ext: omni.isaac.python_app-0.0.0] shutdown
[25.471s] [ext: omni.isaac.range_sensor-0.1.2] shutdown
[25.478s] [ext: omni.isaac.robot_engine_bridge_ui-0.1.0] shutdown
[25.480s] [ext: omni.isaac.synthetic_utils-0.1.0] shutdown
[25.481s] [ext: omni.kit.asset_converter-1.1.22] shutdown
[25.495s] [ext: omni.physx.bundle-1.2.0-5.1] shutdown
[25.495s] [ext: omni.physx.vehicle-1.2.0-5.1] shutdown
[25.513s] [ext: omni.kit.property.physx-0.1.0] shutdown
[25.527s] [ext: omni.kit.property.material-1.6.4] shutdown
[25.527s] [ext: omni.kit.property.usd-3.9.4] shutdown
[25.528s] [ext: omni.kit.window.content_browser-2.3.1] shutdown
[25.529s] [ext: omni.kit.window.profiler-1.0.0] shutdown
[25.530s] [ext: omni.physx.cct-1.2.0-5.1] shutdown
[25.543s] [ext: omni.physx.demos-1.0.0] shutdown
[25.555s] [ext: omni.physx.ui-1.2.0-5.1] shutdown
[25.572s] [ext: omni.syntheticdata-0.1.5] shutdown
[25.572s] [ext: omni.kit.window.viewport-0.0.0] shutdown
2021-08-05 12:51:17 [34,289ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.crashreporter-breakpad.plugin.so!_init
2021-08-05 12:51:17 [34,294ms] [Fatal] [carb.crashreporter-breakpad.plugin] libstdc++.so.6!std::rethrow_exception(std::__exception_ptr::exception_ptr)
2021-08-05 12:51:17 [34,298ms] [Fatal] [carb.crashreporter-breakpad.plugin] libstdc++.so.6!std::terminate()
2021-08-05 12:51:17 [34,303ms] [Fatal] [carb.crashreporter-breakpad.plugin] libstdc++.so.6!__cxa_throw
2021-08-05 12:51:17 [34,305ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.usd.so!std::__throw_bad_weak_ptr()
2021-08-05 12:51:17 [34,307ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.usd.so!std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_add_ref_lock()
2021-08-05 12:51:17 [34,309ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.usd.so!std::basic_string<char, std::char_traits, std::allocator > std::operator+<char, std::char_traits, std::allocator >(std::basic_string<char, std::char_traits, std::allocator >&&, char const*)
2021-08-05 12:51:17 [34,311ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.usd.so!std::basic_string<char, std::char_traits, std::allocator > std::operator+<char, std::char_traits, std::allocator >(std::basic_string<char, std::char_traits, std::allocator >&&, char const*)
2021-08-05 12:51:17 [34,315ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.tasking.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (carb::tasking::ThreadPool::)(), carb::tasking::ThreadPool> > >::~_State_impl()
2021-08-05 12:51:17 [34,318ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.tasking.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (carb::tasking::ThreadPool::)(), carb::tasking::ThreadPool> > >::~_State_impl()
2021-08-05 12:51:17 [34,321ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.tasking.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (carb::tasking::ThreadPool::)(), carb::tasking::ThreadPool> > >::~_State_impl()
2021-08-05 12:51:17 [34,324ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.tasking.plugin.so!make_fcontext
terminate called after throwing an instance of ‘std::bad_weak_ptr’
what(): bad_weak_ptr
2021-08-05 12:51:17 [34,544ms] [Error] [carb.crashreporter-breakpad.plugin] failed to write .dmp to: ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/data/Kit/Isaac-Sim/2021.1/d23f7f13-63d6-4019-cc3c568a-0416b9e1.dmp’
2021-08-05 12:51:17 [34,549ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpthread.so.0!funlockfile
2021-08-05 12:51:17 [34,553ms] [Fatal] [carb.crashreporter-breakpad.plugin] libc.so.6!gsignal
2021-08-05 12:51:17 [34,558ms] [Fatal] [carb.crashreporter-breakpad.plugin] libc.so.6!abort
2021-08-05 12:51:17 [34,563ms] [Fatal] [carb.crashreporter-breakpad.plugin] libstdc++.so.6!+0x8c957
2021-08-05 12:51:17 [34,567ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.crashreporter-breakpad.plugin.so!_init
2021-08-05 12:51:17 [34,572ms] [Fatal] [carb.crashreporter-breakpad.plugin] libstdc++.so.6!std::rethrow_exception(std::__exception_ptr::exception_ptr)
2021-08-05 12:51:17 [34,576ms] [Fatal] [carb.crashreporter-breakpad.plugin] libstdc++.so.6!std::terminate()
2021-08-05 12:51:17 [34,581ms] [Fatal] [carb.crashreporter-breakpad.plugin] libstdc++.so.6!__cxa_throw
2021-08-05 12:51:17 [34,583ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.usd.so!std::__throw_bad_weak_ptr()
2021-08-05 12:51:17 [34,586ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.usd.so!std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_add_ref_lock()
2021-08-05 12:51:17 [34,588ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.usd.so!std::basic_string<char, std::char_traits, std::allocator > std::operator+<char, std::char_traits, std::allocator >(std::basic_string<char, std::char_traits, std::allocator >&&, char const*)
2021-08-05 12:51:17 [34,590ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.usd.so!std::basic_string<char, std::char_traits, std::allocator > std::operator+<char, std::char_traits, std::allocator >(std::basic_string<char, std::char_traits, std::allocator >&&, char const*)
2021-08-05 12:51:17 [34,593ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.tasking.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (carb::tasking::ThreadPool::)(), carb::tasking::ThreadPool> > >::~_State_impl()
2021-08-05 12:51:17 [34,596ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.tasking.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (carb::tasking::ThreadPool::)(), carb::tasking::ThreadPool> > >::~_State_impl()
2021-08-05 12:51:17 [34,600ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.tasking.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (carb::tasking::ThreadPool::)(), carb::tasking::ThreadPool> > >::~_State_impl()
2021-08-05 12:51:17 [34,604ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.tasking.plugin.so!make_fcontext
./python.sh: line 31: 17698 Aborted (core dumped) $python_exe $@
There was an error running python
pg@pg-U-18:~/.local/share/ov/pkg/isaac_sim-2021.1.1$

Flushing the output before shutdown (apparently) fixes!

Before calling:
dataset.kit.shutdown()

I put in a line:
time.sleep(10)

No more crashes. All the viewport data gets populated.

1 Like

Glad that you were able to find a solution.