Syntheticdata.py - rgb retrieves camera grayscale, not color (Solved! ?)

peter.gaston · August 10, 2021, 8:03pm

(Solved? See my response below. tldr; needs to fully wait for scene to render. FYI, on another bug, I also highlighted the need for a wait look for the DataWriter to finish spooling out before killing it.)

Using Isaac Sim, Synthetic data capture I can build the image files nicely, i.e., in color - see below. Leveraging the exact same scenario, with the given file generator.py, the rgb image is in gray scale - instead of color - see the next image. This image is what is returned from the sensors.get_rgb call in syntheticdata.py (211), i.e., it isn’t converted to gray scale later by DataWriter.

The system is a relatively clean ubuntu 18.04 - built for this project a few days ago. GPU is RTX 3070. Isaac Sim seems to run fine. A basic sample ‘visualize_groundtruth.py’ runs fine - no errors, some (different) warnings - and a good output image. generator_stereo.py sort of worked - it showed the physics, but in grayscale. Also crashed with a bad_weak_ptr after about 10 seconds.

(Apologies for reposting this - but this is a show stopper for training object pose detection. Sofar the rest of the pre-training, including outputting the relative pose of the camera to the object is working fine.)

Thought - loading the scenario in Isaac sim takes about 115 seconds. However the python version starts generating data at 30 seconds. Perhaps the offline python needs to wait for full rendering???

There are warnings that show up during the run (from inside VS Code). The full run dump is below, but here’s a summary list:

Warn: dumps uploading isn’t available, Extension with the same id is already registered
Warn: already a module ‘pxr.UsdPhysics’
Warn: [CUDA:RENDER] 0.1 CUDA rend warn : CUDA module initialization failed (version of your CUDA runtime library is 11.0, but 11.1 is required). FYI - I re-did with option ‘install_dependencies.sh --config=x86_64_cuda_11_1’ but no change after restart.
Warn: DcPeekObjectType: Function called while not simulating
and more warnings on shutdown… 436 outstanding handles; texture images not released correctly and could leak; failed to find plugin; leaking 1 carb::tasking::Counter.

From Isaac sim, using the built-in synthetic data recorder

Captured using python code (generator.py):

Full dump of run (from VS Code Terminal window):

pg@pg-U-18:~/.local/share/ov/pkg/isaac_sim-2021.1.1$ cd /home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1 ; /usr/bin/env /usr/bin/python /home/pg/.vscode/extensions/ms-python.python-2021.8.1105858891/pythonFiles/lib/python/debugpy/launcher 46649 – /home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/python_samples/syntheticdata/offline_generation/mPgenerator.py
[Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn’t available.
[Info] [carb] Logging to file: /home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/logs/Kit/Isaac-Sim/2021.1/kit_20210810_150854.log
[Info] [carb] Initializing plugin: carb.events.plugin (interfaces: [carb::events::IEvents v0.2]) (impl: carb.events.plugin)
[0.012s] Loading user config located at: ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/data/Kit/Isaac-Sim/2021.1/user.config.json’
2021-08-10 19:08:54 [41ms] [Warning] [omni.ext.plugin] Extension with the same id is already registered. ext_id: ‘omni.kit.xr-1.0.0’, ext 1: ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/exts/omni.kit.xr’, ext 2: ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/extscore/omni.kit.xr’
[0.058s] [ext: omni.kit.pipapi-0.0.0] startup
[0.113s] [ext: omni.kit.pip_archive-0.0.0] startup
[0.129s] [ext: omni.usd.config-1.0.0] startup
[0.137s] [ext: omni.usd.libs-1.0.0] startup
2021-08-10 19:08:54 [161ms] [Error] [carb.python] Failed to reload python module pxr.Tf. Error: module pxr.Tf._tf not in sys.modules. Traceback:
Traceback (most recent call last):
File “/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py”, line 56, in reload_module
return importlib.reload(sys.modules[name])
File “/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/python/lib/python3.7/importlib/init.py”, line 148, in reload
raise ImportError(msg.format(name), name=name)
ImportError: module pxr.Tf._tf not in sys.modules

2021-08-10 19:08:54 [161ms] [Error] [carb.scripting-python.plugin] Exception: Extension python module: ‘pxr.Tf’ in ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/extscore/omni.usd.libs/.’ failed to load.

At:
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(169): startup_all_extensions_in_module
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(225): startup_all_extensions_in_module
PythonExtension.cpp::startup()(2):
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/exts/omni.isaac.python_app/omni/isaac/python_app/python_app.py(185): _start_app
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/exts/omni.isaac.python_app/omni/isaac/python_app/python_app.py(107): init
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/python_samples/syntheticdata/offline_generation/mPgenerator.py(66): init
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/python_samples/syntheticdata/offline_generation/mPgenerator.py(418):
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/python/lib/python3.7/runpy.py(85): _run_code
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/python/lib/python3.7/runpy.py(96): _run_module_code
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/python/lib/python3.7/runpy.py(263): run_path
/home/pg/.vscode/extensions/ms-python.python-2021.8.1105858891/pythonFiles/lib/python/debugpy/…/debugpy/server/cli.py(285): run_file
/home/pg/.vscode/extensions/ms-python.python-2021.8.1105858891/pythonFiles/lib/python/debugpy/…/debugpy/server/cli.py(444): main
/home/pg/.vscode/extensions/ms-python.python-2021.8.1105858891/pythonFiles/lib/python/debugpy/main.py(45):
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/python/lib/python3.7/runpy.py(85): _run_code
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/python/lib/python3.7/runpy.py(193): _run_module_as_main

2021-08-10 19:08:54 [161ms] [Error] [omni.ext.plugin] [ext: omni.usd.libs-1.0.0] Failed to process python module extension in ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/extscore/omni.usd.libs/.’.
[0.186s] [ext: omni.kit.async_engine-0.0.0] startup
[0.193s] [ext: omni.stats-0.0.0] startup
[0.201s] [ext: omni.assets.plugins-0.0.0] startup
[0.202s] [ext: omni.kit.loop-isaac-0.1.0] startup
[0.205s] [ext: omni.client-0.1.0] startup
[0.221s] [ext: omni.appwindow-1.0.0] startup
[0.238s] [ext: omni.kit.renderer.core-0.0.0] startup

|---------------------------------------------------------------------------------------------|
| Driver Version: 460.91 | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
| | | | | | Device-ID | UUID |
|---------------------------------------------------------------------------------------------|
| 0 | GeForce RTX 3070 | Yes | | 8438 MB | 10de | 0 |
| | | | | | 2484 | 1b228445… |
|=============================================================================================|
| OS: Linux pg-U-18, Version: 4.15.0-153-generic
| XServer Vendor: The X.Org Foundation, XServer Version: 11906000 (1.19.6.0)
| Processor: Intel(R) Core™ i7-7700K CPU @ 4.20GHz | Cores: Unknown | Logical: 8
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 32122 | Free Memory: 22509
| Total Page/Swap (MB): 2047 | Free Page/Swap: 2047
|---------------------------------------------------------------------------------------------|
[0.931s] [ext: omni.kit.test-0.0.0] startup
[0.952s] [ext: omni.ui-2.1.8] startup
[0.979s] [ext: omni.kit.mainwindow-0.0.0] startup
[0.987s] [ext: omni.kit.uiapp-0.0.0] startup
[0.988s] [ext: omni.usd.schema.physics-1.0.0] startup
2021-08-10 19:08:55 [1,078ms] [Warning] [carb.python] There is already a module ‘pxr.UsdPhysics’ in ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/extscore/omni.usd.schema.physics/pxr/UsdPhysics/init.py’. Can’t import another module with the same name in public space from ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/extsPhysics/omni.usd.schema.physics’.
2021-08-10 19:08:55 [1,088ms] [Warning] [carb.python] There is already a module ‘usd.schema.physics’ in ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/extscore/omni.usd.schema.physics/usd/schema/physics/init.py’. Can’t import another module with the same name in public space from ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/extsPhysics/omni.usd.schema.physics’.
[1.098s] [ext: omni.usd.schema.isaac-0.1.0] startup
[1.203s] [ext: omni.usd.schema.semantics-0.0.0] startup
[1.222s] [ext: omni.usd.schema.audio-0.0.0] startup
[1.240s] [ext: omni.usd.schema.physx-0.0.0] startup
[1.304s] [ext: omni.usd.schema.anim-0.0.0] startup
[1.366s] [ext: omni.services.facilities.base-1.0.0] startup
[1.397s] [ext: omni.services.core-1.1.0] startup
[1.819s] [ext: omni.services.transport.server.base-1.0.0] startup
[1.825s] [ext: omni.services.transport.server.http-1.0.0] startup
[2.423s] [ext: omni.kit.notification_manager-1.0.0] startup
[2.434s] [ext: omni.services.transport.client.base-1.0.0] startup
[2.438s] [ext: omni.kit.agent-0.1.0] startup
[2.442s] [ext: omni.kit.window.popup_dialog-2.0.0] startup
[2.455s] [ext: omni.hydra.scene_delegate-0.1.0] startup
[2.464s] [ext: omni.services.client-0.2.0] startup
[2.727s] [ext: carb.audio-0.1.0] startup
[2.749s] [ext: omni.timeline-1.0.0] startup
[2.756s] [ext: omni.kit.commands-0.0.0] startup
[2.773s] [ext: omni.services.transport.client.http_async-1.0.0] startup
[4.310s] [ext: omni.kit.search_core-1.0.0] startup
[4.316s] [ext: omni.kit.widget.path_field-2.0.2] startup
[4.328s] [ext: omni.usd-1.4.2] startup
[4.464s] [ext: omni.kit.thumbnails.images-0.1.1] startup
[4.478s] [ext: omni.kit.widget.filebrowser-2.2.4] startup
[4.490s] [ext: omni.kit.widget.browser_bar-2.0.1] startup
[4.493s] [ext: omni.kit.stage_templates-1.0.4] startup
[4.499s] [ext: omni.kit.window.filepicker-2.3.1] startup
[5.063s] [ext: omni.kit.window.file-1.2.2] startup
[5.392s] [ext: omni.renderer-rtx-0.0.0] startup
[5.393s] [ext: omni.kvdb-0.0.0] startup
[5.424s] [ext: omni.usdphysics-1.2.0] startup
[5.425s] [ext: omni.convexdecomposition-1.2.0] startup
[5.456s] [ext: omni.localcache-0.0.0] startup
[5.486s] [ext: omni.kit.menu.utils-1.0.1] startup
[5.511s] [ext: omni.physx-1.2.0-5.1] startup
[5.555s] [ext: omni.kit.material.library-1.0.4] startup
[5.588s] [ext: omni.kit.menu.create-1.0.1] startup
[5.591s] [ext: omni.kit.window.drop_support-1.0.0] startup
[5.593s] [ext: omni.kit.context_menu-1.1.0] startup
[5.892s] [ext: omni.hydra.rtx-0.1.0] startup
[5.896s] [ext: omni.kit.window.content_browser-2.3.1] startup
[6.182s] [ext: omni.kit.window.viewport-0.0.0] startup
2021-08-10 19:09:00 [6,175ms] [Warning] [omni.ext.plugin] [ext: omni.kit.window.viewport-0.0.0] ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/plugins/rtx/rtx.reshade.plugin’ in ‘[[native.plugin]]’ was not found.
2021-08-10 19:09:00 [6,394ms] [Warning] [rtx.neuraylib.plugin] [CUDA:RENDER] 0.1 CUDA rend warn : CUDA module initialization failed.
2021-08-10 19:09:00 [6,394ms] [Warning] [rtx.neuraylib.plugin] [CUDA:RENDER] 0.1 CUDA rend warn : The version of your CUDA runtime library is 11.0, but 11.1 is required
2021-08-10 19:09:00 [6,394ms] [Warning] [rtx.neuraylib.plugin] [CUDA:RENDER] 0.1 CUDA rend warn : Please verify that you are using the right shared CUDA libraries.
2021-08-10 19:09:00 [6,500ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER] 1.1 IRAY rend warn : CUDA module initialization failed.
2021-08-10 19:09:00 [6,500ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER] 1.1 IRAY rend warn : The version of your CUDA runtime library is 11.0, but 11.1 is required; iray photoreal can only run in CPU mode.
2021-08-10 19:09:00 [6,500ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER] 1.1 IRAY rend warn : Please verify that you are using the right shared CUDA libraries.
2021-08-10 19:09:00 [6,500ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER] 1.1 IRAY rend warn : There is no CUDA-capable GPU available to the iray photoreal renderer.
2021-08-10 19:09:03 [8,871ms] [Warning] [carb] Acquiring non optional plugin interface which is not listed as dependency: [rtx::raytracing::Subsurface v0.1] (plugin: rtx.raytracing.plugin), by client: carb.scenerenderer-rtx.plugin. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client.
[9.208s] [ext: omni.physx.commands-1.0.0] startup
[9.242s] [ext: omni.kit.widget.stage-2.4.0] startup
[9.266s] [ext: omni.kit.window.property-0.0.0] startup
[9.270s] [ext: omni.kit.widget.settings-1.0.0] startup
[9.275s] [ext: omni.kit.property.usd-3.9.4] startup
[9.318s] [ext: omni.kit.window.preferences-1.1.1] startup
[9.378s] [ext: omni.debugdraw-0.0.0] startup
[9.387s] [ext: omni.kit.property.material-1.6.4] startup
[9.396s] [ext: omni.physx.ui-1.2.0-5.1] startup
[9.449s] [ext: omni.kit.numpy.common-0.1.0] startup
[9.739s] [ext: omni.kit.property.physx-0.1.0] startup
[9.832s] [ext: omni.kit.window.toolbar-1.0.0] startup
[9.850s] [ext: omni.physx.demos-1.0.0] startup
[9.880s] [ext: omni.isaac.dynamic_control-0.1.3] startup
[9.925s] [ext: omni.physx.vehicle-1.2.0-5.1] startup
[9.971s] [ext: omni.physx.cct-1.2.0-5.1] startup
[10.005s] [ext: omni.isaac.utils-0.1.4] startup
[10.029s] [ext: omni.syntheticdata-0.1.5] startup
[10.661s] [ext: omni.physx.bundle-1.2.0-5.1] startup
[10.661s] [ext: omni.isaac.range_sensor-0.1.2] startup
[10.678s] [ext: omni.isaac.robot_engine_bridge_ui-0.1.0] startup
[10.689s] [ext: omni.kit.primitive.mesh-1.0.0] startup
[10.701s] [ext: omni.isaac.robot_engine_bridge-0.1.0] startup
[11.359s] [ext: omni.kit.window.console-0.0.0] startup
[11.361s] [ext: omni.isaac.motion_planning-0.1.4] startup
[11.371s] [ext: omni.isaac.occupancy_map-0.1.0] startup
[11.379s] [ext: omni.kit.widget.prompt-1.0.0] startup
[11.386s] [ext: omni.kit.asset_converter-1.1.22] startup
[11.448s] [ext: omni.kit.window.script_editor-1.5.0] startup
[11.449s] [ext: omni.kit.window.status_bar-0.0.0] startup
[11.451s] [ext: omni.rtx.window.settings-0.6.0] startup
[11.464s] [ext: omni.isaac.synthetic_utils-0.1.0] startup
[11.497s] [ext: omni.kit.widget.graph-1.1.2] startup
[11.512s] [ext: omni.kit.window.profiler-1.0.0] startup
[11.524s] [ext: omni.rtx.settings.core-0.5.4] startup
[11.532s] [ext: omni.isaac.dr-0.1.6] startup
[11.543s] [ext: omni.kit.window.extensions-1.0.2] startup
[11.553s] [ext: omni.isaac.urdf-0.1.4] startup
[11.568s] [ext: omni.isaac.contact_sensor-0.1.0] startup
[11.577s] [ext: omni.isaac.python_app-0.0.0] startup
[11.579s] [ext: omni.isaac.sim.python-2021.1.1] startup
[11.579s] app started
OmniKitHelper Starting up …
OmniKitHelper Startup Complete
Loading materials. Will generate data soon…
2021-08-10 19:09:24 [30,657ms] [Warning] [omni.isaac.dynamic_control.plugin] DcPeekObjectType: Function called while not simulating
Writing data to /home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/output
extra updates
ground truth available
Relative pose: (9.66,433.40,32.75) -17.95
ID: 1
2021-08-10 19:09:25 [31,666ms] [Warning] [omni.isaac.dynamic_control.plugin] DcPeekObjectType: Function called while not simulating
extra updates
ground truth available
Relative pose: (76.58,408.49,31.31) -12.20
ID: 2
2021-08-10 19:09:26 [32,455ms] [Warning] [omni.isaac.dynamic_control.plugin] DcPeekObjectType: Function called while not simulating
extra updates
ground truth available
Relative pose: (26.72,429.27,34.31) -18.64
ID: 3
2021-08-10 19:09:27 [33,250ms] [Warning] [omni.isaac.dynamic_control.plugin] DcPeekObjectType: Function called while not simulating
extra updates
ground truth available
Relative pose: (26.62,366.74,29.17) -10.46
ID: 4
2021-08-10 19:09:28 [34,046ms] [Warning] [omni.isaac.dynamic_control.plugin] DcPeekObjectType: Function called while not simulating
extra updates
ground truth available
Relative pose: (93.03,443.58,5.18) -12.08
ID: 5
sleep for 10 - let I/O spool out
shutdown
Shutting Down OmniKitHelper…
[44.864s] [ext: omni.isaac.sim.python-2021.1.1] shutdown
[44.864s] [ext: omni.isaac.contact_sensor-0.1.0] shutdown
[44.911s] [ext: omni.isaac.dr-0.1.6] shutdown
[44.923s] [ext: omni.isaac.motion_planning-0.1.4] shutdown
[44.923s] [ext: omni.isaac.robot_engine_bridge-0.1.0] shutdown
[44.973s] [ext: omni.isaac.occupancy_map-0.1.0] shutdown
[45.019s] [ext: omni.isaac.urdf-0.1.4] shutdown
[45.076s] [ext: omni.isaac.utils-0.1.4] shutdown
[45.289s] [ext: omni.isaac.dynamic_control-0.1.3] shutdown
[45.338s] [ext: omni.isaac.python_app-0.0.0] shutdown
[45.339s] [ext: omni.isaac.range_sensor-0.1.2] shutdown
[45.349s] [ext: omni.isaac.robot_engine_bridge_ui-0.1.0] shutdown
[45.351s] [ext: omni.isaac.synthetic_utils-0.1.0] shutdown
[45.353s] [ext: omni.kit.asset_converter-1.1.22] shutdown
[45.394s] [ext: omni.physx.bundle-1.2.0-5.1] shutdown
[45.394s] [ext: omni.physx.vehicle-1.2.0-5.1] shutdown
[45.442s] [ext: omni.kit.property.physx-0.1.0] shutdown
[45.483s] [ext: omni.kit.property.material-1.6.4] shutdown
[45.484s] [ext: omni.kit.property.usd-3.9.4] shutdown
[45.485s] [ext: omni.kit.window.content_browser-2.3.1] shutdown
[45.486s] [ext: omni.kit.window.profiler-1.0.0] shutdown
[45.487s] [ext: omni.physx.cct-1.2.0-5.1] shutdown
[45.528s] [ext: omni.physx.demos-1.0.0] shutdown
[45.570s] [ext: omni.physx.ui-1.2.0-5.1] shutdown
[45.615s] [ext: omni.syntheticdata-0.1.5] shutdown
[45.615s] [ext: omni.kit.window.viewport-0.0.0] shutdown
2021-08-10 19:09:40 [45,881ms] [Warning] [gpu.foundation.plugin] carb::extras::HandleDatabase<TrueType, HandleType, Allocator>::~HandleDatabase() [with TrueType = rtx::shaderdb::InternalShaderDesc; HandleType = rtx::shaderdb::_ShaderHandle*; Allocator = std::allocatorrtx::shaderdb::InternalShaderDesc]: had 436 outstanding handle(s) at shutdown
[45.893s] [ext: omni.kit.window.drop_support-1.0.0] shutdown
[45.893s] [ext: omni.kit.primitive.mesh-1.0.0] shutdown
[45.895s] [ext: omni.kit.window.preferences-1.1.1] shutdown
[45.901s] [ext: omni.kit.window.status_bar-0.0.0] shutdown
[45.902s] [ext: omni.kit.uiapp-0.0.0] shutdown
[45.902s] [ext: omni.kit.window.file-1.2.2] shutdown
[45.902s] [ext: omni.rtx.settings.core-0.5.4] shutdown
[45.905s] [ext: omni.rtx.window.settings-0.6.0] shutdown
[45.905s] [ext: omni.kit.window.filepicker-2.3.1] shutdown
[45.905s] [ext: omni.kit.notification_manager-1.0.0] shutdown
[45.905s] [ext: omni.kit.widget.stage-2.4.0] shutdown
[45.906s] [ext: omni.kit.context_menu-1.1.0] shutdown
[45.906s] [ext: omni.kit.widget.browser_bar-2.0.1] shutdown
[45.906s] [ext: omni.kit.widget.filebrowser-2.2.4] shutdown
[45.906s] [ext: omni.kit.window.extensions-1.0.2] shutdown
[45.906s] [ext: omni.kit.widget.graph-1.1.2] shutdown
[45.906s] [ext: omni.kit.widget.path_field-2.0.2] shutdown
[45.906s] [ext: omni.kit.widget.prompt-1.0.0] shutdown
[45.906s] [ext: omni.kit.widget.settings-1.0.0] shutdown
[45.907s] [ext: omni.kit.window.console-0.0.0] shutdown
[45.907s] [ext: omni.kit.window.popup_dialog-2.0.0] shutdown
[45.908s] [ext: omni.kit.window.property-0.0.0] shutdown
[45.908s] [ext: omni.kit.window.script_editor-1.5.0] shutdown
[45.908s] [ext: omni.kit.window.toolbar-1.0.0] shutdown
[45.909s] [ext: omni.kit.mainwindow-0.0.0] shutdown
[45.909s] [ext: omni.kit.material.library-1.0.4] shutdown
[45.910s] [ext: omni.kit.menu.create-1.0.1] shutdown
[45.911s] [ext: omni.kit.menu.utils-1.0.1] shutdown
[45.911s] [ext: omni.ui-2.1.8] shutdown
[45.911s] [ext: omni.kit.renderer.core-0.0.0] shutdown
2021-08-10 19:09:40 [46,056ms] [Warning] [gpu.foundation.plugin] Texture:‘/tmp/carb.ljfoo0/bookmark_grey.36x36.png’ is not released correctly and cause leak.
2021-08-10 19:09:40 [46,056ms] [Warning] [gpu.foundation.plugin] Texture:‘/tmp/carb.ljfoo0/filter.28x32.png’ is not released correctly and cause leak.
2021-08-10 19:09:40 [46,056ms] [Warning] [gpu.foundation.plugin] Texture:‘/tmp/carb.ljfoo0/angle_left.32x32.png’ is not released correctly and cause leak.
2021-08-10 19:09:40 [46,056ms] [Warning] [gpu.foundation.plugin] Texture:‘/tmp/carb.ljfoo0/angle_right.32x32.png’ is not released correctly and cause leak.
2021-08-10 19:09:40 [46,056ms] [Warning] [gpu.foundation.plugin] Texture:‘/tmp/carb.ljfoo0/settings.28x32.png’ is not released correctly and cause leak.
2021-08-10 19:09:40 [46,056ms] [Warning] [gpu.foundation.plugin] Release above leaking textures on exit.
[46.108s] [ext: omni.appwindow-1.0.0] shutdown
[46.108s] [ext: omni.kit.search_core-1.0.0] shutdown
[46.108s] [ext: omni.kit.thumbnails.images-0.1.1] shutdown
[46.108s] [ext: omni.services.transport.server.http-1.0.0] shutdown
[46.108s] [ext: omni.services.transport.server.base-1.0.0] shutdown
[46.109s] [ext: omni.services.transport.client.http_async-1.0.0] shutdown
[46.109s] [ext: omni.services.client-0.2.0] shutdown
[46.109s] [ext: omni.services.core-1.1.0] shutdown
[46.109s] [ext: omni.kit.test-0.0.0] shutdown
[46.109s] [ext: omni.kit.loop-isaac-0.1.0] shutdown
[46.109s] [ext: omni.usd.schema.isaac-0.1.0] shutdown
[46.109s] [ext: omni.debugdraw-0.0.0] shutdown
[46.109s] [ext: omni.kit.numpy.common-0.1.0] shutdown
[46.110s] [ext: omni.kit.stage_templates-1.0.4] shutdown
[46.110s] [ext: omni.physx.commands-1.0.0] shutdown
[46.150s] [ext: omni.physx-1.2.0-5.1] shutdown
[46.190s] [ext: omni.convexdecomposition-1.2.0] shutdown
[46.232s] [ext: omni.localcache-0.0.0] shutdown
[46.273s] [ext: omni.kvdb-0.0.0] shutdown
[46.316s] [ext: omni.usd.schema.physx-0.0.0] shutdown
[46.362s] [ext: omni.usdphysics-1.2.0] shutdown
[46.362s] [ext: omni.usd-1.4.2] shutdown
[46.371s] [ext: omni.usd.schema.physics-1.0.0] shutdown
[46.413s] [ext: carb.audio-0.1.0] shutdown
[46.413s] [ext: omni.renderer-rtx-0.0.0] shutdown
[46.414s] [ext: omni.client-0.1.0] shutdown
[46.416s] [ext: omni.assets.plugins-0.0.0] shutdown
[46.416s] [ext: omni.hydra.rtx-0.1.0] shutdown
[46.416s] [ext: omni.hydra.scene_delegate-0.1.0] shutdown
2021-08-10 19:09:40 [46,406ms] [Warning] [carb] unloadPlugin: Failed to find a plugin at: /home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/plugins/libomni.gpucompute-cuda.plugin.so.
2021-08-10 19:09:40 [46,406ms] [Warning] [carb] unloadPlugin: Failed to find a plugin at: /home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/plugins/libomni.gpucompute-d3dvk.plugin.so.
[46.416s] [ext: omni.kit.agent-0.1.0] shutdown
[46.417s] [ext: omni.kit.async_engine-0.0.0] shutdown
[46.417s] [ext: omni.kit.commands-0.0.0] shutdown
[46.417s] [ext: omni.kit.pip_archive-0.0.0] shutdown
[46.417s] [ext: omni.kit.pipapi-0.0.0] shutdown
[46.417s] [ext: omni.services.facilities.base-1.0.0] shutdown
[46.417s] [ext: omni.services.transport.client.base-1.0.0] shutdown
[46.417s] [ext: omni.stats-0.0.0] shutdown
[46.417s] [ext: omni.timeline-1.0.0] shutdown
[46.417s] [ext: omni.usd.schema.anim-0.0.0] shutdown
[46.417s] [ext: omni.usd.schema.semantics-0.0.0] shutdown
[46.417s] [ext: omni.usd.schema.audio-0.0.0] shutdown
[46.417s] [ext: omni.usd.libs-1.0.0] shutdown
[46.419s] [ext: omni.usd.config-1.0.0] shutdown
2021-08-10 19:09:40 [46,683ms] [Warning] [carb.tasking.plugin] Leaking 1 carb::tasking::Counter instances!
Shutting Down Complete
Finish writing data…
Done.

peter.gaston · August 10, 2021, 8:13pm

Very interesting. After “OmniKitHelper Startup Complete” is seen in the log, I then manually threw in wait loops with buried calls to setup_renderer() followed by update(). Seemed wedged on the second iteration to generate data, but eventually looped through.

        for i in range(5):
            print("sleep for 5")
            for j in range(10):
                self.kit.update()
            time.sleep(5)

I’ve got to believe there is a better way to know when the scene is fully rendered and ready to go…

Voila!

peter.gaston · August 11, 2021, 1:57pm

Not to get too deep, but it seems there is a race condition in the loading logic.

When ‘await omni.usd.get_context().open_stage_async(path)’ completes, the function ‘is_loading()’ returns false. This should be true it would seem.
After my 25 second sleep, ‘is_loading()’ returns true
The existing logic waits on ‘is_loading()’ before continuing - which works. (After an additional 30 seconds or so.)

peter.gaston · August 11, 2021, 2:21pm

And - one more area of the dreaded race condition… Notice in the image below the ghosting of the previous position of the pallet. Turns out that more waiting is needed to allow that to settle.

Currently, randomize_once is called, then a wait loop is done waiting for is_loading to be false. then get_groundtruth.

The value returned from is_loading is only valid after the first randomize_once - never again. leading to the ghosting.

(Hack) Solution - toss in more waiting before get_groundtruth

sdebnath · August 11, 2021, 9:34pm

Thanks @peter.gaston for reporting the issues.

is_loading API keeps track of the material loading (similar to the progress bar in bottom right corner). However, textures continue to load even after that for sometime which is not tracked by the API. We are working on this to figure out a proper API that takes into account both.

Also, for subsequent randomizations, is_loading will return false as the scene is already fully loaded with materials and textures. Sometime, it takes more than one frame to collect the groundtruth (gt) properly after the scene randomizes. Hence, additional update() calls will solve it.

Topic		Replies	Views
Generator.py crashes - 'std::bad_weak_ptr' Isaac Sim synthetic-data	3	783	April 5, 2024
Isacc SIm 2022.1.0 Multi GPU dose not work and memory leak is reporeted Isaac Sim cuda , python	4	664	April 5, 2024
Isaac Sim crashed due to inability to load robot description in lula.load_robot() Isaac Sim	2	602	August 17, 2023
Isaac sim launch stuck after "RTX ready" Isaac Sim	16	1150	December 7, 2023
Merely keeping Isaac Sim idle (empty landing screen) pushes RTX 2080 Ti utilization to 68%. Why? Isaac Sim	2	1069	April 5, 2024
Isaac Sim 2022.2.0 freez at startup after first time Isaac Sim	13	1540	March 24, 2023
Can't run 'from omni.isaac.range_sensor import _range_sensor' Isaac Sim lidar	5	813	August 23, 2024
Isaac Sim Replicator crashes Isaac Sim synthetic-data	11	2296	June 21, 2022
Isaac sim install issue Isaac Sim	4	994	May 15, 2023
Isaac Sim - People Simulation broken in 4.1.0 Isaac Sim isaac-sim-v4-1-0	10	415	August 16, 2024

Syntheticdata.py - rgb retrieves camera grayscale, not color (Solved! ?)

Related topics