Syntheticdata.py - rgb retrieves camera grayscale, not color (Solved! ?)

(Solved? See my response below. tldr; needs to fully wait for scene to render. FYI, on another bug, I also highlighted the need for a wait look for the DataWriter to finish spooling out before killing it.)

Using Isaac Sim, Synthetic data capture I can build the image files nicely, i.e., in color - see below. Leveraging the exact same scenario, with the given file generator.py, the rgb image is in gray scale - instead of color - see the next image. This image is what is returned from the sensors.get_rgb call in syntheticdata.py (211), i.e., it isn’t converted to gray scale later by DataWriter.

The system is a relatively clean ubuntu 18.04 - built for this project a few days ago. GPU is RTX 3070. Isaac Sim seems to run fine. A basic sample ‘visualize_groundtruth.py’ runs fine - no errors, some (different) warnings - and a good output image. generator_stereo.py sort of worked - it showed the physics, but in grayscale. Also crashed with a bad_weak_ptr after about 10 seconds.

(Apologies for reposting this - but this is a show stopper for training object pose detection. Sofar the rest of the pre-training, including outputting the relative pose of the camera to the object is working fine.)

Thought - loading the scenario in Isaac sim takes about 115 seconds. However the python version starts generating data at 30 seconds. Perhaps the offline python needs to wait for full rendering???

There are warnings that show up during the run (from inside VS Code). The full run dump is below, but here’s a summary list:

  • Warn: dumps uploading isn’t available, Extension with the same id is already registered
  • Warn: already a module ‘pxr.UsdPhysics’
  • Warn: [CUDA:RENDER] 0.1 CUDA rend warn : CUDA module initialization failed (version of your CUDA runtime library is 11.0, but 11.1 is required). FYI - I re-did with option ‘install_dependencies.sh --config=x86_64_cuda_11_1’ but no change after restart.
  • Warn: DcPeekObjectType: Function called while not simulating
  • and more warnings on shutdown… 436 outstanding handles; texture images not released correctly and could leak; failed to find plugin; leaking 1 carb::tasking::Counter.

From Isaac sim, using the built-in synthetic data recorder

Captured using python code (generator.py):

Full dump of run (from VS Code Terminal window):

pg@pg-U-18:~/.local/share/ov/pkg/isaac_sim-2021.1.1$ cd /home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1 ; /usr/bin/env /usr/bin/python /home/pg/.vscode/extensions/ms-python.python-2021.8.1105858891/pythonFiles/lib/python/debugpy/launcher 46649 – /home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/python_samples/syntheticdata/offline_generation/mPgenerator.py
[Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn’t available.
[Info] [carb] Logging to file: /home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/logs/Kit/Isaac-Sim/2021.1/kit_20210810_150854.log
[Info] [carb] Initializing plugin: carb.events.plugin (interfaces: [carb::events::IEvents v0.2]) (impl: carb.events.plugin)
[0.012s] Loading user config located at: ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/data/Kit/Isaac-Sim/2021.1/user.config.json’
2021-08-10 19:08:54 [41ms] [Warning] [omni.ext.plugin] Extension with the same id is already registered. ext_id: ‘omni.kit.xr-1.0.0’, ext 1: ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/exts/omni.kit.xr’, ext 2: ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/extscore/omni.kit.xr’
[0.058s] [ext: omni.kit.pipapi-0.0.0] startup
[0.113s] [ext: omni.kit.pip_archive-0.0.0] startup
[0.129s] [ext: omni.usd.config-1.0.0] startup
[0.137s] [ext: omni.usd.libs-1.0.0] startup
2021-08-10 19:08:54 [161ms] [Error] [carb.python] Failed to reload python module pxr.Tf. Error: module pxr.Tf._tf not in sys.modules. Traceback:
Traceback (most recent call last):
File “/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py”, line 56, in reload_module
return importlib.reload(sys.modules[name])
File “/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/python/lib/python3.7/importlib/init.py”, line 148, in reload
raise ImportError(msg.format(name), name=name)
ImportError: module pxr.Tf._tf not in sys.modules

2021-08-10 19:08:54 [161ms] [Error] [carb.scripting-python.plugin] Exception: Extension python module: ‘pxr.Tf’ in ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/extscore/omni.usd.libs/.’ failed to load.

At:
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(169): startup_all_extensions_in_module
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(225): startup_all_extensions_in_module
PythonExtension.cpp::startup()(2):
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/exts/omni.isaac.python_app/omni/isaac/python_app/python_app.py(185): _start_app
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/exts/omni.isaac.python_app/omni/isaac/python_app/python_app.py(107): init
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/python_samples/syntheticdata/offline_generation/mPgenerator.py(66): init
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/python_samples/syntheticdata/offline_generation/mPgenerator.py(418):
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/python/lib/python3.7/runpy.py(85): _run_code
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/python/lib/python3.7/runpy.py(96): _run_module_code
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/python/lib/python3.7/runpy.py(263): run_path
/home/pg/.vscode/extensions/ms-python.python-2021.8.1105858891/pythonFiles/lib/python/debugpy/…/debugpy/server/cli.py(285): run_file
/home/pg/.vscode/extensions/ms-python.python-2021.8.1105858891/pythonFiles/lib/python/debugpy/…/debugpy/server/cli.py(444): main
/home/pg/.vscode/extensions/ms-python.python-2021.8.1105858891/pythonFiles/lib/python/debugpy/main.py(45):
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/python/lib/python3.7/runpy.py(85): _run_code
/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/python/lib/python3.7/runpy.py(193): _run_module_as_main

2021-08-10 19:08:54 [161ms] [Error] [omni.ext.plugin] [ext: omni.usd.libs-1.0.0] Failed to process python module extension in ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/extscore/omni.usd.libs/.’.
[0.186s] [ext: omni.kit.async_engine-0.0.0] startup
[0.193s] [ext: omni.stats-0.0.0] startup
[0.201s] [ext: omni.assets.plugins-0.0.0] startup
[0.202s] [ext: omni.kit.loop-isaac-0.1.0] startup
[0.205s] [ext: omni.client-0.1.0] startup
[0.221s] [ext: omni.appwindow-1.0.0] startup
[0.238s] [ext: omni.kit.renderer.core-0.0.0] startup

|---------------------------------------------------------------------------------------------|
| Driver Version: 460.91 | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
| | | | | | Device-ID | UUID |
|---------------------------------------------------------------------------------------------|
| 0 | GeForce RTX 3070 | Yes | | 8438 MB | 10de | 0 |
| | | | | | 2484 | 1b228445… |
|=============================================================================================|
| OS: Linux pg-U-18, Version: 4.15.0-153-generic
| XServer Vendor: The X.Org Foundation, XServer Version: 11906000 (1.19.6.0)
| Processor: Intel(R) Core™ i7-7700K CPU @ 4.20GHz | Cores: Unknown | Logical: 8
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 32122 | Free Memory: 22509
| Total Page/Swap (MB): 2047 | Free Page/Swap: 2047
|---------------------------------------------------------------------------------------------|
[0.931s] [ext: omni.kit.test-0.0.0] startup
[0.952s] [ext: omni.ui-2.1.8] startup
[0.979s] [ext: omni.kit.mainwindow-0.0.0] startup
[0.987s] [ext: omni.kit.uiapp-0.0.0] startup
[0.988s] [ext: omni.usd.schema.physics-1.0.0] startup
2021-08-10 19:08:55 [1,078ms] [Warning] [carb.python] There is already a module ‘pxr.UsdPhysics’ in ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/extscore/omni.usd.schema.physics/pxr/UsdPhysics/init.py’. Can’t import another module with the same name in public space from ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/extsPhysics/omni.usd.schema.physics’.
2021-08-10 19:08:55 [1,088ms] [Warning] [carb.python] There is already a module ‘usd.schema.physics’ in ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/extscore/omni.usd.schema.physics/usd/schema/physics/init.py’. Can’t import another module with the same name in public space from ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/extsPhysics/omni.usd.schema.physics’.
[1.098s] [ext: omni.usd.schema.isaac-0.1.0] startup
[1.203s] [ext: omni.usd.schema.semantics-0.0.0] startup
[1.222s] [ext: omni.usd.schema.audio-0.0.0] startup
[1.240s] [ext: omni.usd.schema.physx-0.0.0] startup
[1.304s] [ext: omni.usd.schema.anim-0.0.0] startup
[1.366s] [ext: omni.services.facilities.base-1.0.0] startup
[1.397s] [ext: omni.services.core-1.1.0] startup
[1.819s] [ext: omni.services.transport.server.base-1.0.0] startup
[1.825s] [ext: omni.services.transport.server.http-1.0.0] startup
[2.423s] [ext: omni.kit.notification_manager-1.0.0] startup
[2.434s] [ext: omni.services.transport.client.base-1.0.0] startup
[2.438s] [ext: omni.kit.agent-0.1.0] startup
[2.442s] [ext: omni.kit.window.popup_dialog-2.0.0] startup
[2.455s] [ext: omni.hydra.scene_delegate-0.1.0] startup
[2.464s] [ext: omni.services.client-0.2.0] startup
[2.727s] [ext: carb.audio-0.1.0] startup
[2.749s] [ext: omni.timeline-1.0.0] startup
[2.756s] [ext: omni.kit.commands-0.0.0] startup
[2.773s] [ext: omni.services.transport.client.http_async-1.0.0] startup
[4.310s] [ext: omni.kit.search_core-1.0.0] startup
[4.316s] [ext: omni.kit.widget.path_field-2.0.2] startup
[4.328s] [ext: omni.usd-1.4.2] startup
[4.464s] [ext: omni.kit.thumbnails.images-0.1.1] startup
[4.478s] [ext: omni.kit.widget.filebrowser-2.2.4] startup
[4.490s] [ext: omni.kit.widget.browser_bar-2.0.1] startup
[4.493s] [ext: omni.kit.stage_templates-1.0.4] startup
[4.499s] [ext: omni.kit.window.filepicker-2.3.1] startup
[5.063s] [ext: omni.kit.window.file-1.2.2] startup
[5.392s] [ext: omni.renderer-rtx-0.0.0] startup
[5.393s] [ext: omni.kvdb-0.0.0] startup
[5.424s] [ext: omni.usdphysics-1.2.0] startup
[5.425s] [ext: omni.convexdecomposition-1.2.0] startup
[5.456s] [ext: omni.localcache-0.0.0] startup
[5.486s] [ext: omni.kit.menu.utils-1.0.1] startup
[5.511s] [ext: omni.physx-1.2.0-5.1] startup
[5.555s] [ext: omni.kit.material.library-1.0.4] startup
[5.588s] [ext: omni.kit.menu.create-1.0.1] startup
[5.591s] [ext: omni.kit.window.drop_support-1.0.0] startup
[5.593s] [ext: omni.kit.context_menu-1.1.0] startup
[5.892s] [ext: omni.hydra.rtx-0.1.0] startup
[5.896s] [ext: omni.kit.window.content_browser-2.3.1] startup
[6.182s] [ext: omni.kit.window.viewport-0.0.0] startup
2021-08-10 19:09:00 [6,175ms] [Warning] [omni.ext.plugin] [ext: omni.kit.window.viewport-0.0.0] ‘/home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/plugins/rtx/rtx.reshade.plugin’ in ‘[[native.plugin]]’ was not found.
2021-08-10 19:09:00 [6,394ms] [Warning] [rtx.neuraylib.plugin] [CUDA:RENDER] 0.1 CUDA rend warn : CUDA module initialization failed.
2021-08-10 19:09:00 [6,394ms] [Warning] [rtx.neuraylib.plugin] [CUDA:RENDER] 0.1 CUDA rend warn : The version of your CUDA runtime library is 11.0, but 11.1 is required
2021-08-10 19:09:00 [6,394ms] [Warning] [rtx.neuraylib.plugin] [CUDA:RENDER] 0.1 CUDA rend warn : Please verify that you are using the right shared CUDA libraries.
2021-08-10 19:09:00 [6,500ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER] 1.1 IRAY rend warn : CUDA module initialization failed.
2021-08-10 19:09:00 [6,500ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER] 1.1 IRAY rend warn : The version of your CUDA runtime library is 11.0, but 11.1 is required; iray photoreal can only run in CPU mode.
2021-08-10 19:09:00 [6,500ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER] 1.1 IRAY rend warn : Please verify that you are using the right shared CUDA libraries.
2021-08-10 19:09:00 [6,500ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER] 1.1 IRAY rend warn : There is no CUDA-capable GPU available to the iray photoreal renderer.
2021-08-10 19:09:03 [8,871ms] [Warning] [carb] Acquiring non optional plugin interface which is not listed as dependency: [rtx::raytracing::Subsurface v0.1] (plugin: rtx.raytracing.plugin), by client: carb.scenerenderer-rtx.plugin. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client.
[9.208s] [ext: omni.physx.commands-1.0.0] startup
[9.242s] [ext: omni.kit.widget.stage-2.4.0] startup
[9.266s] [ext: omni.kit.window.property-0.0.0] startup
[9.270s] [ext: omni.kit.widget.settings-1.0.0] startup
[9.275s] [ext: omni.kit.property.usd-3.9.4] startup
[9.318s] [ext: omni.kit.window.preferences-1.1.1] startup
[9.378s] [ext: omni.debugdraw-0.0.0] startup
[9.387s] [ext: omni.kit.property.material-1.6.4] startup
[9.396s] [ext: omni.physx.ui-1.2.0-5.1] startup
[9.449s] [ext: omni.kit.numpy.common-0.1.0] startup
[9.739s] [ext: omni.kit.property.physx-0.1.0] startup
[9.832s] [ext: omni.kit.window.toolbar-1.0.0] startup
[9.850s] [ext: omni.physx.demos-1.0.0] startup
[9.880s] [ext: omni.isaac.dynamic_control-0.1.3] startup
[9.925s] [ext: omni.physx.vehicle-1.2.0-5.1] startup
[9.971s] [ext: omni.physx.cct-1.2.0-5.1] startup
[10.005s] [ext: omni.isaac.utils-0.1.4] startup
[10.029s] [ext: omni.syntheticdata-0.1.5] startup
[10.661s] [ext: omni.physx.bundle-1.2.0-5.1] startup
[10.661s] [ext: omni.isaac.range_sensor-0.1.2] startup
[10.678s] [ext: omni.isaac.robot_engine_bridge_ui-0.1.0] startup
[10.689s] [ext: omni.kit.primitive.mesh-1.0.0] startup
[10.701s] [ext: omni.isaac.robot_engine_bridge-0.1.0] startup
[11.359s] [ext: omni.kit.window.console-0.0.0] startup
[11.361s] [ext: omni.isaac.motion_planning-0.1.4] startup
[11.371s] [ext: omni.isaac.occupancy_map-0.1.0] startup
[11.379s] [ext: omni.kit.widget.prompt-1.0.0] startup
[11.386s] [ext: omni.kit.asset_converter-1.1.22] startup
[11.448s] [ext: omni.kit.window.script_editor-1.5.0] startup
[11.449s] [ext: omni.kit.window.status_bar-0.0.0] startup
[11.451s] [ext: omni.rtx.window.settings-0.6.0] startup
[11.464s] [ext: omni.isaac.synthetic_utils-0.1.0] startup
[11.497s] [ext: omni.kit.widget.graph-1.1.2] startup
[11.512s] [ext: omni.kit.window.profiler-1.0.0] startup
[11.524s] [ext: omni.rtx.settings.core-0.5.4] startup
[11.532s] [ext: omni.isaac.dr-0.1.6] startup
[11.543s] [ext: omni.kit.window.extensions-1.0.2] startup
[11.553s] [ext: omni.isaac.urdf-0.1.4] startup
[11.568s] [ext: omni.isaac.contact_sensor-0.1.0] startup
[11.577s] [ext: omni.isaac.python_app-0.0.0] startup
[11.579s] [ext: omni.isaac.sim.python-2021.1.1] startup
[11.579s] app started
OmniKitHelper Starting up …
OmniKitHelper Startup Complete
Loading materials. Will generate data soon…
2021-08-10 19:09:24 [30,657ms] [Warning] [omni.isaac.dynamic_control.plugin] DcPeekObjectType: Function called while not simulating
Writing data to /home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/output
extra updates
ground truth available
Relative pose: (9.66,433.40,32.75) -17.95
ID: 1
2021-08-10 19:09:25 [31,666ms] [Warning] [omni.isaac.dynamic_control.plugin] DcPeekObjectType: Function called while not simulating
extra updates
ground truth available
Relative pose: (76.58,408.49,31.31) -12.20
ID: 2
2021-08-10 19:09:26 [32,455ms] [Warning] [omni.isaac.dynamic_control.plugin] DcPeekObjectType: Function called while not simulating
extra updates
ground truth available
Relative pose: (26.72,429.27,34.31) -18.64
ID: 3
2021-08-10 19:09:27 [33,250ms] [Warning] [omni.isaac.dynamic_control.plugin] DcPeekObjectType: Function called while not simulating
extra updates
ground truth available
Relative pose: (26.62,366.74,29.17) -10.46
ID: 4
2021-08-10 19:09:28 [34,046ms] [Warning] [omni.isaac.dynamic_control.plugin] DcPeekObjectType: Function called while not simulating
extra updates
ground truth available
Relative pose: (93.03,443.58,5.18) -12.08
ID: 5
sleep for 10 - let I/O spool out
shutdown
Shutting Down OmniKitHelper…
[44.864s] [ext: omni.isaac.sim.python-2021.1.1] shutdown
[44.864s] [ext: omni.isaac.contact_sensor-0.1.0] shutdown
[44.911s] [ext: omni.isaac.dr-0.1.6] shutdown
[44.923s] [ext: omni.isaac.motion_planning-0.1.4] shutdown
[44.923s] [ext: omni.isaac.robot_engine_bridge-0.1.0] shutdown
[44.973s] [ext: omni.isaac.occupancy_map-0.1.0] shutdown
[45.019s] [ext: omni.isaac.urdf-0.1.4] shutdown
[45.076s] [ext: omni.isaac.utils-0.1.4] shutdown
[45.289s] [ext: omni.isaac.dynamic_control-0.1.3] shutdown
[45.338s] [ext: omni.isaac.python_app-0.0.0] shutdown
[45.339s] [ext: omni.isaac.range_sensor-0.1.2] shutdown
[45.349s] [ext: omni.isaac.robot_engine_bridge_ui-0.1.0] shutdown
[45.351s] [ext: omni.isaac.synthetic_utils-0.1.0] shutdown
[45.353s] [ext: omni.kit.asset_converter-1.1.22] shutdown
[45.394s] [ext: omni.physx.bundle-1.2.0-5.1] shutdown
[45.394s] [ext: omni.physx.vehicle-1.2.0-5.1] shutdown
[45.442s] [ext: omni.kit.property.physx-0.1.0] shutdown
[45.483s] [ext: omni.kit.property.material-1.6.4] shutdown
[45.484s] [ext: omni.kit.property.usd-3.9.4] shutdown
[45.485s] [ext: omni.kit.window.content_browser-2.3.1] shutdown
[45.486s] [ext: omni.kit.window.profiler-1.0.0] shutdown
[45.487s] [ext: omni.physx.cct-1.2.0-5.1] shutdown
[45.528s] [ext: omni.physx.demos-1.0.0] shutdown
[45.570s] [ext: omni.physx.ui-1.2.0-5.1] shutdown
[45.615s] [ext: omni.syntheticdata-0.1.5] shutdown
[45.615s] [ext: omni.kit.window.viewport-0.0.0] shutdown
2021-08-10 19:09:40 [45,881ms] [Warning] [gpu.foundation.plugin] carb::extras::HandleDatabase<TrueType, HandleType, Allocator>::~HandleDatabase() [with TrueType = rtx::shaderdb::InternalShaderDesc; HandleType = rtx::shaderdb::_ShaderHandle*; Allocator = std::allocatorrtx::shaderdb::InternalShaderDesc]: had 436 outstanding handle(s) at shutdown
[45.893s] [ext: omni.kit.window.drop_support-1.0.0] shutdown
[45.893s] [ext: omni.kit.primitive.mesh-1.0.0] shutdown
[45.895s] [ext: omni.kit.window.preferences-1.1.1] shutdown
[45.901s] [ext: omni.kit.window.status_bar-0.0.0] shutdown
[45.902s] [ext: omni.kit.uiapp-0.0.0] shutdown
[45.902s] [ext: omni.kit.window.file-1.2.2] shutdown
[45.902s] [ext: omni.rtx.settings.core-0.5.4] shutdown
[45.905s] [ext: omni.rtx.window.settings-0.6.0] shutdown
[45.905s] [ext: omni.kit.window.filepicker-2.3.1] shutdown
[45.905s] [ext: omni.kit.notification_manager-1.0.0] shutdown
[45.905s] [ext: omni.kit.widget.stage-2.4.0] shutdown
[45.906s] [ext: omni.kit.context_menu-1.1.0] shutdown
[45.906s] [ext: omni.kit.widget.browser_bar-2.0.1] shutdown
[45.906s] [ext: omni.kit.widget.filebrowser-2.2.4] shutdown
[45.906s] [ext: omni.kit.window.extensions-1.0.2] shutdown
[45.906s] [ext: omni.kit.widget.graph-1.1.2] shutdown
[45.906s] [ext: omni.kit.widget.path_field-2.0.2] shutdown
[45.906s] [ext: omni.kit.widget.prompt-1.0.0] shutdown
[45.906s] [ext: omni.kit.widget.settings-1.0.0] shutdown
[45.907s] [ext: omni.kit.window.console-0.0.0] shutdown
[45.907s] [ext: omni.kit.window.popup_dialog-2.0.0] shutdown
[45.908s] [ext: omni.kit.window.property-0.0.0] shutdown
[45.908s] [ext: omni.kit.window.script_editor-1.5.0] shutdown
[45.908s] [ext: omni.kit.window.toolbar-1.0.0] shutdown
[45.909s] [ext: omni.kit.mainwindow-0.0.0] shutdown
[45.909s] [ext: omni.kit.material.library-1.0.4] shutdown
[45.910s] [ext: omni.kit.menu.create-1.0.1] shutdown
[45.911s] [ext: omni.kit.menu.utils-1.0.1] shutdown
[45.911s] [ext: omni.ui-2.1.8] shutdown
[45.911s] [ext: omni.kit.renderer.core-0.0.0] shutdown
2021-08-10 19:09:40 [46,056ms] [Warning] [gpu.foundation.plugin] Texture:‘/tmp/carb.ljfoo0/bookmark_grey.36x36.png’ is not released correctly and cause leak.
2021-08-10 19:09:40 [46,056ms] [Warning] [gpu.foundation.plugin] Texture:‘/tmp/carb.ljfoo0/filter.28x32.png’ is not released correctly and cause leak.
2021-08-10 19:09:40 [46,056ms] [Warning] [gpu.foundation.plugin] Texture:‘/tmp/carb.ljfoo0/angle_left.32x32.png’ is not released correctly and cause leak.
2021-08-10 19:09:40 [46,056ms] [Warning] [gpu.foundation.plugin] Texture:‘/tmp/carb.ljfoo0/angle_right.32x32.png’ is not released correctly and cause leak.
2021-08-10 19:09:40 [46,056ms] [Warning] [gpu.foundation.plugin] Texture:‘/tmp/carb.ljfoo0/settings.28x32.png’ is not released correctly and cause leak.
2021-08-10 19:09:40 [46,056ms] [Warning] [gpu.foundation.plugin] Release above leaking textures on exit.
[46.108s] [ext: omni.appwindow-1.0.0] shutdown
[46.108s] [ext: omni.kit.search_core-1.0.0] shutdown
[46.108s] [ext: omni.kit.thumbnails.images-0.1.1] shutdown
[46.108s] [ext: omni.services.transport.server.http-1.0.0] shutdown
[46.108s] [ext: omni.services.transport.server.base-1.0.0] shutdown
[46.109s] [ext: omni.services.transport.client.http_async-1.0.0] shutdown
[46.109s] [ext: omni.services.client-0.2.0] shutdown
[46.109s] [ext: omni.services.core-1.1.0] shutdown
[46.109s] [ext: omni.kit.test-0.0.0] shutdown
[46.109s] [ext: omni.kit.loop-isaac-0.1.0] shutdown
[46.109s] [ext: omni.usd.schema.isaac-0.1.0] shutdown
[46.109s] [ext: omni.debugdraw-0.0.0] shutdown
[46.109s] [ext: omni.kit.numpy.common-0.1.0] shutdown
[46.110s] [ext: omni.kit.stage_templates-1.0.4] shutdown
[46.110s] [ext: omni.physx.commands-1.0.0] shutdown
[46.150s] [ext: omni.physx-1.2.0-5.1] shutdown
[46.190s] [ext: omni.convexdecomposition-1.2.0] shutdown
[46.232s] [ext: omni.localcache-0.0.0] shutdown
[46.273s] [ext: omni.kvdb-0.0.0] shutdown
[46.316s] [ext: omni.usd.schema.physx-0.0.0] shutdown
[46.362s] [ext: omni.usdphysics-1.2.0] shutdown
[46.362s] [ext: omni.usd-1.4.2] shutdown
[46.371s] [ext: omni.usd.schema.physics-1.0.0] shutdown
[46.413s] [ext: carb.audio-0.1.0] shutdown
[46.413s] [ext: omni.renderer-rtx-0.0.0] shutdown
[46.414s] [ext: omni.client-0.1.0] shutdown
[46.416s] [ext: omni.assets.plugins-0.0.0] shutdown
[46.416s] [ext: omni.hydra.rtx-0.1.0] shutdown
[46.416s] [ext: omni.hydra.scene_delegate-0.1.0] shutdown
2021-08-10 19:09:40 [46,406ms] [Warning] [carb] unloadPlugin: Failed to find a plugin at: /home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/plugins/libomni.gpucompute-cuda.plugin.so.
2021-08-10 19:09:40 [46,406ms] [Warning] [carb] unloadPlugin: Failed to find a plugin at: /home/pg/.local/share/ov/pkg/isaac_sim-2021.1.1/kit/plugins/libomni.gpucompute-d3dvk.plugin.so.
[46.416s] [ext: omni.kit.agent-0.1.0] shutdown
[46.417s] [ext: omni.kit.async_engine-0.0.0] shutdown
[46.417s] [ext: omni.kit.commands-0.0.0] shutdown
[46.417s] [ext: omni.kit.pip_archive-0.0.0] shutdown
[46.417s] [ext: omni.kit.pipapi-0.0.0] shutdown
[46.417s] [ext: omni.services.facilities.base-1.0.0] shutdown
[46.417s] [ext: omni.services.transport.client.base-1.0.0] shutdown
[46.417s] [ext: omni.stats-0.0.0] shutdown
[46.417s] [ext: omni.timeline-1.0.0] shutdown
[46.417s] [ext: omni.usd.schema.anim-0.0.0] shutdown
[46.417s] [ext: omni.usd.schema.semantics-0.0.0] shutdown
[46.417s] [ext: omni.usd.schema.audio-0.0.0] shutdown
[46.417s] [ext: omni.usd.libs-1.0.0] shutdown
[46.419s] [ext: omni.usd.config-1.0.0] shutdown
2021-08-10 19:09:40 [46,683ms] [Warning] [carb.tasking.plugin] Leaking 1 carb::tasking::Counter instances!
Shutting Down Complete
Finish writing data…
Done.

Very interesting. After “OmniKitHelper Startup Complete” is seen in the log, I then manually threw in wait loops with buried calls to setup_renderer() followed by update(). Seemed wedged on the second iteration to generate data, but eventually looped through.

        for i in range(5):
            print("sleep for 5")
            for j in range(10):
                self.kit.update()
            time.sleep(5)

I’ve got to believe there is a better way to know when the scene is fully rendered and ready to go…

Voila!

Not to get too deep, but it seems there is a race condition in the loading logic.

  • When ‘await omni.usd.get_context().open_stage_async(path)’ completes, the function ‘is_loading()’ returns false. This should be true it would seem.
  • After my 25 second sleep, ‘is_loading()’ returns true
  • The existing logic waits on ‘is_loading()’ before continuing - which works. (After an additional 30 seconds or so.)

And - one more area of the dreaded race condition… Notice in the image below the ghosting of the previous position of the pallet. Turns out that more waiting is needed to allow that to settle.

Currently, randomize_once is called, then a wait loop is done waiting for is_loading to be false. then get_groundtruth.

The value returned from is_loading is only valid after the first randomize_once - never again. leading to the ghosting.

(Hack) Solution - toss in more waiting before get_groundtruth

Thanks @peter.gaston for reporting the issues.

is_loading API keeps track of the material loading (similar to the progress bar in bottom right corner). However, textures continue to load even after that for sometime which is not tracked by the API. We are working on this to figure out a proper API that takes into account both.

Also, for subsequent randomizations, is_loading will return false as the scene is already fully loaded with materials and textures. Sometime, it takes more than one frame to collect the groundtruth (gt) properly after the scene randomizes. Hence, additional update() calls will solve it.