CuRobo docker container streaming issues

Omniverse Core Platform Kit Application Streaming
1

hi this is my setup:

  • Issac Sim docker container
  • cuRobo installed in the docker
  • Host omniverse streamer

when I run runheadless.native I can see issacsim running on the omniverse streaming client.

however when I run the curobo example

/isaac-sim/python.sh examples/isaac_sim/collision_checker.py --headless_mode native

this error shows up this is the code for starting the simulation


try:
    # Third Party
    import isaacsim
except ImportError:
    pass


# Third Party
import torch

a = torch.zeros(4, device="cuda:0")

# Standard Library

# Standard Library
import argparse

# Third Party
from omni.isaac.kit import SimulationApp

parser = argparse.ArgumentParser()
parser.add_argument(
    "--nvblox", action="store_true", help="When True, enables headless mode", default=False
)

parser.add_argument(
    "--headless_mode",
    type=str,
    default=None,
    help="To run headless, use one of [native, websocket], webrtc might not work.",
)
args = parser.parse_args()

simulation_app = SimulationApp(
    {
        "headless": args.headless_mode is not None,
        "width": "1920",
        "height": "1080",
    }
)

full log

Starting kit application with the following args:  ['/isaac-sim/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py', '/isaac-sim/apps/omni.isaac.sim.python.kit', '--/app/tokens/exe-path=/isaac-sim/kit', '--/persistent/app/viewport/displayOptions=3094', '--/rtx/materialDb/syncLoads=True', '--/rtx/hydra/materialSyncLoads=True', '--/omni.kit.plugin/syncUsdLoads=True', '--/app/renderer/resolution/width=1920', '--/app/renderer/resolution/height=1080', '--/app/window/width=1440', '--/app/window/height=900', '--/renderer/multiGpu/enabled=True', '--/app/fastShutdown=True', '--ext-folder', '/isaac-sim/exts', '--ext-folder', '/isaac-sim/apps', '--/physics/cudaDevice=0', '--portable', '--no-window', '--/app/window/hideUi=1', '--allow-root']
Passing the following args to the base kit application:  ['--headless_mode', 'native']
[Info] [carb] Logging to file: /isaac-sim/kit/logs/Kit/Isaac-Sim/4.2/kit_20241018_164209.log
2024-10-18 16:42:09 [0ms] [Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
[0.091s] [ext: omni.blobkey-1.1.2] startup
[0.091s] [ext: omni.kit.async_engine-0.0.1] startup
[0.589s] [ext: omni.stats-1.0.1] startup
[0.590s] [ext: omni.datastore-0.0.0] startup
[0.594s] [ext: omni.client-1.2.0] startup
[0.612s] [ext: omni.ujitso.default-1.0.0] startup
[0.613s] [ext: omni.gpu_foundation.shadercache.vulkan-1.0.0] startup
[0.615s] [ext: omni.hsscclient-1.1.1] startup
[0.615s] [ext: omni.assets.plugins-0.0.0] startup
[0.616s] [ext: omni.gpu_foundation-0.0.0] startup
[0.622s] [ext: carb.windowing.plugins-1.0.0] startup
2024-10-18 16:42:09 [556ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-10-18 16:42:09 [556ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[0.623s] [ext: omni.kit.renderer.init-0.0.0] startup
2024-10-18 16:42:09 [579ms] [Warning] [omni.platforminfo.plugin] failed to open the default display.  Can't verify X Server version.

|---------------------------------------------------------------------------------------------|
| Driver Version: 550.107.02    | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA GeForce RTX 4080 SUPER    | Yes: 0 |     | 16376   MB | 10de      | 0          |
|     |                                  |        |     |            | 2702      | e510b4eb.. |
|     |                                  |        |     |            | 1         |            |
|=============================================================================================|
| OS: 22.04.3 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.3, Kernel: 6.8.0-47-generic
| Processor: AMD Ryzen 9 7950X3D 16-Core Processor | Cores: 16 | Logical: 32
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 95681 | Free Memory: 88572
| Total Page/Swap (MB): 8191 | Free Page/Swap: 8191
|---------------------------------------------------------------------------------------------|
2024-10-18 16:42:09 [815ms] [Warning] [gpu.foundation.plugin] IOMMU is enabled.
[1.019s] [ext: omni.kit.pipapi-0.0.0] startup
[1.021s] [ext: omni.kit.pip_archive-0.0.0] startup
[1.021s] [ext: omni.pip.compute-1.4.0] startup
[1.021s] [ext: omni.pip.cloud-1.1.3] startup
[1.023s] [ext: omni.isaac.core_archive-2.3.0] startup
[1.023s] [ext: omni.materialx.libs-1.0.4] startup
[1.029s] [ext: omni.isaac.ml_archive-2.0.1] startup
[1.029s] [ext: omni.kit.telemetry-0.5.0] startup
[1.060s] [ext: omni.usd.config-1.0.4] startup
[1.064s] [ext: omni.gpucompute.plugins-0.0.0] startup
[1.064s] [ext: omni.usd.libs-1.0.1] startup
[1.124s] [ext: omni.iray.libs-0.0.0] startup
[1.128s] [ext: omni.mdl.neuraylib-0.2.8] startup
[1.130s] [ext: omni.mdl-55.0.1] startup
[1.146s] [ext: omni.kit.usd.mdl-1.0.1] startup
[1.224s] [ext: omni.kit.loop-isaac-1.2.0] startup
[1.225s] [ext: omni.kit.test-1.1.0] startup
[1.253s] [ext: omni.appwindow-1.1.8] startup
2024-10-18 16:42:10 [1,187ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-10-18 16:42:10 [1,187ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[1.255s] [ext: omni.kit.renderer.core-1.0.2] startup
2024-10-18 16:42:10 [1,192ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-10-18 16:42:10 [1,192ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-10-18 16:42:10 [1,197ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-10-18 16:42:10 [1,197ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[1.265s] [ext: omni.kit.renderer.capture-0.0.0] startup
[1.266s] [ext: omni.kit.renderer.imgui-1.0.1] startup
2024-10-18 16:42:10 [1,201ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-10-18 16:42:10 [1,201ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-10-18 16:42:10 [1,202ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-10-18 16:42:10 [1,202ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-10-18 16:42:10 [1,203ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-10-18 16:42:10 [1,203ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[1.316s] [ext: omni.ui-2.25.22] startup
[1.324s] [ext: omni.kit.mainwindow-1.0.3] startup
[1.324s] [ext: carb.audio-0.1.0] startup
[1.328s] [ext: omni.uiaudio-1.0.0] startup
[1.329s] [ext: omni.kit.uiapp-0.0.0] startup
[1.329s] [ext: omni.usd.schema.isaac-2.1.0] startup
[1.335s] [ext: omni.usd.schema.audio-0.0.0] startup
[1.337s] [ext: omni.usd.schema.anim-0.0.0] startup
[1.351s] [ext: omni.usd.schema.geospatial-0.0.0] startup
[1.353s] [ext: omni.usd.schema.semantics-0.0.0] startup
[1.355s] [ext: omni.usd.schema.omnigraph-1.0.0] startup
[1.359s] [ext: omni.anim.navigation.schema-106.1.0] startup
[1.361s] [ext: omni.usd.schema.scene.visualization-2.0.2] startup
[1.363s] [ext: omni.usd.schema.omniscripting-1.0.0] startup
[1.366s] [ext: omni.anim.graph.schema-106.1.0] startup
[1.370s] [ext: omni.usd.schema.physx-106.1.9] startup
[1.392s] [ext: omni.usd.schema.forcefield-106.1.9] startup
[1.395s] [ext: omni.graph.exec-0.9.4] startup
[1.396s] [ext: omni.kit.usd_undo-0.1.8] startup
[1.397s] [ext: omni.kit.actions.core-1.0.0] startup
[1.398s] [ext: omni.kit.exec.core-0.13.4] startup
[1.400s] [ext: omni.usd_resolver-1.0.0] startup
[1.402s] [ext: omni.kit.commands-1.4.9] startup
[1.405s] [ext: omni.activity.core-1.0.1] startup
[1.406s] [ext: omni.usd.core-1.4.1] startup
[1.408s] [ext: omni.resourcemonitor-105.0.1] startup
[1.409s] [ext: omni.kit.window.popup_dialog-2.0.24] startup
[1.412s] [ext: omni.timeline-1.0.10] startup
[1.413s] [ext: omni.kit.widget.nucleus_connector-1.1.9] startup
[1.414s] [ext: usdrt.scenegraph-7.5.0] startup
[1.443s] [ext: omni.kit.audiodeviceenum-1.0.1] startup
[1.444s] [ext: omni.hydra.usdrt_delegate-7.5.1] startup
[1.453s] [ext: omni.hydra.scene_delegate-0.3.3] startup
[1.457s] [ext: omni.usd-1.12.2] startup
[1.478s] [ext: omni.kit.menu.core-1.0.4] startup
[1.479s] [ext: omni.kit.collaboration.telemetry-1.0.0] startup
[1.479s] [ext: omni.kit.property.adapter.core-1.0.1] startup
[1.481s] [ext: omni.kit.menu.utils-1.5.27] startup
[1.486s] [ext: omni.kit.clipboard-1.0.4] startup
[1.487s] [ext: omni.kit.collaboration.channel_manager-1.0.12] startup
[1.487s] [ext: omni.kit.property.adapter.usd-1.0.1] startup
[1.488s] [ext: omni.kit.widget.options_menu-1.1.6] startup
[1.490s] [ext: omni.kit.widget.graph-1.12.15] startup
[1.493s] [ext: omni.kit.widget.searchfield-1.1.8] startup
[1.494s] [ext: omni.kit.widget.filter-1.1.4] startup
[1.495s] [ext: omni.kit.property.adapter.fabric-1.0.1] startup
[1.495s] [ext: omni.ui_query-1.1.4] startup
[1.496s] [ext: omni.kit.window.extensions-1.4.11] startup
[1.500s] [ext: omni.kit.usd.layers-2.1.36] startup
[1.505s] [ext: omni.kit.widget.context_menu-1.2.2] startup
[1.506s] [ext: omni.hydra.rtx.shadercache.vulkan-1.0.0] startup
[1.507s] [ext: omni.kit.notification_manager-1.0.9] startup
[1.508s] [ext: omni.kit.hotkeys.core-1.3.5] startup
[1.508s] [ext: omni.kit.context_menu-1.8.1] startup
[1.509s] [ext: omni.kit.widget.prompt-1.0.7] startup
[1.510s] [ext: omni.kit.helper.file_utils-0.1.8] startup
[1.511s] [ext: omni.kit.widget.nucleus_info-1.0.2] startup
[1.511s] [ext: omni.kit.widget.filebrowser-2.10.51] startup
[1.514s] [ext: omni.kit.search_core-1.0.5] startup
[1.515s] [ext: omni.kit.widget.path_field-2.0.10] startup
[1.515s] [ext: omni.ui.scene-1.10.3] startup
[1.518s] [ext: omni.kit.widget.search_delegate-1.0.5] startup
[1.519s] [ext: omni.kit.widget.options_button-1.0.3] startup
[1.520s] [ext: omni.kit.widget.browser_bar-2.0.10] startup
[1.520s] [ext: omni.volume-0.5.0] startup
[1.522s] [ext: omni.kit.window.filepicker-2.10.40] startup
[1.531s] [ext: omni.ujitso.processor.texture-1.0.0] startup
[1.531s] [ext: omni.kit.window.file_exporter-1.0.30] startup
[1.533s] [ext: omni.ujitso.client-0.0.0] startup
[1.533s] [ext: omni.kit.widget.stage-2.11.2] startup
[1.540s] [ext: omni.kit.widget.live_session_management.ui-1.0.1] startup
[1.541s] [ext: omni.kit.collaboration.presence_layer-1.0.9] startup
[1.543s] [ext: omni.hydra.rtx-1.0.0] startup
[1.555s] [ext: omni.inspect-1.0.1] startup
[1.557s] [ext: omni.kit.widget.searchable_combobox-1.0.6] startup
[1.557s] [ext: omni.kit.window.file_importer-1.1.12] startup
[1.558s] [ext: omni.kit.stage_template.core-1.1.22] startup
[1.558s] [ext: omni.kit.widget.settings-1.2.2] startup
[1.560s] [ext: omni.kit.window.file-1.3.54] startup
[1.562s] [ext: omni.kit.window.drop_support-1.0.3] startup
[1.562s] [ext: omni.kit.window.content_browser_registry-0.0.6] startup
[1.562s] [ext: omni.kit.window.preferences-1.6.0] startup
[1.566s] [ext: omni.kit.window.content_browser-2.9.18] startup
[1.576s] [ext: omni.kit.widget.live_session_management-1.2.20] startup
[1.578s] [ext: omni.kit.widget.text_editor-1.0.2] startup
[1.579s] [ext: omni.kit.hydra_texture-1.3.9] startup
[1.581s] [ext: omni.kit.viewport.legacy_gizmos-1.1.0] startup
[1.583s] [ext: omni.kit.raycast.query-1.0.5] startup
[1.587s] [ext: omni.kit.material.library-1.5.6] startup
[1.591s] [ext: omni.kit.widget.viewport-106.1.0] startup
[1.594s] [ext: omni.kit.viewport.registry-104.0.6] startup
[1.594s] [ext: omni.hydra.engine.stats-1.0.2] startup
[1.596s] [ext: omni.kit.widget.highlight_label-1.0.2] startup
[1.596s] [ext: omni.kit.viewport.window-107.0.6] startup
[1.608s] [ext: omni.kit.window.property-1.11.3] startup
[1.610s] [ext: omni.graph.core-2.179.2] startup
[1.611s] [ext: omni.graph.tools-1.79.0] startup
[1.626s] [ext: omni.kit.viewport.utility-1.0.17] startup
[1.627s] [ext: omni.kit.property.usd-4.2.8] startup
[1.633s] [ext: omni.debugdraw-0.1.3] startup
[1.635s] [ext: omni.graph-1.140.0] startup
[1.665s] [ext: omni.graph.ui-1.70.2] startup
[1.677s] [ext: omni.kvdb-106.1.9] startup
[1.680s] [ext: omni.graph.image.core-0.4.5] startup
[1.681s] [ext: omni.localcache-106.1.9] startup
[1.683s] [ext: omni.graph.image.nodes-1.1.0] startup
[1.685s] [ext: omni.usdphysics-106.1.9] startup
[1.688s] [ext: omni.physx.foundation-106.1.9] startup
[1.689s] [ext: omni.convexdecomposition-106.1.9] startup
[1.693s] [ext: omni.graph.nodes-1.146.1] startup
[1.700s] [ext: omni.physx.cooking-106.1.9] startup
[1.708s] [ext: omni.kit.primitive.mesh-1.0.17] startup
[1.710s] [ext: omni.physx-106.1.9] startup
[1.719s] [ext: omni.kit.stage_templates-1.2.5] startup
[1.721s] [ext: omni.isaac.version-1.1.0] startup
[1.722s] [ext: omni.kit.widget.material_preview-1.0.16] startup
[1.723s] [ext: omni.isaac.nucleus-0.3.1] startup
[1.724s] [ext: omni.physics.tensors-106.1.9] startup
[1.729s] [ext: omni.physx.stageupdate-106.1.9] startup
[1.730s] [ext: omni.graph.action_core-1.1.6] startup
[1.733s] [ext: omni.warp.core-1.2.1] startup
[1.804s] [ext: omni.physx.tensors-106.1.9] startup
[1.808s] [ext: omni.graph.scriptnode-1.20.1] startup
[1.809s] [ext: omni.kit.manipulator.viewport-107.0.0] startup
[1.810s] [ext: omni.graph.action_nodes-1.24.0] startup
[1.815s] [ext: omni.kit.ui_test-1.3.2] startup
[1.820s] [ext: omni.isaac.core-3.19.5] startup
[2.055s] [ext: omni.graph.visualization.nodes-2.1.1] startup
[2.060s] [ext: omni.graph.action-1.102.1] startup
[2.062s] [ext: omni.kit.window.cursor-1.1.2] startup
[2.064s] [ext: omni.isaac.ui-0.16.0] startup
[2.072s] [ext: omni.kit.viewport.menubar.core-106.1.0] startup
[2.085s] [ext: omni.kit.graph.delegate.default-1.2.2] startup
[2.086s] [ext: omni.syntheticdata-0.6.9] startup
[2.106s] [ext: omni.kit.graph.usd.commands-1.3.1] startup
[2.109s] [ext: omni.kit.graph.editor.core-1.5.3] startup
[2.113s] [ext: omni.kit.numpy.common-0.1.2] startup
[2.115s] [ext: omni.warp-1.2.1] startup
[2.126s] [ext: omni.kit.window.material_graph-1.8.18] startup
[2.181s] [ext: omni.isaac.dynamic_control-1.3.8] startup
[2.196s] [ext: omni.replicator.core-1.11.20] startup
2024-10-18 16:42:11 [2,146ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template
Warp 1.2.1 initialized:
   CUDA Toolkit 11.8, Driver 12.4
   Devices:
     "cpu"      : "x86_64"
     "cuda:0"   : "NVIDIA GeForce RTX 4080 SUPER" (16 GiB, sm_89, mempool enabled)
   Kernel cache:
     /root/.cache/warp/1.2.1
[2.306s] [ext: omni.isaac.lula-3.0.1] startup
[2.313s] [ext: omni.isaac.surface_gripper-1.0.1] startup
[2.317s] [ext: omni.isaac.core_nodes-1.16.3] startup
[2.333s] [ext: omni.isaac.motion_generation-7.1.0] startup
[2.346s] [ext: omni.isaac.manipulators-2.1.0] startup
[2.348s] [ext: omni.isaac.franka-0.4.1] startup
[2.355s] [ext: omni.kit.graph.widget.variables-2.1.0] startup
[2.362s] [ext: omni.kit.graph.delegate.modern-1.10.6] startup
[2.368s] [ext: omni.isaac.cortex-0.3.9] startup
[2.370s] [ext: omni.graph.window.core-1.113.1] startup
[2.391s] [ext: omni.isaac.wheeled_robots-2.3.3] startup
[2.406s] [ext: omni.isaac.cortex.sample_behaviors-1.0.6] startup
[2.406s] [ext: omni.graph.window.generic-1.26.0] startup
[2.418s] [ext: omni.isaac.menu-0.7.3] startup
[2.422s] [ext: omni.kit.widget.live-2.1.8] startup
[2.429s] [ext: omni.kit.actions.window-1.1.1] startup
[2.438s] [ext: omni.kit.widget.cache_indicator-2.0.10] startup
[2.494s] [ext: omni.kit.hotkeys.window-1.4.5] startup
2024-10-18 16:42:11 [2,427ms] [Warning] [omni.kit.widget.cache_indicator.cache_state_menu] Unable to detect Omniverse Cache Server. Consider installing it for better IO performance.
[2.504s] [ext: omni.kit.ui.actions-1.0.2] startup
[2.506s] [ext: omni.kit.widget.stage_icons-1.0.5] startup
[2.510s] [ext: omni.kit.menu.common-1.1.7] startup
[2.515s] [ext: omni.kit.selection-0.1.4] startup
[2.518s] [ext: omni.isaac.kit-1.13.2] startup
[2.519s] [ext: omni.isaac.block_world-1.0.0] startup
[2.526s] [ext: omni.kit.window.stage-2.5.10] startup
[2.529s] [ext: omni.isaac.debug_draw-1.1.0] startup
[2.532s] [ext: omni.kit.menu.edit-1.1.24] startup
[2.536s] [ext: omni.kit.menu.file-1.1.14] startup
[2.537s] [ext: omni.kit.profiler.window-2.2.3] startup
[2.543s] [ext: omni.kit.menu.stage-1.2.5] startup
[2.545s] [ext: omni.isaac.occupancy_map-1.0.3] startup
[2.549s] [ext: omni.importer.mjcf-1.1.1] startup
[2.554s] [ext: omni.graph.window.action-1.28.0] startup
[2.559s] [ext: omni.kit.property.camera-1.0.8] startup
[2.561s] [ext: omni.kit.property.light-1.0.10] startup
[2.562s] [ext: omni.kit.property.geometry-1.3.1] startup
[2.565s] [ext: omni.hydra.scene_api-0.1.2] startup
[2.568s] [ext: omni.kit.property.audio-1.0.14] startup
[2.570s] [ext: omni.kit.property.render-1.1.2] startup
[2.571s] [ext: omni.kit.property.transform-1.5.9] startup
[2.574s] [ext: omni.kit.property.material-1.10.8] startup
[2.578s] [ext: omni.kit.widget.layers-1.8.0] startup
[2.586s] [ext: omni.kit.property.isaac-0.2.3] startup
[2.587s] [ext: omni.kit.property.bundle-1.3.1] startup
[2.588s] [ext: omni.sensors.nv.common-1.2.2-isaac-1] startup
[2.596s] [ext: omni.kit.property.layer-1.1.8] startup
[2.598s] [ext: omni.kit.stage_column.payload-2.0.0] startup
[2.599s] [ext: omni.isaac.scene_blox-0.1.2] startup
[2.600s] [ext: omni.sensors.nv.materials-1.2.1-isaac-1] startup
[2.602s] [ext: omni.sensors.nv.wpm-1.2.1-isaac-1] startup
[2.602s] [ext: omni.kit.viewport.actions-106.0.2] startup
[2.605s] [ext: omni.sensors.nv.lidar-1.2.2-isaac-1] startup
[2.608s] [ext: omni.sensors.nv.radar-1.2.1-isaac-1] startup
[2.613s] [ext: omni.kit.viewport.menubar.display-107.0.2] startup
[2.614s] [ext: omni.kit.manipulator.transform-104.7.4] startup
[2.616s] [ext: omni.kit.widget.toolbar-1.7.2] startup
[2.620s] [ext: omni.sensors.tiled-0.0.6] startup
[2.622s] [ext: omni.sensors.nv.ids-1.1.0-isaac-1] startup
[2.625s] [ext: omni.isaac.range_sensor-3.1.2] startup
[2.633s] [ext: omni.usdphysics.ui-106.1.9] startup
[2.648s] [ext: omni.physx.commands-106.1.9] startup
[2.651s] [ext: omni.kit.manipulator.tool.snap-1.5.11] startup
[2.655s] [ext: omni.isaac.sensor-12.9.1] startup
[2.701s] [ext: omni.kit.manipulator.selector-1.1.1] startup
[2.704s] [ext: omni.physx.ui-106.1.9] startup
[2.731s] [ext: omni.isaac.quadruped-2.0.1] startup
[2.737s] [ext: omni.kit.viewport.manipulator.transform-107.0.2] startup
[2.739s] [ext: omni.physx.demos-106.1.9] startup
[2.752s] [ext: omni.kit.property.physx-106.1.9] startup
[2.803s] [ext: omni.kit.widget.calendar-1.0.8] startup
[2.806s] [ext: omni.kit.manipulator.prim.core-107.0.4] startup
[2.811s] [ext: omni.fabric.commands-1.1.5] startup
[2.813s] [ext: omni.physx.vehicle-106.1.9] startup
[2.826s] [ext: omni.kit.widget.extended_searchfield-1.0.28] startup
[2.833s] [ext: omni.kit.manipulator.prim.fabric-107.0.3] startup
[2.835s] [ext: omni.kit.manipulator.prim.usd-107.0.2] startup
[2.836s] [ext: omni.kit.widget.timeline-105.0.1] startup
[2.838s] [ext: omni.kit.window.commands-0.2.6] startup
[2.839s] [ext: omni.physx.camera-106.1.9] startup
[2.848s] [ext: omni.kit.manipulator.camera-105.0.5] startup
[2.850s] [ext: omni.kit.viewport.menubar.camera-107.0.2] startup
[2.853s] [ext: omni.kit.manipulator.prim-107.0.0] startup
[2.854s] [ext: omni.kit.manipulator.selection-106.0.1] startup
[2.855s] [ext: omni.kit.window.toolbar-1.6.1] startup
[2.856s] [ext: omni.kit.stage_column.variant-1.0.13] startup
[2.858s] [ext: omni.kit.viewport.menubar.render-106.1.1] startup
[2.859s] [ext: omni.kit.viewport.menubar.settings-107.0.3] startup
[2.862s] [ext: omni.physx.cct-106.1.9] startup
[2.870s] [ext: omni.physx.graph-106.1.9] startup
[2.885s] [ext: omni.physx.supportui-106.1.9] startup
[2.899s] [ext: omni.physx.telemetry-106.1.9] startup
[2.902s] [ext: omni.graph.ui_nodes-1.26.0] startup
[2.907s] [ext: omni.kit.viewport.bundle-104.0.1] startup
[2.907s] [ext: omni.isaac.universal_robots-0.3.5] startup
[2.908s] [ext: omni.kit.window.console-0.2.13] startup
[2.922s] [ext: omni.kit.window.script_editor-1.7.6] startup
[2.924s] [ext: omni.kit.menu.create-1.0.16] startup
[2.926s] [ext: omni.graph.bundle.action-2.4.1] startup
[2.928s] [ext: omni.physx.bundle-106.1.9] startup
[2.929s] [ext: omni.rtx.window.settings-0.6.17] startup
[2.933s] [ext: omni.kit.window.status_bar-0.1.7] startup
[2.937s] [ext: omni.kit.window.title-1.1.5] startup
[2.940s] [ext: omni.replicator.isaac-1.15.2] startup
[2.958s] [ext: omni.replicator.replicator_yaml-2.0.6] startup
[2.974s] [ext: omni.rtx.settings.core-0.6.3] startup
[2.983s] [ext: omni.kit.viewport.menubar.lighting-106.0.2] startup
[2.990s] [ext: semantics.schema.editor-0.3.8] startup
[2.994s] [ext: semantics.schema.property-1.0.4] startup
[2.995s] [ext: omni.kit.viewport.rtx-104.0.1] startup
[2.995s] [ext: omni.isaac.utils-1.0.1] startup
[2.998s] [ext: omni.isaac.cloner-0.8.1] startup
[3.000s] [ext: omni.importer.urdf-1.14.1] startup
[3.029s] [ext: omni.kit.window.stats-0.1.6] startup
[3.033s] [ext: omni.isaac.sim.python-4.2.0] startup
[3.034s] Simulation App Starting
[4.413s] app ready
2024-10-18 16:42:13 [4,842ms] [Warning] [omni.kit.imgui_renderer.plugin] _createExtendCursor: No windowing.
2024-10-18 16:42:13 [4,842ms] [Warning] [omni.kit.imgui_renderer.plugin] _createExtendCursor: No windowing.
[5.088s] Simulation App Startup Complete
2024-10-18 16:42:16 [7,187ms] [Warning] [curobo] Running in headless mode: native
[7.263s] [ext: omni.kit.asset_converter-2.1.21] startup
[7.279s] [ext: omni.kit.tool.asset_importer-2.5.5] startup
[7.328s] [ext: omni.kit.widget.zoombar-1.0.5] startup
[7.330s] [ext: omni.kit.browser.core-2.3.11] startup
[7.333s] [ext: omni.kit.browser.folder.core-1.9.13] startup
[7.336s] [ext: omni.isaac.asset_browser-0.5.1] startup
[7.380s] [ext: omni.kit.livestream.core-4.3.6] startup
[7.383s] [ext: omni.kit.streamsdk.plugins-4.5.3] startup
[7.383s] [ext: omni.kit.livestream.native-4.1.0] startup
Warning: Possible version incompatibility. Attempting to load carb::cudainterop::CudaInterop with version v0.13 against v0.11.

Failed to open [/var/run/utmp]
Active user not found. Using default user [kiosk]Streaming server started.
**** Click Play to start simulation *****
**** Click Play to start simulation *****
**** Click Play to start simulation *****
**** Click Play to start simulation *****
**** Click Play to start simulation *****
main: thread_init: already added for thread
main: thread_init: already added for thread
main: thread_init: already added for thread
2024-10-18 16:42:26 [17,150ms] [Warning] [carb.livestream.plugin] onClientEventRaised: NVST_CCE_DISCONNECTED when m_connectionCount 0 != 1  <-- happens when I try to connect 
**** Click Play to start simulation *****

streaming client

image
image728×447 23.8 KB

any help is greatly appreciated, thank you!

2

Hello and thanks for your question. So, you are saying that you can get Isaac Sim to stream correctly, but not the CuRobo example ? I not familiar with this example.

3

@rdarbha

4

Please provide the exact command how you run IsaacSim in the docker.
Do you run the OV streaming client on the same physical/virtual server/PC?

5

Hello, thank you for getting back. It looks like the cuRobo code base is old and has not been update. I have figured a way to fix this issue.

I have gone through the documentation and rewrote the code for headless streaming from docker.


try:
    # Third Party
    import isaacsim
except ImportError:
    pass


# Third Party
import torch

a = torch.zeros(4, device="cuda:0")

# Standard Library
import argparse

from scipy.spatial.transform import Rotation as R

from isaacsim import SimulationApp

# This sample enables a livestream server to connect to when running headless
CONFIG = {
    "width": 1280,
    "height": 720,
    "window_width": 1920,
    "window_height": 1080,
    "headless": True,
    "hide_ui": False,  # Show the GUI
    "renderer": "RayTracedLighting",
    "display_options": 3286,  # Set display options to show default grid
}


# Start the omniverse application
kit = SimulationApp(launch_config=CONFIG)

from omni.isaac.core.utils.extensions import enable_extension

# Default Livestream settings
kit.set_setting("/app/window/drawMouse", True)
kit.set_setting("/app/livestream/proto", "ws")
kit.set_setting("/ngx/enabled", False)

# Note: Only one livestream extension can be enabled at a time

# Enable Native Livestream extension
# Default App: Streaming Client from the Omniverse Launcher
enable_extension("omni.kit.streamsdk.plugins-3.2.1")
enable_extension("omni.kit.livestream.core-3.2.0")
enable_extension("omni.kit.livestream.native")

# Enable WebRTC Livestream extension
# Default URL: http://localhost:8211/streaming/webrtc-client/
# enable_extension("omni.services.streamclient.webrtc")
##-------------------------------## 
# Third Party
import carb
import numpy as np
from omni.isaac.core import World
from omni.isaac.core.materials import OmniPBR
from omni.isaac.core.objects import sphere

# CuRobo
# from curobo.wrap.reacher.ik_solver import IKSolver, IKSolverConfig
from curobo.geom.sdf.world import CollisionCheckerType
from curobo.geom.types import WorldConfig, Cuboid, Mesh
from curobo.types.base import TensorDeviceType
from curobo.types.math import Pose

########### OV #################
from curobo.util.logger import setup_curobo_logger
from curobo.util.usd_helper import UsdHelper
from curobo.util_file import get_world_configs_path, join_path, load_yaml
from curobo.wrap.model.robot_world import RobotWorld, RobotWorldConfig

from omni.isaac.core.robots import Robot


from omni.isaac.core.objects import DynamicCuboid
from omni.isaac.core.objects.ground_plane import GroundPlane
from omni.isaac.core.physics_context import PhysicsContext
from omni.isaac.core.objects import VisualSphere


# CuRobo
from curobo.util.logger import log_warn
from curobo.util.usd_helper import set_prim_transform
from curobo.types.state import JointState


from typing import Dict, List

ISAAC_SIM_23 = False
try:
    # Third Party
    from omni.isaac.urdf import _urdf  # isaacsim 2022.2
except ImportError:
    # Third Party
    from omni.importer.urdf import _urdf  # isaac sim 2023.1 or above
    ISAAC_SIM_23 = True

# Standard Library
from typing import Optional

# Third Party
from omni.isaac.core.utils.extensions import enable_extension

# CuRobo
from curobo.util_file import get_assets_path, get_filename, get_path_of_dir, join_path
from curobo.wrap.reacher.motion_gen import MotionGen, MotionGenConfig, MotionGenPlanConfig



def quaternion_to_euler(q):
    """Convert quaternion to Euler angles (roll, pitch, yaw)"""
    r = R.from_quat(q)
    return r.as_euler('xyz', degrees=True)

def euler_to_quaternion(euler):
    """Convert Euler angles to quaternion"""
    r = R.from_euler('xyz', euler, degrees=True)
    return r.as_quat()


def calculate_face_centers(position, orientation, scale):
    # Create rotation matrix from quaternion
    rot_matrix = R.from_quat(orientation).as_matrix()
    
    # Half extents of the cuboid
    half_extents = scale / 2

    # Define the face centers in local coordinates
    local_centers = np.array([
        [half_extents[0], 0, 0],  # +X face
        [-half_extents[0], 0, 0], # -X face
        [0, half_extents[1], 0],  # +Y face
        [0, -half_extents[1], 0], # -Y face
        [0, 0, half_extents[2]],  # +Z face
        [0, 0, -half_extents[2]]  # -Z face
    ])

    # Transform local centers to world coordinates
    world_centers = []
    for local_center in local_centers:
        world_center = np.dot(rot_matrix, local_center) + position
        world_centers.append(world_center)

    return np.array(world_centers)

def add_robot_to_scene(
    robot_config: Dict,
    my_world: World,
    load_from_usd: bool = False,
    subroot: str = "",
    robot_name: str = "robot",
    position: np.array = np.array([0, 0, 0]),
):

    urdf_interface = _urdf.acquire_urdf_interface()
    # Set the settings in the import config
    import_config = _urdf.ImportConfig()
    import_config.merge_fixed_joints = False
    import_config.convex_decomp = False
    import_config.fix_base = True
    import_config.make_default_prim = True
    import_config.self_collision = False
    import_config.create_physics_scene = True
    import_config.import_inertia_tensor = False
    import_config.default_drive_strength = 1047.19751
    import_config.default_position_drive_damping = 52.35988
    import_config.default_drive_type = _urdf.UrdfJointTargetType.JOINT_DRIVE_POSITION
    import_config.distance_scale = 1
    import_config.density = 0.0

    asset_path = get_assets_path()
    if (

    "external_asset_path" in robot_config["kinematics"]

    and robot_config["kinematics"]["external_asset_path"] is not None

    ):

        asset_path = robot_config["kinematics"]["external_asset_path"]

    full_path = join_path(asset_path, robot_config["kinematics"]["urdf_path"])
    robot_path = get_path_of_dir(full_path)
    filename = get_filename(full_path)
    imported_robot = urdf_interface.parse_urdf(robot_path, filename, import_config)
    dest_path = subroot
    robot_path = urdf_interface.import_robot(
        robot_path,
        filename,
        imported_robot,
        import_config,
        dest_path,
    )

    base_link_name = robot_config["kinematics"]["base_link"]

    robot_p = Robot(
        prim_path=robot_path + "/" + base_link_name,
        name=robot_name,
    )

    robot_prim = robot_p.prim
    stage = robot_prim.GetStage()
    linkp = stage.GetPrimAtPath(robot_path)
    set_prim_transform(linkp, [position[0], position[1], position[2], 1, 0, 0, 0])

    robot = my_world.scene.add(robot_p)
    return robot, robot_path


def main(): 

    usd_help = UsdHelper()
    act_distance = 0.4
    ignore_list = ["/World/target", "/World/defaultGroundPlane"]

    # assuming obstacles are in objects_path:
    my_world = World(stage_units_in_meters=1.0)
    my_world.scene.add_default_ground_plane()

    
    
    dynamicCuboid = DynamicCuboid(prim_path="/World/cube", position=np.array([-.5, -.2, 1.0]), scale=np.array([.5, .5, .5]), color=np.array([.2,.3,0.]))

    cuboid = my_world.scene.add(
        dynamicCuboid
    )

    face_markers = []
    for i in range(6):
        marker = VisualSphere(prim_path=f"/World/face_marker_{i}", 
                              name=f"face_marker_{i}",
                              radius=0.05, 
                              color=np.array([1.0, 0.0, 0.0]))
        my_world.scene.add(marker)
        face_markers.append(marker)


    stage = my_world.stage
    usd_help.load_stage(stage)
    xform = stage.DefinePrim("/World", "Xform")
    stage.SetDefaultPrim(xform)
    stage.DefinePrim("/curobo", "Xform")

    tensor_args = TensorDeviceType()
    world_config_1 = WorldConfig(cuboid=[dynamicCuboid])

    initial_quat = [0, 0, 0, 1]
    initial_euler = quaternion_to_euler(initial_quat)
    print(f"Initial Euler angles: {initial_euler}")

    # Keep roll and pitch constant, iterate over yaw
    roll, pitch, yaw = initial_euler[0], initial_euler[1], initial_euler[2]
    yaw_range = np.linspace(0, 360, 37)  # 0 to 360 degrees in 10-degree steps

    print("Loading robot...")
    robot_cfg = load_yaml("/root/Project/config/curobo.yml")["robot_cfg"]
    print(robot_cfg["kinematics"])
    print("Robot config loaded.")


    j_names = robot_cfg["kinematics"]["cspace"]["joint_names"]
    default_config = robot_cfg["kinematics"]["cspace"]["retract_config"]

    robot, robot_prim_path = add_robot_to_scene(robot_cfg, my_world, position=np.array([-1.5, -.2, 1.0]))

    j_names = robot_cfg["kinematics"]["cspace"]["joint_names"]
    idx_list = [robot.get_dof_index(x) for x in j_names]

    robot.set_joint_positions(default_config, idx_list)

    robot._articulation_view.set_max_efforts(
        values=np.array([5000 for i in range(len(idx_list))]), joint_indices=idx_list
    )

    robot._articulation_view.set_max_efforts(
        values=np.array([5000 for i in range(len(idx_list))]), joint_indices=idx_list
    )
    
    
    world_cfg_table = WorldConfig.from_dict(
        load_yaml("/root/curobo/src/curobo/curobo/content/configs/world/collision_table.yml")
    )
    world_cfg_table.cuboid[0].pose[2] -= 0.02

    world_cfg1 = WorldConfig.from_dict(
        load_yaml("/root/curobo/src/curobo/curobo/content/configs/world/collision_table.yml")
    ).get_mesh_world()
    
    world_cfg1.mesh[0].name += "_mesh"
    world_cfg1.mesh[0].pose[2] = -10.5

    world_cfg = WorldConfig(cuboid=world_cfg_table.cuboid, mesh=world_cfg1.mesh)

    setup_curobo_logger("warn")
    past_pose = None
    n_obstacle_cuboids = 30
    n_obstacle_mesh = 10

    motion_gen_config = MotionGenConfig.load_from_robot_config(
        robot_cfg,
        world_cfg,
        tensor_args,
        collision_checker_type=CollisionCheckerType.MESH,
        use_cuda_graph=True,
        interpolation_dt=0.03,
        collision_cache={"obb": n_obstacle_cuboids, "mesh": n_obstacle_mesh},
        collision_activation_distance=0.025,
        fixed_iters_trajopt=True,
        maximum_trajectory_dt=0.5,
    )

    motion_gen = MotionGen(motion_gen_config)
    
    # Initialize motion generators for both arms
    motion_gen_left = MotionGen(motion_gen_config)
    motion_gen_right = MotionGen(motion_gen_config)

    articulation_controller = robot.get_articulation_controller()

    # Get initial joint positions
    initial_joint_positions = robot.get_joint_positions()

    link_names = motion_gen.kinematics.link_names
    ee_link_name = motion_gen.kinematics.ee_link

    kin_state = motion_gen.kinematics.get_state(motion_gen.get_retract_config().view(1, -1))

    link_retract_pose = kin_state.link_pose
    print("ee_link_name: ", ee_link_name)

    while kit._app.is_running() and not kit.is_exiting():
        # Update the simulation
        position = np.array([-.5, -.2, 1.0])
        quat = euler_to_quaternion([roll, pitch, yaw])
        
        face_centers = calculate_face_centers(position, quat, np.array([.25, .25, .25]))
        for i, center in enumerate(face_centers):
            face_markers[i].set_world_pose(position=center)

        # Choose two face markers as targets (e.g., face 0 for left arm, face 5 for right arm)
        left_target = face_centers[0]
        #load left target on cuda tensor
        left_target = torch.tensor(left_target, device="cuda:0", dtype=torch.float32)
        right_target = face_centers[5]
        #load right target on cuda tensor
        right_target = torch.tensor(right_target, device="cuda:0", dtype=torch.float32)
        
        quat = torch.tensor(np.array([1, 0, 0, 0]), device="cuda:0", dtype=torch.float32)

        sim_js = robot.get_joints_state()
        sim_js_names = robot.dof_names

        cu_js = JointState(
            position=tensor_args.to_device(sim_js.positions),
            velocity=tensor_args.to_device(sim_js.velocities) * 0.0,
            acceleration=tensor_args.to_device(sim_js.velocities) * 0.0,
            jerk=tensor_args.to_device(sim_js.velocities) * 0.0,
            joint_names=sim_js_names,
        )

        cu_js = cu_js.get_ordered_joint_state(motion_gen.kinematics.joint_names)


        # compute curobo solution:
        ik_goal = Pose(
            position=tensor_args.to_device(ee_translation_goal),
            quaternion=tensor_args.to_device(ee_orientation_teleop_goal),
        )
        # add link poses:
        link_poses = {}
        for i in target_links.keys():
            c_p, c_rot = target_links[i].get_world_pose()
            link_poses[i] = Pose(
                position=tensor_args.to_device(c_p),
                quaternion=tensor_args.to_device(c_rot),
            )
        result = motion_gen.plan_single(
            cu_js.unsqueeze(0), ik_goal, plan_config.clone(), link_poses=link_poses
        )

        # Generate motion for left arm
        left_result = motion_gen_left.plan_single(
            initial_joint_positions,
            Pose(position=left_target, quaternion=quat),
        )

        # Generate motion for right arm
        right_result = motion_gen_right.plan_single(
            initial_joint_positions,
            Pose(position=right_target, quaternion=quat),

        )

        # Check if both plans were successful
        if left_result.success and right_result.success:
            # Interpolate and apply joint positions for both arms
            left_traj = left_result.get_interpolated_plan()
            right_traj = right_result.get_interpolated_plan()
            
            for i in range(len(left_traj)):
                # Combine left and right arm positions
                combined_positions = np.concatenate([left_traj[i], right_traj[i]])
                
                # Apply the joint positions to the robot
                articulation_controller.apply_action(combined_positions)
                
                # Step the simulation
                my_world.step(render=True)
                kit.update()

        # Update initial joint positions for the next iteration
        initial_joint_positions = robot.get_joints_positions()

        # Step the world and render
        my_world.step(render=True)
        kit.update()

    # Cleanup
    kit.close()


if __name__ == "__main__":
    main()

However I’m unable to make the robot move with cuRoBo. which may not be directly linked to isaacsim.