Isaac v4.0.0 Container cannot load our YAML scenarios from v2023.1.1

Dear Team,

We are testing 4.0.0 in order to switch to using it, as it solves many of the problems we have been previously reported. Thank you and the team for such amazing work.

That said, unfortunately, our YAML scenarios that are fine on 2023.1.1 do not even load on 4.0.0. Both the Isaac Sim Window App and Container (on Ubuntu) are stuck while parsing the YAML scenarios. It doesn’t even reach loading any of the assets. It seems like the randomizer graph is blowing up to infinity as the CPU is going to 100% on a single core for the kit process. We have tried waiting 24 hours to see if it will load but it won’t. It seems like adding a certain amount of randomizer actions inside the on_frame causes the issue.

Below you can find:

  • reproduction scenario,
  • reproduction command,
  • log

In the scenario, removing uniformm0001_instance4, uniformm0001_instance3, uniformm0001_instance2 “fixes” the problem. Basically there seems to be a limit to how many groups one can have. In the previous versions of Isaac that wasn’t the problem as we have 100s of code blocks like that. We need this sorted out quick.

Reproduction Scenario

# Default scene settings. This script default is 1 (IsaacSim), so if you change
# the meters per unit or up axis, you need to alter the coordinates and rotations.

# IsaacSim default is 1, Code is 0.01
stage_unit_setting:
  settings.set_stage_meters_per_unit:
    meters_per_unit: 1

# IsaacSim default is "Z"", Code is "Y"
stage_up_axis_setting:
  settings.set_stage_up_axis:
    up_axis: Z

# Enable Path Tracing
path_tracing:
  settings.set_render_pathtraced:
    samples_per_pixel: 32

# Load the stage
load_stage:
  create.from_usd:
    usd: http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/Isaac/Environments/Simple_Warehouse/full_warehouse.usd

# Select the objects in the stage
forklift:
  get.prim_at_path:
    path: /Replicator/Ref_Xform/Ref/forklift

floor:
  get.prims:
    semantics: [[class, floor]]

racks_and_walls:
  get.prims:
    semantics: [[class, rack], [class, wall]]

default_light:
  get.prim_at_path:
    path: /Environment/defaultLight

light:
  create.group:
    items:
      - /Replicator/Ref_Xform/Ref/RectLight
      - /Replicator/Ref_Xform/Ref/RectLight_01
      - /Replicator/Ref_Xform/Ref/RectLight_02
      - /Replicator/Ref_Xform/Ref/RectLight_03
      - /Replicator/Ref_Xform/Ref/RectLight_04
      - /Replicator/Ref_Xform/Ref/RectLight_05
      - /Replicator/Ref_Xform/Ref/RectLight_06

camera:
  create.camera:
    position: [0, 0, 0]
    rotation: [0, 0, 0]

# Create output
render_product:
  create.render_product:
    camera: camera
    resolution: [1280, 720]

writer:
  writers.get:
    name: BasicWriter
    init_params:
      rgb: True
      bounding_box_2d_tight: True
      semantic_filter_predicate: class:sign|cone|person
      output_dir: /workspaces/Surveily.Datasets.Generator.Omniverse/.out/Issues/stuck
      #output_dir: /home/wojtek/stuck

writer_attach:
  writer.attach:
    render_products: render_product

# Load businessf0002 animations
businessf0002_load_animations:
  create.from_dir:
    recursive: True
    semantics: [[animation, businessf0002]]
    dir_path: http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/ActorCore/Business_F_0002/Motion/

businessf0002_animations:
  get.prims:
    semantics: [[animation, businessf0002]]

# Load partym0001 animations
partym0001_load_animations:
  create.from_dir:
    recursive: True
    semantics: [[animation, partym0001]]
    dir_path: http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/ActorCore/Party_M_0001/Motion/

partym0001_animations:
  get.prims:
    semantics: [[animation, partym0001]]

# Load uniformf0001 animations
uniformf0001_load_animations:
  create.from_dir:
    recursive: True
    semantics: [[animation, uniformf0001]]
    dir_path: http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/ActorCore/Uniform_F_0001/Motion/

uniformf0001_animations:
  get.prims:
    semantics: [[animation, uniformf0001]]

# Load uniformm0001 animations
uniformm0001_load_animations:
  create.from_dir:
    recursive: True
    semantics: [[animation, uniformm0001]]
    dir_path: http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/ActorCore/Uniform_M_0001/Motion/

uniformm0001_animations:
  get.prims:
    semantics: [[animation, uniformm0001]]

# Load worker animations
worker_load_animations:
  create.from_dir:
    recursive: True
    semantics: [[animation, worker]]
    dir_path: http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/Worker/Motions/

worker_animations:
  get.skelanimation:
    semantics: [[animation, worker]]

# Set the trigger as on_frame
trigger:
  trigger.on_frame:
    max_execs: 20
    rt_subframes: 5

# When the trigger executes, run the following logic
with_trigger:
  with.trigger:
    # Disable default light
    with.default_light:
      modify.visibility:
        value: False

    # Set sun intensity
    sun_intensity:
      with.light:
        modify.attribute:
          name: inputs:intensity
          value:
            distribution.uniform:
              lower: 500
              upper: 5000

    # Set sun temperature
    sun_temperature_on:
      with.light:
        modify.attribute:
          name: inputs:enableColorTemperature
          value: True

    # Set sun temperature
    sun_temperature:
      with.light:
        modify.attribute:
          name: inputs:colorTemperature
          value:
            distribution.uniform:
              lower: 4000
              upper: 7000

    # move the camera around and rotate
    with.camera:
      modify.pose:
        position:
          distribution.uniform:
            lower: [-23.6, -3, 2.5]
            upper: [3.03, 3, 7.5]
        rotation:
          distribution.uniform:
            lower: [0, -30, -120]
            upper: [0, -10, -60]

    # Create and scatter businessf0002
    businessf0002_instance:
      randomizer.instantiate:
        paths:
          - http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/ActorCore/Business_F_0002/Actor/business-f-0002/business-f-0002.usd
        size: 5
        mode: reference

    with.businessf0002_instance:
      modify.pose:
        scale: 0.01
        rotation:
          distribution.uniform:
            lower: [0, 0, -180]
            upper: [0, 0, 180]
      modify.semantics:
        semantics: [[class, person]]
      modify.animation:
        values:
          distribution.sequence:
            items: businessf0002_animations
      randomizer.scatter_2d:
        seed: 1
        surface_prims: floor
        check_for_collisions: True

    # Create and scatter partym0001
    partym0001_instance:
      randomizer.instantiate:
        paths:
          - http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/ActorCore/Party_M_0001/Actor/party-m-0001/party-m-0001.usd
        size: 5
        mode: reference

    with.partym0001_instance:
      modify.pose:
        scale: 0.01
        rotation:
          distribution.uniform:
            lower: [0, 0, -180]
            upper: [0, 0, 180]
      modify.semantics:
        semantics: [[class, person]]
      modify.animation:
        values:
          distribution.sequence:
            items: partym0001_animations
      randomizer.scatter_2d:
        seed: 2
        surface_prims: floor
        check_for_collisions: True

    # Create and scatter uniformf0001
    uniformf0001_instance:
      randomizer.instantiate:
        paths:
          - http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/ActorCore/Uniform_F_0001/Actor/uniform_f_0001/uniform_f_0001.usd
        size: 5
        mode: reference

    with.uniformf0001_instance:
      modify.pose:
        scale: 0.01
        rotation:
          distribution.uniform:
            lower: [0, 0, -180]
            upper: [0, 0, 180]
      modify.semantics:
        semantics: [[class, person]]
      modify.animation:
        values:
          distribution.sequence:
            items: uniformf0001_animations
      randomizer.scatter_2d:
        seed: 3
        surface_prims: floor
        check_for_collisions: True

    # Create and scatter uniformm0001
    uniformm0001_instance:
      randomizer.instantiate:
        paths:
          - http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/ActorCore/Uniform_M_0001/Actor/uniform_m_0001/uniform_m_0001.usd
        size: 5
        mode: reference

    with.uniformm0001_instance:
      modify.pose:
        scale: 0.01
        rotation:
          distribution.uniform:
            lower: [0, 0, -180]
            upper: [0, 0, 180]
      modify.semantics:
        semantics: [[class, person]]
      modify.animation:
        values:
          distribution.sequence:
            items: uniformm0001_animations
      randomizer.scatter_2d:
        seed: 4
        surface_prims: floor
        check_for_collisions: True

    # Create and scatter uniformm00012
    uniformm0001_instance2:
      randomizer.instantiate:
        paths:
          - http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/ActorCore/Uniform_M_0001/Actor/uniform_m_0001/uniform_m_0001.usd
        size: 5
        mode: reference

    with.uniformm0001_instance2:
      modify.pose:
        scale: 0.01
        rotation:
          distribution.uniform:
            lower: [0, 0, -180]
            upper: [0, 0, 180]
      modify.semantics:
        semantics: [[class, person]]
      modify.animation:
        values:
          distribution.sequence:
            items: uniformm0001_animations
      randomizer.scatter_2d:
        seed: 4
        surface_prims: floor
        check_for_collisions: True

    # Create and scatter uniformm00013
    uniformm0001_instance3:
      randomizer.instantiate:
        paths:
          - http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/ActorCore/Uniform_M_0001/Actor/uniform_m_0001/uniform_m_0001.usd
        size: 5
        mode: reference

    with.uniformm0001_instance3:
      modify.pose:
        scale: 0.01
        rotation:
          distribution.uniform:
            lower: [0, 0, -180]
            upper: [0, 0, 180]
      modify.semantics:
        semantics: [[class, person]]
      modify.animation:
        values:
          distribution.sequence:
            items: uniformm0001_animations
      randomizer.scatter_2d:
        seed: 4
        surface_prims: floor
        check_for_collisions: True

    # Create and scatter uniformm00014
    uniformm0001_instance4:
      randomizer.instantiate:
        paths:
          - http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/ActorCore/Uniform_M_0001/Actor/uniform_m_0001/uniform_m_0001.usd
        size: 5
        mode: reference

    with.uniformm0001_instance4:
      modify.pose:
        scale: 0.01
        rotation:
          distribution.uniform:
            lower: [0, 0, -180]
            upper: [0, 0, 180]
      modify.semantics:
        semantics: [[class, person]]
      modify.animation:
        values:
          distribution.sequence:
            items: uniformm0001_animations
      randomizer.scatter_2d:
        seed: 4
        surface_prims: floor
        check_for_collisions: True

    # Create and scatter worker
    worker_instance:
      randomizer.instantiate:
        paths:
          - http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/Worker/Props/Worker.usd
        size: 5
        mode: reference

    with.worker_instance:
      modify.pose:
        scale: 0.01
        rotation:
          distribution.uniform:
            lower: [0, 0, -180]
            upper: [0, 0, 180]
      modify.semantics:
        semantics: [[class, person]]
      # modify.animation:
      #   values:
      #     distribution.sequence:
      #       items: worker_animations
      randomizer.scatter_2d:
        seed: 5
        surface_prims: floor
        check_for_collisions: True

    # create and scatter cones
    cone_instance:
      randomizer.instantiate:
        paths:
          - http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/Isaac/Environments/Simple_Warehouse/Props/S_TrafficCone.usd
        size: 20
        mode: scene_instance

    with.cone_instance:
      modify.pose:
        rotation:
          distribution.uniform:
            lower: [0, 0, -180]
            upper: [0, 0, 180]
      randomizer.scatter_2d:
        seed: 101
        surface_prims: floor
        no_coll_prims: racks_and_walls
        check_for_collisions: True

    # create and scatter wet floor signs
    wet_floor_sign_instance:
      randomizer.instantiate:
        paths:
          - http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/Isaac/Environments/Simple_Warehouse/Props/S_WetFloorSign.usd
        size: 20
        mode: scene_instance

    with.wet_floor_sign_instance:
      modify.pose:
        rotation:
          distribution.uniform:
            lower: [0, 0, -180]
            upper: [0, 0, 180]
      randomizer.scatter_2d:
        seed: 102
        surface_prims: floor
        no_coll_prims: racks_and_walls
        check_for_collisions: True

    # move the forklift around and rotate
    with.forklift:
      modify.pose:
        rotation:
          distribution.uniform:
            lower: [0, 0, -180]
            upper: [0, 0, 180]
      # randomizer.scatter_2d:
      #   seed: 0
      #   surface_prims: floor
      #   no_coll_prims: racks_and_walls
      #   check_for_collisions: True

    # Randomize floor
    with.floor:
      modify.material:
        value:
          - /Replicator/Ref_Xform/Ref/SM_floor27/Looks/MI_Floor_02b
          - /Replicator/Ref_Xform/Ref/SM_WallA_3M42/SM_WallA_6M/Looks/MI_WallA_01

Reproduction Command

#!/bin/bash

set -e

FILE=$1

echo $FILE

if [ -f "$FILE" ]; then
    /isaac-sim/runheadless.native.sh  --no-window \
                                      --/app/viewport/grid/enabled=false \
                                      --/app/viewport/defaults/guide/grid/visible=false \
                                      --/omni/replicator/replicatorYaml/yamlPath=$FILE
    exit $?
else
    echo "File not found: $FILE"
    exit 1
fi

Log

The NVIDIA Omniverse License Agreement (EULA) must be accepted before
Omniverse Kit can start. The license terms for this product can be viewed at
https://docs.omniverse.nvidia.com/app_isaacsim/common/NVIDIA_Omniverse_License_Agreement.html

Omniverse Software collects installation and configuration details about your software, hardware, and network
configuration (e.g., version of operating system, applications installed, type of hardware, network speed, IP
address) based on our legitimate interest in improving your experience. To improve performance, troubleshooting
and diagnostic purposes of our software, we also collect session behavior, error and crash logs.

Data Collection in container mode is completely anonymous unless specified. You may opt-out of this collection
anytime by not setting the OMNI_ENV_PRIVACY_CONSENT environment variable.

To opt-in set the OMNI_ENV_PRIVACY_CONSENT environment variable when running the container. Set the 
OMNI_ENV_PRIVACY_USERID environment variable tag the telemetry data with a user ID.
Loading user config located at: '/root/.local/share/ov/data/Kit/Isaac-Sim/4.0/user.config.json'
[Info] [carb] Logging to file: /root/.nvidia-omniverse/logs/Kit/Isaac-Sim/4.0/kit_20240704_133835.log
2024-07-04 13:38:35 [1ms] [Warning] [carb.crashreporter-breakpad.plugin] Failed to parse toml file '/root/.local/share/ov/data/Kit/Isaac-Sim/4.0/b1fb1dda-d815-49dd-c713c5b9-b7a01f5c.dmp.toml': /root/.local/share/ov/data/Kit/Isaac-Sim/4.0/b1fb1dda-d815-49dd-c713c5b9-b7a01f5c.dmp.toml could not be opened for parsing
2024-07-04 13:38:35 [1ms] [Warning] [carb.crashreporter-breakpad.plugin] [previous crash] preventing upload of minidump due to user opt-out: '/root/.local/share/ov/data/Kit/Isaac-Sim/4.0/b1fb1dda-d815-49dd-c713c5b9-b7a01f5c.dmp'
2024-07-04 13:38:35 [1ms] [Warning] [carb.crashreporter-breakpad.plugin] [previous crash] preventing upload of minidump due to user opt-out: '/root/.local/share/ov/data/Kit/Isaac-Sim/4.0/566d147c-a3ef-4b9c-1d8b74b4-f4e53ce6.dmp.zip'
2024-07-04 13:38:35 [2ms] [Warning] [carb.crashreporter-breakpad.plugin] [previous crash] preventing upload of minidump due to user opt-out: '/root/.local/share/ov/data/Kit/Isaac-Sim/4.0/32e7b4f7-856e-4ac8-d44f8a9a-cb31077e.dmp.zip'
2024-07-04 13:38:35 [2ms] [Warning] [carb.crashreporter-breakpad.plugin] [previous crash] preventing upload of minidump due to user opt-out: '/root/.local/share/ov/data/Kit/Isaac-Sim/4.0/c978f5e5-9abd-45aa-763052b7-b485711a.dmp.zip'
2024-07-04 13:38:35 [2ms] [Warning] [carb.crashreporter-breakpad.plugin] [previous crash] preventing upload of minidump due to user opt-out: '/root/.local/share/ov/data/Kit/Isaac-Sim/4.0/6bcaf074-cf4f-4d55-82818798-a7c72cde.dmp.zip'
2024-07-04 13:38:35 [3ms] [Warning] [carb.crashreporter-breakpad.plugin] [previous crash] preventing upload of minidump due to user opt-out: '/root/.local/share/ov/data/Kit/Isaac-Sim/4.0/e268bb75-ddfd-410b-26f1bb80-f0d713ac.dmp.zip'
2024-07-04 13:38:35 [3ms] [Warning] [carb.crashreporter-breakpad.plugin] [previous crash] preventing upload of minidump due to user opt-out: '/root/.local/share/ov/data/Kit/Isaac-Sim/4.0/90449153-9f7f-41fb-56f54c84-6b9fefaf.dmp.zip'
[0.196s] [ext: omni.kit.async_engine-0.0.0] startup
[0.804s] [ext: omni.stats-0.0.0] startup
[0.805s] [ext: omni.client-1.1.0] startup
[0.828s] [ext: omni.datastore-0.0.0] startup
[0.828s] [ext: omni.blobkey-1.1.0] startup
[0.828s] [ext: omni.ujitso.default-1.0.0] startup
[0.829s] [ext: omni.hsscclient-0.0.0] startup
[0.829s] [ext: omni.rtx.shadercache.vulkan-1.0.0] startup
[0.830s] [ext: omni.assets.plugins-0.0.0] startup
[0.831s] [ext: omni.gpu_foundation-0.0.0] startup
[0.841s] [ext: carb.windowing.plugins-1.0.0] startup
[0.870s] [ext: omni.kit.renderer.init-0.0.0] startup

|---------------------------------------------------------------------------------------------|
| Driver Version: 550.90.07     | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA GeForce RTX 3060 Laptop.. | Yes: 0 |     | 6144    MB | 10de      | 0          |
|     |                                  |        |     |            | 2520      | 1ddb812e.. |
|     |                                  |        |     |            | 1         |            |
|=============================================================================================|
| OS: 22.04.3 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.3, Kernel: 6.8.0-36-generic
| XServer Vendor: The X.Org Foundation, XServer Version: 12101011 (1.21.1.11)
| Processor: AMD Ryzen 7 6800HS with Radeon Graphics | Cores: 8 | Logical: 16
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 39434 | Free Memory: 31546
| Total Page/Swap (MB): 8191 | Free Page/Swap: 8191
|---------------------------------------------------------------------------------------------|
2024-07-04 13:38:37 [1,219ms] [Warning] [gpu.foundation.plugin] IOMMU is enabled.
[1.409s] [ext: omni.kit.pipapi-0.0.0] startup
[1.411s] [ext: omni.kit.pip_archive-0.0.0] startup
[1.411s] [ext: omni.pip.compute-1.3.1] startup
[1.411s] [ext: omni.mtlx-0.1.0] startup
[1.412s] [ext: omni.usd.config-1.0.3] startup
[1.416s] [ext: omni.gpucompute.plugins-0.0.0] startup
[1.416s] [ext: omni.usd.libs-1.0.0] startup
[1.485s] [ext: omni.pip.cloud-1.1.3] startup
[1.488s] [ext: omni.isaac.core_archive-2.3.0] startup
[1.488s] [ext: omni.kit.telemetry-0.5.0] startup
[1.516s] [ext: omni.kit.loop-isaac-1.2.0] startup
[1.517s] [ext: omni.kit.test-0.0.0] startup
[1.571s] [ext: omni.isaac.ml_archive-2.0.1] startup
[1.571s] [ext: omni.appwindow-1.1.8] startup
[1.573s] [ext: omni.kit.renderer.core-1.0.1] startup
[1.584s] [ext: omni.kit.renderer.capture-0.0.0] startup
[1.586s] [ext: omni.kit.renderer.imgui-1.0.1] startup
[1.660s] [ext: omni.ui-2.23.11] startup
[1.670s] [ext: omni.kit.mainwindow-1.0.3] startup
[1.671s] [ext: carb.audio-0.1.0] startup
[1.675s] [ext: omni.uiaudio-1.0.0] startup
[1.676s] [ext: omni.kit.uiapp-0.0.0] startup
[1.676s] [ext: omni.usd.schema.metrics.assembler-106.0.1] startup
[1.780s] [ext: omni.usd.schema.omnigraph-1.0.0] startup
[1.786s] [ext: omni.usd.schema.geospatial-0.0.0] startup
[1.790s] [ext: omni.usd.schema.semantics-0.0.0] startup
[1.807s] [ext: omni.usd.schema.anim-0.0.0] startup
[1.823s] [ext: omni.usd.schema.audio-0.0.0] startup
[1.826s] [ext: omni.anim.navigation.schema-106.0.2] startup
[1.829s] [ext: omni.usd.schema.isaac-2.0.2] startup
[1.835s] [ext: omni.usd.schema.physx-106.0.16] startup
[1.878s] [ext: omni.anim.graph.schema-106.0.2] startup
[1.884s] [ext: omni.usd.schema.scene.visualization-2.0.2] startup
[1.886s] [ext: omni.usd.schema.omniscripting-1.0.0] startup
[1.890s] [ext: omni.graph.exec-0.9.3] startup
[1.891s] [ext: omni.kit.actions.core-1.0.0] startup
[1.892s] [ext: omni.kit.usd_undo-0.1.8] startup
[1.892s] [ext: omni.kit.exec.core-0.13.0] startup
[1.894s] [ext: omni.kit.commands-1.4.9] startup
[1.898s] [ext: omni.usd_resolver-1.0.0] startup
[1.902s] [ext: omni.resourcemonitor-105.0.1] startup
[1.904s] [ext: omni.kit.window.popup_dialog-2.0.24] startup
[1.907s] [ext: omni.activity.core-1.0.1] startup
[1.909s] [ext: omni.usd.core-1.2.11] startup
[1.912s] [ext: omni.timeline-1.0.10] startup
[1.914s] [ext: omni.kit.widget.nucleus_connector-1.1.5] startup
[1.916s] [ext: usdrt.scenegraph-7.4.8] startup
[1.963s] [ext: omni.kit.audiodeviceenum-1.0.1] startup
[1.965s] [ext: omni.hydra.usdrt_delegate-7.4.7] startup
[1.981s] [ext: omni.hydra.scene_delegate-0.3.3] startup
[1.986s] [ext: omni.iray.libs-0.0.0] startup
[1.991s] [ext: omni.usd-1.10.39] startup
[2.025s] [ext: omni.mdl.neuraylib-0.2.4] startup
[2.026s] [ext: omni.kit.asset_converter-2.1.10] startup
[2.145s] [ext: omni.usd.schema.forcefield-106.0.16] startup
[2.150s] [ext: omni.kit.collaboration.telemetry-1.0.0] startup
[2.151s] [ext: omni.kit.collaboration.channel_manager-1.0.11] startup
[2.152s] [ext: omni.kit.widget.path_field-2.0.9] startup
[2.153s] [ext: omni.kit.usd.layers-2.1.30] startup
[2.161s] [ext: omni.kit.clipboard-1.0.3] startup
[2.162s] [ext: omni.kit.widget.browser_bar-2.0.9] startup
[2.163s] [ext: omni.kit.notification_manager-1.0.8] startup
[2.165s] [ext: omni.kit.collaboration.presence_layer-1.0.8] startup
[2.166s] [ext: omni.kit.helper.file_utils-0.1.7] startup
[2.168s] [ext: omni.kit.widget.nucleus_info-1.0.2] startup
[2.169s] [ext: omni.kit.widget.options_menu-1.1.4] startup
[2.171s] [ext: omni.kit.widget.filebrowser-2.3.47] startup
[2.176s] [ext: omni.kit.search_core-1.0.5] startup
[2.176s] [ext: omni.kit.widget.options_button-1.0.2] startup
[2.177s] [ext: omni.kit.widget.versioning-1.4.6] startup
[2.179s] [ext: omni.kit.widget.search_delegate-1.0.4] startup
[2.180s] [ext: omni.kit.widget.context_menu-1.2.1] startup
[2.180s] [ext: omni.convexdecomposition-106.0.16] startup
[2.184s] [ext: omni.ui.scene-1.9.3] startup
[2.189s] [ext: omni.kit.menu.utils-1.5.23] startup
[2.200s] [ext: omni.kit.window.filepicker-2.10.32] startup
[2.210s] [ext: omni.kvdb-106.0.16] startup
[2.212s] [ext: omni.physx.foundation-106.0.16] startup
[2.212s] [ext: omni.kit.widget.searchable_combobox-1.0.5] startup
[2.213s] [ext: omni.usdphysics-106.0.16] startup
[2.215s] [ext: omni.localcache-106.0.16] startup
[2.216s] [ext: omni.kit.widget.prompt-1.0.7] startup
[2.217s] [ext: omni.kit.widget.settings-1.1.0] startup
[2.219s] [ext: omni.physx.cooking-106.0.16] startup
[2.230s] [ext: omni.kit.context_menu-1.8.0] startup
[2.231s] [ext: omni.kit.window.preferences-1.5.1] startup
[2.234s] [ext: omni.kit.widget.live_session_management.ui-1.0.1] startup
[2.237s] [ext: omni.kit.widget.filter-1.1.3] startup
[2.237s] [ext: omni.inspect-1.0.1] startup
[2.239s] [ext: omni.kit.window.file_exporter-1.0.29] startup
[2.242s] [ext: omni.kit.hotkeys.core-1.3.3] startup
[2.242s] [ext: omni.kit.widget.graph-1.12.6] startup
[2.247s] [ext: omni.graph.core-2.170.0] startup
[2.250s] [ext: omni.kit.widget.stage-2.10.26] startup
[2.260s] [ext: omni.kit.widget.live_session_management-1.2.18] startup
[2.262s] [ext: omni.graph.tools-1.77.0] startup
[2.282s] [ext: omni.kit.window.extensions-1.4.7] startup
[2.287s] [ext: omni.ujitso.processor.texture-1.0.0] startup
[2.288s] [ext: omni.kit.window.file_importer-1.1.11] startup
[2.289s] [ext: omni.kit.primitive.mesh-1.0.16] startup
[2.292s] [ext: omni.volume-0.5.0] startup
[2.294s] [ext: omni.ujitso.client-0.0.0] startup
[2.295s] [ext: omni.mdl-52.0.1] startup
[2.313s] [ext: omni.kit.stage_template.core-1.1.20] startup
[2.314s] [ext: omni.kit.widget.highlight_label-1.0.1] startup
[2.314s] [ext: omni.kit.stage_templates-1.2.2] startup
[2.316s] [ext: omni.hydra.rtx-0.2.0] startup
[2.328s] [ext: omni.kit.material.library-1.4.0] startup
[2.333s] [ext: omni.hydra.engine.stats-1.0.1] startup
[2.336s] [ext: omni.kit.widget.searchfield-1.1.5] startup
[2.337s] [ext: omni.kit.window.file-1.3.52] startup
[2.339s] [ext: omni.kit.hydra_texture-1.2.6] startup
[2.344s] [ext: omni.kit.window.drop_support-1.0.1] startup
[2.344s] [ext: omni.kit.widget.viewport-106.0.2] startup
[2.348s] [ext: omni.kit.viewport.registry-104.0.6] startup
[2.348s] [ext: omni.kit.raycast.query-1.0.5] startup
[2.354s] [ext: omni.kit.window.content_browser_registry-0.0.4] startup
[2.355s] [ext: omni.kit.viewport.window-106.0.8] startup
[2.371s] [ext: omni.kit.window.property-1.11.1] startup
[2.373s] [ext: omni.kit.window.content_browser-2.9.13] startup
[2.386s] [ext: omni.kit.widget.text_editor-1.0.2] startup
[2.387s] [ext: omni.kit.viewport.utility-1.0.17] startup
[2.388s] [ext: omni.kit.property.usd-3.21.26] startup
[2.396s] [ext: omni.graph-1.135.0] startup
[2.502s] [ext: omni.physx-106.0.16] startup
[2.513s] [ext: omni.graph.ui-1.67.1] startup
[2.534s] [ext: omni.graph.action_core-1.1.4] startup
[2.541s] [ext: omni.graph.image.core-0.3.2] startup
[2.545s] [ext: omni.graph.action_nodes-1.21.3] startup
[2.553s] [ext: omni.graph.image.nodes-1.0.2] startup
[2.555s] [ext: omni.kit.numpy.common-0.1.2] startup
[2.557s] [ext: omni.graph.nodes-1.141.2] startup
[2.568s] [ext: omni.physics.tensors-106.0.16] startup
[2.579s] [ext: omni.physx.stageupdate-106.0.16] startup
[2.583s] [ext: omni.graph.action-1.101.1] startup
[2.584s] [ext: omni.warp.core-1.1.0] startup
Warp 1.1.0 initialized:
   CUDA Toolkit 11.5, Driver 12.4
   Devices:
     "cpu"      : "x86_64"
     "cuda:0"   : "NVIDIA GeForce RTX 3060 Laptop GPU" (6 GiB, sm_86, mempool enabled)
   Kernel cache:
     /root/.cache/warp/1.1.0
[2.701s] [ext: omni.isaac.dynamic_control-1.3.7] startup
[2.722s] [ext: omni.warp-1.1.0] startup
[2.758s] [ext: omni.kit.viewport.actions-105.2.8] startup
[2.777s] [ext: omni.kit.widget.imageview-1.0.3] startup
[2.780s] [ext: omni.debugdraw-0.1.3] startup
[2.785s] [ext: omni.physx.tensors-106.0.16] startup
[2.789s] [ext: omni.kit.window.cursor-1.1.2] startup
[2.791s] [ext: omni.isaac.version-1.1.0] startup
[2.792s] [ext: omni.kit.widget.toolbar-1.6.2] startup
[2.837s] [ext: omni.kit.manipulator.transform-104.7.5] startup
[2.857s] [ext: omni.kit.viewport.menubar.core-107.0.0] startup
[3.047s] [ext: omni.isaac.nucleus-0.1.2] startup
[3.049s] [ext: omni.kit.manipulator.tool.snap-1.4.5] startup
[3.054s] [ext: omni.kit.viewport.menubar.display-106.0.0] startup
[3.057s] [ext: omni.kit.window.quicksearch-2.4.4] startup
[3.062s] [ext: omni.physx.commands-106.0.16] startup
[3.067s] [ext: omni.usdphysics.ui-106.0.16] startup
[3.089s] [ext: omni.isaac.core-3.12.0] startup
[4.126s] [ext: omni.syntheticdata-0.6.7] startup
[4.152s] [ext: omni.kit.property.material-1.9.3] startup
[4.161s] [ext: omni.kit.manipulator.selector-1.1.1] startup
[4.164s] [ext: omni.physx.ui-106.0.16] startup
[4.199s] [ext: omni.fabric.commands-1.1.4] startup
[4.204s] [ext: omni.kit.manipulator.viewport-107.0.0] startup
[4.207s] [ext: omni.kit.viewport.manipulator.transform-106.0.0] startup
ViewportTransformManipulatorExt on_startup
[4.209s] [ext: omni.kit.property.physx-106.0.16] startup
[4.281s] [ext: omni.physx.demos-106.0.16] startup
[4.304s] [ext: omni.ui_query-1.1.2] startup
[4.308s] [ext: omni.kit.manipulator.prim.core-106.0.2] startup
[4.315s] [ext: omni.graph.visualization.nodes-2.1.1] startup
[4.327s] [ext: omni.physx.vehicle-106.0.16] startup
[4.356s] [ext: omni.kit.ui_test-1.2.18] startup
[4.360s] [ext: omni.kit.manipulator.prim.fabric-106.0.1] startup
[4.362s] [ext: omni.kit.manipulator.prim.usd-106.0.1] startup
[4.364s] [ext: omni.physx.camera-106.0.16] startup
[4.376s] [ext: omni.physx.cct-106.0.16] startup
[4.387s] [ext: omni.kit.manipulator.selection-104.0.9] startup
[4.395s] [ext: omni.kit.manipulator.prim-106.0.0] startup
[4.396s] [ext: omni.kit.window.toolbar-1.6.1] startup
[4.403s] [ext: omni.kit.widget.layers-1.7.9] startup
[4.423s] [ext: omni.kit.graph.delegate.default-1.2.2] startup
[4.426s] [ext: omni.physx.graph-106.0.16] startup
[4.441s] [ext: omni.graph.ui_nodes-1.24.1] startup
[4.447s] [ext: omni.physx.supportui-106.0.16] startup
[4.474s] [ext: omni.kit.graph.usd.commands-1.3.1] startup
[4.478s] [ext: omni.kit.graph.editor.core-1.5.3] startup
[4.483s] [ext: omni.physx.telemetry-106.0.16] startup
[4.490s] [ext: omni.graph.bundle.action-2.0.4] startup
[4.490s] [ext: omni.command.usd-1.0.3] startup
[4.491s] [ext: omni.kit.widget.material_preview-1.0.16] startup
[4.495s] [ext: omni.isaac.debug_draw-0.7.1] startup
[4.502s] [ext: omni.kit.window.material_graph-1.8.15] startup
[4.553s] [ext: omni.sensors.tiled-0.0.3] startup
[4.556s] [ext: omni.physx.bundle-106.0.16] startup
[4.556s] [ext: omni.graph.scriptnode-1.18.2] startup
[4.558s] [ext: omni.isaac.ui-0.15.1] startup
[4.573s] [ext: omni.replicator.core-1.11.8] startup
2024-07-04 13:38:40 [4,582ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template
[4.696s] [ext: omni.sensors.nv.common-1.0.1] startup
[4.708s] [ext: omni.isaac.range_sensor-3.0.1] startup
[4.745s] [ext: omni.isaac.core_nodes-1.15.4] startup
[4.770s] [ext: omni.sensors.nv.wpm-1.0.0] startup
[4.773s] [ext: omni.sensors.nv.materials-1.0.0] startup
[4.776s] [ext: omni.isaac.lula-3.0.1] startup
[4.787s] [ext: omni.isaac.surface_gripper-1.0.1] startup
[4.796s] [ext: omni.isaac.motion_generation-7.0.1] startup
[4.839s] [ext: omni.isaac.manipulators-2.1.0] startup
[4.851s] [ext: omni.sensors.nv.radar-1.0.1] startup
[4.859s] [ext: omni.sensors.nv.lidar-1.0.1] startup
[4.865s] [ext: omni.isaac.franka-0.4.1] startup
[4.869s] [ext: omni.isaac.dofbot-0.3.2] startup
[4.881s] [ext: omni.isaac.sensor-11.3.0] startup
[4.989s] [ext: omni.isaac.cortex-0.3.8] startup
[4.992s] [ext: omni.isaac.universal_robots-0.3.5] startup
[4.999s] [ext: omni.isaac.quadruped-1.4.5] startup
[5.040s] [ext: omni.isaac.cortex.sample_behaviors-1.0.5] startup
[5.044s] [ext: omni.kit.graph.widget.variables-2.1.0] startup
[5.055s] [ext: omni.kit.graph.delegate.modern-1.10.6] startup
[5.058s] [ext: omni.isaac.wheeled_robots-2.1.3] startup
[5.061s] [ext: omni.graph.window.core-1.107.1] startup
[5.080s] [ext: omni.isaac.examples-2.0.0] startup
[5.213s] [ext: omni.importer.mjcf-1.1.0] startup
[5.246s] [ext: omni.graph.window.generic-1.23.1] startup
[5.259s] [ext: omni.importer.onshape-0.7.1] startup
[5.270s] [ext: omni.isaac.manipulators.ui-1.2.1] startup
[5.273s] [ext: omni.importer.urdf-1.14.1] startup
[5.324s] [ext: omni.kit.streamsdk.plugins-3.2.1] startup
[5.324s] [ext: omni.isaac.robot_description_editor-2.4.0] startup
[5.347s] [ext: omni.isaac.examples_nodes-0.1.3] startup
[5.349s] [ext: omni.isaac.cloner-0.8.1] startup
[5.355s] [ext: omni.isaac.extension_templates-1.7.0] startup
[5.359s] [ext: omni.isaac.gain_tuner-1.1.1] startup
[5.368s] [ext: omni.isaac.gym-0.11.3] startup
[5.371s] [ext: omni.isaac.import_wizard-0.5.2] startup
[5.378s] [ext: omni.isaac.kit-1.10.0] startup
[5.387s] [ext: omni.kit.mesh.raycast-105.4.0] startup
[5.393s] [ext: omni.anim.curve.core-1.1.13] startup
[5.409s] [ext: omni.isaac.block_world-1.0.0] startup
[5.413s] [ext: omni.anim.shared.core-106.0.0] startup
[5.422s] [ext: omni.anim.curve.ui-1.3.16] startup
[5.432s] [ext: omni.isaac.occupancy_map-1.0.1] startup
[5.442s] [ext: omni.isaac.merge_mesh-1.0.1] startup
[5.450s] [ext: omni.kit.widget.stage_icons-1.0.3] startup
[5.461s] [ext: omni.anim.curve.bundle-1.2.3] startup
[5.462s] [ext: omni.isaac.occupancy_map.ui-1.1.0] startup
[5.470s] [ext: omni.kit.widget.zoombar-1.0.5] startup
[5.476s] [ext: omni.kit.window.stage-2.5.9] startup
[5.481s] [ext: omni.kit.usd.collect-2.2.21] startup
[5.487s] [ext: omni.kit.browser.core-2.3.11] startup
[5.496s] [ext: omni.kit.menu.stage-1.2.5] startup
[5.498s] [ext: omni.kit.tool.collect-2.2.12] startup
[5.505s] [ext: omni.kit.browser.folder.core-1.9.12] startup
[5.512s] [ext: omni.kit.search.files-1.0.4] startup
[5.517s] [ext: omni.isaac.physics_inspector-0.1.4] startup
[5.520s] [ext: omni.isaac.asset_browser-0.4.6] startup
[5.527s] [ext: omni.kit.usda_edit-1.1.11] startup
[5.539s] [ext: omni.scene.optimizer.core-106.0.4] startup
[5.553s] [ext: omni.isaac.scene_blox-0.1.1] startup
[5.556s] [ext: omni.isaac.surface_gripper.ui-1.0.0] startup
[5.558s] [ext: omni.isaac.synthetic_recorder-1.7.3] startup
[5.566s] [ext: omni.kit.converter.common-201.0.2] startup
[5.571s] [ext: omni.isaac.throttling-1.1.0] startup
[5.574s] [ext: omni.isaac.utils-1.0.1] startup
[5.578s] [ext: omni.isaac.wheeled_robots.ui-1.1.1] startup
[5.584s] [ext: omni.kit.converter.cad_core-201.0.2] startup
[5.596s] [ext: omni.kit.converter.jt_core-201.0.2] startup
[5.614s] [ext: omni.kit.tool.asset_importer-2.5.5] startup
[5.623s] [ext: omni.kit.actions.window-1.1.1] startup
[5.630s] [ext: omni.isaac.window.about-1.1.0] startup
[5.634s] [ext: omni.kit.browser.asset-1.3.9] startup
[5.638s] [ext: omni.kit.selection-0.1.4] startup
[5.642s] [ext: omni.kit.converter.cad-201.0.2] startup
[5.648s] [ext: omni.kit.hotkeys.window-1.4.5] startup
[5.658s] [ext: omni.kit.manipulator.camera-105.0.5] startup
[5.664s] [ext: omni.kit.menu.aov-1.1.3] startup
[5.666s] [ext: omni.kit.menu.edit-1.1.19] startup
[5.673s] [ext: omni.kit.property.audio-1.0.11] startup
[5.676s] [ext: omni.hydra.scene_api-0.1.2] startup
[5.683s] [ext: omni.kit.property.camera-1.0.6] startup
[5.685s] [ext: omni.kit.property.geometry-1.3.0] startup
[5.692s] [ext: omni.kit.property.light-1.0.8] startup
[5.695s] [ext: omni.kit.property.render-1.1.1] startup
[5.698s] [ext: omni.kit.property.transform-1.5.1] startup
[5.704s] [ext: omni.kit.menu.file-1.1.9] startup
[5.709s] [ext: omni.kit.core.collection-0.1.7] startup
[5.711s] [ext: omni.kit.preferences.animation-1.1.7] startup
[5.716s] [ext: omni.isaac.menu-0.4.1] startup
[5.721s] [ext: omni.kit.property.bundle-1.2.11] startup
[5.727s] [ext: omni.kit.menu.create-1.0.12] startup
[5.732s] [ext: omni.isaac.physics_utilities-0.1.4] startup
[5.739s] [ext: omni.kit.property.collection-0.1.17] startup
[5.744s] [ext: omni.kit.property.isaac-0.2.3] startup
[5.748s] [ext: omni.mdl.usd_converter-1.0.18] startup
[5.751s] [ext: omni.kit.property.layer-1.1.5] startup
[5.757s] [ext: omni.kit.stagerecorder.core-105.0.5] startup
[5.762s] [ext: omni.kit.scripting-106.0.1] startup
[5.773s] [ext: semantics.schema.property-1.0.2] startup
[5.776s] [ext: omni.kit.stage.copypaste-1.2.2] startup
[5.780s] [ext: omni.kit.stage.mdl_converter-1.0.5] startup
[5.783s] [ext: omni.kit.widgets.custom-1.0.8] startup
[5.796s] [ext: omni.kit.stagerecorder.ui-105.0.6] startup
[5.802s] [ext: omni.kit.stage_column.variant-1.0.13] startup
[5.806s] [ext: omni.kit.stage_column.payload-2.0.0] startup
[5.810s] [ext: omni.isaac.range_sensor.examples-1.0.0] startup
[5.833s] [ext: omni.graph.window.action-1.25.2] startup
[5.850s] [ext: omni.kit.variant.editor-106.0.0] startup
[5.868s] [ext: omni.kit.stagerecorder.bundle-105.0.2] startup
[5.869s] [ext: omni.kit.thumbnails.usd-1.0.9] startup
[5.949s] [ext: omni.isaac.range_sensor.ui-1.0.0] startup
[5.953s] [ext: omni.kit.variant.presenter-105.1.2] startup
[5.962s] [ext: omni.kit.viewport.menubar.camera-105.1.8] startup
[5.973s] [ext: omni.kit.viewport.menubar.lighting-106.0.2] startup
[5.989s] [ext: omni.rtx.window.settings-0.6.15] startup
[6.003s] [ext: omni.kit.viewport.menubar.render-106.1.2] startup
[6.011s] [ext: omni.kit.widget.live-2.1.6] startup
[6.015s] [ext: omni.kit.widget.calendar-1.0.8] startup
[6.019s] [ext: omni.kit.viewport.menubar.settings-106.0.0] startup
[6.029s] [ext: omni.kit.widget.collection-0.1.18] startup
[6.038s] [ext: omni.kit.viewport.rtx-104.0.1] startup
[6.039s] [ext: omni.kit.viewport_widgets_manager-1.0.8] startup
[6.045s] [ext: omni.kit.widget.cache_indicator-2.0.8] startup
[6.238s] [ext: omni.kit.widget.extended_searchfield-1.0.27] startup
2024-07-04 13:38:42 [6,237ms] [Warning] [omni.kit.widget.cache_indicator.cache_state_menu] Unable to detect Omniverse Cache Server. Consider installing it for better IO performance.
[6.647s] [ext: omni.kit.widget.timeline-105.0.1] startup
[6.652s] [ext: omni.kit.window.collection-0.1.22] startup
[6.655s] [ext: omni.kit.window.commands-0.2.5] startup
[6.656s] [ext: omni.kit.window.console-0.2.12] startup
[6.666s] [ext: omni.kit.window.imageviewer-1.0.7] startup
[6.672s] [ext: omni.kit.window.privacy-0.1.0] startup
[6.673s] [ext: omni.kit.tool.asset_exporter-1.3.3] startup
[6.676s] [ext: omni.kit.window.script_editor-1.7.6] startup
[6.678s] [ext: omni.kit.window.status_bar-0.1.6] startup
[6.683s] [ext: omni.kit.window.usd_paths-1.0.6] startup
[6.686s] [ext: omni.usd.metrics.assembler-106.0.1] startup
[6.693s] [ext: omni.replicator.isaac-1.15.0] startup
[6.708s] [ext: omni.kit.browser.material-1.5.2] startup
[6.715s] [ext: omni.replicator.replicator_yaml-2.0.5] startup
[6.725s] [ext: omni.kit.ui.actions-1.0.1] startup
[6.730s] [ext: omni.rtx.settings.core-0.6.0] startup
[6.737s] [ext: omni.simready.explorer-1.0.26] startup
[6.754s] [ext: omni.usd.metrics.assembler.physics-106.0.2] startup
[6.758s] [ext: omni.kit.tool.remove_unused.core-0.1.2] startup
[6.761s] [ext: omni.kit.window.material-1.5.7] startup
[6.769s] [ext: omni.usd.metrics.assembler.ui-106.0.2] startup
[6.774s] [ext: omni.kit.menu.common-1.1.5] startup
[6.776s] [ext: omni.kit.renderer.cuda_interop-1.0.1] startup
[6.778s] [ext: omni.kit.quicklayout-1.0.6] startup
[6.779s] [ext: omni.kit.window.title-1.1.3] startup
[6.781s] [ext: semantics.schema.editor-0.3.4] startup
[6.784s] [ext: omni.kit.tool.remove_unused.controller-0.1.3] startup
[6.785s] [ext: omni.isaac.robot_assembler-1.5.0] startup
[6.790s] [ext: omni.isaac.camera_inspector-0.2.1] startup
[6.791s] [ext: omni.kit.livestream.core-3.2.0] startup
[6.796s] [ext: omni.kit.window.stats-0.1.6] startup
[6.798s] [ext: omni.isaac.app.setup-0.13.2] startup
[6.800s] Isaac Sim Headless Native Version: 4.0.0-rc.21
[6.807s] [ext: omni.isaac.assets_check-0.3.4] startup
[6.809s] [ext: omni.kit.profiler.window-2.2.1] startup
2024-07-04 13:38:42 [6,791ms] [Warning] [omni.kit.profiler.window] remove _SpanInstance.__lt__ and use insort 'key' arg instead
[6.817s] [ext: omni.isaac.sim.base-4.0.0] startup
[6.819s] [ext: omni.kit.livestream.native-4.1.0] startup

Failed to open [/var/run/utmp]
Active user not found. Using default user [kiosk]Streaming server started.
[12.254s] [ext: omni.isaac.sim.headless.native-4.0.0] startup
2024-07-04 13:38:51 [15,596ms] [Warning] [omni.fabric.plugin] No source has valid data array=0x22db80e0 usdValid=0 cpuValid=0
2024-07-04 13:38:51 [15,823ms] [Warning] [omni.fabric.plugin] No source has valid data array=0x22fbee40 usdValid=0 cpuValid=0

It hangs here indefinitely.

cc @pcallender @jiehanw @dennis.lynch @hclever

1 Like

One more thing, if you remove all other actor core instances from the reproduction scenario and only keep uniformm0001_instance4 , uniformm0001_instance3 , uniformm0001_instance2 it will work, so its not like they are breaking it. It’s the amount of nodes that is causing the problem.

also: manipulating duplicate scatter seed doesn’t make a difference

Thanks for putting this together. I’ve asked dev to look into this.

Has anyone tried to reproduce the issue?

Repro?

Still nothing? It’s been ages @pcallender and I have provided so much details to quickly reproduce the issue.

Hello @Turowicz, I want to first apologize for the delay in getting back to you. We will be improving our process to reduce delays in responding going forward.

We are indeed able to reproduce the hanging issue, though our logs look somewhat different from yours (seems to hang further down the process). We’re continuing to investigate this issue. We’re aiming to bring many improvements to ReplicatorYAML to make it more robust and more intuitive to work with. Thank you for bringing this bug to light and for all the details, it’s incredibly helpful and we’re grateful for the time you’ve taken in putting this together.

1 Like

Thanks again @Turowicz, there is indeed a bug in utils.py that as you suspected causes the graph connection code to explode. This is due to new logic in 4.0 that avoids nodes to be executed out of order. The suggested workaround for now is to avoid re-using the same reference across characters - so rather than:

surface_prims: floor,

write:

surface_prims:
  get.prims:
    semantics: [[class, floor]]

If this is not a workable solution and modifying an extension file is practical for your application, I can also walk you through the change in utils.py that addresses the issue.

Workaround code:
(note: I also made a small change that fixed an issue scattering the forklift)

# Default scene settings. This script default is 1 (IsaacSim), so if you change
# the meters per unit or up axis, you need to alter the coordinates and rotations.

# IsaacSim default is 1, Code is 0.01
stage_unit_setting:
  settings.set_stage_meters_per_unit:
    meters_per_unit: 1

# IsaacSim default is "Z"", Code is "Y"
stage_up_axis_setting:
  settings.set_stage_up_axis:
    up_axis: Z

# Enable Path Tracing
path_tracing:
  settings.set_render_pathtraced:
    samples_per_pixel: 32

# Load the stage
load_stage:
  create.from_usd:
    usd: http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/Isaac/Environments/Simple_Warehouse/full_warehouse.usd

forklift_og:
  get.prim_at_path:
    path: /Replicator/Ref_Xform/Ref/forklift

hide_forklift:
  with.forklift_og:
    modify.visibility:
      value: False

# Select the objects in the stage
forklift:
  create.from_usd:
      usd: http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/Isaac/Props/Forklift/forklift.usd

floor:
  get.prims:
    semantics: [[class, floor]]

racks_and_walls:
  get.prims:
    semantics: [[class, rack], [class, wall]]

default_light:
  get.prim_at_path:
    path: /Environment/defaultLight

light:
  create.group:
    items:
      - /Replicator/Ref_Xform/Ref/RectLight
      - /Replicator/Ref_Xform/Ref/RectLight_01
      - /Replicator/Ref_Xform/Ref/RectLight_02
      - /Replicator/Ref_Xform/Ref/RectLight_03
      - /Replicator/Ref_Xform/Ref/RectLight_04
      - /Replicator/Ref_Xform/Ref/RectLight_05
      - /Replicator/Ref_Xform/Ref/RectLight_06

camera:
  create.camera:
    position: [0, 0, 0]
    rotation: [0, 0, 0]

# Create output
render_product:
  create.render_product:
    camera: camera
    resolution: [1280, 720]

writer:
  writers.get:
    name: BasicWriter
    init_params:
      rgb: True
      bounding_box_2d_tight: True
      semantic_filter_predicate: class:sign|cone|person
      output_dir: /workspaces/Surveily.Datasets.Generator.Omniverse/.out/Issues/stuck
      #output_dir: /home/wojtek/stuck

writer_attach:
  writer.attach:
    render_products: render_product

# Load businessf0002 animations
businessf0002_load_animations:
  create.from_dir:
    recursive: True
    semantics: [[animation, businessf0002]]
    dir_path: http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/ActorCore/Business_F_0002/Motion/

businessf0002_animations:
  get.prims:
    semantics: [[animation, businessf0002]]

# Load partym0001 animations
partym0001_load_animations:
  create.from_dir:
    recursive: True
    semantics: [[animation, partym0001]]
    dir_path: http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/ActorCore/Party_M_0001/Motion/

partym0001_animations:
  get.prims:
    semantics: [[animation, partym0001]]

# Load uniformf0001 animations
uniformf0001_load_animations:
  create.from_dir:
    recursive: True
    semantics: [[animation, uniformf0001]]
    dir_path: http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/ActorCore/Uniform_F_0001/Motion/

uniformf0001_animations:
  get.prims:
    semantics: [[animation, uniformf0001]]

# Load uniformm0001 animations
uniformm0001_load_animations:
  create.from_dir:
    recursive: True
    semantics: [[animation, uniformm0001]]
    dir_path: http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/ActorCore/Uniform_M_0001/Motion/

uniformm0001_animations:
  get.prims:
    semantics: [[animation, uniformm0001]]

# Load worker animations
worker_load_animations:
  create.from_dir:
    recursive: True
    semantics: [[animation, worker]]
    dir_path: http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/Worker/Motions/

worker_animations:
  get.skelanimation:
    semantics: [[animation, worker]]

# Set the trigger as on_frame
trigger:
  trigger.on_frame:
    max_execs: 20
    rt_subframes: 5

# When the trigger executes, run the following logic
with_trigger:
  with.trigger:
    # Disable default light
    with.default_light:
      modify.visibility:
        value: False

    # Set sun intensity
    sun_intensity:
      with.light:
        modify.attribute:
          name: inputs:intensity
          value:
            distribution.uniform:
              lower: 500
              upper: 5000

    # Set sun temperature
    sun_temperature_on:
      with.light:
        modify.attribute:
          name: inputs:enableColorTemperature
          value: True

    # Set sun temperature
    sun_temperature:
      with.light:
        modify.attribute:
          name: inputs:colorTemperature
          value:
            distribution.uniform:
              lower: 4000
              upper: 7000

    # move the camera around and rotate
    with.camera:
      modify.pose:
        position:
          distribution.uniform:
            lower: [-23.6, -3, 2.5]
            upper: [3.03, 3, 7.5]
        rotation:
          distribution.uniform:
            lower: [0, -30, -120]
            upper: [0, -10, -60]

    # Create and scatter businessf0002
    businessf0002_instance:
      randomizer.instantiate:
        paths:
          - http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/ActorCore/Business_F_0002/Actor/business-f-0002/business-f-0002.usd
        size: 5
        mode: reference

    with.businessf0002_instance:
      modify.pose:
        scale: 0.01
        rotation:
          distribution.uniform:
            lower: [0, 0, -180]
            upper: [0, 0, 180]
      modify.semantics:
        semantics: [[class, person]]
      modify.animation:
        values:
          distribution.sequence:
            items: businessf0002_animations
      randomizer.scatter_2d:
        seed: 1
        surface_prims:
          get.prims:
            semantics: [[class, floor]]
        no_coll_prims:
          get.prims:
            semantics: [[class, rack], [class, wall]]
        check_for_collisions: True

    # Create and scatter partym0001
    partym0001_instance:
      randomizer.instantiate:
        paths:
          - http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/ActorCore/Party_M_0001/Actor/party-m-0001/party-m-0001.usd
        size: 5
        mode: reference

    with.partym0001_instance:
      modify.pose:
        scale: 0.01
        rotation:
          distribution.uniform:
            lower: [0, 0, -180]
            upper: [0, 0, 180]
      modify.semantics:
        semantics: [[class, person]]
      modify.animation:
        values:
          distribution.sequence:
            items: partym0001_animations
      randomizer.scatter_2d:
        seed: 2
        surface_prims:
          get.prims:
            semantics: [[class, floor]]
        no_coll_prims:
          get.prims:
            semantics: [[class, rack], [class, wall]]
        check_for_collisions: True

    # Create and scatter uniformf0001
    uniformf0001_instance:
      randomizer.instantiate:
        paths:
          - http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/ActorCore/Uniform_F_0001/Actor/uniform_f_0001/uniform_f_0001.usd
        size: 5
        mode: reference

    with.uniformf0001_instance:
      modify.pose:
        scale: 0.01
        rotation:
          distribution.uniform:
            lower: [0, 0, -180]
            upper: [0, 0, 180]
      modify.semantics:
        semantics: [[class, person]]
      modify.animation:
        values:
          distribution.sequence:
            items: uniformf0001_animations
      randomizer.scatter_2d:
        seed: 3
        surface_prims:
          get.prims:
            semantics: [[class, floor]]
        no_coll_prims:
          get.prims:
            semantics: [[class, rack], [class, wall]]
        check_for_collisions: True

    # Create and scatter uniformm0001
    uniformm0001_instance:
      randomizer.instantiate:
        paths:
          - http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/ActorCore/Uniform_M_0001/Actor/uniform_m_0001/uniform_m_0001.usd
        size: 15
        mode: reference

    with.uniformm0001_instance:
      modify.pose:
        scale: 0.01
        rotation:
          distribution.uniform:
            lower: [0, 0, -180]
            upper: [0, 0, 180]
      modify.semantics:
        semantics: [[class, person]]
      modify.animation:
        values:
          distribution.sequence:
            items: uniformm0001_animations
      randomizer.scatter_2d:
        seed: 4
        surface_prims:
          get.prims:
            semantics: [[class, floor]]
        no_coll_prims:
          get.prims:
            semantics: [[class, rack], [class, wall]]
        check_for_collisions: True

    # Create and scatter worker
    worker_instance:
      randomizer.instantiate:
        paths:
          - http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/NVIDIA/Assets/Characters/Reallusion/Worker/Props/Worker.usd
        size: 5
        mode: reference

    with.worker_instance:
      modify.pose:
        scale: 0.01
        rotation:
          distribution.uniform:
            lower: [0, 0, -180]
            upper: [0, 0, 180]
      modify.semantics:
        semantics: [[class, person]]
      modify.animation:
        values:
          distribution.sequence:
            items: worker_animations
      randomizer.scatter_2d:
        seed: 7
        surface_prims:
          get.prims:
            semantics: [[class, floor]]
        no_coll_prims:
          get.prims:
            semantics: [[class, rack], [class, wall]]
        check_for_collisions: True

    # create and scatter cones
    cone_instance:
      randomizer.instantiate:
        paths:
          - http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/Isaac/Environments/Simple_Warehouse/Props/S_TrafficCone.usd
        size: 20
        mode: scene_instance

    with.cone_instance:
      modify.pose:
        rotation:
          distribution.uniform:
            lower: [0, 0, -180]
            upper: [0, 0, 180]
      randomizer.scatter_2d:
        seed: 101
        surface_prims:
          get.prims:
            semantics: [[class, floor]]
        no_coll_prims:
          get.prims:
            semantics: [[class, rack], [class, wall]]
        check_for_collisions: True

    # create and scatter wet floor signs
    wet_floor_sign_instance:
      randomizer.instantiate:
        paths:
          - http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1/Isaac/Environments/Simple_Warehouse/Props/S_WetFloorSign.usd
        size: 20
        mode: scene_instance

    with.wet_floor_sign_instance:
      modify.pose:
        rotation:
          distribution.uniform:
            lower: [0, 0, -180]
            upper: [0, 0, 180]
      randomizer.scatter_2d:
        seed: 102
        surface_prims:
          get.prims:
            semantics: [[class, floor]]
        no_coll_prims:
          get.prims:
            semantics: [[class, rack], [class, wall]]
        check_for_collisions: True

    # move the forklift around and rotate
    with.forklift:
      modify.pose:
        rotation:
          distribution.uniform:
            lower: [0, 0, -180]
            upper: [0, 0, 180]
      randomizer.scatter_2d:
        seed: 0
        surface_prims:
          get.prims:
            semantics: [[class, floor]]
        no_coll_prims:
          get.prims:
            semantics: [[class, rack], [class, wall]]
        check_for_collisions: True

    # Randomize floor
    with.floor:
      modify.material:
        value:
          - /Replicator/Ref_Xform/Ref/SM_floor27/Looks/MI_Floor_02b
          - /Replicator/Ref_Xform/Ref/SM_WallA_3M42/SM_WallA_6M/Looks/MI_WallA_01
1 Like

Dear @jlafleche, thank you for the workaround. I will test this later this week.

yep, it works!

for python: all you need is a factory function that you pass into the surface prims

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.