Freespace Demo Not Working in Isaac Sim 4.1 and Later

[k_m]

I want to create a demo using Isaac Sim 4.1 based on the following link, but there is an issue with the dataset: GitHub - NVIDIA-AI-IOT/robot_freespace_seg_Isaac_TAO: In this workflow we demonstrate using SDG + TAO for a freespace segmentation application

Previously, the dataset could be created correctly using Isaac Sim 2023.1.1. However, with Isaac Sim 4.1 or later versions, the floor no longer appears in the semantic segmentation images.

Below is what I have done so far. Could you confirm if the issue can be reproduced on your end? What changes should I make to fix this issue?

Tasks Performed:

• Opened the simple_room.usd file.
  omniverse://localhost/NVIDIA/Assets/Isaac/4.1/Isaac/Environments/Simple_Room/
• Labeled the floor asset and executed the script to create the dataset.
  
  

import omni.replicator.core as rep

camera = rep.create.camera() 
render_product = rep.create.render_product(camera, (1024, 1024))

active_vp_window = omni.kit.viewport.utility.get_active_viewport()
active_vp_window.camera_path = "/Replicator/Camera_Xform/Camera" 

PROPS = "omniverse://localhost/NVIDIA/Assets/Isaac/2022.2.1/Isaac/Props/YCB/Axis_Aligned/"
instances = rep.randomizer.instantiate(rep.utils.get_usd_files(PROPS, recursive=True), size=20)

with rep.trigger.on_frame(num_frames=1000):

    rect_light = rep.create.light(
        position=(0, -1.25829, 3.28544),
        scale=(5, 1, 1),
        color=rep.distribution.uniform((0, 0, 0), (2, 2, 2)),
        intensity=rep.distribution.uniform((5000), (45000)),
        temperature=rep.distribution.uniform((0), (10000)),
        light_type="rect"
    )

    with instances:
        rep.modify.pose(
            scale=rep.distribution.uniform((1), (3)),
            position=rep.distribution.uniform((1, 1, -0.75), (4, 4, -0.4)), 
            rotation=rep.distribution.uniform((0, 0, 0), (360, 360, 360)),
        )

    with camera:
        rep.modify.pose(
            position=(2, 2, 0),
            rotation=rep.distribution.uniform((0, -15, 0), (0, -10, 360))
        )

# Initialize and attach writer
writer = rep.WriterRegistry.get("BasicWriter")
writer.initialize(output_dir="simpleroom_output", rgb=True, semantic_segmentation=True)
writer.attach([render_product])

# Specify rendering options for preview
rep.orchestrator.preview()

Dataset:

• The first few semantic segmentation images are created correctly.
• However, some images are generated where the floor appears but is not annotated.
• When checking the JSON file for the semantic segmentation, the “floor” label is not output.

Additional Information:

• Changing the simpleroom.usd asset to another version (e.g., 4.2 or 4.0) results in the same issue.
• Placing a labeled plane mesh instead of the simpleroom.usd seems to work correctly, suggesting a potential issue with the simpleroom.usd.
• When performing labeling, the following error/warning is output:

2024-12-11 00:44:22 [115,216ms] [Warning] [omni.ui] Container::addChild attempting to add a child during a draw callback
  File "/home/nvidia/.local/share/ov/pkg/isaac-sim-4.1.0/extscache/semantics.schema.editor-0.3.6/semantics/schema/editor/ext.py", line 508, in <lambda>
    "Add Entry On All Selected Prims", height=30, clicked_fn=lambda: self._add_to_all_entries(entries)
  File "/home/nvidia/.local/share/ov/pkg/isaac-sim-4.1.0/extscache/semantics.schema.editor-0.3.6/semantics/schema/editor/ext.py", line 430, in _add_to_all_entries
    e.add_entry(n_type, n_data)
  File "/home/nvidia/.local/share/ov/pkg/isaac-sim-4.1.0/extscache/semantics.schema.editor-0.3.6/semantics/schema/editor/ext.py", line 282, in add_entry
    self._update_single_entry(new_name, type, data)
  File "/home/nvidia/.local/share/ov/pkg/isaac-sim-4.1.0/extscache/semantics.schema.editor-0.3.6/semantics/schema/editor/ext.py", line 295, in _update_single_entry
    self.update_ui_func()
  File "/home/nvidia/.local/share/ov/pkg/isaac-sim-4.1.0/extscache/semantics.schema.editor-0.3.6/semantics/schema/editor/ext.py", line 675, in _build_window_ui
    with ui.ScrollingFrame():
  File "/home/nvidia/.local/share/ov/pkg/isaac-sim-4.1.0/kit/kit", line 2, in <module>

Isaac Sim Version:

• 4.2.0
• 4.1.0

Operating System:

• Ubuntu 20.04.6 LTS

GPU Information:

• Model: NVIDIA A40-16Q
• Driver Version: 550.54.15

[k_m]

I’m not sure why, but the issue was resolved by loading the asset using rep.create.from_usd instead of rep.randomizer.instantiate.