Issue with Semantic Segmentation using semanticData and semanticColor in Isaac Sim

Omniverse Isaac Sim
1

Isaac Sim Version

4.2.0
4.1.0
4.0.0
2023.1.1
2023.1.0-hotfix.1
Other (please specify):

Isaac Lab Version (if applicable)

1.2
1.1
1.0
Other (please specify):

Operating System

Ubuntu 22.04
Ubuntu 20.04
Windows 11
Windows 10
Other (please specify):

GPU Information

  • Model:
  • Driver Version:

Topic Description

Detailed Description

I am working on implementing semantic segmentation in Isaac Sim, but I’m encountering issues with applying semantic labels and colors as expected. I’d appreciate any help with this situation. Here’s a summary of my setup and the issues I’m facing:

  1. Objective:
  • I aim to assign unique semanticId and semanticColor values to each object from a predefined YAML file and apply these attributes to objects in Isaac Sim.
  • When capturing segmentation images, I want each object to be visualized using the pre-defined semanticColor.
  1. Current Setup:
  • I load semanticId and semanticColor mappings from a YAML file and assign them to each object. For each prim, I directly set attributes like semanticData and semanticColor using prim.CreateAttribute.
  • I’m using rep.AnnotatorRegistry.get_annotator("semantic_segmentation") to retrieve segmentation data, but the expected colors are not displayed correctly; only some objects are updated, or default colors are shown instead.
  • Even after setting semanticData and semanticColor, the segmentation capture does not seem to reflect these settings. To address this, I tried setting attributes in the semantic:Semantics:<params> format, but the issue persists.
  1. Current Issues:
  • The semanticColor I assigned is not reflected in the actual segmentation image, and it seems like Isaac Sim might be assigning colors randomly.
  • Although I manually assigned semanticData and semanticColor attributes, I’m unable to accurately retrieve the color values through rep.AnnotatorRegistry.get_annotator("semantic_segmentation").
  • The segmentation image does not accurately visualize objects with the predefined colors associated with their semanticData.
  1. Questions:
  • How can I ensure that Isaac Sim correctly uses the user-defined color (semanticColor) for each semantic label (semanticData) in segmentation?
  • Is there an alternative configuration needed besides the semantic:Semantics:<params> format for these attributes?
  • Are there any recommended settings or steps I should check to ensure that the semanticColor values are applied correctly in the segmentation capture?