I’ve encountered a bug in Omniverse Replicator where the output for semantic segmentation data is incorrect.
Problem Description
The labels and colors do not match between the two primary semantic segmentation outputs:
-
The mapping file:
semantic_segmentation_labels_xx.json -
The image data:
semantic_segmentation_xx.png
When using the BasicWriter and executing orchestrator steps asynchronously, the semantic segmentation data retrieved after the Replicator step appears to be swapped or mismatched with the scene.
Observed Error
The NumPy array from annotator containing the semantic IDs for each pixel does not correspond to the current stage setup. In a scene configured with two distinct semantic IDs (e.g., type0_red and type0_yellow), the output IDs could be:
-
Identical (e.g., both are labeled as the same ID).
-
Swapped (e.g., the IDs for the two objects are inverted).
-
Incorrectly set (e.g., an altogether wrong ID is assigned).
Replication
I have attached a full example that consistently reproduces this bug for debugging purposes.
Steps to reproduce:
- Unzip the ReplicatorSemanticBug.zip
- Load the stage: stage.usd
- Open Script Editor window
- File→Open→ semantic_test.py (located in the same folder as stage.usd)
- Run Script and wait for the results in output folder
NOTE: Although you can reproduce it with this example, mismatching could appears in different sample output
Sample Stage:
- There are 3 red planes meshes under a parent prim with semantic label “TYPE0_RED”
- There is a folder with externals USD files with a yellow plane that will be added to the stage under a parent prim with semantic label “TYPE1_YELLOW”
Tested environment:
- Windows 10 (KIT 107.3.1) and Ubuntu 22.04 (107.3.0)
- omni.replicator.core extension
- Using RTX - Real-Time 2.0 render
Expected output: (For example found in samples 0000, 0003, …)
Current output with mismatching label colors: (Example 0001, 0005, 0007, …)
ReplicatorSemanticBug.zip (12.0 KB)