Replicator's Semantic Segmentation has a lot of noise

Please provide all relevant details below before submitting your post. This will help the community provide more accurate and timely assistance. After submitting, you can check the appropriate boxes. Remember, you can always edit your post later to include additional information if needed.

Isaac Sim Version

4.5.0
4.2.0
4.1.0
4.0.0
4.5.0
2023.1.1
2023.1.0-hotfix.1
Other (please specify):

Operating System

Ubuntu 24.04
Ubuntu 22.04
Ubuntu 20.04
Windows 11
Windows 10
Other (please specify):

GPU Information

  • Model: RTX 4090
  • Driver Version: 550.120

Topic Description

Detailed Description

(Describe the issue in detail, including what you were trying to do, what you expected to happen, and what actually happened)

Steps to Reproduce

GOAL: Use the semantic segmentation to extract just the arm and lemon from the rgb image.

  1. Created a simple world in isaac sim with a robot arm and lemon on a table, and some walls and floor around it.
  2. Used replicator to record the scene when I start the simulation. I recorded rgb, semantic segmentation, and depth from image plane
  3. In post-processing, I used the semantic segmentation output to create a mask, that I used to overlay the robot and lemon from the rgb footage onto the semantic segmentation output. The overlaying was mainly a gut-check on if the segmentation is working correctly.

I saw all this noise around the lemon and robot and I don’t know where its coming from. In this simulated world, I would expect perfect segmentation, and so I expect I should be able to perfectly mask out the arm from the rgb image.

Screenshots or Videos

(If applicable, add screenshots or links to videos that demonstrate the issue)
I’ve included the semantic segmentation output, the rgb output, and the eventual overlayed output with the rgb arm and lemon overlayed into the sem-seg video.

@hardik15 just out of curiosity, what’s your output render size set at and format? i noticed from the screenshot that it’s a mp4 file, was that encoded in post as well from the images?

Yes, the video was made in post. With replicator I’m outputting (640, 480) sized png’s.

a few more inquiries to dig into your process a bit deeper:

  1. could you attach one frame and all of the corresponding output for further assessment?
  2. could you also elaborate a bit more on your post-process to do the masking, including but not limited to the program of choice (After Effects, Resolve, etc) and tools used (color keying, transparency detection, etc)?
  3. provide the snippet for your output so we can examine whether you have set certain render settings (Antialiasing, denoising, etc)