Isaac Sim 2022.1.1 Replicator Composer Missing Amodal & Occluded Instance Segmentation Sensor

Omniverse Isaac Sim
, ,
1

Is there any way I can obtain the groundtruth instance segmentation mask of each individual object (including amodal and occluded masks) in a rendered scene of a camera in isaac sim?

Currently, the API only provides sensors such as instance segmentation, depth, RGB etc but there is no sensor that can extract the groundtruth amodal and occluded masks of USD objects in a rendered scene.

  • How can I create my own custom sensor for this use case of generating amodal and occluded instance segmentation masks for each object in each rendered scene?
  • Is it possible to hide objects (reduce opacity to 0) for each scene and obtain the instance segmentation mask for each object in the scene using replicator composer?

Thank you so much in advance for your help!

3

Hi there,

I am currently looking into the possibility of accessing amodal and occlusion mask data from the occlusion calculation, I will get back to you as soon as I have an answer. If that however is currently not possible a workaround would be as you mentioned to hide objects in the scene, read out the instance segmentation and then compare it with the original visible scene.

UPDATE:
It is currently not possible to get amodal and occlusion segementation mask, it is however something that is actively investigated to provide it as default annotators.

Cheers,
Andrei

1 Like
4

Hi ahaidu,

Thank you so much for your help.

I would like to ask how can I iteratively hide objects in each scene so that I can iteratively read and save the instance segmentation mask for each object for the same scene using replicator composer.
Can this be done by editing the replicator composer yaml config or the python api extension source codes directly?

Thank you so much in advance!

5

Hi there,

UPDATE:
It is currently not possible to get amodal and occlusion segementation mask, it is however something that is actively investigated to provide it as default annotators.

As a current workaround you can change the visibility of prims using the python isaac or usd api:

  • using isaac.core.utils:
from omni.isaac.core.utils import prims

cube1_prim = prims.create_prim(prim_path="/World/Cube1", prim_type="Cube", position=(0., 0., 2.))
cube2_prim = prims.create_prim(prim_path="/World/Cube2", prim_type="Cube", position=(0., 0., -1.))
cube2_ref = prims.get_prim_at_path("/World/Cube2")

prims.set_prim_visibility(cube1_prim, False)
prims.set_prim_visibility(cube2_ref, False)
  • using USD:
from pxr import UsdGeom

stage = omni.usd.get_context().get_stage()
cube3_prim = stage.DefinePrim("/World/Cube3", "Cube")
cube4_prim = stage.DefinePrim("/World/Cube4", "Cube")
cube4_ref = stage.GetPrimAtPath("/World/Cube4")

imageable3 = UsdGeom.Imageable(cube3_prim)
imageable3.MakeInvisible()
# imageable3.MakeVisible()

imageable4 = UsdGeom.Imageable(cube4_prim)
imageable4.MakeInvisible()
# imageable4.MakeVisible()

Cheers,
Andrei

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.