How to get Instance ID segmentation annotator to work from my extension

Omniverse Isaac Sim
, , ,
1

I’m trying to understand how to make the annotator for instance ID segmentation work in my extension. My extension does not directly use replicator, and I’m not sure if it’s needed. Examples such as this or this or this are not directly useful. I am currently instantiating a Camera from omni.isaac.sensor and I am able to access RGB and Depth. I called initialize() on the camera in my “on play” callback, and I’ve registered annotators using the methods on the camera object, like so:

    def on_play(self):
        for camera in self.cameras:
            camera.initialize()
            camera.add_instance_id_segmentation_to_frame()
            camera.add_distance_to_image_plane_to_frame()

But the segmentation data is all zeros or all ones! Below is a snippet of code showing how I get the images. This happens inside of on_physics_step. I can tell that something is almost working because the ‘idToLabels’ values change as the camera moves and different objects go in and out of view, but the values are something like: {'19': 'INVALID', '40': 'INVALID', '66': 'INVALID', '71': 'INVALID', '65': 'INVALID', '70': 'INVALID', '80': 'INVALID', '29': 'INVALID', '21': 'INVALID'}

        for camera in self.cameras:
            frame_dict = camera.get_current_frame()
            rgba = frame_dict.get('rgba')
            instance_id_seg = frame_dict.get('instance_id_segmentation')
            depth = frame_dict.get('distance_to_image_plane')
            if rgba is not None:
                rgb = rgba[..., :3].astype(np.uint8)
                rr.log(f"camera/{camera.name}/rgb", rr.Image(rgb).compress(95))
            if instance_id_seg is not None:
                segmentation = instance_id_seg['data']
                id_to_labels = instance_id_seg['info']['idToLabels']
                print(len(id_to_labels), np.unique(segmentation))

One thing that makes this work to:

  1. Load my plugin
  2. Load the script editor and paste in the replicator demo from here
  3. Click “Run” on the script, which creates a few new things in the scene including a SGDPipeline (what is this?)
  4. Click Play to run isaac sim, which seems to run both the replicator code and my extension. This results in a valid instance id segmentation! But why?

Some of my questions are:

  • What is this replicator script doing that I need to be doing in my extension?
  • Why does the rgb and depth annotators work without any fuss? What makes the instance id segmentation annotator different?
  • Is there an example code showing how to use annotators without base the project entirely around replicator? My project is primarily based around cumotion and use a custom extension, and I’m not using the replicator workflow.

Isaac Sim Version

4.2.0

Operating System

Ubuntu 22.04

2

Sorry to say that I can’t answer any of the questions at the end, since I have no experience with the replicator workflow, but you linked to a post of mine where I was working on a standalone app, which should be similar to the extension workflow, as to my understanding the main difference is that you start the latter from within the isaac GUI and the former from a CLI. So instead, I have a question of my own: have you tried adding add_update_semantics(your_prim_path, 'someprim') after adding the prims you want to be annotated?
Code context is in one of the posts you linked, but a couple of replies below the accepted answer here.

For some reason, instanceId requires you to “register” the prims you want to ID (counter-intuitive behavior as an instanceId sounds like something you automatically get when instantiating an object in your scene, especially considering the other annotators work out of the box so to say).

One question I can answer however: SDGPipeline (not “SGDPipeline” as you wrote) should be the Synthetic Data Generation Pipeline.

Note: I think I’ve used isaacsim 2023.x.x when posting linked post, so might be that something changed in the newest major releases.

3

Thanks! Yes the key was to add semantic labels. I guess this was somehow being done by creating/adding the replicator code. Adding semantic labels myself via the UI and saving them to my USD files worked. Very unintuitive like you say, I thought each prim just got some random Id or something.