Isaac sim semantic label

Omniverse Isaac Sim
1

Hello, When I use isaac sim to give semantic label to some objects, I also using python to output the semantic map. However, there are two different kind of semantic label. Some objects have apple:apple, and also apple two semantic map, while others only have banana one semantic map.
My question are

  1. what the difference of these two kind of semantic map, apple:apple and apple.
  2. why some objects have two (both apple:apple and apple) and some only hve one
    I also check the semantic label output numpy array and I find these seems not number corresponding to the number of apple:apple.
3

@thuaj can you provide a simple usd file that reproduces the issue?

5

label.usdc (2.4 KB)
Hello, I upload a usd sample.
Q1: In this usd, bowl only have the semantic of bowl, while clock has both clock and clock::clock.
I get the semantic maps by python code

            lst = list(range(1, 50))
            semantic_map = self.sd_helper.get_semantic_label_map(lst)

Q2:When I using the python to get the semantic and instance groundtruth output. I can set the instance mapping. I set it like this.

[0, "/World/lroom_origin_final/lroom/lroom/ceiling", 17, "ceiling", [10, 4, 9], ""], \
[21, "/World/lroom_origin_final/bowl", 15, "bowl", [144], ""]

I can get the correct instance label number by setting this ceiling instance to number 0.
My question is how can I set the semantic number for the ceiling to a certain number?
Although I set it to number 17 , the semantic output of ceiling is not 17.
Q3: If I set this instance mapping, the semantic output will become clock::clock and bowl. The clock semantic label disappears.

Overall, my needs are set instance and semantic number to certain object and semantic class.

7

I have the same question, how to add custom number for semantic labels and publish it ?

8

@thuaj can you also attach bowl.usd and clock.usd, or file->save flattened as and attach that.

@fzg7919 the number for the semantic label is just the index into the actual list of semantic classes. When you add a semantic schema API to a prim that is the label and you can use the semantic id that is returned by synthetic data as an index to retrieve it.

Here is a simple example you can paste in the script editor and then press play to start publishing semantic labels for a stage (if it has any) to ROS

import omni.graph.core as og
og.Controller.edit(
    {"graph_path": "/ActionGraph", "evaluator_name": "execution"},
    {
        og.Controller.Keys.CREATE_NODES: [
            ("OnPlaybackTick", "omni.graph.action.OnPlaybackTick"),
            ("CameraHelper", "omni.isaac.ros_bridge.ROS1CameraHelper"),
            ("IsaacClock", "omni.isaac.core_nodes.IsaacReadSimulationTime"),
            ("ClockPublisher", "omni.isaac.ros_bridge.ROS1PublishClock"),
        ],
        og.Controller.Keys.SET_VALUES: [
            ("CameraHelper.inputs:viewport", "Viewport"),
            ("CameraHelper.inputs:topicName", "semantic_segmentation"),
            ("CameraHelper.inputs:type", "semantic_segmentation"),
            ("CameraHelper.inputs:enableSemanticLabels", True),
            ("CameraHelper.inputs:semanticLabelsTopicName", "semantic_labels"),
        ],
        og.Controller.Keys.CONNECT: [
            ("OnPlaybackTick.outputs:tick", "CameraHelper.inputs:execIn"),
            ("OnPlaybackTick.outputs:tick", "ClockPublisher.inputs:execIn"),
            ("IsaacClock.outputs:simulationTime", "ClockPublisher.inputs:timeStamp"),
        ],
    },
)
10

So there is no way to use custom numbers as labels instead of index ?

11

Right, the internal semantic IDs cannot be set, they are automatically computed based on the class/instance.

12

Can you elaborate on the usecase where you would need to set custom IDs? vs specifying classes?

13

Hello
For my case.
I have multiple usd files which consitsit same objects instances but in different places on the stage. I generate semantic images using these usd files. However I find different files have different semantic numbers, which making it difficult to generate a class to number tables suitable for all the files.

14

clock.usd (27.7 KB)
This is clock usd.

15

bowl.usd (9.7 KB)
This is the bowl usd.

16

Yes , we use custom ids for specifying semantic class , and I find a new problem when I use semantic schema editor to attach semantic labels to some prims like floor, wall, human, and there are some prims I dont attach semantic labels to them , but the semantic publisher of roscamerahelper will publish all the prims , it seems like all prims have semantic data, but i only want to have the target prims’ semantic data that I attach semantic data using semantic schema editor but not all prims.

17

here is the semantic labes ros topic output:
data: “{"0": {"class": "UNLABELLED"}, "15": {"class": "curtain"}, "2": {"
class": "Wall,wall"}, "3": {"class": "Wall,floor"}, "9": {"class":
\ "carpet"}, "1": {}, "time_stamp": {"secs": "1877", "nsecs": "266764574"
}}”
it seems like the all instance in the camera viewport have semantic data, if i do not set semantic data to the instance , it‘s semantic data will be {"class": "UNLABELLED"} , that is right ?