How to synchronize published ROS2 topics from ISAAC Sim with Semantic Segmentation Images got from Replicator

Omniverse Isaac Sim
,
1

Hi,

I have recorded a rosbag file consisting of the following topics published by the Nova Carter Robot in ISAAC Sim’s provided example (Isaac Examples → ROS → Navigation) warehouse simulation environment:

/chassis/odom
/clock
/cmd_vel
/front_stereo_camera/left/camera_info
/front_stereo_camera/left/image_raw
/plan
/tf

I have to parse the contents of these topics to get necessary fields (ex: camera image, robot speed, motion command, route), time-synchronize them, add semantic segmentation labels for each corresponding camera image and dump these contents into a pandas Dataframe.

The problem I am running into is that segmentation data are not being published to a ROS2 topic, but instead, are being manually generated from ISAAC Sim’s Replicator and dumped into a local folder in form of a synchronized RGB image, a semantic image and a.json with only labels information.

How do I get time synchronized entries for the above ros topics i.e. /chassis/odom, /cmd_vel, /front_stereo_camera/left/image_raw, /plan, /tf, along with the semantic labels – either in the form of ROS topics or dumped directly into a Dataframe or json on the fly-- so that I can generate a Dataframe at my end that looks something like this:-

|–timestamp–| camera image |-robot speed-|-------motion command----|--------route-----------|semantic labels|
7599.669830 |------image -----|–speed value–| linear & angular velocities| array of route points| semantic mask

3

Hi @shreshta.shetty! You can try ROS2 camera helper. Based on your configuration, it can output metadata.

allowedTokens = rgb,depth,depth_pcl,instance_segmentation,semantic_segmentation,bbox_2d_tight,bbox_2d_loose,bbox_3d,camera_info

4

Hi, @zhengwang !

Thanks for the reply!

Using the ROS2 Camera Helper you mentioned gives the semantic image with each pixel being populated with the class number and the image shape over a specified ROS2 topic.

For each image, however, the SDG also gives a .json file, which maps the class number to the class name for each image. Following is an example of such a .json file for a random semantic image:-
{"0": {"class": "BACKGROUND"}, "1": {"class": "UNLABELLED"}, "2": {"class": "furniture"}, "3": {"class": "human"}, "4": {"class": "navigable"}, "5": {"class": "wall"}}.
I need this data on a ROS2 topic too. The aforementioned solution doesn’t give this.

Any help on how to get this class number to class name mapping on a ROS2 topic?

5

@shreshta.shetty Let me reach out to the internal team.

7

Hi @shreshta.shetty! You can look into the configuration Enable Semantic Labels and Semantic Labels Topic Name. This would give you labels for each class label and publish to a ROS2 topic.