Bounding box generation for robot actor

Omniverse Isaac Sim
1

Hello!

I am using the carter_sim application and trying to generate training data for my robot’s object recognition, adding a class from static meshes was pretty easy, as I just added the class in the json config file, following this documentation: https://docs.nvidia.com/isaac/isaac_sim/plugins/isaac_sim/sensors.html?highlight=detection#enabling-bounding-boxes.

However, when I try to do the same for a spawned robot it is not that simple, I spawned a carter robot, named ‘carter_2’, from URDF by adding it to the apps/carter_sim/bridge_config/carter_full_config.json file, I changed the camera config to look as follows:

"segmentation_classes": {
  "static_mesh": [
    ...
    {
      "name": "table"
    },
    {
      "name": "carter_2"
    }
   ]
},
"bounding_box_settings": {
  "all_bounding_boxes": false,
  "occlusion_check": true,
  "output_occlusion_ratio": true,
  "ray_spacing_in_pixels": 10,
  "max_rays_per_box": 16,
  "occlusion_threshold": 0.8,
  "draw_debug": false,
  "classes": [
    ...
    {
      "name": "table"
    },
    {
      "name": "carter_2"
    }
  ],
  "exclude_classes": [
    {
      "name": "dummycharacter"
    }
  ]
}
...

When I check on the Issac Sight app, the segmentation seems to work properly, showing the robot’s shade on a different color, but on the detection window, there is no bounding box for the carter robot, as you can see on the attached image.

Screenshot.jpg
Screenshot.jpg2492×1378 566 KB

Screenshot.jpg
Screenshot.jpg2492×1378 571 KB

2

Hi Leo,

I was able to reproduce your issue. Currently cameras are unable to detect bounding boxes for the parent actor they are attached to.

If you create a separate ActorGroup in the graph and config files and add a simple cube actor like below

{
   "name": "alt_parent",
   "reference": "Blueprint'/Game/Actors/CubeActor.CubeActor'",
   "pose": [1, 0, 0, 0, -11.01, 63.47, 0.92]
}

And change the camera parent to “alt_parent”, the bounding box should appear. You might have to change the initial pose a bit depending on where you spawn the carter urdf. This does not solve your issue directly but should allow you to get the bounding box for carter.

We will work on a fix for this to support your use case.