ROS1 Camera Helper: 2D and 3D bounding box ROS messages do not work

Omniverse Isaac Sim
,
1

I am following the ROS1/Cameras tutorial in order to create a camera publisher with 3D bounding box data.

I have a simple scene with a cylinder for which I have attached a semantic label. In the viewport, I am able to see the semantic segmentation and 2D or 3D bounding boxes.

3d_bbox
3d_bbox825×564 28.7 KB

view_semantic
view_semantic1460×1197 109 KB

However, the ROS messages are not working properly.

In ROS1 Camera Helper, I have tried the following types:

  1. Semantic segmentation
  • Using rostopic list, I see the topic /semantic (✓)
  • Using rostopic echo /semantic, I see a bunch of zeros and ones (seems ok)
  • In rviz though, I get unsupported image encoding [32SC1]

rviz_semantic
rviz_semantic1220×901 90.4 KB

  1. 2D bounding box
  • Using rostopic list, I see the topic /bbox_2d (✓)
  • Using rostopic echo /bbox_2d, I get:
ERROR: Cannot load message class for [vision_msgs/Detection2DArray]. Are your messages built?
  1. 3D bounding box
  • No topic is declared at all. (rostopic list does not show the topic)

rgb and depth types are fine and I can visualize them in rviz.

Could you please let me know how to create a valid 3D bounding box camera publisher?

2

Hi @elena_e

  1. According to the rviz documentation the 32SC1 format is not supported.

    I may be missing something, but I have not been able to find a ROS command line terminal tool that supports 32SC1 conversion to other formats. Then, I created a simple script to transform 32SC1 to 8UC1 (mono/gray scale image): convert_32SC1_to_8UC1.py (1.4 KB)

    You can use it to visualize the semantic topic

    Screenshot from 2023-03-06 10-57-18
    Screenshot from 2023-03-06 10-57-181788×960 232 KB

  2. Install vision_msgs package as follow:

    apt-get install ros-${ROS_DISTRO}-vision-msgs

  3. I have not encountered any problems when publishing the bbox_3d. The following .usd defines an action graph for it: bbox_3d.usd (24.8 KB)

1 Like
3

Hi Toni,

Thank you so very much for your response!

After installing the vision-msgs package, both the 2D and 3D bounding box messages work!

Also, using your script, I was able to visualize the semantic segmentation in rviz as well. Thank you very much for sharing!

4

Hi @toni.sm,

I have been looking at the 3D bounding box data but it seems that something is off.

  1. I wasn’t sure whether the pose of the bounding box is w.r.t. the camera or the world (actual pose equal to the prim’s pose shown in properties) but after moving the camera, I get the same pose for the bounding boxes, so I assume it is w.r.t. the world. Is this accurate?

  2. The issue I have is that the bounding boxes seem accurate only for cubes and not for spheres or cylinders (haven’t tested anything else).

In the .usd example given above, I have added a cylinder and translated the cube.

cube
cube1414×851 161 KB

For the cube, the center of the bbox matches the cube’s center, there is no rotation and the size is correct:
bbox_cube

However, for the cylinder, the bbox’s center position is not the same as the prim’s translation, it shows a rotation that does not exist and the size also seems wrong.

cylinder
cylinder571×645 30.7 KB

bbox_cylinder

I would expect to get:

  • position: x:3.26, y:0, z:0
  • orientation: x:0, y:0, z:0, w:1
  • size: x:1, y:1, z:1

Same for the sphere.
The center’s position numbers are the same but they refer to different axes, there is a rotation that shouldn’t exist and the size is different.

sphere
sphere493×599 27.9 KB

bbox_sphere

Since the cube’s bbox matches the prim’s characteristics, I would expect that to happen for the other objects as well. So, are the other data indeed wrong? If not, how should I interpret them?

Edit: After examining the bbox data in another scene with two cubes and a cylinder, I get “correct” results that match the prim’s attributes for the cylinder and “wrong” results for the cubes. So, it is not the shape that matters but I cannot figure out why only one shape gives “correct” results while the others don’t.

5

Hi @elena_e

As a user, I lack sufficient information to answer these questions.
@Hammad_M could you please help us identify what might be wrong?

7

Oh, I see. Thank you very much for replying though!

8

Due to a memory layout change bbox2d loose, tight and bbox3d were all incorrect in 2022.2.0
I have added better tests to catch this and it is fixed in 2022.2.1, coming out this week (fingers crossed)

tested with the warehouse and ROS2 humble and it is correct in 2022.2.1:

image
image1920×699 126 KB

1 Like
9

Oh, okay, so I will wait for the new version release and hopefully, it will work with ROS 1 as well.

Thank you for your response!

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.