Why doesn't Nvblox work with Realsense RSD455 in Isaac Sim?

[pu9730962]

As the title suggests, when using the “RealSense RSD455” camera in Isaac Sim 4.5 and running ros2 launch nvblox_examples_bringup isaac_sim_example.launch.py to test NVBlox, I found that NVBlox fails to construct the TSDF map. However, when I use the “Leopard Hawk camera”, NVBlox works as expected. Could you please advise on how to resolve this issue?

Rviz2

I performed the following three experiments to assist in analyzing the problem.

Case1:
Create two “Isaac Create Render Products” and set cameraPrim to “/World/hawk_manipulator/left/camera_right” and “/World/hawk_manipulator/left/camera_left” respectively. Also, in the “ROS2 Publish Transform Tree”, set the targetPrims to “/World/hawk_manipulator/left/camera_right” and “/World/hawk_manipulator/left/camera_left” . With this setup, NVblox in Rviz2 can display correctly.

Case2:
Create two “Isaac Create Render Products” and set cameraPrim to “/World/rsd455/RSD455/Camera_OmniVision_OV9782_Right” and “/World/rsd455/RSD455/Camera_OmniVision_OV9782_Left” respectively. Also, in the “ROS2 Publish Transform Tree”, set the targetPrims to “/World/rsd455/RSD455/Camera_OmniVision_OV9782_Right” and “/World/rsd455/RSD455/Camera_OmniVision_OV9782_Left” . With this setup, NVblox in Rviz2 can’t display correctly.

Case3:
Create two “Isaac Create Render Products” and set cameraPrim to “/World/rsd455/RSD455/Camera_OmniVision_OV9782_Right” and “/World/rsd455/RSD455/Camera_OmniVision_OV9782_Left” respectively. Also, in the “ROS2 Publish Transform Tree”, set the targetPrims to “/World/hawk_manipulator/left/camera_right” and “/World/hawk_manipulator/left/camera_left”. With this setup, NVblox in Rviz2 can display correctly.

[Ahung]

Hi @pu9730962 ,

Thanks for reporting. Here is the nvblox/isaacsim docs.

It seems that you are changing the camera in Isaac Sim. Nvblox is only tested on the cameras in the example scene. localhost/NVIDIA/Assets/Isaac/4.2/Isaac/Samples/NvBlox/nvblox_sample_scene.usd .

Can you check whether the topics from Sim to nvblox match? Nvblox needs a depth image + camera pose to update the tsdf. A full log would help.

Best,
Ahung

[pu9730962]

I checked a section of code in nvblox.launch.py, and it requires the user to publish the following topics. I published the required topics as specified, but as long as I use the RSD455 in Isaac Sim, NVblox fails to display.

def get_isaac_sim_remappings(mode: NvbloxMode, num_cameras: int,
                             lidar: bool) -> List[Tuple[str, str]]:
    remappings = []
    camera_names = ['front_stereo_camera', 'left_stereo_camera',
                    'right_stereo_camera'][:num_cameras]
    for i, name in enumerate(camera_names):
        remappings.append((f'camera_{i}/depth/image', f'{name}/depth/ground_truth'))
        remappings.append((f'camera_{i}/depth/camera_info', f'{name}/left/camera_info'))
        remappings.append((f'camera_{i}/color/image', f'{name}/left/image_raw'))
        remappings.append((f'camera_{i}/color/camera_info', f'{name}/left/camera_info'))
    if mode is NvbloxMode.people_segmentation:
        remappings.append(
            ('camera_0/mask/image', '/semantic_conversion/front_stereo_camera/semantic_mono8'))
        remappings.append(('camera_0/mask/camera_info', '/front_stereo_camera/left/camera_info'))
    if lidar:
        remappings.append(('pointcloud', '/front_3d_lidar/point_cloud'))
    return remappings