Infrared vs RGB cameras for Isaac ROS Visual SLAM

Autonomous Machines Robotics - Isaac Isaac ROS
1

Hi, I would like to run the Isaac ROS Visual SLAM on the NVIDIA Carter robot with RealSense D455 cameras attached to it.

From this package’s launch files, the Isaac Sim version uses two RGB cameras'front_stereo_camera/left_rgb/image_raw' and 'front_stereo_camera/right_rgb/image_raw', while the RealSense version uses the Infrared camera outputs camera/infra1/image_rect_raw and camera/infra2/image_rect_raw.

Is there any way to align these, because ideally I want to train a VSLAM model in Isaac Sim with RGB and then apply it on the real robot with RGB images as well or train using Infrared and apply it using the actual RealSense camera Infrared outputs. (i.e. train and deploy on the same type of images)

I have searched around and it seems like RealSense D455 only has 1x RGB camera and thus only 1x RGB topics; also there seems to be no option for Infrared camera in Isaac Sim from this forum post. Any help will be greatly appreciated. Thank you.

2

Hi @ychua041

The best way is to create a new ROS 2 package and rewrite the remapping you want on the Realsense D455. This approach will provide more flexibility for the tests you are doing with the Nova Carter robot.

If you need more help, let me know.
Raffaello

3

Hi @Raffaello, thanks for the reply. Previously I rewrote the mapping in the Visual Slam launch file to use the same RGB topic from the camera, but it seems like it doesn’t work as I assume the package requires a pair of images at an offset.

As the training environment on Isaac Sim uses RGB inputs but the actual deployment of the package uses Infrared inputs, regarding creating my own ros2 package, would that mean creating a digital twin of infrared cameras on Isaac Sim itself? Would it be possible to point me to some resources for further information? Thanks!

4

Hi @ychua041

At this moment, it is not possible to simulate an infrared camera on Isaac Sim.