I am running isaac_ros_bi3d GEM on realsense D455 infra pair in AGX Orin. The realsense camera has an infra stereo pair input. The output does not look as good as the output for a rgb stereo pair.
The /camera/infra1/image_rect_raw and /camera/infra2/image_rect_raw were first flipped and converted into rgb8 using ImageFlipNode and ImageFormatConveterNode respectively before passing to bi3d node. Please find a sample infra1 raw image and disparity output from the bi3d model below,
The output Occupancy Grid from the isaac_ros_freespace_segmentation is also blank. Does the performance gets affected with the infra images? Do you recommend a general setting for realsense infra images?
Note: I’m running the GEM’s on a input rosbag recorded from a robot already equipped with Realsense D455 (ros1 noetic). I have not installed realsense using the dockerfile from ISAAC ROS.
Please let me know if you need more information from my side.
You could also replace your rosbag with the one provided in step 3 of the Freespace Segmentation Quickstart (bi3dnode_rosbag). This would help narrow it down to an input data/preprocessing issue.
The setup works as expected with the bi3d_rosbag provided by ISAAC. The provided rosbag however has rgb stereo input. In my case, its an infra image stereo pair with height=480 and width=848.
Is there a rosbag povided by ISAAC with infra stereo pair input? If yes, I can run my setup on that.
I tried to visualize occupancy grid from the rgb_storage bag. And, the output I obtain (attached) does not have a full range of view as shown for the RGB pair.
After modifying the launch file with correct topic re-mappings and image dimensions, I have 4 nodes ( Bi3DNode, ImageFormatConverterNode(left image), ImageFormatConverterNode (right image) and FreespaceSegmentationNode. However, the output obtained is just a square which is not intuitive.
Do you have any pointers why is this the case? The tf2 frame corresponding to the camera optical center in this case is the D455_1:left_ir, right?
Note: I run this to publish a static TF for the camera frame: ros2 run tf2_ros static_transform_publisher 0 0 0.1 1 0 0 0 base_link D455_1:left_ir
Hi! Bi3D is trained on rgb8 color images, so mono8 grayscale will result in degraded model performance. Could you please elaborate on the use-case and why infra images are required instead of rgb8 for input?
I am using the realsense D455 camera which only provides infra stereo pair images. This tutorial here shows the results of freespace segmentation on infra stereo pair from Realsense camera, right? As Realsense D455 is listed in the supported cameras from ISAAC, I assumed the freespace segmentation also works as for RGB color images.
