Use SAM for Initial Mask as Input to FoundationPose in Isaac ROS instead of SynDETR

[xyzuzyasidoado]

Hello everyone!

In Tutorial for FoundationPose with Isaac Sim — isaac_ros_docs documentation it is stated that one can replace the SyntheticaDETR initial object detection with other models like SAM/Yolov8 to enable first frame bbox generation for FoundationPose as an input to be able to use tracking for all subsequent frames. Is it possible to help me in editing the launch file isaac_ros_foundationpose_isaac_sim.launch.py that is located in isaac_ros_pose_estimation/isaac_ros_foundationpose/launch/isaac_ros_foundationpose_isaac_sim_tracking.launch.py at main · NVIDIA-ISAAC-ROS/isaac_ros_pose_estimation · GitHub for FoundationPose tracking with SAM/Yolov8 instead of SyntheticaDETR?

I found Tutorial for Segment Anything with Isaac Sim — isaac_ros_docs documentation to use as a starting point and the corresponding launch file isaac_ros_segment_anything_isaac_sim.launch.py at isaac_ros_image_segmentation/isaac_ros_segment_anything/launch/isaac_ros_segment_anything_isaac_sim.launch.py at main · NVIDIA-ISAAC-ROS/isaac_ros_image_segmentation · GitHub and tried to substitute SyntheticaDETR with the SAM nodes with no success.

Best regards
xyz

[ashwinvk]

@xyzuzyasidoado

o be able

One of the inputs for foundation_pose is the segmentation mask on the topic. Can ensure this is being published with exact time synced timestamps with respect to the other inputs? Can you share a screen shot of your ROS nodes using rqt_graph.

[xyzuzyasidoado]

Hello @ashwinvk
thanks for your answer and sorry for the late response!

The tip to use rqt_graph helped me a lot in understanding the launch files and what nodes they are starting and to what topics they are subscribed to. I noticed that the 2d detection topic wasn’t configured correctly in my configuration. I overlooked the API documentation for that completely.

If I encounter new errors, I will start a new topic for this.