Full pipeline tutorial for isaac_ros_foundationpose with custom fine-tuned YOLOV8 model, running on Jetson Orin AGX

Robotics & Edge Computing Jetson Systems Jetson Projects
, , , , ,
1

Hi everyone!

I just published a tutorial for the full isaac_ros_foundationpose pipeline with custom fine-tuned YOLOV8 model

Here you find the YOUTUBE VIDEO!

KEY FEATURES

The pipeline includes:

  • NVIDIA Isaac SIM 5.0 to generate synthetic data for fine-tuning the YoloV8s model and for a digital twin of the scene to publish a virtual RealSense camera stream and robot data to ROS2
  • Ultralytics to fine-tune the YoloV8s model
  • isaac_ros_foundationpose to estimate the 6D pose of the object, using custom fine-tuned YoloV8s for object detection
  • ManyMove to handle the ROS2 logic and motion planning leveraging MoveIt2 and BehaviorTree.CPP

Hardware:

  • NVIDIA Jetson Orin Agx Developer Kit for isaac_ros_foundationpose and ManyMove on ROS2 Humble
  • Laptop with RTX card for Isaac SIM

HIGHLIGHTS

  • All assets provided to complete the pipeline, from .usd files for SDG and scene to .obj and mesh for FoundationPose
  • Example executable with ManyMove with behavior tree nodes to rectify FoundationPose output to allow grasping of symmetric objects and to limit pose validity to a specific bounding box: these features stabilize the output and enhance reliability on bin picking applications.
  • Full Isaac Sim scene with Ufactory Lite6 cobot and gripper, customized to provide a realistic pneumatic gripper simulation while keeping coherent ROS2 interaction and MoveIt2 path planning.

LINKS

1 Like