How do you turn a human demonstration into a training episode a robot can use?
RLWRLD’s technical walkthrough follows the steps needed to align robot actions, camera observations, and task instructions for dexterous manipulation:
- Capture the interaction. See why synchronized head-mounted and external camera views matter when hands or objects are occluded.
- Reconstruct hand motion and contact. RLHND uses NVIDIA Cosmos 3 video features to estimate hand pose, contact, and force.
- Retarget motion to a robot. Map the reconstructed interaction to the target robot’s joints and limits to produce action trajectories.
- Build matching observations. Replay those trajectories in NVIDIA Isaac Sim, align action labels for control latency, and export training episodes in LeRobot format.
The blog also explores using a world model to generate observations for garment packing and different robot configurations. These are research examples; evaluating their effect on policy learning and real-world task performance is the next step.
Explore the workflow and bring your questions:
📖 Read the Technical Blog
🤗 Explore the Cosmos 3 Model Collection
🧑💻 Explore NVIDIA Cosmos on GitHub
📺 Tune in to the Cosmos Labs Livestream on November 19 at 9 AM PT
Where does your human-to-robot data workflow get stuck: capture, retargeting, or matching observations to actions? Share your task, robot hand, and current blocker here or in the Cosmos Discord channel.