I’m working on a thesis project involving humanoid chess manipulation with a Unitree G1 robot in Isaac Lab Arena. My goal is to collect teleoperation demonstrations using a Meta Quest 3 headset to build a dataset for VLA post-training (GR00T pipeline).
My current setup:
Isaac Lab Arena with IsaacSim 5.1
G1 WBC joint embodiment, fixed-base, with a chess board environment
Ubuntu Linux with NVIDIA GPU
Meta Quest 3 headset available
I’ve seen several possible paths for Quest 3 teleoperation:
CloudXR Early Access — I understand Quest 3 is supported via the CloudXR EA program. Is this the recommended path? How do I apply for access?
ALVR + SteamVR + OSC — The community solution described in PR #4419 and the simintel blog post using body tracking via OSC protocol. Has anyone gotten this working with the G1 specifically?
IsaacLab’s built-in OpenXR + G1 retargeters — IsaacLab already has G1TriHandUpperBodyRetargeterCfg and G1LowerBodyStandingRetargeterCfg. Can these work with Quest 3 hand tracking via CloudXR, or are they only tested with Apple Vision Pro?
My specific needs are:
Hand tracking (both hands) for upper body control of the G1
Fixed-base setup (no locomotion needed)
Ability to record demonstrations for dataset collection
Working with IsaacSim 4.5 / Isaac Lab Arena
What would be the recommended approach for getting Quest 3 teleop working with the G1 in the current ecosystem? Any guidance on the CloudXR EA program timeline for Quest would also be appreciated.
Thank you for your interest in Isaac Lab. To ensure efficient support and collaboration, please submit your topic to its GitHub repo following the instructions provided on Isaac Lab’s Contributing Guidelines regarding discussions, submitting issues, feature requests, and contributing to the project.
We appreciate your understanding and look forward to assisting you.
Just to share some practical feedback — I’ve actually tested the next setup with the Unitree G1 in Isaac Lab, and it works.
I was able to run the official Isaac Lab teleoperation examples from:
using:
ALVR v20.14.1
SteamVR (OpenXR runtime)
Meta Quest 2
Isaac Sim 5.1 + Isaac Lab
G1 upper-body retargeting
I successfully controlled the G1 upper body in a fixed-base setup using VR hand tracking.
Here is a short demo video:
Given that Quest 3 has improved tracking and hardware compared to Quest 2, I would expect it to work at least as well using the same ALVR + SteamVR pipeline — without necessarily requiring CloudXR Early Access.
For fixed-base teleoperation and dataset collection, the ALVR + SteamVR route seems fully viable in the current ecosystem.
I managed to get things working using the CloudXR beta. I initially tried SteamVR but didn’t have much luck, although I also tried a slightly different approach than what you suggested. I’m going to give your method another shot.
Quick question—do you have any experience using Groot in Isaac Lab? I have a few workflow questions and can go more in-depth about what I’m trying to do and where I’m currently getting stuck, if that’s okay.
Oh yes — I’m familiar with Arena. It’s actually the only documentation I’ve found that shows a full end-to-end workflow for implementing Groot.
Right now I’m trying to modify the environment so it works for my chess problem with the G1. I’ll admit I’m struggling a bit to rework the repo and properly adapt it for chess. I also wasn’t able to get teleoperation working there, so for the moment I’m using the standard Isaac Lab setup until I can figure that part out.
