How to Integrate GR00T Inference with Isaac Sim for Visualization?

Omniverse Isaac Sim
1

Isaac Sim Version

5.1.0
☑ 5.0.0
4.5.0
4.2.0
4.1.0
4.0.0
4.5.0
2023.1.1
2023.1.0-hotfix.1
Other (please specify):

Operating System

Ubuntu 24.04
☑ Ubuntu 22.04
Ubuntu 20.04
Windows 11
Windows 10
Other (please specify):

GPU Information

  • Model: NVIDIA GeForce RTX 5080
  • Driver Version:570

Topic Description

How to interface GR00T with Isaac Sim for inference visualization and control?

Detailed Description

I’m currently working on integrating GR00T (a transformer-based humanoid robot foundation model) with Isaac Sim. My goal is to visualize and interact with GR00T’s inference results inside Isaac Sim, ideally through GUI or Extension-based interfaces.

I’ve already set up GR00T’s inference server outside Isaac Sim using a ZMQ client-server architecture. The inference works correctly, and I can obtain action predictions from GR00T. Now, I want to bring these predictions into Isaac Sim to control a humanoid robot and visualize the results.

I have the following questions:

  1. Is there an existing Extension or interface in Isaac Sim that supports integration with external inference engines like GR00T?
  2. What is the recommended way to send GR00T’s inference results (e.g., joint positions or actions) to Isaac Sim in real-time?
  3. Are there any examples or best practices for integrating transformer-based models with Isaac Sim for humanoid control?

Error Messages

N/A

Screenshots or Videos

N/A

Additional Information

What I’ve Tried

  • Verified GR00T inference server works correctly outside Isaac Sim.
  • Explored Isaac Sim Extensions but couldn’t find a direct interface for external inference engines.
  • Considered writing a custom Extension to receive ZMQ messages.

Related Issues

N/A

Additional Context

N/A

3

I also have the same query. I couldn’t find any documentation on the integration between Isaac Sim and GR00T.

1 Like
4

Hello, you may want to use IsaacLab for this. IsaacLab is designed for ML and will have the inference tools you need. Here’s a tutorial on using a trained policy for humanoid robot control: Policy Inference in USD Environment

For more advance examples specifically using Gr00t in IsaacLab check out: https://github.com/isaac-sim/IsaacLabEvalTasks

If you need greater abstraction, like via ZMQ as you suggest, you could also consider implementing a custom controller. However YMMV WRT ZMQ I/o performance.

HTH

1 Like