Integrating Isaac Sim with a Python Middleware for local & remote (headless) use as a simulator connected to custom controllers

Hello from Nimble One ๐Ÿ‘‹
We have an important question regarding the integration of Isaac into our software, as a robotic simulator.

Our goal :

  1. Have Isaac Sim instantiated and controlled with a python process, creating the simulation, and stepping the physics it as we see fit (ie. in sync with our controller). This python process would communicated through our middleware with other process like our controller.
  2. Ideally this would also work over the network, with Isaac running on another (more powerfull) machine, with the visual feedback getting transmitted somehow (using the websocket kit ?).
  3. Ideally this would also be with Isaac installed with the docker container method to allow easy packaging & upgrades.

The problems :

  1. Isaac comes with its own python distribution (3.7), not compatible with ours (3.8).
  2. Our software is already packaged in a docker container. โ€œMergingโ€ the two is doable (if you ignore the python clash) but painful to maintain.

Solutions ?
The cleanest solution would therefore be to have isaac installed in its own docker container and our software in another.
What about the interface then ?

With our current understanding of how omniverse is built, the following solution is what we have in mind :
Have Isaac installed in a docker, and make a custom kit app with the python kit app and websocket kit app merged into one.
โ†’ This way we would have the ability to start isaac with a python process, communicating over network to other process (over network so eventually processes on another machine).
โ†’ And we would have visual feedback and UI interactivity through the websocket UI.

Is that the way we are supposed to achieve our goals ?

Hi! Very sorry for the delay in responding here. From your description, I believe that what you are trying to do is possible. You should be able to replace the built-in Python distribution with your own 3.8 system. Communication between containers should be possible, but you might need to write a custom extension to do that. In case you havenโ€™t seen the documentation related to the Python integration I have included some links here. I will speak to some experts on our team and come back to you with more specific advice. I believe we had a tutorial or example regarding replacement of the default Python system with your own.

(Python Environment โ€” Omniverse Robotics documentation)

(Python Environment โ€” Omniverse Robotics documentation)

1 Like

Thanks !
So far we are using python code injection using http.server, very simple but not necessarily the intended way.

Iโ€™ve been informed by the lead engineer that we will offer Python 3.8 capability later this year, but at present weโ€™re stuck with 3.7 because of the bindings for the modules comprising the application. However you can replace the internal environment with your own via conda etc.