Communication between Isaac Sim and a custom local server

Hey guys,

I am developing the picking and placing of a robot depending on the results of an algorithm on a server (server on the local network). The robot and pick and place task is based on omni.isaac.manipulators. Also I have the USD scene file with a camera in it.

The ideal or final concept would be that the camera gets a frame, sends it to a local server and then the servers gives back the coordinates of picking (in a mqtt or rest communication). But as first step I want to being able to load a picture from Isaac Sim and then send it to server.

I was wondering which is the best path forward or methodology to use to achieve that goal.

  • Can I directly send a file to localhost port or local server from Isaac Sim?
  • Do I have to use an already created extension? (I have created a custom extension for the robot and the picking). Do I have to create a new one?
  • Do I need a custom connector?

If any of those possibilities is possible, how exactly would be the process?

Lately, I would like to apply Isaac Gym to test and check different positions and patterns.

Thank you very much for all the support and insight on the forums and I am sorry for my lack of knowledge.

Hi. Yes, you can read and write files to your local Nucleus.
See the omni.client api. We do have some examples in Isaac Sim Python code. omni.client is part of the Kit SDK.
You can set a custom Nucleus server URL too as the default instead of localhost.

Hi, thank you!

The omni.client.api also works with local servers that are not Nucleus?

No, it only works with Nucleus (omniverse://), S3 buckets (http://) and local disk (e.g. C:).

Take a look at the ROS bridge extension too.

Perfect, I will take a look at it.
Thank you so much.

1 Like