Captured devices with WebRTC

Omniverse Isaac Sim
,
1

I have a server that provides Isaac Sim 5.1 over WebRTC.

I can successfully use my keyboard and mouse with my client.

However, the Kaya Gamepad example is not working with my gamepad (I have tested that the gamepad is working), even though the WebRTC extension documentation states:

Input Handling: Processing of keyboard, mouse, and gamepad input from clients

I would like to ask the following:

  1. Are gamepads supported by the WebRTC extension used by Isaac Sim?
  2. If not, is there any way to enable it?
  3. I would also be interested in controlling a simulated SO-101 follower in my server by capturing the joints of my leader. Would it be feasible for me to add this functionality, if not provided?
3

Gamepad should work automatically after being connected to the system. No extra setup is required. Can you check if it is recognized by the client system?

For #3, what you try to do is unclear to me. Could you elaborate?

4

@lhuang thanks for the reply. Indeed, I can successfully use the gamepad in the client system (I checked that with AntiMicroX).

As of #3, I would like to do the following:

  • Have a real SO-101 robot next to me
  • Have a simulated S0-101 robot in an Isaac Sim server
  • Teleoperate the simulated robot based on the movements of the real one

That is, I would need to send a custom message over WebRTC the same way that a keyboard or a gamepad does, by reading the joint positions of the real robot. I assume this is not possible by default, and wonder if I could do some kind of development to achieve this.

Hope it is clear now.

5

Thanks for the detail! Had a discussion with the dev team. Teleop Core device support will be available on Isaac Lab 3.0. release. It should be able to support your requirements to teleoperate the simulated robot based on the movements of the real robot.

6

Great! Would you be able to tell which version of Isaac Sim would that correspond to?

Just for clarification, would both gamepad and teleoperation support be native for that release?

7

It might be Isaac Sim 6.0 GA.

And yes, that should be the case.