IsaacSim with WebRTC renders very slowly compared to the local mode

Omniverse Isaac Sim
, ,
1

Isaac Sim Version

4.5.0

Operating System

Ubuntu 22.04

GPU Information

  • Model: RTX A6000
  • Driver Version: 550.127.05

Topic Description

Our Isaac Sim simulations, which run smoothly on a local server, experience significant slowdowns in simulation progress when executed in headless mode and streamed via WebRTC.

Detailed Description

We primarily use Python scripts to simulate various scenarios in Isaac Sim. Our script architecture involves two main processes:

  1. Rendering Process: This process continuously calls SimulationApp.update(). It runs irrespective of active flags or signals, essentially simulating a static environment when no specific actions are being performed.
  2. Signal Process: This process is responsible for defining and executing the simulation scenario. It sends commands such as set_joint_velocities(), GetTargetVelocityAttr().Set(), or set_angular_velocity. Temporal pauses between operations (e.g., moving ‘robot A’ then turning it) are currently implemented using time.sleep(), as the rendering process is separated.

We are encountering the following issues when running the script in headless mode and connecting via WebRTC, which are not present in local, non-headless execution:

  1. Early Operation Starts: Subsequent operations in the Signal Process often begin before previous actions (e.g., a robot reaching its target position) have visually or physically completed in the simulation. While this is partly expected due to time.sleep() not being a robust synchronization mechanism, the problem is significantly exacerbated in headless mode. Increasing time.sleep() values temporarily mitigates this, but it’s not a sustainable solution and points to an underlying performance issue.
  • Note: Currently, we cannot use SimulationApp.update() for several iteration to wait for the finish of previous operation because we are using the separated restless rendering process.
  1. Severe Rendering Latency (Main Problem): The primary concern is the drastically reduced rendering speed compared to local execution. This slowdown directly contributes to the first problem, as the physical simulation time steps appear to increase due to the rendering bottleneck. We suspect that SimulationApp.update() might be incurring significant overhead by continuously sending every frame to the WebRTC client for streaming, thereby heavily impacting overall simulation performance.

We are seeking solutions or suggestions to address the slow simulation phenomenon when using headless Isaac Sim with WebRTC, particularly concerning rendering speed and its impact on simulation progression.

Steps to Reproduce

  • Please provide a minimal reproducible example (e.g., a simplified script that demonstrates the slowdown). This is crucial for debugging. For example:
    1. Start Isaac Sim in headless mode with WebRTC enabled.
    2. Load a simple scene (e.g., a single robot).
    3. Execute a script that continuously calls SimulationApp.update() in one process and attempts to move the robot with time.sleep() pauses in another process.
    4. Observe the rendering performance and the desynchronization between planned actions and visual completion via the WebRTC client.

(Add more steps as needed)

Additional Information

What I’ve Tried

We tried connecting the headless server with different client computers. But it shows the same phenomenon.

4

Thanks for bringing up this issue! Let me reach out to the internal team.

1 Like
7

There have been several updates to Isaac Sim, and it might be worth trying to reproduce this issue with the latest version. Have you had a chance to test your setup with Isaac Sim 5.0? It’s possible that some of the performance bottlenecks related to WebRTC streaming have been addressed in the newer release.

8

Hello!

We noticed that this topic hasn’t received any recent responses, so we are closing it for now to help keep the forum organized.

If you’re still experiencing this issue or have additional questions, please feel free to create a new topic with updated details. When doing so, we recommend mentioning or linking to this original topic in your new post—this helps provide context and makes it easier for others to assist you.

Thank you for being part of the NVIDIA Isaac Sim community.

Best regards,
The NVIDIA Isaac Sim Forum Team