Improving RTF with high physics time steps

[leolamigo0628]

6.0.0
5.1.0
5.0.0
4.5.0
4.2.0
4.1.0
4.0.0
4.5.0
2023.1.1
2023.1.0-hotfix.1
Other (please specify):

Operating System

Ubuntu 24.04
Ubuntu 22.04
Ubuntu 20.04
Windows 11
Windows 10
Other (please specify):

GPU Information

• Model: RTX 5080 16GB VRAM
• Driver Version: 580.126.20

CPU Information

• Model: Intel Ultra 7 265KF

Topic Description

I’m running the OHT project, with multi OHT and rails. I try to keep RTF close to 1.0. However, it consistently remained between 0.5 and 0.75.

Detailed Description

In the stage, there are 5 OHT on the rails. OHT includes the front wheel module and the rear wheel module. Each wheel module have two main rubber wheels at the top of rail, four guide plastic wheels inside the rail. Every wheel is using the Shape-Cylinder collider. Rails are use the convex hull as a collider.

To make the OHT run stably on the track, I have tried many physics scene parameters. Specifically, I tested time steps of 60, 90, 100, 120, and 150 per second, and also adjusted the minimum position and velocity iteration counts. Finally, I set 150 time step per second and 2 minimum position and velocity iteration counts. The current parameters allow the OHT to run smoothly.

Each OHT has 3 Action Graphs to be controlled by ROS 2 Humble, and 1 Action Graph to publish TF transforms. The OHT doesn’t have any cameras.

After getting it to run, I’m confused about why the OHT is moving so slowly on the viewport. I observed the Real Time Factor published via a ROS topic. However, I noticed that the RTF was only hovering between 0.3 and 0.4. I then followed the steps in the Isaac Sim Performance Optimization Handbook to see if I could improve it. Unfortunately, it only reached a maximum of 0.5 to 0.6. If I use Python to launch Isaac Sim, the RTF can reach a maximum of 0.7 RTF.

I would like to ask if there are any other methods to stably maintain the RTF at 1.0 under my current environment, or if you have any recommendations on how to further tune the physics scene parameters.

An Articulation Root has been added to each OHT’s base_link.

[VickNV]

Hi @leolamigo0628,

Thank you for the detailed report with your physics scene settings and screenshots.

The RTF of 0.3–0.7 you’re observing at 150 Hz with 5 articulated OHTs is a known performance characteristic when running high physics rates with multiple articulated bodies. Each physics step has a minimum compute cost that becomes the bottleneck when the per-step simulated time (1/150 = 6.67 ms) is smaller than the per-step wall-clock cost.

What you can do now to improve RTF:

1. Reduce physics rate to 90–100 Hz. This is the single highest-impact change. At 100 Hz you should see RTF close to or above 1.0. To maintain stability at the lower rate, increase solver iterations:

• Position iterations: 4
• Velocity iterations: 2

2. Disable articulation self-collisions on each OHT
   (Property panel → PhysxArticulationAPI → Enabled Self Collisions = False).
   For rail-guided vehicles this is safe and reduces contact pair generation.
3. Audit guide wheel colliders. If not all 8 guide wheels per OHT need active collision (e.g., some are redundant for lateral constraint), disabling collision on a few will reduce per-step cost.
4. Simplify rail colliders where possible — box approximation is cheaper than convex hull for long straight sections.
5. CPU governor (if applicable):
   sudo cpupower frequency-set -g performance
6. Python launch gives better RTF than GUI — you’ve already confirmed this. For digital twin sync, running headless with disable_viewport_updates=True is optimal.

The Performance Optimization Handbook covers rendering and asset optimization well, but doesn’t currently document the physics-rate-vs-articulation-count trade-off you’ve hit. We appreciate you surfacing this — it helps us understand what the industrial automation community needs from the simulation performance side.

Let us know if reducing to 100 Hz maintains acceptable stability for your OHT rail contacts.

[leolamigo0628]

I have followed the steps as suggested.

Here are the adjustments I’ve made so far:

1. Physics rate: 100 Hz, Position iterations: 4, Velocity iterations: 2

2. Disabled self-collisions.

3. I couldn’t reduce the number of guide wheels (currently 4) because they are required to keep the OHT (Overhead Hoist Transport) moving along the tracks.

4. The collisions for all the track segments are now generated using Shape -> Cube.

5. This method was already in use previously.

As a result, the simulation runs very unsmoothly, and I am encountering the following issues:

1. At some turns, there are suspended sections where the OHT relies on small upper wheels for control. These small wheels support the OHT to prevent it from derailing. However, in these areas, the wheels clip straight through the meshes (penetration), causing the OHT to lose control.

2. The straight sections consist of multiple track segments, all spliced together with Shape -> Cube collisions without any gaps. Yet, when the OHT moves across these joints, it stutters (as if it’s getting caught on something) and bounces.

3. Because the curved sections are irregular shapes, I used Triangle Mesh for their collision generation. When the OHT passes through these areas, it slows down significantly, as if there is a strong resistance blocking its forward movement.

4. Likely due to the OHT derailing, the RTF (Real Time Factor) drops quite a bit.

Interestingly, these issues did not happen with my previous settings (Physics rate: 150 Hz, Position iterations: 2, Velocity iterations: 1). I also tried applying all your suggested parameters first, but then simply changing the Physics rate back to 150 Hz makes it run smoothly again.

Could you please help me identify what the root cause might be and how I should adjust the settings?

Image 1: nvidia-smi, htop, and RTF when running via Python API with Physics rate: 150 Hz, Position iterations: 2, Velocity iterations: 1.

Image 2: Profiler when running via ./isaac_sim.sh with Physics rate: 150 Hz, Position iterations: 2, Velocity iterations: 1.

Image 3: nvidia-smi, htop, and RTF when running via Python API with Physics rate: 100 Hz, Position iterations: 4, Velocity iterations: 2.

Image 4: Profiler when running via ./isaac_sim.sh with Physics rate: 100 Hz, Position iterations: 4, Velocity iterations: 2.

I apologize that, for certain reasons, I am unable to upload a video recording of the simulation runtime. Thank you for your understanding.

[leolamigo0628]

Where can I find documentation or tutorials to make the simulation environment run more smoothly?
Are there any optimization strategies?

[VickNV]

Hi @leolamigo0628,

Thank you for the detailed follow-up with profiler screenshots and the thorough testing of the recommended settings. The stability issues you’re seeing at 100 Hz are expected and help clarify the trade-off in your scenario.

Why 100 Hz Causes Instability for Your OHTs

Your OHT rail system depends on high temporal resolution (many contact checks per second) more than high solver iteration counts. Here’s what’s happening at each problem area:

1. Wheel penetration at suspended sections: At 100 Hz (10 ms steps), your small guide wheels travel further between collision checks. If a guide wheel radius is ~2-3 cm, it can move a significant fraction of its radius in a single step at speed, allowing it to tunnel through thin rail geometry. Higher solver iterations don’t help – they improve constraint convergence, not collision detection frequency.

2. Stuttering at rail segment joints (box collider seams): This is a known PhysX behavior with adjacent box colliders. When a cylinder wheel rolls across the seam between two boxes, it can catch on the internal edge between them. At 150 Hz this is imperceptible because the wheel barely moves between steps, but at 100 Hz the larger per-step displacement makes the edge-catching pronounced.

3. High friction on curved triangle mesh sections: Triangle mesh colliders generate contact normals per-triangle-face. On a curved surface, the contact normal changes discontinuously at triangle edges, creating artificial “speed bumps.” Higher physics rates mask this because smaller displacements mean smoother contact normal transitions between frames.

Bottom line: 150 Hz is the correct physics rate for your use case. The challenge is making it fast enough for RTF 1.0.

Optimizations to Improve RTF at 150 Hz

Collision geometry changes (highest impact):

• Replace convex hull rail colliders with SDF (Signed Distance Field) colliders for your static rails. SDF colliders provide smooth contact normals on curved geometry and eliminate the triangle-edge friction problem entirely. Since your rails are static, SDF is the ideal collider type. In the Property panel: Collision > Approximation > SDF Mesh.

• Merge adjacent rail segments into fewer, longer meshes. Instead of many small box-collider segments spliced together, merge each straight section into a single mesh. This eliminates the seam-catching problem and reduces the total number of broadphase collision pairs.

• Consider using sphere colliders for guide wheels instead of cylinders. Sphere-vs-mesh contact is both cheaper and generates smoother contact normals than cylinder-vs-mesh.

Physics settings:

• Disable scene query support if you are not using raycasts or overlap queries:

  physx_scene_api.CreateEnableSceneQuerySupportAttr(False)
  

• Try single-threaded physics. With only 5 OHTs, thread synchronization overhead may exceed the parallelism benefit:

  --/persistent/physics/numThreads=0
  

• Enable Fabric to avoid expensive USD readback each step:

  carb.settings.get_settings().set_bool("/physics/fabricEnabled", True)
  

Action Graph optimization:

• You have 4 Action Graphs per OHT (3 ROS 2 + 1 TF) = 20 graphs total. Each graph adds per-step evaluation overhead. Consider:
  • Combining the 3 ROS 2 control graphs into a single graph per OHT
  • Using Lazy Graph evaluation so graphs only execute when they have new ROS messages
  • Publishing TF at a lower rate (e.g., 30 Hz) rather than every physics step

If RTF still cannot reach 1.0:

If after these optimizations you’re at RTF 0.7-0.8 but not 1.0, you have two options:

1. Accept a fixed sub-1.0 RTF and compensate in your ROS 2 controller. Many digital twin deployments run at a stable RTF < 1.0 and scale controller timing accordingly. A stable 0.7 RTF is more useful than an unstable 1.0.

2. Reduce the wheel count by checking whether all 4 guide wheels per module are mechanically necessary for lateral constraint. If 2 guide wheels per module provide sufficient stability, cutting from 8 to 4 guide wheels per OHT would halve the guide wheel contact pairs.

Let us know how the SDF collider and mesh merging changes affect your RTF and stability. Those two changes together should have the largest impact.