PhysX float32 precision limit — need nanometer-level positioning in Isaac Sim, rigid body freezes below ~1 μm

Omniverse Isaac Sim
1

Important: Isaac Sim support

Note: For Isaac Sim support, the community is gradually transitioning from this forum to the Isaac Sim GitHub repository so that questions and issues can be tracked, searched, and resolved more efficiently in one place. Whenever possible, please create a GitHub Discussion or Issue there instead of starting a new forum topic.

Note: For any Isaac Lab topics, please submit your topic to its GitHub repo ( GitHub - isaac-sim/IsaacLab: Unified framework for robot learning built on NVIDIA Isaac Sim · GitHub ) following the instructions provided on Isaac Lab’s Contributing Guidelines ( Contribution Guidelines — Isaac Lab Documentation ).

Please provide all relevant details below before submitting your post. This will help the community provide more accurate and timely assistance. After submitting, you can check the appropriate boxes. Remember, you can always edit your post later to include additional information if needed.

6.0.0
5.1.0
[ ] 5.0.0
4.5.0
4.2.0
4.1.0
4.0.0
4.5.0
2023.1.1
2023.1.0-hotfix.1
Other (please specify):

Operating System

Ubuntu 24.04
Ubuntu 22.04
Ubuntu 20.04
[ ] Windows 11
Windows 10
Other (please specify):

I’m using Isaac Sim as a digital twin to train an RL agent (PPO) for precision motion control of a 12-inch wafer AOI (Automated Optical Inspection) linear motor stage. In semiconductor/display manufacturing, AOI stages require nanometer-level positioning accuracy (typically ±10–100 nm) to align wafers or panels for high-resolution optical inspection. My goal is to use the Isaac Sim digital twin to train an RL controller that achieves this level of precision, so the learned policy can later be transferred to the real hardware.

The agent has successfully learned to position within the 1–10 μm tolerance band, but when I try to push accuracy below ~1 μm toward the target nanometer range (±100 nm or better), the rigid body appears to “freeze” — the applied force produces no measurable change in position. I suspect this is a float32 precision limitation in PhysX, but I’d like to confirm and ask if there are any workarounds that would allow nanometer-scale simulation fidelity.

3

Hi, You’ve correctly identified the issue — this is a float32 precision limitation in PhysX, not a bug. At nanometer scales, you’re hitting the representable precision limit of single-precision floating-point.

The workaround: Use scale transformation

Work at 1:1,000,000 scale where 1 mm in simulation represents 1 μm in reality. This keeps you in PhysX’s usable precision range (float32 has ~119 nm precision at meter scale, which becomes 0.119 nm at your scaled 1 μm scale).

# Define scale factor (1 simulation unit = 1 μm)
SCALE_FACTOR = 1e-6  # 1:1,000,000 scale

# When setting up your stage/prims:
# Work in millimeters in simulation (mm precision is achievable)
# but interpret 1 mm in simulation as 1 μm in real world

# Example: Target 100 nm real-world precision
target_position_real = 100e-9  # 100 nm in meters
target_position_sim = target_position_real / SCALE_FACTOR  # 0.1 mm in simulation

# Apply position in scaled space
prim.GetAttribute("xformOp:translate").Set((target_position_sim, 0, 0))

# When reading back for RL agent:
measured_position_sim = prim.GetAttribute("xformOp:translate").Get()[0]
measured_position_real = measured_position_sim * SCALE_FACTOR

This gives you effective sub-micron control while staying within float32’s usable precision range.

For RL training, I’d also recommend:

  1. Train at 1-10 μm precision (which you’ve achieved)
  2. Use domain randomization with position noise
  3. Fine-tune on real hardware with your actual nanometer-precision sensors

PhysX won’t give you true nanometer simulation, but with proper scaling and sim-to-real transfer, your learned policy should work on real AOI stages.