CPU bottleneck

Isaac Sim Version

4.2.0
4.1.0
4.0.0
2023.1.1
2023.1.0-hotfix.1
Other (please specify):

Operating System

Ubuntu 22.04
Ubuntu 20.04
Windows 11
Windows 10
Other (please specify): A cloud VM running Ubuntu 22.04 and Isaac Sim in a docker container.

GPU Information

  • Model: 2xL40
  • Driver Version: 535.183.03

CPU information

  • AMD EPYC 7763 64-Core Processor | Cores: 60 | Logical: 60

Topic Description

Detailed Description

When running the quadruped from the examples, I noticed a CPU bottleneck. This was confirmed using the profiler, and also by setting the lowest rendering resolution and still getting the same fps.

For reference, I used the same container on my local machine with an RTX 3060 Ti and an AMD Ryzen 5800x CPU, and the performance was significantly better (~15 more fps) on the local machine for the same resolution.
I also had the issue on this local machine, but was able to mitigate it by setting the cpu governor to performance (was on powersave by default). But this cannot be done on the VM since it has to be changed on the host.

Is there a workaround to avoid this CPU bottleneck ?

Thanks

Steps to Reproduce

  1. Run the Isaac Sim docker
  2. Run the isaac-sim.headless.native.sh
  3. Open Isaac Examples > Quadruped

Screenshots or Videos

htop

profiler

Additional Information

What I’ve Tried

Here is what I tried without success:

Additional context

Native Livestream is enabled, and I connect to the sim using the streaming client.

Could you please verify whether this issue is still present in Isaac Sim version 4.5.0?

Hi, although I am not the original poster, I can confirm that with Isaac Sim version 4.5.0 and a similar server platform (AMD 7763 CPU, docker, etc), this issue is still present.

Once the GUI of Isaac Sim is started, it creates way too many threads (256 in my case) and consumes 20% of the total 256 cores, which is a bit confused for me.

If I limit the FPS in the developer settings to 15 fps, the CPU usage goes down a bit. However, I think it’s just a walkaround than a perfect solution.

I would be glad if you developers are interested in this issue and I am willing to provide more info if it helps. Thanks!

Hi @User12003301 and @hammond-tang,

Apologies for the long silence on this.

This issue has been partially addressed. Starting with Isaac Sim 5.0, SimulationApp automatically caps thread counts to min(cpu_cores, 32) via the limit_cpu_threads config parameter (defaults to 32). This prevents the runaway thread spawning
on high-core-count server CPUs.

If you’re using SimulationApp from Python, this is already handled:

  # Default: capped at 32 threads                                                                                                                                                                                                                  
  app = SimulationApp({"headless": True})                                                                                                                                                                                                          
                                                                                                                                                                                                                                                   
  # Or set a custom limit                                                                                                                                                                                                                          
  app = SimulationApp({"headless": True, "limit_cpu_threads": 16})                                                                                                                                                                                 

However, if you’re launching via isaac-sim.sh or isaac-sim.headless.native.sh directly (which was your workflow), the .kit app files don’t include this cap. You’ll need to pass the flags manually:

  ./isaac-sim.headless.native.sh \                                                                                                                                                                                                                 
    --/plugins/carb.tasking.plugin/threadCount=12 \                                                                                                                                                                                                
    --/plugins/omni.tbb.globalcontrol/maxThreadCount=12                                                                                                                                                                                            

Additional tuning for server CPUs:

  • –/physics/numThreads=4 — cap PhysX threads
  • taskset -c 0-15 ./isaac-sim.sh … — pin to a core subset for better cache locality on multi-NUMA systems

For more details on modifying Carb settings, see the Modify Carb Settings (Modify Carb Settings — Isaac Sim Documentation) documentation.

Closing as the core issue is addressed in 5.0+ for SimulationApp, with manual flags available for shell launches.