Poor Performance of Robot Loading and Simulation

Isaac Sim Version

4.2.0
4.1.0
4.0.0
2023.1.1
2023.1.0-hotfix.1
Other (please specify):

Operating System

Ubuntu 22.04
Ubuntu 20.04
Windows 11
Windows 10
Other (please specify):

GPU Information

  • Model: NVIDIA GeForce RTX 4090
  • Driver Version: 535.183.01
  • CUDA: 12.2

Topic Description

Detailed Description

I’m attempting to load and simulate robots. However, when I try to run a large number of robots (such as 2048 or 4096 environments) in parallel, the process becomes very slow.

The average simulation speed is 432.75 steps per second, meaning it takes 5 to 6 seconds to complete one simulation step. The loading time is also very slow, it takes 262.78 seconds to load 2048 h1 robots.

I expect each simulation step of 4096 robots to take less than 1 second. I have tried both the h1 and go2 robots, but they are both very slow.

Is this simulation speed normal? Where might the problem be?

Key code

I’m using SimulationApp to launch Omniverse Toolkit

Launch Code:

        simulation_app = SimulationApp({"headless": True})

...
        self.world = World(stage_units_in_meters=1.0)
        
        # Add default ground plane
        ground_plane_path = '/World/ground_plane'
        self.world.scene.add_default_ground_plane(prim_path=ground_plane_path)
        collision_filter_global_paths = [ground_plane_path]

        # Add light
        create_prim("/DistantLight", "DistantLight")
    
        # Add the source robot to the world
        source_prim_path = f'/World/envs/{self.prim_path_expr}_0'
        add_reference_to_stage(usd_path=self.usd_path, prim_path=source_prim_path)
        
        # Add cloner
        self.cloner = GridCloner(spacing=self.spacing)
        
        # Generate paths for clones
        target_paths = self.cloner.generate_paths(f"/World/envs/{self.prim_path_expr}", self.num_envs)

        # Clone
        self.position_offsets = np.array([[0, 0, self.initial_z_translation]] * self.num_envs)
        self.cloner.clone(
            source_prim_path=source_prim_path,
            prim_paths=target_paths,
            position_offsets=self.position_offsets,
            replicate_physics=True,
            base_env_path="/World/envs",
        )
        
        # Filter the collision groups
        self.cloner.filter_collisions(
            physicsscene_path=self.world.get_physics_context().prim_path,
            collision_root_path="/World/collisions",
            prim_paths=target_paths,
            global_paths=collision_filter_global_paths
        )

        # Wrap all robots
        self.system = RobotView(prim_paths_expr=f"/World/envs/{self.prim_path_expr}_*", 
                                name=f"{self.prim_path_expr}_view")
        self.world.scene.add(self.system)
        self.world.reset()

Step code:

action = ArticulationActions(joint_positions=actions)
for i in range(self.sub_control_frequency):
        self.system.apply_action(action)
        self.world.step(render= False)

Error Messages

I came across a possible warning that might be the cause, but I can’t find any related information about it.

2024-11-21 02:27:57 [33,186ms] [Warning] [omni.physx.plugin] Large amount of replications running inefficiently with USD updates enabled.

Additional Information

What I’ve Tried

I conducted an experiment to compare the performance of 512-robot simulations with and without collision avoidance.

The results are as follows:

  • Without collision avoidance: 428.45 simulation steps per second
  • With collision avoidance: 297.67 simulation steps per second

While collision avoidance is beneficial, it is not the primary factor affecting performance.

Related Issues

(If you’re aware of any related issues or forum posts, please link them here)

Additional Context

(Add any other context about the problem here)

Why do you need such large number of environments? For RL? then use Isaac Lab?
If you insist of using the Cloner, have a look at Isaac Lab git. They are solving this problem too.

Thank you for your response! I think IsaacLab is also using the Cloner, so I am curious why IsaacLab performs well with so many environments, while mine performs poorly.

Solved:

self.world = World(device='cuda', backend='torch', stage_units_in_meters=1.0)

instead of:

self.world = World(stage_units_in_meters=1.0)