Issue with Environment spacing and PxgDynamicsMemoryConfig::foundLostAggregatePairsCapacity

Autonomous Machines Robotics - Isaac Isaac Gym
1

I have an simulation with 4000 environments in parallel. Every environment contains one robot in its own separate collision group. The zeroth environment contains an additional 29205 rigid bodies (boxes) in collision group -1 (serving as terrain for all the robots). I am using preview release 2 on Ubuntu 20.04.

The coordinate systems of all of the environments are aligned. i.e. this is how I initialize my environments:
upper = g.Vec3(0.0, 0.0, 0.0)
lower = lower = g.Vec3(0.0, 0.0, 0.0)
envs_per_row = 1
env = self.gym.create_env(self.sim, lower, upper, envs_per_row)

Whenever I start training my RL agent, I get the following error:

/buildAgent/work/f3416cf82e3cf1ba/source/gpubroadphase/src/PxgAABBManager.cpp (904) : invalid parameter : The application needs to increase PxgDynamicsMemoryConfig::foundLostAggregatePairsCapacity to 328226000 , otherwise, the simulation will miss interactions

This error occurs if I run on a 1080Ti (with 12 Gb VRAM)or an A40 (with 40 Gb VRAM). In either case, Iā€™m not close to running out of memory (program only uses about 7 Gb VRAM).

However, this error goes away if I simply space my environments out, even just by one centimeter. i.e.
upper = g.Vec3(0.0, 0.01, 0.0)
lower = lower = g.Vec3(0.0, 0.00, 0.0)
envs_per_row = 1
env = self.gym.create_env(self.sim, lower, upper, envs_per_row)

In doing so, I encounter another strange behavior: the first two environments spawn on top of each other. All the environments have the same x-position and are 0.01 m apart in the y-dimension, except for the first two environments, which are right on top of each other.

My questions:

  1. what is PxgDynamicsMemoryConfig::foundLostAggregatePairsCapacity? Does it have to do with the PhysX param max_gpu_contact_pairs? How can I increase this buffer? Why does setting envs on top of each other cause problems with it?
  2. Why do the first two envs spawn on top of each other, even when upper is nonzero?

My work around is just to space out my envs and dealing with the identical position of the zeroth and first env manually in my code.

2

Iā€™m getting a similar error, it does not stop the simulation from running though, and I canā€™t tell if it actually is effecting the simulation outputs.

3

I have been testing different settings for the sim on GPU, and found that the following line appears to prevent that PxgDynamicsMemoryConfig warning:
sim_params.physx.max_gpu_contact_pairs = sim_params.physx.max_gpu_contact_pairs*20
My guess is that the solver limits how many contacts it considers in the batch physics computations based on max_gpu_contact_pairs, but hopefully future documentation will confirm or clarify this. Not sure why spacing out the environments would effect this, though.

It seems like a reasonable idea to also increase the default_buffer_size_multiplier based on this other isaacgym forum thread,

Hope this helps.

1 Like
4

Thanks for your response. I believe I may have tried that, but found that increasing max_gpus_contact_pairs greatly increases the amount of VRAM that IssacGym uses. Was that the case for you?

5

You are right, it does appear to increase the amount of VRAM used, even with a constant number of actors and all other parameters kept constant. Maybe it pre-allocates a contact buffer in GPU memory based on this parameter, and then if there are more contacts than would fit in the buffer, it drops the remainder. That would be consistent with the warning the simulation will miss interactions, but does not explain why spacing out the environments would impact it.

6

Makes sense. Just wondering, are you using preview release 2 as well?

7

I am currently using 1.0.preview3 (Version: 1.0rc3) downloaded Feb.3.

8

Hi, yes increasing max_gpu_contact_pairs will allow physx to allocate more memory for contacts.

9

Hey. I want to follow up on this. I have an application in which agents are moving around in a large scene and it is convenient to initialize all agents in the same location.

However, doing this leads to the same issues @jcoholich and @jwhitman were seeing. I also find that initializing all agents in the same location drastically slows down the simulation (5-10x).

My understanding is that each environment should be independent of each other, so it is concerning that placing agents/environments in the same location results in more contact computations. My guess is that the simulation is precomputing some information about contact pairs of objects that are close to each other, even if they exist in separate environments.

My question: Is there a ā€˜correctā€™ way to place a large number of agents in the same location that resolves this issue?

10

Hi,

I got around this issue by initializing actors at different locations and then resetting them at (slightly) different locations (every time I reset my environment), but not at the same point. I found that spawning them at the same location (even when in different environments, with environment spacing set to zero) causes missed collisions without increasing maximum gpu contact pairs. Let me know if you find a solution to spawning them at exactly the same location.

1 Like
11

Interesting suggestion! I tried resetting the agents to the same location and got the same result as initializing them in the same location. Resetting to slightly different locations does fix the contact issue, but doesnā€™t solve my need of placing all robots in the same location.

Iā€™ll post if I find a solution!

12

Sorry to revive this thread, but I am facing the exact same problem and encountered an additional issue. Any non-zero environment spacing makes environments behave different, even if the environments are supposed to be exactly the same. However, setting it to zero makes the simulation slow. Has anyone encountered this same problem as well?

See thread Different results based on environment spacing

13

Hi @aart.stuurman,

While the simulations donā€™t interact with one another, the parts of PhysX that handle collision run first, and have to assume collisions between all the objects if they are all in the same spot before these are filtered out.

Iā€™d recommend spacing the actors at reasonable distances, like we do in the various examples.

Iā€™ll respond in your other thread separately.

Take care,
-Gav

1 Like