I have an simulation with 4000 environments in parallel. Every environment contains one robot in its own separate collision group. The zeroth environment contains an additional 29205 rigid bodies (boxes) in collision group -1 (serving as terrain for all the robots). I am using preview release 2 on Ubuntu 20.04.
The coordinate systems of all of the environments are aligned. i.e. this is how I initialize my environments:
upper = g.Vec3(0.0, 0.0, 0.0)
lower = lower = g.Vec3(0.0, 0.0, 0.0)
envs_per_row = 1
env = self.gym.create_env(self.sim, lower, upper, envs_per_row)
Whenever I start training my RL agent, I get the following error:
/buildAgent/work/f3416cf82e3cf1ba/source/gpubroadphase/src/PxgAABBManager.cpp (904) : invalid parameter : The application needs to increase PxgDynamicsMemoryConfig::foundLostAggregatePairsCapacity to 328226000 , otherwise, the simulation will miss interactions
This error occurs if I run on a 1080Ti (with 12 Gb VRAM)or an A40 (with 40 Gb VRAM). In either case, Iām not close to running out of memory (program only uses about 7 Gb VRAM).
However, this error goes away if I simply space my environments out, even just by one centimeter. i.e.
upper = g.Vec3(0.0, 0.01, 0.0)
lower = lower = g.Vec3(0.0, 0.00, 0.0)
envs_per_row = 1
env = self.gym.create_env(self.sim, lower, upper, envs_per_row)
In doing so, I encounter another strange behavior: the first two environments spawn on top of each other. All the environments have the same x-position and are 0.01 m apart in the y-dimension, except for the first two environments, which are right on top of each other.
My questions:
- what is
PxgDynamicsMemoryConfig::foundLostAggregatePairsCapacity
? Does it have to do with the PhysX parammax_gpu_contact_pairs
? How can I increase this buffer? Why does setting envs on top of each other cause problems with it? - Why do the first two envs spawn on top of each other, even when
upper
is nonzero?
My work around is just to space out my envs and dealing with the identical position of the zeroth and first env manually in my code.