I did a little more digging into why GRB were crashing (using the exact setup above, with the only change being the collision shapes are switched to boxes instead of spheres, and only 10k rigidbodies are spawned).
I spawn all 10k boxes from frames 0-50, and then get a repeatable crash around frame 250, long after the boxes settle on the ground. The crash only occurs in GRB/GPU mode and disappears when switching back to CPU mode. It happens in PsPool.h and it seems like it’s caused by a reinterpret_cast to a corrupted pointer. Here’s a screencap of the error in VS2015:
The interesting thing is that I didn’t enable ePCM in my sceneDesc, but the crash happens in an ePCM block of code. Turns out that even though the docs state that ePCM is off by default, when you instantiate a PxSceneDesc using a PxToleranceScale, ePCM is turned on in the constructor. By manually disabling ePCM after construction of the sceneDesc, I avoid the crash and GRB/GPU mode works fine and prints out error messages when memory limits are reached instead of crashing. I guess the downside is that ePCM can’t be used.
Once ePCM is off, I’ve tested 150k box rigidbodies spawning over various static colliders across a 300 frame interval and haven’t had any crashes occur…even without increasing the GPU memory limits. I get buffer overflow error limit messages and missed contacts (because I didn’t increase GPU memory limits), but not hard crashing. So that’s good.
Edit: scratch the last part. If I replay my scene enough times CUDA will still crash. At the start of each play I initialize all my scene variables (foundation, physics, cudacontextmanager, etc)…and at the end of each play I release them all…is there something else I need to do to clean up CUDA memory?