Different results based on environment spacing

Performing the exact same simulation in two environments yields different results. If environment spacing is set to zero the differences disappear(lower bounds and upper bounds from create_env set to 0). With large environment spacing robots clearly drift even on a flat plane with no actuation. The drifting increases with changing position on the environment in the environment grid.

How can I resolve this problem? My workaround is setting environment size to 0, but this makes the simulation incredibly slow. See Issue with Environment spacing and PxgDynamicsMemoryConfig::foundLostAggregatePairsCapacity