This looks pretty normal from the point of view of a stepped rigid body physics simulation. I don’t know how this is done in the Gym, but if it works like other physics simulations I know about, here’s what I suggest.
You can think of this as chatter, vibration, or dragging that will happen with rigid objects in the real world, too – the reason a table leg or chair leg makes a humming sound when being dragged across the floor, or a piece of chalk makes that screechy noise on the blackboard.
Depending on the specifics of the contact generation and physics simulation, you can attempt to reduce this effect in a few ways.
The most effective would be a special “sliding” kind of contact joint, that simply constrains the object movement in the plane of the contact surface. This requires a special contact joint implementation, which I don’t think is available in the simulator (or the library it uses.)
The second would be to switch the simulation to “continuous collision” mode, but I don’t think this is available in the gym (or the library it uses.)
The third would be to average the readings out of the simulation over time. Run the physics engine at a higher rate than what you use for the rest of the simulation (say, 500 Hz) and average the last 50 milliseconds or so for each reading output. This can work really well, and is achievable in most physics engines. It might still need some updates to the software, and to the sensor implementation.
Finally, you can make the surfaces softer. Different physics engines have different tunables for depenetration movement/force and joint softness. Turn up the joint softness, if you have access to it, and turn down restitution, ideally to 0, making the contact entirely inelastic. I know that the physics engine allows you to limit the impulse applied at each contact by limiting the maximum impulse applied at each contact, which can generate soft contacts, but I don’t know if there’s an easy way to enable that in the Gym without updating the software.
I am comparing it to other simulation engines I’ve worked with before. Some engines end up “boucing” or “dragging” rigid bodies very similar to this. The contact penetrates the object a little bit, a force gets generated to depenetrate; next step, the object has moved out of the contact and there’s no depenetration force, but gravity starts accelerating the object back into contact, so the frame after that it penetrates again …
Physics engines that do not use a fixed time step size (in virtual time) suffer this even worse. I don’t know if the gym uses a fixed or a variable time step, though; that might be something to look into to see if it can be affected.
And, yes, CPU and GPU may give different results, As far as I understand it, two successive runs of the CPU may give different results, as this physics engine is not guaranteed to be deterministic.