Robot Bumpy when reducing the physics step

I have a robot that contains lidars, and a camera. I need to publish the laser scans and the camera frames alongside the odometry and tf. When doing so on render frame, the fps drops to around 5 frames per second. I have decoupled the physics step and rendering (I have now an extension that runs on physics_callback) and I set the time step per second to 20.
This is still fine for me as I need to get data at least every 100ms (with 20 steps per second, I am getting it every 50ms).
With that I was able to increase the frame rate significantly.

I then tried to move the robot using the articulation class, and the robots bounces (weirdly) when doing so. Increasing the time step per second back to 60 does the trick but decreases the fps.
What do I need to do for the robot to move smoothly?
Any pointers here would be much appreciated.