I created my own AI environment, attempting to base it off of the RL games examples.
Physics seems to be rendering accurately, but pauses for several frames while the iteration completes and I believe the average reward is calculated. I have a ball dropping onto a cube, similarly to the ball dropping onto the balance bot, and it stops in midair as the simulation pauses and the iteration line detailing FPS etc. is printed in the terminal. This does not happen when I run balance bot, even when it is using the exact same rewards calculations. This causes the entire program to run very slowly.
I looked into this some more - it seems to be related to the fact that my number of environments was too close to my minibatch size? Increasing minibatch size improved but did not eliminate the jerking motions.
I’m not sure what the minibatch size does or why it would causing jerking motions in the simulation, if someone can enlighten me.
The rendering, simulation and RL all run in sequence with each other. When the RL is busy collecting rollouts and performing training, rendering and simulation will freeze and wait for the RL pieces to complete. This will be less noticeable in tasks where RL runs fast.
What kinds of things cause the RL to run slowly? The complexity of the rewards function, or the total number of possible solutions? Total number of actions given?
Many parameters may affect the runtime of RL, such as mini_epochs, horizon_length during rollout collection, or the architecture of your network. More complex reward function and observation may also add more pauses into simulation.