Camera output lags due to rendering speed

I am currently training an RL agent using the input from a camera. At each step of the training, the camera is moved. Before getting the output from the camera, I call world.step(). However, I have noticed that the camera output does not always match the expected position of the camera. I believe this is due to the slow rendering speed, but assumed that world.step() would block until rendering is finished.
I then moved to rep.orchestrator.step(), which after running multiple tests by comparing images of moving the camera over the same path multiple times, seems to block until rendering is finished.
There are two issues with rep.orchestrator.step():

  1. As training continues, iterations per second drops from 6 to 3.
  2. At around 2500 steps, the output from the camera no longer updates (it is stuck on one image frame) even if the camera position changes (this behaviour continued over 10k steps before I quit it).

Are there any suggestions? And why does world.step() not block for one rendering_dt?

Here is the camera setup:

camera = Camera(
        orientation = camera_orientation

# at each RL step
rep.orchestrator.step() # or world.step(), or both

Hi @replicator.user.123 - It seems like you’re encountering a couple of issues related to rendering and camera updates in your reinforcement learning (RL) simulation. Here are a few suggestions that might help:

  1. Ensure the camera is updated before stepping the simulation: Make sure you’re updating the camera’s position before calling world.step() or rep.orchestrator.step(). If the camera’s position is updated after stepping the simulation, the rendered image might not reflect the new position.
  2. Check if the simulation is running too fast: If the simulation is running faster than the rendering, the rendered image might not always reflect the current state of the simulation. You can try slowing down the simulation or synchronizing it with the rendering to ensure they’re in sync.
  3. Use a blocking call to get the camera image: Instead of immediately getting the camera image after stepping the simulation, you can try using a blocking call that waits until the rendered image is ready. This might be a function like camera.get_current_frame(blocking=True).
  4. Check for resource leaks or bottlenecks: The decreasing iterations per second and the camera getting stuck on one frame might indicate a resource leak or a bottleneck in your code. Check if there’s a part of your code that’s using up more and more resources over time, or if there’s a part that’s getting slower over time.

Hi thank you for responding.

  1. I have been using rep.orchestrator.step() after moving the camera position, which works well but is much slower than world.step(). I believe world.step() does not block properly for rendering, which is why I am unable to use it.
  2. Where can I find resources to synchronize the simulation with the rendering?
  3. How would I know when the rendered image is ready? I have checked the source code for rep.orchestrator.step() and it does some frame waiting in the background, would this be the same use case?
  4. I have been encountering this issue Isaac Sim OS (driver?) persistent memory leak, it might be related.

This is exactly the problem I’ve been having for a long time now and that I’ve seen many issues about.
The rendering time depends on the complexity of the scene for RTX, and (even if IDK about the orchestrator) for path tracing in how many times stage/world steps are being called. If the scene is complex and the “fps” of the viewport is higher than the rendering time, you are screwed. A solution I found is this one which takes into account both situations and manually do the multiple rendering steps required.

Hi @replicator.user.123 and @marc2002 ,

There’s a few separate things we need to investigate here, but in the meantime:

For getting camera position, it is likely best to use the camera_params annotator output instead of querying the camera object