The simulation time is desynchronized with the playback time

Omniverse Isaac Sim
1

Isaac Sim Version

5.0.0

Operating System

Windows 11

GPU Information

  • Model: RTX 3080 Mobile
  • Driver Version: 577.0

Topic Description

Detailed Description

2 years ago somebody detected problem when an annotator output is always delayed for 2 frames.
I ran into the same problem when tried to get rendered frames in a graph powered by the On Physics Step node. Actually, there were 2 problems:

  1. On Physics Step node evaluates an OnDemand graph containing it during the PhysX step event:
// Inside OgnOnPhysicsStep class of isaacsim.core.nodes extension
static void onPhysicsStep(float timeElapsed, void* userData)
{
   ...
   graphObj.iGraph->evaluate(graphObj);
}

static void initialize(const NodeObj& nodeObj, GraphInstanceID instanceId)
{
   ...  
   g_graphsWithPhysxStepNode[graphObj.graphHandle].stepSubscription = g_physXInterface->subscribePhysicsOnStepEvents(
                    false, 0, onPhysicsStep, reinterpret_cast<void*>(&state.m_graphHandlePair));
   ...
}
static void initInstance(NodeObj const& nodeObj, GraphInstanceID instanceId)
{
    initialize(nodeObj, instanceId);
}

Physics is executed before rendering (Pre-Render, Render, Post-Render). Replicator’s PostProcess graph where attached annotator nodes are placed is evaluated just after render products are ready:

# Inside syntheticdata.py of omni.syntheticdata extension
def post_process_graph_tick_order() -> int:
        """Return the post process graph tick order."""
        # eCheckForHydraRenderComplete + 1: after the render products are available
        return carb.settings.get_settings().get("/app/updateOrder/checkForHydraRenderComplete") + 1

If you try to get a data from an annotator inside a graph powered by On Physics Step node, you will always get it lagged by at least a frame because at this moment a PostRender graph with GPU Interop: Render Product Entry node (which gives rendered frames on GPU for an annotator) for your render product has not been evaluated for the current simulation step yet.

  1. The simulation time by default is ahead of the playback time by 1-2 frame because the isaacsim.core.simulation_manager.SimulationManger.initialize_physics method which is called when its warmup is needed (usually during isaacsim.core.api.world.World.reset()call or a simulation reset) does two simulation steps (the second step in _create_simulation_view, see comment 2):
# Inside simulation_manager.py of isaacsim.core.simulation_manager extension
@classmethod
def initialize_physics(cls) -> None:
    if SimulationManager._warmup_needed:
        ...
        SimulationManager._physx_interface.update_simulation(SimulationManager.get_physics_dt(), 0.0)
    ...
# The second step in _create_simulation_view, see post 2

Moreover, when you stop a simulation in an app with GUI, the playback time is reseted to startTimeCode of a root layer (usually it is 0). The simulation time is reseted to 0th frame, and then on the timeline play eventinitialize_physics is performed what moves simulation two frames more forward:

# Inside simulation_manager.py of isaacsim.core.simulation_manager extension
def _warm_start(event) -> None:
    if SimulationManager._carb_settings.get_as_bool("/app/player/playSimulations"):
        SimulationManager.initialize_physics()

def _on_stop(event) -> None:
    SimulationManager._warmup_needed = True
    ...

@classmethod
def _setup_warm_start_callback(cls) -> None:
    if cls._callbacks_enabled["warm_start"] and cls._warm_start_callback is None:
        cls._warm_start_callback = cls._timeline.get_timeline_event_stream().create_subscription_to_pop_by_type(
            int(omni.timeline.TimelineEventType.PLAY), cls._warm_start
        )

    @classmethod
def _setup_on_stop_callback(cls) -> None:
    if cls._callbacks_enabled["on_stop"] and cls._on_stop_callback is None:
        cls._on_stop_callback = cls._timeline.get_timeline_event_stream().create_subscription_to_pop_by_type(
            int(omni.timeline.TimelineEventType.STOP), cls._on_stop
        )

Now there is 2 frame desynchonization between the main timeline and the simulation, and therefore between the rendering and the simulation too.

I wrote a demo that shows it, here is a typical output for a stage with startTimeCode set to 0:

-------Before World.reset-------
Timeline   time: 0.0
Simulation time: 0.0
Render     time: 0.0

Going to World.reset

-------TIMELINE PLAY-------
Timeline   time: 0.0
Simulation time: 0.03333333507180214
Render     time: 0.0

-------CURRENT_TIME_TICKED-------
Timeline   time: 0.01666666753590107
Simulation time: 0.03333333507180214
Render     time: 0.0

Resetted

-------After World.reset-------
Timeline   time: 0.01666666753590107
Simulation time: 0.03333333507180214
Render     time: 0.01666666753590107

Going to step 0
-------CURRENT_TIME_TICKED-------
Timeline   time: 0.03333333507180214
Simulation time: 0.03333333507180214
Render     time: 0.01666666753590107

-------POST_PHYSICS_STEP-------
Timeline   time: 0.03333333507180214
Simulation time: 0.05000000260770321
Render     time: 0.01666666753590107

Stepped 0. All graphs including the PostRender and PostProcess ones have been evaluated

----------------STEP 0----------------
Timeline   time: 0.03333333507180214
Simulation time: 0.05000000260770321
Render     time: 0.03333333507180214

Going to step 1
-------CURRENT_TIME_TICKED-------
Timeline   time: 0.05000000260770321
Simulation time: 0.05000000260770321
Render     time: 0.03333333507180214

-------POST_PHYSICS_STEP-------
Timeline   time: 0.05000000260770321
Simulation time: 0.06666667014360428
Render     time: 0.03333333507180214

Stepped 1. All graphs including the PostRender and PostProcess ones have been evaluated

----------------STEP 1----------------
Timeline   time: 0.05000000260770321
Simulation time: 0.06666667014360428
Render     time: 0.05000000260770321

Now stop the playback in GUI and start it again
-------TIMELINE STOP-------
Timeline   time: 0.0
Simulation time: 0.06666667014360428
Render     time: 0.05000000260770321

-------TIMELINE PLAY-------
Timeline   time: 0.0
Simulation time: 0.03333333507180214
Render     time: 0.0

-------CURRENT_TIME_TICKED-------
Timeline   time: 0.01666666753590107
Simulation time: 0.03333333507180214
Render     time: 0.0

-------POST_PHYSICS_STEP-------
Timeline   time: 0.01666666753590107
Simulation time: 0.05000000260770321
Render     time: 0.0 # will be 0.01666666753590107 at the end of the kit's update cycle

Steps to Reproduce

  1. Create a stage in Isaac Sim 5.0.0 with a camera prim at the path “/World/Camera”

  2. Download my demo script txt file, remove .txt in the filename and inside change the stage path to yours:

...
# Open a USD Stage with a camera prim on the path /World/Camera
omni.usd.get_context().open_stage(your_stage_path)
...
  1. Being in a console, run it by python.bat/python.sh like any other standalone python scripts.
  2. When the playback has been paused, stop it and play again for a few frames.
  3. See time output logs in the console.

Additional Information

What I’ve Tried

  1. Set startTimeCode of a stage to 1. There is still a small desync that will be accumulated during a simulation:
----------------STEP 0----------------
Timeline   time: 0.0500000017384688
Simulation time: 0.05000000260770321
Render     time: 0.0500000017384688
  1. Set physics_dt for initialize_physics to 0.0:
# Inside simulation_manager.py of isaacsim.core.simulation_manager extension
@classmethod
def initialize_physics(cls) -> None:
    ...
    SimulationManager._physx_interface.update_simulation(0.0, 0.0)
    ...

It perfectly synchronizes the time at a startup, but I’m not sure if it doesn’t break a simulation overall:

----------------STEP 0----------------
Timeline   time: 0.03333333507180214
Simulation time: 0.03333333507180214
Render     time: 0.03333333507180214

They both don’t fix the problem with GUI simulation reset though.
test_timeline_vs_simulaiton.py.txt (2.9 KB)

2

Just looked insimulation_manager.py again and I found that during an initialization the simulation is always stepped twice:

@classmethod
def _setup_post_warm_start_callback(cls) -> None:
    if cls._callbacks_enabled["post_warm_start"] and cls._post_warm_start_callback is None:
        cls._post_warm_start_callback = cls._message_bus.observe_event(
            event_name=IsaacEvents.PHYSICS_WARMUP.value,
            on_event=cls._create_simulation_view,
            observer_name="SimulationManager._post_warm_start_callback",
        )

def _create_simulation_view(event) -> None:
    ...
    SimulationManager._physx_interface.update_simulation(SimulationManager.get_physics_dt(), 0.0)
    ...

@classmethod
def initialize_physics(cls) -> None:
    if SimulationManager._warmup_needed:
        SimulationManager._physx_interface.update_simulation(SimulationManager.get_physics_dt(), 0.0)
        ... 
        SimulationManager._message_bus.dispatch_event(IsaacEvents.PHYSICS_WARMUP.value, payload={})
        # _create_simulation_view is called after it

If I block simulation stepping in initialize_physics, it still happens in _create_simulation_view. Is this doubled stepping really the intended attitude?

4

Hi, The Simulation Time and Render Time is independent and you can set the physics_dt and render_dt to control the step size of them.
If you set physics_dt to 0.0 I think physics simulation is skipped so that maybe not what you want.

5

Hello. In my case physics_dt is equal to rendering_dt and I am expecting the frame-perfect behaivor, i.e. simulate → render the results → change some physical parameters before the next simulation step.

As there is no any direct access and documentation to the RTX Render, I am tied to the SyntheticData’s PostProcess graph. I haven’t tried to evaluate my logic right after the PostRender graph for my Render Product. When I was digging in built-in Profiler traces, I noticed that the render is always completed before the post-render stage where PostRender graphs are evaluated andPostProcess graphs are evaluated at the beginning of the kit’s update. I am not sure why the execution order for PostProcess graphs is -100 (the very begining of the kit update stage) then, there might be the constant 1 frame lag between Render Products render results and the simulation, but… (see the figure 2 and 3).

I have no free unpaid time and can not do a complex research now, but I can share some results I got earlier. Excuse me for .jpeg screenshots:

1
11280×720 126 KB

Here you can see the 2 frame lag between the render results (58 frame) and the simulaton (60 frame). physics_dt = rendering_dt = 60, startTimeCode = 0 for the stage and the Simulation Manager is doing two physics steps during an initialization. My graph is driven by the On Physics Step node. When the simulation time equals to 1 second or more, it gets a frame from an omni.syntheticdata.SdRenderVarPtr LdrColor node annotator related to a Render Product and stops the playback. The application is running in the GUI + Extension workflow so there was at least 1 GUI simulation reset.

2
21280×720 95 KB

Here the left window shows the render result after 60 World.step(), the right one shows the render result during 60th World.step() right after the 61th physics step. The application is run by a Python script similar to the one I uploaded in the first post. In the script I’m doing a non-pausing Replicator’s step, World.reset() and then 60 World.step(). There is no a GUI simulation reset, but the right render result has a 1 frame lag. And I am honestly do not know why the left one is frame-perfect if the PostProcess graph still must contain a result of the previous frame.

3
31280×720 154 KB

The previous setup but with only one step during the Simulation Manager initialization. The results the same. Do not have any clues about it.

6

Thanks for reporting this. I am closing the ticket. Please open new ones if you have any issues with this in your tests.