Pose and image capturing does not synchronize in the extension

I am currently developing the environment to capture an image and pose of a falling object in every two second using the extension. However, the image and the pose I got is un-synchronized, and it is capturing the pose that is few milisecond before the the timing of image.

code is something like follows.

from omni.isaac.dynamic_control import _dynamic_control
dc = _dynamic_control.acquire_dynamic_control_interface()

def _update(self, dt):

    # capture every two second
    if time>2.0:
       # capture image
       img = get_rgb_numpy()

       # capture current pose of the object
       body_handle = dc.get_rigid_body(prim_path)
       tr = dc.get_rigid_body_pose(body_handle)

       # save the output
       save_img(img)
       save_pose(tr)

       # reset the environment and re-drop the objects
       reset_and_drop_objects()
       time=0

    time+=dt

It seems like this problem happens since the physics engine is running independently in the different thread from the renderer. This problem does not happen when I use Python script and manually control the simulation step. However, I want to run using extension because it runs faster then the Python script.

How can I capture the pose and image exactly at the same timing??

What I have tried is the following, but I have no luck with it.

  1. use editor.pause()

     # capture every two second
     if time>2.0:
        self._editor.pause()
    
        # capture image
        img = get_rgb_numpy()
    
        # capture current pose of the object
        body_handle = dc.get_rigid_body(prim_path)
        tr = dc.get_rigid_body_pose(body_handle)
    
        # save the output
        save_img(img)
        save_pose(tr)
    
        self._editor.play()
    
        # reset the environment and re-drop the objects
        reset_and_drop_objects()
        time=0
    
     time+=dt

Because extensions run in a separate thread it is not possible to directly synchronize things without also causing the main thread to run locked with the extension thread.
This would result in it running at the same performance as the python script.

The python script method is currently the best way to get synchronized data because you have guarantees about how time-stepping occurs

1 Like