How to sync physics scene to the renderer

Omniverse Isaac Sim
,
1

Hello, I’m trying to record the robot trajectory performed in the experiment with IsaacSim 4.0.0 and the IsaacLab, but I found that the point cloud rendered by the camera, especially the gripper camera, is not synchronous with the physics scene, which seems that the rendered result is the state of the scene a few steps ago, and as a result the point cloud from the gripper camera is tilted, as the red point cloud shown below (and the gripper camera axes is also drawn in the figure, and obsviously it’s far away from the corresponding position in the point cloud). Only after calling env.sim.render() manually for several times the rendering result can be correct, so I wander if there are some APIs that can force the renderer to get the scene of the current time.

image
image1013×994 95 KB

The simulation configuration is shown below:

@configclass
class SceneCfg(InteractiveSceneCfg):
    ...

    gripper_camera = CameraCfg(
        prim_path="{ENV_REGEX_NS}/robot/panda_hand/camera",
        height=480,
        width=640,
        data_types=["rgb", "distance_to_image_plane"],
        spawn=sim_utils.PinholeCameraCfg(
            focal_length=24.0,
            focus_distance=400.0,
            horizontal_aperture=53.7,
            clipping_range=(0.01, 1.0e5),
        ),
        offset=CameraCfg.OffsetCfg(
            pos=(0.05, 0, 0.05), rot=(0.70441603, -0.06162842, -0.06162842, 0.70441603), convention="ros"
        ),
    )

    static_camera = CameraCfg(
        prim_path="{ENV_REGEX_NS}/static_camera",
        height=480,
        width=640,
        data_types=["rgb", "distance_to_image_plane"],
        spawn=sim_utils.PinholeCameraCfg(
            focal_length=24.0,
            focus_distance=400.0,
            horizontal_aperture=53.7,
            clipping_range=(0.01, 1.0e5),
        ),
        offset=CameraCfg.OffsetCfg(
            pos=(2., 2.25, 2), rot=(-0.16793911, 0.3131961, 0.82376817, -0.44171333), convention="ros"
        ),
    )

@configclass
class EnvCfg(ManagerBasedEnvCfg):
    sim = sim_utils.SimulationCfg(dt=1 / 120.0, substeps=5, device="cpu", use_gpu_pipeline=False, use_fabric=True)
    scene = SceneCfg(num_envs=1, env_spacing=50)
    actions = ActioinsCfg()
    observations = DefaultObservationsCfg()
    events = RandomRobotSceneEvents()

    def __post_init__(self):
        self.decimation = 10
3

Hello, this seems to be related to how you are running your simulation. Have you followed the tutorials here, for example? are you including a sim.reset() for your simulation? it would be helpful if you can elaborate on your simulation step setup. Thanks!

4

Thank you very much! This works for me. During the sumulation I loaded some new models to the stage but forget to call the reset(), and after adding the env.sim.reset(soft=True) this promblem seems to be solved.

The code I used is as follows

env_cfg: ManagerBasedEnvCfg = EnvCfg()
env = ManagerBasedEnv(env_cfg)
...
# load new objects
env.sim.pause()
obj_cfg = RigidObjectCfg(...)
new_obj = RigidObject(obj_cfg)
env.scene.rigid_objects['new_obj'] = new_obj
env.sim.play()
env.sim.reset(soft=True)
...
action = policy()
obs, _ = env.step(action)
image = obs['images']['base_camera_rgb']
...
1 Like