MultiMeshRayCaster data (pos_w, ray_hits_w) fails to update after dynamic teleportation in _apply_action

Omniverse Isaac Sim
, , ,
1

Environment

  • Isaac Sim: 5.1.0

  • Isaac Lab: v2.2.1

  • nvblox: v4.2.0

  • ROS2: Jazzy

Problem Description

I am experiencing a critical synchronization issue between the USD Stage and the MultiMeshRayCaster sensor. When moving a Lidar prim dynamically using XFormPrim (or set_world_poses) within the _apply_action() method, the sensor’s internal data does not reflect the change immediately.

Even though the prim has physically moved in the stage, lidar.data.pos_w still returns the coordinates from the previous step. Consequently, ray_hits_w are calculated using the stale origin.

Impact on nvblox:

This desynchronization causes nvblox to generate incorrect and incomplete meshes. Since the pointcloud is published with a world frame ID but contains points relative to an outdated sensor pose, the TSDF integration fails. We observe a “ghosting” effect where the mesh is only partially reconstructed in the original position, and the pointcloud centroid does not follow the actual Lidar movement as seen in the simulator.

Evidence & Logs

Env 0: pos=[ 3.6766016   1.4495581  -0.05967574], yaw=-1.761
Env 1: pos=[3.6638875  5.9535947  0.14356598], yaw=-2.749
Env 2: pos=[0.13727075 1.6849464  0.14911243], yaw=1.567
Env 3: pos=[-0.19420904  5.301439    0.19386607], yaw=0.199
Env 4: pos=[-3.8345873   1.6949377  -0.03933457], yaw=-3.022
Env 5: pos=[-4.8244386   5.2764564   0.02837813], yaw=1.357
hits_w sample: [[inf inf inf]
 [inf inf inf]
 [inf inf inf]]
lidar_pos: [6.   1.5  1.25]
[DEBUG] lidar_pos env_0: [6.   1.5  1.25]
[DEBUG] maiz pos env_0:  [ 3.6766016   1.4495581  -0.05967574]
[DEBUG] env_0 origin:    [ 4. -2.  0.]
[DEBUG] rayos validos:   0/13440
[INFO] No se detectaron puntos validos en este step.
  0%|                                                                                                                          | 0/4800 [00:00<?, ?it/s]==================================================
LIDAR pos:  [6.   1.5  1.25]
LIDAR quat: [ 7.0710671e-01  5.9604645e-08  7.0658284e-01 -2.7213871e-02] (wxyz)
MAIZ pos:   [ 3.6766016   1.4495581  -0.05967574]
Vector lidar->maiz: [-2.3233984  -0.05044186 -1.3096757 ]
Distancia al maiz: 2.668 m
Rayos totales: 13440, Validos: 0, Inf: 13440
NO HAY HITS VALIDOS -- lidar no apunta al maiz
==================================================
  2%|██▎                                                                                                              | 99/4800 [00:06<05:32, 14.14it/s]hits_w sample: [[inf inf inf]
 [inf inf inf]
 [inf inf inf]]
lidar_pos: [6.   1.5  1.25]
[DEBUG] lidar_pos env_0: [6.   1.5  1.25]
[DEBUG] maiz pos env_0:  [ 4.487348    1.4044088  -0.06461412]
[DEBUG] env_0 origin:    [ 4. -2.  0.]
[DEBUG] rayos validos:   0/13440
[INFO] No se detectaron puntos validos en este step.
  4%|████▋                                                                                                           | 199/4800 [00:12<04:53, 15.68it/s]hits_w sample: [[inf inf inf]
 [inf inf inf]
 [inf inf inf]]
lidar_pos: [6.   1.5  1.25]
[DEBUG] lidar_pos env_0: [6.   1.5  1.25]
[DEBUG] maiz pos env_0:  [3.5156274  1.5681864  0.17093834]
[DEBUG] env_0 origin:    [ 4. -2.  0.]
[DEBUG] rayos validos:   0/13440
[INFO] No se detectaron puntos validos en este step.
  6%|██████▉                                                                                                         | 299/4800 [00:19<04:39, 16.10it/s]hits_w sample: [[inf inf inf]
 [inf inf inf]
 [inf inf inf]]
lidar_pos: [6.   1.5  1.25]
[DEBUG] lidar_pos env_0: [6.   1.5  1.25]
[DEBUG] maiz pos env_0:  [3.780462   2.1472208  0.16228068]
[DEBUG] env_0 origin:    [ 4. -2.  0.]
[DEBUG] rayos validos:   35/13440

[PC WORLD] step=300
  plant_pos world: [3.780462   2.1472208  0.16228068]
  lidar_pos world: [6.   1.5  1.25]
  cloud centroid:  [3.870013   2.310655   0.65109015]
  dist centroid→plant: 0.523m
  7%|███████▎

Relevant Code

1. Sensor Configuration:

Python

MultiMeshRayCasterSensor = MultiMeshRayCasterCfg(
    prim_path="/World/envs/env_.*/Lidar",
    mesh_prim_paths=["/World/envs/env_.*/Maiz"],
    update_period=0.02,
    pattern_cfg=LidarPatternCfg(channels=70, vertical_fov_range=(-45, 45.0), horizontal_res=0.625, horizontal_fov_range=(-60.0, 60.0)),
)

2. Dynamic Movement & Observation (nbv_ppo_env.py):

Python

def _apply_action(self) -> None:
    # ... pose calculation ...
    XFormPrim(
        f"/World/envs/env_{env_idx}/Lidar",
        positions=torch.tensor(world_pos, device=self.device).unsqueeze(0),
        orientations=torch.tensor(quat, device=self.device).unsqueeze(0)
    )

def _get_observations(self) -> dict:
    # These calls return stale data from the previous transform state
    hits_w = self.lidar.data.ray_hits_w  
    lidar_pos = self.lidar.data.pos_w[env_idx]

Questions

  1. How can I force the MultiMeshRayCaster to synchronize its internal XFormView and GPU buffers immediately after a pose update in the same control loop?

  2. Is there a specific method to trigger a manual update of the Raycaster’s transform buffers without waiting for the next physics stepping?

  3. Why does lidar.data.pos_w lag behind the actual USD Prim transform even if update() is called?

    image from the front

image
image1455×861 99.1 KB

image from the back

image
image1267×824 82.6 KB

3

A1: Call reset() then update() with force_recompute=True immediately after the pose change:

env_ids = torch.tensor([env_idx], device=self.device, dtype=torch.long)

self.lidar.reset(env_ids)

self.lidar.update(dt=0.0, force_recompute=True)

reset() sets _is_outdated[env_ids] = True. update(force_recompute=True) then calls _update_buffers_impl() unconditionally, which internally calls _update_ray_infos() — that’s where pos_w, ray_starts_w, and ray_directions_w are all recomputed from the current USD transform. Without force_recompute=True, the update is gated by update_period and may not fire immediately.

A2: Yes. update(dt=0.0, force_recompute=True) does exactly this — it flushes the sensor buffers without advancing the physics simulation:

self.lidar.update(dt=0.0, force_recompute=True)

Passing dt=0.0 means the sensor’s internal timestamp does not advance, so it has no side effects on the simulation clock. The force_recompute=True flag bypasses the update_period timer check entirely. The underlying pose read (obtain_world_pose_from_view()) queries the USD/PhysX transform state directly — no physics step required.

A3: Three layered reasons:

a) _is_outdated is never set by an external pose change.
pos_w is only rewritten inside _update_buffers_impl(), which only runs when _is_outdated[env_id] == True. Moving the prim via XFormPrim or set_world_poses() outside the sensor’s own lifecycle has no mechanism to set that flag — the sensor is unaware the prim moved.

b) The update cycle is periodic, not event-driven.
Even when _is_outdated is correctly set, _update_buffers_impl() only fires when the elapsed time since last update exceeds cfg.update_period. If you read .data in the same step you moved the prim, the period check may not have elapsed yet.

c) XformPrimView has a known stale-read issue in headless mode.
The sensor resolves its tracked prim to an XformPrimView (since a bare Lidar XForm has no physics body). The source code explicitly warns that XformPrimView “will most likely not be updated correctly when running in headless mode.” In contrast, RigidBodyView and ArticulationView read directly from PhysX’s internal transform buffer, which is always authoritative after a step. This is why maiz_pos (likely on a physics body) updates correctly while lidar_pos (on a pure XForm) does not.

4

closing this thread since there is no activity for more than two weeks. Please re-open or open a new thread if you still need any support. Thanks!