Nvblox: sensor delay turns camera into eraser

[edward.bingham]

I’ve been trying to integrate a camera with NVblox for a while now. However, the camera’s localization estimation goes through PX4 EKF2 to fuse localization with other sensors, so there is a bit of a sensor delay between the depth image from the camera and the transform from localization. This sensor delay is turning the camera into an eraser when I try to integrate the depth image into NVblox. Whenever the camera looks away, it deletes many of the voxels on the edge of the view. I can do some work to tighten the sensor delay, but I’d prefer if the mapping solution were much more robust to this issue. What steps can I take to fix this?

Here are my current parameters for NVblox. I’ve tried to tune the parameters as much as possible to keep voxels in place, but there seems to be a fundamental limitation.

multi_mapper:
  connected_mask_component_size_threshold: 2000
  remove_small_connected_components: true

  static_mapper:
    do_depth_preprocessing: false
    depth_preprocessing_num_dilations: 3

    projective_integrator:
      projective_integrator_max_integration_distance_m: 20.0
      projective_integrator_truncation_distance_vox: 10.0
      projective_integrator_weighting_mode: “constant”
      projective_integrator_max_weight: 100.0
      projective_tsdf_integrator_invalid_depth_decay_factor: -1.0

    occupancy_integrator:
      free_region_occupancy_probability: 0.3
      occupied_region_occupancy_probability: 0.7
      unobserved_region_occupancy_probability: 0.5
      occupied_region_half_width_m: 1.5

    occupancy_decay_integrator:
      occupied_region_decay_probability: 0.45
      free_region_decay_probability: 0.55
      occupancy_decay_to_free: false

    freespace_integrator:
      max_tsdf_distance_for_occupancy_m: 1.0
      max_unobserved_to_keep_consecutive_occupancy_ms: 10000
      min_duration_since_occupied_for_freespace_ms: 2000
      min_consecutive_occupancy_duration_for_reset_ms: 10
      check_neighborhood: true

    view_calculator:
      raycast_subsampling_factor: 4
      workspace_bounds_type: “unbounded”
      workspace_bounds_min_corner_x_m: 0.0
      workspace_bounds_max_corner_x_m: 0.0
      workspace_bounds_min_corner_y_m: 0.0
      workspace_bounds_max_corner_y_m: 0.0
      workspace_bounds_min_height_m: -100.0
      workspace_bounds_max_height_m: 100.0

    esdf_integrator:
      esdf_integrator_min_weight: 0.1
      esdf_integrator_max_site_distance_vox: 2.0
      esdf_integrator_max_distance_m: 2.0
      esdf_slice_min_height: -0.1
      esdf_slice_max_height: 0.3
      esdf_slice_height: 0.0

    mesh_integrator:
      mesh_integrator_min_weight: 0.1
      mesh_integrator_weld_vertices: true

    tsdf_decay_integrator:
      tsdf_decay_factor: 0.95
      tsdf_decayed_weight_threshold: 0.0001
      tsdf_set_free_distance_on_decayed: false
      tsdf_decayed_free_distance_vox: 4.0

    decay_integrator_base:
      decay_integrator_deallocate_decayed_blocks: true

    dynamic_mapper:
      esdf_integrator:
      esdf_slice_min_height: -0.1
      esdf_slice_max_height: 0.3
      esdf_slice_height: 0.0

[vchuang]

Hello @edward.bingham,

Welcome to the Isaac ROS forum and thanks for your post!

If you’d like to make NVBlox less aggressive at clearing free space, you can try adjusting the following parameters in your freespace_integrator and occupancy_integrator:

• Lower free_region_occupancy_probability (closer to 0.5), so free observations are less decisive.

• Raise occupied_region_occupancy_probability, so occupied evidence dominates over occasional free readings.

• Increase min_duration_since_occupied_for_freespace_ms (you already have 2000 ms, you can try larger values like 3000–5000) so NVBlox only clears cells that haven’t been reaffirmed as occupied for a while.

• Increase max_unobserved_to_keep_consecutive_occupancy_ms to keep voxels around longer when they go unobserved (you already have 10000 ms, you can try even higher if memory allows).

These changes will make it harder for a single misaligned frame to immediately clear previously occupied voxels. When verifying the improvement, try moving the camera slowly at first and check that voxels don’t disappear at the edges. Then, gradually increase the motion speed to see where artifacts start to appear.

Good luck with your implementation!