Feasibility of using NVBlox with RealSense for 3D reconstruction and feature mapping

Robotics & Edge Computing Robotics - Isaac Isaac ROS
1

Hi everyone,

I’m currently trying to use NVBlox with a robot-mounted Intel RealSense for 3D reconstruction of a workspace, and creating a featuremap of the workpieces.

Most of the examples and documentation I’ve seen so far focus on using NVBlox primarily for 2D mapping and navigation with Nav2 (e.g., costmaps and obstacle avoidance). I’m wondering how well NVBlox supports more general 3D reconstruction use cases beyond navigation.

Specifically, I have a few questions:

  1. Is it feasible to use NVBlox with a robot-mounted RealSense camera for dense 3D reconstruction of a workspace?

  2. Is it possible to integrate deep, feature-based reconstruction or semantic features into the current NVBlox pipeline, or would this require a parallel perception pipeline alongside NVBlox?

Any insights, recommended approaches, or pointers to relevant examples or documentation would be greatly appreciated.

Thanks!

2

Hello @MossProphet,

Welcome to the Isaac ROS forum and thanks for the post!
I’ll answer your questions below.

  1. Yes, that’s a supported use case of using NVBlox for dense 3D reconstruction with RealSense. NVBlox builds a 3D TSDF voxel grid (and optional color, mesh, and ESDF layers) from depth images and poses. The RealSense tutorials in the docs shows exactly this setup with a robot‑mounted RealSense camera.
    The Nav2 costmap is just one consumer of that 3D map, you can use the TSDF/mesh directly for workspace reconstruction and inspection.

  2. Integrating deep / feature‑based / semantic information
    NVBlox already supports:
    Static reconstruction (TSDF + color + ESDF),
    People reconstruction using DNN segmentation or detection to separate people into a dedicated occupancy layer,
    Dynamic reconstruction of generic moving objects using a freespace‑based method (Dynablox).
    For more general “feature maps of workpieces” or rich learned features, the typical approach is to:
    Use NVBlox as the geometric backbone. (TSDF/mesh/ESDF)
    Run a parallel DNN perception pipeline (e.g. detectors or segmentors) on the RGB/depth.
    Project those results into the NVBlox frame and maintain your own per‑voxel or per‑object semantic/feature layer alongside NVBlox.

So yes, you can absolutely use NVBlox + RealSense for dense 3D workspace reconstruction. If you want richer, feature‑based or semantic mapping, you’d currently build that on top of NVBlox’s TSDF/mesh, reusing the existing people/dynamic reconstruction modes where they fit and adding your own feature fusion for workpieces.

3

Thank you for the Reply,
I started working on it, and the people reconstruction workflow is helpful for a semantic mapping approach.