[NuRec] Failing to reproduce r2b_galileo 3D Reconstruction

[bittencourtpedro]

Operating System:
Linux
Kit Template:
USD Composer
GPU Hardware:
40 series
GPU Driver:
Latest

Hello,

I am currently running the NuRec pipeline described here:

Reconstruct Scenes from Stereo Camera Data — NVIDIA Omniverse NuRec

Strictly following the documented specifications and parameters.

Using the official dataset r2b_galileo

Environment

• Container: nvcr.io/nvidia/isaac/ros:noble-ros2_jazzy_d3e84470d576702a380478a513fb3fc6-amd64
• nvblox installed following the official instructions

Step 4 – Mesh Generation with nvblox

Command used:

/packages/nvblox/build/nvblox/executables/fuse_cusfm \
    --color_image_dir images/ \
    --depth_image_dir depth/ \
    --frames_meta_file cusfm/kpmap/keyframes/frames_meta.json \
    --save_2d_occupancy_map_path nvblox_mesh/occupancy_map \
    --mapping_type_dynamic \
    --projective_integrator_max_integration_distance_m=2.5 \
    --esdf_slice_min_height=0.09 \
    --esdf_slice_max_height=0.65 \
    --esdf_slice_height=0.3 \
    --mesh_output_path nvblox_mesh/nvblox_mesh.ply \
    --nouse_2d_esdf_mode \
    --fit_to_z0 2>&1 | tee logs/step4.log

Output Generated

nvblox_mesh/
├── nvblox_mesh.ply
├── occupancy_map.png
├── occupancy_map.yaml
└── T_world_to_z0.json

Issue

The generated occupancy_map.png has significantly less detail compared to the official result provided by NVIDIA for the same dataset:

Generated by reproducing the pipeline:

Official reference (on HuggingFace):

This discrepancy propagates to Step 5, resulting in a much lower-quality 3D reconstruction compared to the official demo.
The behavior is similar to the issue discussed here:

[NuRec] Poor reconstruction (Spikes) & Coordinate Mismatch with Official r2b_galileo dataset

Operating System: Linux Kit Version: 107 (Kit App Template) Kit Template: USD Composer GPU Hardware: 50 series - 5070Ti GPU Driver: Latest Hello, I am attempting to run the Neural Reconstruction (Stereo Camera Flow) pipeline by strictly following the official documentation: https://docs.nvidia.com/nurec/robotics/neural_reconstruction_stereo.html I am using the official sample dataset (r2b_galileo) provided by NVIDIA, not custom data. However, I am encountering significant issues…

What I Tried

• Adjusted multiple nvblox parameters (integration distance, ESDF slicing bounds, voxel_size, etc.)
• Some improvements were observed, but results are still far from the official output

Reproducibility Data (Logs + Outputs)

I have collected:

• Full outputs and logs for all pipeline steps
• Several resulting occupancy maps for comparison

These are available here:

3d_reconstruction - Google Drive

Questions

1. What exact parameters/configuration were used to generate the official r2b_galileo reconstruction (especially for nvblox / Step 4)?

2. Are there additional preprocessing steps or implicit assumptions (e.g., filtering, coordinate normalization, depth scaling) not documented in the pipeline?

3. For custom datasets, what is the recommended methodology to systematically tune nvblox parameters beyond trial-and-error? Are they related to the camera or the recorded scene?

   • Are there guidelines based on sensor characteristics, scene scale, or motion profile?
   • Any diagnostic metrics or intermediate outputs that should be monitored?

Any guidance would be appreciated, especially if there is a reference configuration or reproducible setup matching the official results.

Thanks.

[Richard3D]

Thank you for your post. I will reach out to a NuRec expert and have them get back to you.

[dkoy8927]

From https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-NuRec/blob/main/nova_carter-galileo/occupancy_map.yaml

image: “occupancy_map.png”
mode: “trinary”
resolution: 0.02 # ← this is the nvblox voxel_size
origin: [-8.32, -5.92, 0]
negate: 0
occupied_thresh: 0.65
free_thresh: 0.25

Following command I use with https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-NuRec/blob/main/nova_carter-galileo/raw_images.zip

fuse_cusfm
–color_image_dir /workspace/images/
–depth_image_dir /workspace/depth/
–frames_meta_file /workspace/cusfm/kpmap/keyframes/frames_meta.json
–save_2d_occupancy_map_path /workspace/nvblox_mesh/occupancy_map
–mesh_output_path /workspace/nvblox_mesh/nvblox_mesh.ply
–mapping_type_dynamic
–voxel_size=0.02
–projective_integrator_max_integration_distance_m=5.0
–esdf_slice_min_height=0.09
–esdf_slice_max_height=0.65
–esdf_slice_height=0.3
–nouse_2d_esdf_mode
–fit_to_z0

[bittencourtpedro]

Thanks for the detailed clarification and for sharing the exact command/configuration used.

I reran the pipeline using the HuggingFace version of the dataset. Initially I assumed it was identical to the dataset referenced in the documentation, but apparently there are relevant differences.

Using that dataset version, and your parameters, the generated occupancy map became much closer to the official HuggingFace result.

I then continued the pipeline and trained 3DGUT using the exact parameters from the documentation:

python train.py \
  --config-name apps/cusfm_3dgut.yaml \
  path=/workspace \
  out_dir=/workspace/3dgrut \
  initialization.fused_point_cloud_path=/workspace/nvblox_mesh/nvblox_mesh.ply \
  experiment_name=3dgut_gs \
  export_usdz.enabled=true \
  export_usdz.apply_normalizing_transform=true

However, the final reconstruction quality is still significantly below the expected result and also far from the reconstruction/gif published on HuggingFace:

nvidia/PhysicalAI-Robotics-NuRec at main

My reconstruction result:

video.webm

At this point I am trying to understand whether there are still undocumented differences in the pipeline.

My 3DGUT output and logs are available here:

3dgut_output - Google Drive

Any additional guidance would help a lot.

[dkoy8927]

You might want to try

--config-name apps/cusfm_3dgut_mcmc.yaml or apps/colmap_3dgrt_mcmc.yaml

copy & paste from doc.
We use the apps/cusfm_3dgut.yaml configuration in this tutorial, but you can also use apps/cusfm_3dgut_mcmc.yaml, which pairs 3DGRUT with an MCMC (Markov Chain Monte Carlo) densification strategy. In practice, this approach samples and densifies Gaussians in regions where the reconstruction is uncertain, sharpening thin structures and edges while improving overall fidelity, with only a modest increase in training time compared to the baseline configuration

[bittencourtpedro]

Thanks for the information.

Tested the reconstruction using MCMC, but the results were actually worse than the default configuration.

For example, the resulting mean_psnr dropped from ~22 to ~14, which is significantly lower. The final reconstruction quality was also visually degraded.

For reference, I uploaded the complete outputs and logs here:

https://drive.google.com/drive/folders/1bFzjX4gTrVB_zv9CSsZHYbU0xhvVX48l?usp=sharing

If there are any additional parameters or recommended settings for MCMC with the r2b_galileo dataset, I would appreciate any guidance.

[dkoy8927]

Compare the log files,

   \[INFO\] Initializing from accumulated point cloud: minha_cena/nvblox_mesh1-default/nvblox_mesh.ply                                        logger.py:67
   \[INFO\] Loading fused point cloud from minha_cena/nvblox_mesh1-default/nvblox_mesh.ply...                                                 logger.py:67

[ 13:47:13] [INFO] Loaded 112930 points from accumulated point cloud logger.py:67

  \[INFO\] Initializing from accumulated point cloud: minha_cena/nvblox_mesh4-max5-voxel2/nvblox_mesh.ply                                     logger.py:67
   \[INFO\] Loading fused point cloud from minha_cena/nvblox_mesh4-max5-voxel2/nvblox_mesh.ply...                                              logger.py:67

[01:42:59] [INFO] Loaded 2987747 points from accumulated point cloud logger.py:67

I think one of them uses the intended mesh, but it loads 2987747 points and hits the CUDA OOM error.

[3dgut][ERROR] ::: cudaMallocAsync(&ptr, size, stream) failed: 2: cudaErrorMemoryAllocation - out of memory at /workspace/PEDRO/humanoid/humanoid/packages/3dgrut/threedgut_tracer/include/3dgut/utils/cuda/cudaMemoryAllocator.h:44
[3dgut][ERROR] ::: CUDA error 2: cudaErrorMemoryAllocation - out of memory)