3-D reconstruction using Colmap and 3DGUT (Monocular Image Pipeline)

Omniverse Core Platform OpenUSD
, ,
1

Hello team,

I have been trying to reconstruct the 3-D scenes using multiple monocular images of scene captured from different positions/angles (tried with video frames also). I am aiming to generate the usdz file which I can use with the Nvidia NuRec (Carla + Nurec).

Here are the articles I have followed:

Sharing some of my outputs.

Please suggest how can I improve my reconstruction quality to achieve same quality output as that of examples present in Nvidia Physical AI repository. Also if there are other ways to generate the same (For example Depth Anything 3 also offers 3-d reconstruction and Gaussian Splatting based novel view which seems to provide better reconstruction but no integration with NuRec).

2

Hi and thanks for posting the question. You are certainly getting the hang of it. The second one, with the office chair looks great. For tips you just need to stick to the recommendations given right there in the “Workflow” section, like:

  • Ensure steady lighting and proper focus
  • Avoid fast motion and blur
  • Use faster shutter speed (1/100 second or faster recommended)
  • Lock focus and exposure to maintain consistency across shots
  • Use a tripod or stabilization when possible

Camera Settings

For DSLR/Mirrorless Cameras:

  • Manual mode: Use manual mode (M) for full control
  • Shutter speed: 1/120–1/250s outdoors, ≥1/100s indoors
  • ISO: Keep as low as possible (ISO 100-400) while maintaining proper exposure
  • Aperture: f/8-f/11 for good depth of field
  • White balance: Lock white balance to avoid color shifts
  • Focus: Use manual focus or single-point autofocus, then lock focus
  • Format: Shoot in RAW or highest quality JPEG

On top of that MORE data is better. So take many more photos. I would do at least 100, but you really may need more.

3

Thanks for the advice. Issue seems to be from colmap side tho I am not sure why. I was using colmap with GUI and results were bad but using the cli commands with same input images I got much much better results. Thanks for the support.

4

Hi @Richard3D I was able to generate the USDZ file. I want to run my custom USDZ file for CARLA + NuRec rendering as of now I have tried using the USDZ output of 3dgrut but it is not compatible as the files inside the USDZ are different (3dgrut USDZ vs Nvidia Physical AI repository). Can you help me guiding how can I generate USDZ file compatible for Carla + NuRec rendering.

5

I have run reconstruction the reconstruction on KITTI dataset video and the construction but when trying to get a novel view I am seeing noise/artifacts. Please suggest how can I improve Novel view synthesis without artifacts.

6

Looks really good. I don’t think you can get much better than that at the current level of technology. It is just a question of working with the limits of the NuRec and GS. Again, the only thing you can do to improve is more and higher quality input imagery. This is not really an Omniverse issue. This is just the current processing limit of the workflow outside of Nvidia.