Questions about Compatibility between 3DGRUT Reconstruction Data and NuRec Format, and Understanding the NuRec Checkpoint Structure

Omniverse Core Platform RTX Renderer
1

Hi everyone,
I’m currently trying to integrate my 3DGRUT reconstructed scene data with NuRec (NVIDIA Unified Reconstruction) in CARLA, and I’ve encountered several issues that I’d appreciate some guidance on.

Here are my main questions:

  1. Data Compatibility
    I reconstructed a scene using 3DGRUT and obtained standard Gaussian Splatting outputs.
    However, these cannot be directly loaded by NuRec.
    1. Do I need to convert the 3DGRUT output into a specific NuRec-compatible format or auxiliary data structure?
    2. If yes, are there any public conversion scripts or references available?
  2. Checkpoint Files in the NuRec Dataset
    Thanks for providing the PhysicalAI-Autonomous-Vehicles-NuRec dataset.
    I noticed that within the .usdz file, there’s a checkpoint file — something not present in 3DGRUT reconstruction outputs.
    1. How is this checkpoint file generated? What algorithm or training process produces it?
  3. Structure of the Checkpoint File
    I loaded the checkpoint via torch.load and inspected the file content:
    1. It includes several branches such as background, road, and dynamic. How are these divisions generated? Do they require additional label input during reconstruction or training?
    2. The features_albedo tensor has the shape (n, 5, 3), while 3DGRUT’s output has (n, 3). What does this extra dimension (5) represent — multi-view features, hierarchical levels, or additional color channels?

Any reference documentation about NuRec’s data format, checkpoint training pipeline, or example scripts for converting 3DGRUT outputs to NuRec-compatible formats would be greatly appreciated.
Thank you!

2

This topic is not supported directly here, because this is CARLA but have you downloaded the new CARLA nIghtly build with NuRec build in?

Get Ready for CARLA 0.9.16 this summer! - CARLA Simulator

3

Thank you very much for your assistance!

We have successfully downloaded and integrated the Carla version with NuRec, and we are able to load the PhysicalAI-Autonomous-Vehicles-NuRec dataset and render driving scene images without any issues. However, we have encountered a problem when working with the data generated by 3DGRUT. We noticed that this data contains non-USD formatted elements, which prevents it from being directly loaded by NuRec.

Thus, I have two questions:

  1. Is the data that NuRec can load the result of some conversion applied to the 3DGRUT data? If so, could you provide any reference materials or publicly available scripts to help us understand and replicate this process?
  2. The checkpoint.ckpt file in the PhysicalAI-Autonomous-Vehicles-NuRec dataset – we have noticed that this file contains information related to a trained model. Could you provide more details about which model this file is from and how it was trained?

Thank you again for your help, and I look forward to your response!