NuRec expects well-calibrated, time-synchronized multi‑sensor logs (cameras, optionally LiDAR, plus ego pose) and outputs a USDZ scene package that can be reused across simulators, with Fixer improving reconstruction artifacts rather than raw video resolution. carla.readthedocs
Log data preparation
To prepare logs for NuRec you typically need:
- Calibrated camera intrinsics and extrinsics for all views, with stable poses over the log. voxel51
- Time synchronization between cameras, LiDAR (if present), and ego pose/IMU so each frame bundle aligns in time. carla.readthedocs
- A consistent world / vehicle coordinate system and complete metadata (timestamps, sensor IDs, vehicle pose trajectory). voxel51
- Sufficient parallax and coverage: multiple viewpoints over the area you want reconstructed, not just a single static camera. developer.nvidia
A typical example is an AV-style log with 4–8 surround cameras, optional LiDAR, and GNSS/IMU over a 10–30 s segment covering the scene of interest. developer.nvidia
Is LiDAR necessary?
- NuRec can reconstruct scenes from camera-only logs; LiDAR is not strictly required. docs.nvidia
- When available, LiDAR point clouds help constrain geometry, improving structure, depth consistency, and robustness in texture‑poor regions. arxiv
- For indoor or short‑range robotics scenes, multi‑view RGB plus accurate poses are often sufficient; for long‑range AV highway/urban scenes, LiDAR tends to improve large‑scale layout and distant structures. sciencedirect
So you can run NuRec without LiDAR, but you should expect better metric accuracy and fewer geometric artifacts when LiDAR is included.
NuRec outputs and USDZ contents
NuRec writes a USDZ bundle as its primary artifact. This package normally contains: docs.nvidia
- A USD scene graph (the neural scene asset with geometry/volume, materials, and bindings to the neural renderer). docs.nvidia
- A trained reconstruction checkpoint (e.g., NeRF / 3DGS / 3DGUT weights) referenced by the USD schema. docs.nvidia
- A JSON track file with ego and rig trajectories and sequence metadata (frame ordering, timestamps, etc.). docs.nvidia
- Optionally, an OpenDRIVE (XODR) map if you provide one; NuRec does not infer XODR itself. docs.nvidia
NVIDIA notes that these USDZ assets are designed to load directly into Omniverse‑based apps (Isaac Sim, CARLA integration via NuRec, robotics/AV platforms with OmniNuRecVolumeAPI). Whether you can “directly” use them in AlpaSim depends on AlpaSim’s USD/USDZ and NuRec integration; if AlpaSim is an Omniverse Kit–based app with OmniNuRecVolumeAPI support, you can generally load the USDZ and control rendering, otherwise you may need an import or conversion step. linkedin
Typical scene components in the USDZ
- Neural volume representation of the environment (road, buildings, foliage, static objects). carla.readthedocs
- Materials/appearance parameters used by the NuRec renderer (neural radiance field or 3D Gaussians configuration). developer.nvidia
- Coordinate frames and transforms for world, ego vehicle, and sensor rigs. voxel51
- Optional semantic groupings or prims used for simulation hooks (e.g., driveable area volumes, occluder tags), depending on your pipeline. edge-ai-vision
These are scene assets; dynamic agents (cars, pedestrians) are usually simulated separately, not baked as controllable USD prims in the NuRec volume. edge-ai-vision
Per‑scene reconstruction / training
- Each distinct scene you want to reconstruct from logs is processed separately: you pick a segment of data and run a reconstruction (which internally trains the neural representation for that segment). carla.readthedocs
- You do not typically run a global “once for all scenes” 3DGUT/NeRF training; the training is per scene/log, producing one checkpoint and USDZ per scenario. developer.nvidia
- Workflows can be automated to run many scenes in batch, but conceptually it is still one reconstruction per scene region you care about. voxel51
Some pipelines may reuse pre‑trained world models for priors, but NuRec’s shipped process still optimizes each reconstruction to its specific sensor logs. edge-ai-vision
Changing weather and environment in simulation
- NuRec itself reconstructs the scene as observed (lighting, weather, foliage, etc.); the neural asset encodes that appearance. carla.readthedocs
- To change weather, time of day, or fog during simulation, NVIDIA pairs NuRec scenes with world foundation models and tools like Cosmos Transfer, which can generate scene variations (e.g., different weather and lighting) on top of reconstructed assets. edge-ai-vision
- Within Omniverse/Isaac Sim, you can also adjust sensor rendering parameters (exposure, noise, fog volumes, rain particle systems) at the simulator level, effectively changing rendered conditions even when the underlying neural asset was trained under one condition. voxel51
So yes, you can modify environmental conditions for simulation, but it is done via simulator and generative‑model tooling around NuRec rather than retuning the original reconstruction.
Fixer model purpose
- NuRec Fixer is a generative post‑processing model that removes reconstruction artifacts and restores missing detail in under‑constrained regions of the neural scene. paralleldomain
- It is diffusion/transformer‑based, built on NVIDIA Cosmos, and is applied either during reconstruction or as a post‑process on rendered frames to inpaint, sharpen, and clean up surfaces and textures (e.g., road markings, building facades). paralleldomain
- While perceived resolution and sharpness improve, Fixer is not a generic “video super‑resolution” model; its objective is simulation‑grade scene fidelity and artifact removal rather than arbitrary video upscaling. paralleldomain
An example workflow is: reconstruct with NuRec → render sensor views from the neural scene → run Fixer on those rendered frames to reduce smearing, fill in occluded or sparse regions, and make the scene more realistic for AV or robotics training. paralleldomain
To tailor this to your pipeline, what simulator stack are you planning to target first (Isaac Sim, CARLA, AlpaSim only, or a custom Omniverse app)?