[Bug] NRE `mode=val` fails with PPISP post-processing checkpoint — "uninitialized parameter" error

Omniverse Core Platform NuRec
1

Environment

  • NRE Container: nvcr.io/nvidia/nre/nre-ga:26.02.00
  • GPU: NVIDIA RTX PRO 6000 Blackwell Max-Q (98GB)
  • Dataset: NVIDIA PAI (PhysicalAI-Autonomous-Vehicles), clip 4848b5fd-72a8-4594-a2eb-9d5702f694e0
  • Config: Based on configs/apps/prod/Hyperion-8.1/car2sim_6cam.yaml (PPISP post-processing)

Summary

When training with PPISP post-processing (override /model/post_processing@model.post_processing.b: ppisp) and then attempting to render from the saved checkpoint using mode=val, the following error occurs:

RuntimeError
Can't access the shape of an uninitialized parameter or buffer. 
This error usually happens in `load_state_dict` when trying to load an 
uninitialized parameter into an initialized one. Call `forward` to 
initialize the parameters before accessing their attributes.

This prevents any mode=val rendering from a PPISP-trained checkpoint, making it impossible to render novel views or ego views after training completes.

Steps to Reproduce

1. Train with PPISP post-processing (succeeds)

docker run --shm-size=64g --rm --gpus "device=0" \
    -e NGC_API_KEY="${NGC_API_KEY}" \
    -e WANDB_MODE=disabled \
    --volume "${DATA_DIR}:/workdir/dataset" \
    --volume "${OUTPUT_DIR}:/workdir/output" \
    --volume "${CUSTOM_CONFIG_DIR}:/app/internal/scripts/pycena/runtime/pycena_run.runfiles/_main/configs/custom" \
    nvcr.io/nvidia/nre/nre-ga:26.02.00 \
    mode=train \
    out_dir=/workdir/output \
    --config-name=configs/custom/nv_prod.yaml \
    dataset.path="/workdir/dataset/${MANIFEST}" \
    trainer.max_epochs=1 \
    trainer.world_size=1 \
    +log_level=2

Training completes successfully. Checkpoint saved at ${RUN_ID}/checkpoints/last.ckpt.

2. Render from checkpoint using mode=val (fails)

docker run --shm-size=64g --rm --gpus "device=0" \
    -e NGC_API_KEY="${NGC_API_KEY}" \
    -e WANDB_MODE=disabled \
    --volume "${DATA_DIR}:/workdir/dataset" \
    --volume "${OUTPUT_DIR}:/workdir/output" \
    nvcr.io/nvidia/nre/nre-ga:26.02.00 \
    mode=val \
    out_dir=/workdir/output \
    --config-name="/workdir/output/${RUN_ID}/config/parsed.yaml" \
    dataset.path="/workdir/dataset/${MANIFEST}" \
    dataset.val_camera_frame_step=1 \
    system.test.save_results=true \
    system.test.save_videos=true

Error:

Validation DataLoader 0:   0%|          | 0/2396 [00:00<?, ?it/s]
RuntimeError: Can't access the shape of an uninitialized parameter or buffer.

3. Workarounds attempted (all fail with same error)

  • model.calib.name=skip-calib — same error
  • dataset.precompute_rendering_data=false — same error
  • ~model.post_processing (Hydra delete) — same error

Expected Behavior

mode=val should successfully load the PPISP-trained checkpoint and render frames, just as it does for checkpoints trained with bilateral_grid_per_camera post-processing.

Actual Behavior

The PPISP module has parameters that are only initialized during a forward pass in training. When mode=val tries to load the checkpoint via load_state_dict(), these parameters are still uninitialized, causing the RuntimeError.

Impact

  • Cannot render novel views from a PPISP-trained checkpoint
  • Cannot iterate on camera rigs without retraining from scratch
  • mode=trainval workaround does produce renders, but:
    • Only renders what was configured at training time
    • Cannot resume from checkpoint — starts training over again
    • Makes it impossible to experiment with different novel view parameters post-training

Working Comparison

Training with bilateral_grid_per_camera post-processing (default in configs/apps/AV/ configs) produces checkpoints that work correctly with mode=val:

# This works:
defaults:
  - /model/post_processing@model.post_processing.b: bilateral_grid_per_camera
  - /model/post_processing@model.post_processing.c: bilateral_grid_per_frame

# This does NOT work with mode=val:
defaults:
  - override /model/post_processing@model.post_processing.b: ppisp
  - override /model/post_processing@model.post_processing.c: null

Suggested Fix

The PPISP module should either:

  1. Initialize all parameters in __init__() rather than lazily during forward(), or
  2. Handle uninitialized parameters gracefully during load_state_dict(), or
  3. Support a mode=val code path that initializes parameters before loading the checkpoint

Config Used

Custom config based on car2sim_6cam.yaml, with all NVIDIA production settings (MCMC strategy, road semantic, calib, 6cam Hyperion 8.1 sensors):

defaults:
  - /apps/AV/_base_3dgut_dynamic_road_semantic.yaml
  - /apps/prod/options/artifact: default
  - /apps/prod/options/nrend: disabled
  - /apps/prod/options/difix/3dgut: disabled
  - /apps/prod/options/mesh: disabled
  - /apps/prod/options/ground: enabled
  - /apps/prod/options/subsampling/3dgut: batch_sampler_4
  - /apps/prod/Hyperion-8.1/options/sensors: hyperion8.1_6cam_1lidar
  - /apps/prod/Hyperion-8.1/options/validation: default
  - /apps/prod/options/calib: enabled
  - /apps/AV/module_mixins/_mcmc_mixin.yaml
  - override /model/gaussians/strategy@model.strategy: mcmc
  - override /model/post_processing@model.post_processing.b: ppisp
  - override /model/post_processing@model.post_processing.c: null
  - override /logger: dummy

dataset:
  n_samples_per_epoch: 40000
  lidar_ids:
    - lidar_top_360fov
  # ... (full config available on request)
2

We are looking into this bug and will get back to you. Sorry for the inconvenience.

3

Can you check parsed.yaml whetherresume is null or not. (check both working one and not working one)

if it has resume null, then try resume=/workdir/output/${RUN_ID}/checkpoints/last.ckpt

Relevant command in that section:

--config-name=/workdir/output/config/parsed.yaml
mode=val
resume=/workdir/output/checkpoints/last.ckpt
out_dir=/workdir/output

Use NuRec for Autonomous Vehicles — NVIDIA Omniverse NuRec