NuRec validation mode (mode=val) issue

Omniverse Core Platform NuRec
,
1

Hi, I encountered two issues when using nurec’s val mode:

  1. When specifying parameters: OverrideParseException.
  2. Unrepaired parameters appeared in parsed.yaml.

environment

System Environment

Distributor ID: Ubuntu
Description:    Ubuntu 20.04.6 LTS
Release:        20.04
Codename:       focal

NVIDIA-SMI 550.127.08             Driver Version: 550.127.08     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100 80GB PCIe          Off |   00000000:34:00.0 Off |                    0 |
| N/A   77C    P0            311W /  300W |   63391MiB /  81920MiB |    100%      Default |
|                                         |                        |             Disabled

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Tue_Feb_27_16:19:38_PST_2024
Cuda compilation tools, release 12.4, V12.4.99
Build cuda_12.4.r12.4/compiler.33961263_0

NuRrec environment

NuRec v26.2.158
Model v$26.2.158 

nvcr.io/nvidia/nre/nre-tools-ga:latest
nvcr.io/nvidia/nre/nre-ga:latest

Execution process

Auxiliary data command

docker run --shm-size=2g -it --rm --gpus '"device=6"' \
   --volume /share/home/hcy/NuRec/data/docker/dataset:/workdir/dataset \
   --volume /share/home/hcy/NuRec/data/docker/output:/workdir/output \
   nvcr.io/nvidia/nre/nre-tools-ga:latest \
   --dataset-path=/workdir/dataset/yj/tj_short/my_custom_drive_001.json \
   --output-dir=/workdir/output/yj/tj_short \
   --store-meta \
   --no-seg-logits \
   --lidar-seg-camvis

log:
AuxiliaryData.log (4.3 KB)

Reconstruct command

docker run --shm-size=64g --rm --gpus '"device=6"' \
  --volume /share/home/hcy/NuRec/data/docker/dataset:/workdir/dataset \
  --volume /share/home/hcy/NuRec/data/docker/output:/workdir/output \
  nvcr.io/nvidia/nre/nre-ga:latest \
  mode=train \
  out_dir=/workdir/output/reconstruction/yj/tj_short \
  --config-name=configs/apps/prod/Hyperion-8.1/car2sim_6cam.yaml \
  dataset.path=/workdir/dataset/yj/tj_short/my_custom_drive_001.json \
  dataset.camera_ids=[cam_front] \
  dataset.lidar_ids=[lidar_middle] \
  dataset.aux_data=True \
  model.layers.dynamic_deformables.tracks.label_classes.0=car \
  dataset.cuboid_tracks_params.track_label_sources=null \
  checkpoint.artifact.enabled=True \
  dataset.n_samples_per_epoch=3000

log:
Reconstruct.log (69.9 KB)

parsed.yaml

parsed.yaml.txt (206.5 KB)
dataset.val_camera_ids

image
image1206×342 30.7 KB

Validation(mode=val)

docker run --shm-size=64g --rm --gpus '"device=7"' \
  --volume /share/home/hcy/NuRec/data/docker/dataset:/workdir/dataset \
  --volume /share/home/hcy/NuRec/data/docker/output:/workdir/output \
  nvcr.io/nvidia/nre/nre-ga:latest \
  --config-name=/workdir/output/reconstruction/yj/tj_short/iNWfKMyAewtQ97FBHQZyE3/config/parsed.yaml \
  mode=val  \
  dataset.path=/workdir/dataset/yj/tj_short/my_custom_drive_001.json \
  resume=/workdir/output/reconstruction/yj/tj_short/iNWfKMyAewtQ97FBHQZyE3/checkpoints/last.ckpt \
  out_dir=/workdir/output/reconstruction/yj/tj_short/iNWfKMyAewtQ97FBHQZyE3/test \
  dataset.val_camera_frame_step=1

Exception:

OverrideParseException
LexerNoViableAltException: dataset.path=/workdir/dataset/yj/tj_short/my_custom_drive_001.json

Temporary solution

Removing the specified parameters and moving them to parsed.yaml allows the program to run normally.

dataset.path=/workdir/dataset/yj/tj_short/my_custom_drive_001.json \
resume=/workdir/output/reconstruction/yj/tj_short/iNWfKMyAewtQ97FBHQZyE3/checkpoints/last.ckpt \
out_dir=/workdir/output/reconstruction/yj/tj_short/iNWfKMyAewtQ97FBHQZyE3/test \
dataset.val_camera_frame_step=1

docker run --shm-size=64g --rm --gpus '"device=7"' \
  --volume /share/home/hcy/NuRec/data/docker/dataset:/workdir/dataset \
  --volume /share/home/hcy/NuRec/data/docker/output:/workdir/output \
  nvcr.io/nvidia/nre/nre-ga:latest \
  --config-name=/workdir/output/reconstruction/yj/tj_short/iNWfKMyAewtQ97FBHQZyE3/config/parsed.yaml \
  mode=val

Validation.log (12.6 KB)

image
image1434×686 121 KB

2

It could be the hidden/trailing white-space after .json. likely from line-continuation/copy-paste.

you might want to use explicit, shell-quoted overrides :

‘mode=val’ \
‘dataset.path=/workdir/dataset/yj/tj_short/my_custom_drive_001.json’ \
‘resume=/workdir/output/reconstruction/yj/tj_short/iNWfKMyAewtQ97FBHQZyE3/checkpoints/last.ckpt’ \
‘out_dir=/workdir/output/reconstruction/yj/tj_short/iNWfKMyAewtQ97FBHQZyE3/test’ \
‘dataset.val_camera_ids=[cam_front]’ \
‘dataset.val_lidar_ids=[lidar_middle]’ \
‘dataset.val_camera_frame_step=1’

3

It’s working normally now, thank you very much.

4

Great to hear !