## Summary
`nre-ga main mode=train` (and `nre-tools-ga ncore-aux-data
--segmentation-backend=mask2former`) raise a bare `KeyError: ‘<camera_id>’`
immediately after the three component stores finish loading, before any
training step runs. The store is a fully custom NCore V4 sequence built with
the public `nvidia-ncore==19.0.0` Python API; `SequenceComponentGroupsReader`
and `SequenceLoaderV4` round-trip the store correctly and expose the expected
`sequence_id`, `camera_ids`, `lidar_ids`, intrinsics, and extrinsics.
The error appears to originate inside the `pycena_run_cc` binary in
a `TimestampFrameSampler.sample_frame` codepath that indexes
`cameras_frame_timestamps_us[unique_sensor_id]`. Whatever populates that dict
uses a different key than the bare `camera_id` we put into the store.
We’ve reproduced the failure with:
- multiple sensor-name conventions (Kodiak native, Hyperion-8.1
`camera_front_wide_120fov` / `camera_front_tele_30fov` /
`camera_cross_right_120fov` / `lidar_gt_top_p128_v4p5`, and Waymo
`camera_front_50fov` / `lidar_top`);
- multiple training configs (`apps/prod/Hyperion-8.1/car2sim_6cam.yaml`,
`apps/prod/Hyperion-8.1/sqa_default.yaml`, `apps/prod/Waymo/sqa_default.yaml`);
- 1, 2, and 3 cameras in the store;
- with and without `rig_properties.platform_name = “hy8.1_*”` in the manifest
`generic_meta_data`.
We’d like guidance on (a) which exact NCore V4 keying/metadata the trainer
expects beyond what the public NCore docs spell out, and ideally (b) a tiny
reference NCore V4 dataset we can diff against.
Environment
| Component | Version |
| --------------------- | ------------------------------------------------------------------------------------------------ |
| Host OS | Ubuntu 22.04 (`Linux 5.15.0-176-generic x86_64`) |
| GPU / Driver | NVIDIA GeForce RTX 3090 / 580.105.08 |
| Docker | 29.3.0 |
| `nre-ga` image | `nvcr.io/nvidia/nre/nre-ga:latest` (`sha256:dbb6be50cabc…`, built 2026-02-04) |
| `nre-tools-ga` image | `nvcr.io/nvidia/nre/nre-tools-ga:latest` (`sha256:0ffe8dccab71…`, built 2026-02-04) |
| `nvidia-ncore` (PyPI) | `19.0.0` |
| Python | 3.11 (matching the runfiles interpreter inside the image) |
## What works
1. Building the store with the public NCore V4 API
(`SequenceComponentGroupsWriter` + `PosesComponent.Writer`,
`IntrinsicsComponent.Writer`, `CameraSensorComponent.Writer`,
`LidarSensorComponent.Writer`, plus empty `MasksComponent`/`CuboidsComponent`
groups required by `nre-tools-ga`).
2. Reading the store back with `SequenceComponentGroupsReader` and
`SequenceLoaderV4` — `loader.sequence_id`, `loader.camera_ids`,
`loader.lidar_ids`, `loader.pose_graph`, and `loader.get_camera_sensor(id)`
all return the values we put in.
3. `nre-tools-ga ncore-aux-data --segmentation-backend=none --no-ego-mask
--no-lidar-seg-camvis --depth-backend=none --store-meta` accepts the
manifest without errors. This confirms the V4 schema validates against
NVIDIA’s loader.
## What fails
```text
SequenceStoreReader: Loading component store …/kodiak_.ncore4-camera_front_50fov.zarr.itar
SequenceStoreReader: Loading component store …/kodiak_.ncore4-lidar_top.zarr.itar
SequenceStoreReader: Loading component store …/kodiak_.ncore4.zarr.itar
KeyError
‘camera_front_50fov’
```
(Exit 255, no Python traceback even with `HYDRA_FULL_ERROR=1` and
`PYTHONFAULTHANDLER=1` — see “Traceback suppression” below.)
The same error is produced by `nre-tools-ga ncore-aux-data
--segmentation-backend=mask2former` immediately after the semantic
segmentation checkpoint loads, with the same `KeyError` keyed by the same
`<camera_id>`.
## Minimal repro
### 1. Build a single-camera NCore V4 sequence
This script depends only on `nvidia-ncore==19.0.0`, `numpy`, `scipy`, `opencv-python`,
`typed-argument-parser`, and `universal_pathlib`. It writes one image, one
identity rig pose, identity sensor extrinsics, and a trivial intrinsics block:
```python
# tiny_ncore_repro.py
import io
import numpy as np
import cv2
from upath import UPath
from ncore.data.v4 import (
CameraSensorComponent, CuboidsComponent, IntrinsicsComponent,
LidarSensorComponent, MasksComponent, PosesComponent,
SequenceComponentGroupsWriter,
)
from ncore.impl.common.transformations import HalfClosedInterval
from ncore.impl.data.types import (
OpenCVPinholeCameraModelParameters,
RowOffsetStructuredSpinningLidarModelParameters,
ShutterType,
)
OUT = UPath(“/tmp/ncore_repro”)
OUT.mkdir(parents=True, exist_ok=True)
SEQ = “tiny_seq_0001”
CAM = “camera_front_50fov” # also tried Hyperion + Kodiak-native names; same error
LID = “lidar_top”
T0 = 1_700_000_000_000_000
INTERVAL = HalfClosedInterval(T0, T0 + 1_000_000)
writer = SequenceComponentGroupsWriter(
output_dir_path=OUT,
store_base_name=f"{SEQ}",
sequence_id=SEQ,
sequence_timestamp_interval_us=INTERVAL,
store_type=“itar”,
generic_meta_data={
# Tried with and without; same KeyError either way.
“rig_properties”: {
“platform_name”: “hy8.1_tiny”, “layout”: “hyperion_8.1_tiny”,
},
},
)
# Identity rig trajectory (poses are written rebased so the first frame is
# the world origin; this matches the fix from NVIDIA/ncore#77).
poses = writer.register_component_writer(PosesComponent.Writer, “default”, group_name=None)
poses.store_dynamic_pose(
source_frame_id=“rig”, target_frame_id=“world”,
poses=np.tile(np.eye(4, dtype=np.float64)[None], (2, 1, 1)),
timestamps_us=np.array([T0, T0 + 500_000], dtype=np.uint64),
)
# (world, world_global) anchor — required as of NCore #76*, identity here.*
poses.store_static_pose(“world”, “world_global”, np.eye(4, dtype=np.float32))
poses.store_static_pose(CAM, “rig”, np.eye(4, dtype=np.float32))
poses.store_static_pose(LID, “rig”, np.eye(4, dtype=np.float32))
# Trivial pinhole intrinsics
intr = writer.register_component_writer(IntrinsicsComponent.Writer, “default”, group_name=None)
intr.store_camera_intrinsics(
CAM,
OpenCVPinholeCameraModelParameters(
resolution=np.array([640, 480], dtype=np.uint64),
shutter_type=ShutterType.ROLLING_TOP_TO_BOTTOM,
principal_point=np.array([320.0, 240.0], dtype=np.float32),
focal_length=np.array([500.0, 500.0], dtype=np.float32),
radial_coeffs=np.zeros(6, dtype=np.float32),
tangential_coeffs=np.zeros(2, dtype=np.float32),
thin_prism_coeffs=np.zeros(4, dtype=np.float32),
),
)
intr.store_lidar_intrinsics(
LID,
RowOffsetStructuredSpinningLidarModelParameters(
spinning_frequency_hz=10.0, spinning_direction=“ccw”,
n_rows=128, n_columns=3600,
row_elevations_rad=np.linspace(0.25, -0.43, 128, dtype=np.float32),
column_azimuths_rad=np.linspace(-np.pi, np.pi, 3600, endpoint=False, dtype=np.float32),
row_azimuth_offsets_rad=np.zeros(128, dtype=np.float32),
),
)
# nre-tools-ga requires these groups to exist even if empty
writer.register_component_writer(MasksComponent.Writer, “default”, group_name=None)
writer.register_component_writer(CuboidsComponent.Writer, “default”, group_name=None)
# One camera frame
img = (np.random.rand(480, 640, 3) * 255).astype(np.uint8)
ok, png = cv2.imencode(“.png”, img)
assert ok
cam = writer.register_component_writer(CameraSensorComponent.Writer, CAM, group_name=CAM)
cam.store_frame(png.tobytes(), “png”, np.array([T0 + 200_000, T0 + 200_000], dtype=np.uint64), {}, {})
# One lidar frame
lidar = writer.register_component_writer(LidarSensorComponent.Writer, LID, group_name=LID)
n = 1024
direction = np.random.randn(n, 3).astype(np.float32)
direction /= np.linalg.norm(direction, axis=1, keepdims=True)
lidar.store_frame(
direction,
(T0 + 300_000) * np.ones(n, dtype=np.uint64),
None,
np.full((1, n), 5.0, *dtype*=np.float32),
np.full((1, n), 0.5, *dtype*=np.float32),
np.array(\[T0 + 300_000, T0 + 300_000\], *dtype*=np.uint64),
{}, {},
)
stores = writer.finalize()
import json
from ncore.data.v4 import SequenceComponentGroupsReader
reader = SequenceComponentGroupsReader(stores, open_consolidated=True)
with open(OUT / f"{SEQ}.json", “w”) as f:
json.dump(reader.get_sequence_meta().to_dict(), f, *indent*=2, *sort_keys*=True)
print(“wrote”, OUT / f"{SEQ}.json")
```
### 2. Run training
```bash
docker run --rm --gpus all \
-e NGC_API_KEY=“$NGC_API_KEY” \
-e HYDRA_FULL_ERROR=1 \
--volume /tmp/ncore_repro:/workdir/dataset \
--volume /tmp/ncore_repro_out:/workdir/output \
nvcr.io/nvidia/nre/nre-ga:latest \
main \
--config-name=configs/apps/prod/Waymo/sqa_default.yaml \
mode=train \
out_dir=/workdir/output \
dataset.path=/workdir/dataset/tiny_seq_0001.json \
‘dataset.camera_ids=[camera_front_50fov]’ \
‘dataset.val_camera_ids=[camera_front_50fov]’ \
‘dataset.lidar_ids=[lidar_top]’ \
dataset.aux_data=False
```
### 3. Observed output (last 12 lines)
```text
[INFO] World Size: 1
[INFO] Node Count: 1
[INFO] Device Count per Node: 1
[INFO] Log every 50 steps
Seed set to 42
[INFO] Setting up TensorProber with config
RUN id: …
[INFO] Number of checkpoints to save: 30
[INFO] SODataModule: train_num_workers=24 val_num_workers=8 test_num_workers=8
[INFO] SequenceStoreReader: Loading component store …/tiny_seq_0001.ncore4-camera_front_50fov.zarr.itar
[INFO] SequenceStoreReader: Loading component store …/tiny_seq_0001.ncore4-lidar_top.zarr.itar
[INFO] SequenceStoreReader: Loading component store …/tiny_seq_0001.ncore4.zarr.itar
KeyError
‘camera_front_50fov’
```
Exit code 255. No traceback.
## What we’ve verified about the store
```python
from upath import UPath
from ncore.data.v4 import SequenceComponentGroupsReader, SequenceLoaderV4
import glob
paths = sorted(UPath(p) for p in glob.glob(“/tmp/ncore_repro/*.zarr.itar”))
loader = SequenceLoaderV4(SequenceComponentGroupsReader(paths))
print(loader.sequence_id) # tiny_seq_0001
print(loader.camera_ids) # [‘camera_front_50fov’]
print(loader.lidar_ids) # [‘lidar_top’]
cam = loader.get_camera_sensor(“camera_front_50fov”)
print(cam.frames_count) # 1
print(cam.T_sensor_rig.shape) # (4, 4)
print(type(cam.model_parameters)._name_)
# OpenCVPinholeCameraModelParameters
```
All three lookups (`camera_front_50fov` in `IntrinsicsComponent`, in
`CameraSensorComponent` group names, and in `PosesComponent.get_static_pose(
“camera_front_50fov”, “rig”)`) succeed and return self-consistent metadata. The
JSON manifest reflects the same key — see “Manifest snippet” below.
## Manifest snippet
`tiny_seq_0001.json` — produced by `SequenceComponentGroupsReader(…).get_sequence_meta()`:
```json
{
“version”: “v4”,
“sequence_id”: “tiny_seq_0001”,
“sequence_timestamp_interval_us”: {“start”: 1700000000000000, “stop”: 1700000001000000},
“generic_meta_data”: {
“rig_properties”: {“platform_name”: “hy8.1_tiny”, “layout”: “hyperion_8.1_tiny”}
},
“component_stores”: [
{"path": "tiny_seq_0001.ncore4-camera_front_50fov.zarr.itar",
“components”: {“cameras”: {“camera_front_50fov”: {“generic_meta_data”: {}, “version”: “v1”}}}},
{"path": "tiny_seq_0001.ncore4-lidar_top.zarr.itar",
“components”: {“lidars”: {“lidar_top”: {“generic_meta_data”: {}, “version”: “v1”}}}},
{"path": "tiny_seq_0001.ncore4.zarr.itar",
“components”: {
“poses”: {“default”: {“generic_meta_data”: {}, “version”: “v1”}},
“intrinsics”: {“default”: {“generic_meta_data”: {}, “version”: “v1”}},
“masks”: {“default”: {“generic_meta_data”: {}, “version”: “v1”}},
“cuboids”: {“default”: {“generic_meta_data”: {}, “version”: “v1”}}
}}
]
}
```
## Findings from inspecting `pycena_run_cc`
The error message format (`KeyError\n’'` with no traceback) is produced by
a custom `sys.excepthook` installed by the obfuscated runner — see
`pycena_run_cc` ≈ offset 849069, function `11lllllll1lI1IlIIl11lllI1` calling
`sys.excepthook(BaseException, …)`. Setting `HYDRA_FULL_ERROR=1`,
`PYTHONFAULTHANDLER=1`, and injecting a `sitecustomize.py` via `PYTHONPATH` all
fail to surface the underlying traceback because the obfuscation layer
replaces `sys.excepthook` before user code runs.
Two relevant classes recovered from strings + bytecode shape:
```python
# pycena_run_cc, around offset 1716375
class SensorCalibProvider:
\_camera_ids: List\[str\]
\_sequence_id: str | None
def _init_(self, rig_trajectories):
self._camera_ids, self._sequence_id = _split(rig_trajectories)
def get_unique_sensor_id(self, camera_id: str) → str:
return f"{camera_id}@{self._sequence_id}" if self._sequence_id else camera_id
# pycena_run_cc, around offset 2273259
class TimestampFrameSampler:
cameras_frame_timestamps_us: Dict\[str, torch.Tensor\] *# populated from rig_trajectory*
lidars_frame_timestamps_us: Dict\[str, torch.Tensor\]
def _init_(self, dataset, …):
rt = dataset.get_datasource().get_rig_trajectories()
…
self.cameras_frame_timestamps_us = rt.cameras_frame_timestamps_us
def sample_frame(self, rng, batch_idx, frame_range, unique_sensor_id):
sensor_timestamps = self.cameras_frame_timestamps_us\[unique_sensor_id\]\[:, 0\].numpy()
# ^^^ KeyError: ‘<camera_id>’ here
…
```
The `KeyError`'s argument is the bare `camera_id` (no `@<sequence_id>`
suffix), so on this code path `_sequence_id` was `None`. That points at either
(a) some metadata field on the manifest / rig that NCore reads to populate
`_sequence_id`, which we’re not setting, or (b) the dict was populated under a
different key (e.g. logical sensor name vs. unique sensor id) and the
`sample_frame` lookup uses the bare form regardless.
Around offsets 986471 / 1865134 the rig-trajectory loader does iterate
`for uci in rig_trajectories: trajectory.camera_calibrations[uci].logical_sensor_name`,
which suggests the trajectory dict is keyed by some “unique camera id” derived
from the sequence. We haven’t been able to locate where that key is
materialized from the on-disk store.
## What we’ve tried that did not change the failure
| # | Variation | Result |
| - | ------------------------------------------------------------------------------------------------------------------- | ------------------------------------- |
| 1 | 3 cameras, Kodiak-native names (`camera_front_left`, …) | `KeyError: ‘camera_front_left’` |
| 2 | 3 cameras, Hyperion-8.1 names (`camera_front_tele_30fov`, `camera_front_wide_120fov`, `camera_cross_right_120fov`) | `KeyError: ‘camera_front_tele_30fov’` |
| 3 | 1 camera, Waymo names (`camera_front_50fov` + `lidar_top`) — minimal repro above | `KeyError: ‘camera_front_50fov’` |
| 4 | Added `rig_properties.platform_name = “hy8.1_kodiak_research”` + `layout` in `generic_meta_data` | unchanged |
| 5 | Training configs: `apps/prod/Hyperion-8.1/car2sim_6cam.yaml`, `…/sqa_default.yaml`, `apps/prod/Waymo/sqa_default.yaml` | all same error |
| 6 | `dataset.aux_data=False`, plus `–no-ego-mask`, `–no-lidar-seg-camvis` | unchanged |
| 7 | `HYDRA_FULL_ERROR=1` + `PYTHONFAULTHANDLER=1` + sitecustomize-injected `sys.excepthook` | traceback still suppressed |
| 8 | Applied the [#76 / #77]( Waymo converter produces poses incompatible with NuRec training: missing `(world, world_global)` static pose · Issue #76 · NVIDIA/ncore · GitHub ) fix — rebased `(rig, world)` dynamic poses to be local to the first frame and added `(world, world_global)` static pose | **unchanged** — same `KeyError`. Distinct from the `ValueError: World-to-world_global poses are currently required…` reported in #76. |
## Questions
1. **What populates `cameras_frame_timestamps_us`?** Is it keyed by the
`CameraSensorComponent` group name, the `IntrinsicsComponent` camera id,
the `PosesComponent` static-pose source frame id, or something else
entirely (e.g. a `logical_sensor_name` field we haven’t set)?
2. **How is `SensorCalibProvider._sequence_id` populated?** Is it pulled from
`sequence_meta.sequence_id`, from `rig_properties`, or from a separate
`RigTrajectoriesComponent`-style block we should be emitting? In our
repro, `SequenceLoaderV4(…).sequence_id` returns the right value, but
the `SensorCalibProvider` clearly didn’t pick it up (the `KeyError` key
has no `@` suffix).
3. **Are `MasksComponent` / `CuboidsComponent` / a `PointCloudsComponent`
required to be non-empty for `nre-ga` to construct its rig trajectory?**
We register empty `default` groups for the first two (required by
`nre-tools-ga`) and emit no point cloud component.
4. **Is there a published `tiny_dataset` / reference NCore V4 sample we can
download and diff against?** The NCore tutorial
([data_loading.html]( NCore V4 Data Loading — NCore ))
references a `ncore-demo/sequence-ncore4.json` but doesn’t ship a
downloadable copy of the underlying `.zarr.itar` files.
5. Is there a documented `–config` or env var to **disable the
short-exception `sys.excepthook`** inside `pycena_run_cc` so we can see
the actual traceback? We tried `NRE_DEBUGPY_*` but couldn’t get a normal
traceback path even with `debugpy` attached.
## Why we care
We’re building a research prototype that converts logs from a custom AV
platform (Kodiak Robotics’ autonomous semi truck) into NCore V4 so we can
evaluate NuRec for closed-loop simulation. Everything upstream of the trainer
works — image and lidar capture, calibration, pose data, NCore loader
round-trip, `nre-tools-ga` aux-data generation in minimal mode — but we can’t
get past the dataset-construction step inside `nre-ga`. A pointer to the right
rig metadata, a documented contract for what the loader expects beyond the
public NCore V4 schema, or a diffable reference dataset (e.g. the
`ncore-demo/sequence-ncore4.json` referenced in the
[data_loading tutorial]( NCore V4 Data Loading — NCore ))
would unblock us. We’re happy to share our converter if that helps diagnose.
Thanks!