nvcr.io/nim/nvidia/corrdiff:1.1.0 fails at startup with ManifestDownloadError
because the internal model manifest references the nvstaging NGC org, which
external NVAIE subscribers cannot access.
EXACT ERROR (from docker logs):
tokio.rs: “nvstaging/nim/earth2-corrdiff-us-gefs-hrrr:1.1.0”:
fetching filemap from:
https://api.ngc.nvidia.com/v2/org/nvstaging/team/nim/models/earth2-corrdiff-us-gefs-hrrr/1.1.0/files
nimlib.exceptions.ManifestDownloadError: Error downloading manifest:
Permission error: The requested operation requires permissions that the
user does not have. This may be due to the user not being a member of
the organization that owns the repo.
ENVIRONMENT:
- Org: Substance AI (NVAIE Essentials active via AWS AMI Redemption,
term valid through 2026-05-08) - Host: AWS g6e.2xlarge, NVIDIA GPU Cloud VMI Enterprise 7.0.0
(AMI ami-06b20703476428553) - GPU: L40S 48 GB (meets 40 GB minimum)
- VM registered via ngc-token.sh -n: “Request status: SUCCESS”
- Docker pull nvcr.io/nim/nvidia/corrdiff:1.1.0 succeeds (digest
sha256:cc4a92a26defe92bf62bd0425125a91ecd119825fd479a6ef5e7a34321c54c52) - Launch command: verbatim from official deployment guide
WHAT WE VERIFIED:
- Multiple Personal API Keys (with NGC Catalog + Public API Endpoints +
Secrets Manager scopes): all fail identically - Same failure on standard DLAMI and on NVAIE VMI
- list-model-profiles returns only one profile, pointing at nvstaging
- ngc registry model list ‘corrdiff’ shows production org has only
nim/nvidia/earth2-corrdiff-us-gefs-hrrr:1.0.0 — version 1.1.0 does not
exist in the production model registry - NIM 1.0.0 works end-to-end on the same host (confirms environment OK)
ROOT CAUSE HYPOTHESIS:
The 1.1.0 NIM container was published to the public container registry,
but the corresponding model weights were not promoted from nvstaging to
the production model registry. External subscribers cannot pull the model
manifest at runtime.
REQUEST:
Please either (a) publish earth2-corrdiff-us-gefs-hrrr:1.1.0 to the
production nim/nvidia/ org, or (b) rebuild and republish
nvcr.io/nim/nvidia/corrdiff:1.1.0 with a manifest that references
the production org.
IMPACT:
Using 1.0.0 as a workaround but seeing severe suppression of extreme
precipitation and wind — insufficient for our insurance catastrophe
modeling use case. 1.1.0 is blocked.