Riva 2.16 quick start error - riva_init.sh - invalid API key

Please provide the following information when requesting support.

Hardware - GPU (L40S)
Hardware - CPU(AMD EPYC 9654)
Operating System - Ubuntu 22.04 LTS Server
Riva Version - 2.16
TLT Version (if relevant)
How to reproduce the issue ? (This is for errors. Please share the command and the detailed log here)

I have successfully installed and tested with sample commands using url of the 3 Riva NIM from NGC - ASR, TTS, NMT.

My $NGC_API_KEY is valid and I can login successfully using either docker or ngc_cli:

echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin

However, I run into api key issue while followig Quick Start Guide — NVIDIA Riva

~/NVIDIA/riva_quickstart_v2.16.0$ bash riva_init.sh
Logging into NGC docker registry if necessary…
Pulling required docker images if necessary…
Note: This may take some time, depending on the speed of your Internet connection.

Pulling Riva Speech Server images.
Image nvcr.io/nvidia/riva/riva-speech:2.16.0 exists. Skipping.
Image nvcr.io/nvidia/riva/riva-speech:2.16.0-servicemaker exists. Skipping.

Downloading models (RMIRs) from NGC…
Note: this may take some time, depending on the speed of your Internet connection.
To skip this process and use existing RMIRs set the location and corresponding flag in config.sh.

==========================
=== Riva Speech Skills ===

NVIDIA Release (build 99025723)
Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Copyright (c) 2018-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:

To install Python sample dependencies, run /opt/tensorrt/python/python_setup.sh

To install the open-source samples corresponding to this TensorRT release version
run /opt/tensorrt/install_opensource.sh. To build the open source parsers,
plugins, and samples for current top-of-tree on master or a different branch,
run /opt/tensorrt/install_opensource.sh -b
See GitHub - NVIDIA/TensorRT: NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT. for more information.

2024-08-07 21:20:06 URL:https://files.ngc.nvidia.com/org/nvidia/team/ngc-apps/recipes/ngc_cli/versions/3.26.0/files/ngccli_linux.zip?versionId=45HwDObJnPZppQsIVrIVZy1ZDl9g1OA0&Expires=1723152006&Signature=CAztWW3PVFyzWI5g9KeChmkA3fHz-lGkCI3rmADVSrZI86fDvq1T0zLKu~8DgTiwKizFqZrj2DxeehQ7pvSYhbcBFAcvSDmMIntqEfM-whDVVTuDFfgN0NOirxCxu2ACT9XYR3XKt00b6JO9vB-87FbMewaCLLMIwY~oVqxna8XLkk3ipKSvD14WfB4dQAEDzHoVIQCS2l6lVVLpkSXhQZ5YtF1AEgcqN10~7hhVzqVIvTQ6aLQ5vK6zXVyO-QQUgrDhNgwxcHOnxYp-E~zvGXjJmfetdM6NWlFDb6jtENhnhQJ-uMl4xIrv9QHlbUszF7IgBNhKny3wZ-wIhTxzgw__&Key-Pair-Id=KCX06E8E9L60W [45878930/45878930] → “ngccli_linux.zip” [1]
/opt/riva
/data/rmir /opt/riva

Downloading nvidia/riva/rmir_asr_conformer_en_us_str:2.16.0…
Error: Invalid apikey
Attempt 1 out of 3 failed
Trying again…
Error: Invalid apikey
Attempt 2 out of 3 failed
Trying again…
Error: Invalid apikey
Attempt 3 out of 3 failed
Error occurred downloading ‘nvidia/riva/rmir_asr_conformer_en_us_str:2.16.0’. Exiting.
Error in downloading models.

My NGC_API_KEY is fine and I can use my key to pull Riva NIMs without issue.

$ echo “$NGC_API_KEY” | docker login nvcr.io --username ‘$oauthtoken’ --password-stdin
WARNING! Your password will be stored unencrypted in /home/user/.docker/config.json.
Configure a credential helper to remove this warning. See
docker login | Docker Docs

Login Succeeded

However, seems my key does not work for the following containers as I extracted the docker command from riva_init.sh:

$ docker run
–init -it --rm --gpus “device=0” -v riva-model-repo:/data -e NGC_CLI_API_KEY=$NGC_API_KEY
-e NGC_CLI_ORG=nvidia --name riva-service-maker nvcr.io/nvidia/riva/riva-speech:2.16.0-se
rvicemaker download_ngc_models nvidia/riva/rmir_asr_conformer_en_us_str:2.16.0 nvidia/riva/rmir
_asr_conformer_en_us_ofl:2.16.0 nvidia/riva/rmir_nlp_punctuation_bert_base_en_us:2.16.0 nvidia/r
iva/rmir_nlp_punctuation_bert_base_en_us:2.16.0 nvidia/riva/rmir_tts_fastpitch_hifigan_en_us_ipa
:2.16.0

==========================
=== Riva Speech Skills ===

NVIDIA Release (build 99025723)
Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Copyright (c) 2018-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:

To install Python sample dependencies, run /opt/tensorrt/python/python_setup.sh

To install the open-source samples corresponding to this TensorRT release version
run /opt/tensorrt/install_opensource.sh. To build the open source parsers,
plugins, and samples for current top-of-tree on master or a different branch,
run /opt/tensorrt/install_opensource.sh -b
See GitHub - NVIDIA/TensorRT: NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT. for more information.

2024-08-07 21:48:04 URL:https://files.ngc.nvidia.com/org/nvidia/team/ngc-apps/recipes/ngc_cli/versions/3.26.0/files/ngccli_linux.zip?versionId=45HwDObJnPZppQsIVrIVZy1ZDl9g1OA0&Expires=1723153683&Signature=fmLRROc5Urbr3HDVVgYPOFf1YrgsEyasrfWKpPu08YZm~QKVGqRagVDnOAz3pDSGfiy2CAblyGxxYP5xirKfjZKgvJxlF35XAjPhi8wZqYR2IvxrKOlgvR-NaQwmEZaGzIwumlbx0pTmwt4NvIGAeDb9~qaoub-DDc6xJEHawvXB6A37xHlSTRz911EProOIUPkNZsYt5-L1z-4ESpHjNAjJBwFQGDiNBhLlHn4NFcJhchCcc9m~97gDr~82xH2aeYHd9vxKBhDaQJlQTg5OPa8SSIa4mH2LRpZ-eUOY~ZLPGv8GT3Z7dozgIQBUucCHWexGdBXaght5-G~1UdowLg__&Key-Pair-Id=KCX06E8E9L60W [45878930/45878930] → “ngccli_linux.zip” [1]
/opt/riva
You can now run: /tmp/aws/aws --version
/opt/riva
/data/rmir /opt/riva

Downloading nvidia/riva/rmir_asr_conformer_en_us_str:2.16.0…
Error: Invalid apikey
Attempt 1 out of 3 failed
Trying again…
Error: Invalid apikey
Attempt 2 out of 3 failed
Trying again…
Error: Invalid apikey
Attempt 3 out of 3 failed
Error occurred downloading ‘nvidia/riva/rmir_asr_conformer_en_us_str:2.16.0’. Exiting.

Now I narrowed the error to the download_ngc_models command inside the container:

I can pull the container of nvcr.io/nvidia/riva/riva-speech:2.16.0-servicemaker

$docker run --init -it --rm --gpus “device=0”
-v riva-model-repo:/data
-e NGC_CLI_API_KEY=nvapi-aXXXXXX1zg35qFZZ
-e NGC_CLI_ORG=nvidia
–name riva-service-maker
nvcr.io/nvidia/riva/riva-speech:2.16.0-servicemaker

Then I entered the container and run download_ngc_models nvidia/riva/rmir_asr_conformer_en_us_str:2.16.0

Note that my NGC_CLI_API_KEY was passed to the container successfully:

root@a440d114e35f:/opt/riva# echo $NGC_CLI_API_KEY
nvapi-aXXXXXX1zg35qFZZ

root@a440d114e35f:/opt/riva#ngc registry nvidia/riva/rmir_asr_confdownload_ngc_models
/tmp/ngc-cli/ngc
/tmp/aws/aws
/data/rmir /opt/riva
/opt/riva
root@a440d114e35f:/opt/riva# download_ngc_models nvidia/riva/rmir_asr_conformer_en_us_str:2.16.0
/tmp/ngc-cli/ngc
/tmp/aws/aws
/data/rmir /opt/riva

Downloading nvidia/riva/rmir_asr_conformer_en_us_str:2.16.0…
Error: Invalid apikey
Attempt 1 out of 3 failed
Trying again…
Error: Invalid apikey
Attempt 2 out of 3 failed
Trying again…
Error: Invalid apikey
Attempt 3 out of 3 failed
Error occurred downloading ‘nvidia/riva/rmir_asr_conformer_en_us_str:2.16.0’. Exiting.

Turns out I can download those models outside the riva-service-maker container, which makes the api error while downloading inside the container interesting…

$ echo “$NGC_API_KEY” | docker login nvcr.io --username ‘$oauthtoken’ --password-stdin
WARNING! Your password will be stored unencrypted in /home/user/.docker/config.json.
Configure a credential helper to remove this warning. See
docker login | Docker Docs

Login Succeeded

$ ngc-cli/ngc registry model download-version nvidia/riva/rmir_asr_conformer_en_us_str:2.16.0
{
“download_end”: “2024-08-07 15:20:00”,
“download_start”: “2024-08-07 15:19:26”,
“download_time”: “33s”,
“files_downloaded”: 1,
“local_path”: “/home/user/NVIDIA/rmir_asr_conformer_en_us_str_v2.16.0”,
“size_downloaded”: “929.76 MB”,
“status”: “COMPLETED”
}

$ ngc-cli/ngc registry model download-version nvidia/riva/rmir_nlp_punctuation_bert_base_en_us:2.16.0
{
“download_end”: “2024-08-07 15:25:42”,
“download_start”: “2024-08-07 15:25:33”,
“download_time”: “8s”,
“files_downloaded”: 1,
“local_path”: “/home/user/NVIDIA/rmir_nlp_punctuation_bert_base_en_us_v2.16.0”,
“size_downloaded”: “384.79 MB”,
“status”: “COMPLETED”
}

Okay so I ended up by using ngc to download those 4 models directly then pass them over to riva_init.sh by editing the config.sh:

nvidia/riva/rmir_asr_conformer_en_us_str:2.16.0
nvidia/riva/rmir_asr_conformer_en_us_ofl:2.16.0
nvidia/riva/rmir_nlp_punctuation_bert_base_en_us:2.16.0
nvidia/riva/rmir_tts_fastpitch_hifigan_en_us_ipa:2.16.0

$ ngc-cli/ngc registry model download-version nvidia/riva/rmir_asr_conformer_en_us_str:2.16.0
{
“download_end”: “2024-08-07 15:20:00”,
“download_start”: “2024-08-07 15:19:26”,
“download_time”: “33s”,
“files_downloaded”: 1,
“local_path”: “/home/user/NVIDIA/rmir_asr_conformer_en_us_str_v2.16.0”,
“size_downloaded”: “929.76 MB”,
“status”: “COMPLETED”
}

riva_model_loc=“/home/user/NVIDIA/riva_2.16_models_for_docker”
use_existing_rmirs=true

then run $bash config.sh && bash riva_init.sh

2 Likes

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.