Hi
Thanks for your reply, I have set all the service enable flags to false and use_existing_rmirs to true. I also comment out the portion of checking NGC API key in riva_init.sh, here I attached my new config.sh and riva_init.sh
However, I still cannot initialize Riva properly due to the following error after running riva_init.sh
find: '/data/rmir': No sush file or directly
It came from the last docker command in riva_init.sh
docker run --init -it --rm --gpus '"'$gpus_to_use'"' \
-v $riva_model_loc:/data \
-e "MODEL_DEPLOY_KEY=${MODEL_DEPLOY_KEY}" \
--name riva-service-maker \
$image_init_speech deploy_all_models /data/rmir /data/models
Here are my questions:
- Where should I create the folder
$riva_model_loc? Is it under riva_quickstart_v1.6.0-beta?
- Should I download the models first before running riva_init.sh?
- Where could I find theses models on NGC? (i.e. I see
"${riva_ngc_org}/${riva_ngc_team}/rmir_nlp_punctuation_bert_base:${riva_ngc_model_version}") in the config.sh as one of the ASR models, how could I find it on NGC? Querying “Punctuation model” gives me too many results and I have no way to filter further
- and where should I store these models locally? Should I keep them in the folder
$riva_model_loc/rmir ?
Config.sh
# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
#
# NVIDIA CORPORATION and its licensors retain all intellectual property
# and proprietary rights in and to this software, related documentation
# and any modifications thereto. Any use, reproduction, disclosure or
# distribution of this software and related documentation without an express
# license agreement from NVIDIA CORPORATION is strictly prohibited.
# Enable or Disable Riva Services
service_enabled_asr=false
service_enabled_nlp=false
service_enabled_tts=false
# Specify one or more GPUs to use
# specifying more than one GPU is currently an experimental feature, and may result in undefined behaviours.
gpus_to_use="device=0"
# Specify the encryption key to use to deploy models
MODEL_DEPLOY_KEY="tlt_encode"
# Locations to use for storing models artifacts
#
# If an absolute path is specified, the data will be written to that location
# Otherwise, a docker volume will be used (default).
#
# riva_init.sh will create a `rmir` and `models` directory in the volume or
# path specified.
#
# RMIR ($riva_model_loc/rmir)
# Riva uses an intermediate representation (RMIR) for models
# that are ready to deploy but not yet fully optimized for deployment. Pretrained
# versions can be obtained from NGC (by specifying NGC models below) and will be
# downloaded to $riva_model_loc/rmir by `riva_init.sh`
#
# Custom models produced by NeMo or TLT and prepared using riva-build
# may also be copied manually to this location $(riva_model_loc/rmir).
#
# Models ($riva_model_loc/models)
# During the riva_init process, the RMIR files in $riva_model_loc/rmir
# are inspected and optimized for deployment. The optimized versions are
# stored in $riva_model_loc/models. The riva server exclusively uses these
# optimized versions.
riva_model_loc="riva-model-repo"
# The default RMIRs are downloaded from NGC by default in the above $riva_rmir_loc directory
# If you'd like to skip the download from NGC and use the existing RMIRs in the $riva_rmir_loc
# then set the below $use_existing_rmirs flag to true. You can also deploy your set of custom
# RMIRs by keeping them in the riva_rmir_loc dir and use this quickstart script with the
# below flag to deploy them all together.
use_existing_rmirs=true
# Ports to expose for Riva services
riva_speech_api_port="50051"
riva_vision_api_port="60051"
# NGC orgs
riva_ngc_org="nvidia"
riva_ngc_team="riva"
riva_ngc_image_version="1.6.0-beta"
riva_ngc_model_version="1.6.0-beta"
# Pre-built models listed below will be downloaded from NGC. If models already exist in $riva-rmir
# then models can be commented out to skip download from NGC
########## ASR MODELS ##########
models_asr=(
### Punctuation model
"${riva_ngc_org}/${riva_ngc_team}/rmir_nlp_punctuation_bert_base:${riva_ngc_model_version}"
### Citrinet-1024 Streaming w/ CPU decoder, best latency configuration
"${riva_ngc_org}/${riva_ngc_team}/rmir_asr_citrinet_1024_asrset3p0_streaming:${riva_ngc_model_version}"
### Citrinet-1024 Streaming w/ CPU decoder, best throughput configuration
# "${riva_ngc_org}/${riva_ngc_team}/rmir_asr_citrinet_1024_asrset3p0_streaming_throughput:${riva_ngc_model_version}"
### Citrinet-1024 Offline w/ CPU decoder,
"${riva_ngc_org}/${riva_ngc_team}/rmir_asr_citrinet_1024_asrset3p0_offline:${riva_ngc_model_version}"
### Jasper Streaming w/ CPU decoder, best latency configuration
# "${riva_ngc_org}/${riva_ngc_team}/rmir_asr_jasper_english_streaming:${riva_ngc_model_version}"
### Jasper Streaming w/ CPU decoder, best throughput configuration
# "${riva_ngc_org}/${riva_ngc_team}/rmir_asr_jasper_english_streaming_throughput:${riva_ngc_model_version}"
### Jasper Offline w/ CPU decoder
# "${riva_ngc_org}/${riva_ngc_team}/rmir_asr_jasper_english_offline:${riva_ngc_model_version}"
### Quarztnet Streaming w/ CPU decoder, best latency configuration
# "${riva_ngc_org}/${riva_ngc_team}/rmir_asr_quartznet_english_streaming:${riva_ngc_model_version}"
### Quarztnet Streaming w/ CPU decoder, best throughput configuration
# "${riva_ngc_org}/${riva_ngc_team}/rmir_asr_quartznet_english_streaming_throughput:${riva_ngc_model_version}"
### Quarztnet Offline w/ CPU decoder
# "${riva_ngc_org}/${riva_ngc_team}/rmir_asr_quartznet_english_offline:${riva_ngc_model_version}"
### Jasper Streaming w/ GPU decoder, best latency configuration
# "${riva_ngc_org}/${riva_ngc_team}/rmir_asr_jasper_english_streaming_gpu_decoder:${riva_ngc_model_version}"
### Jasper Streaming w/ GPU decoder, best throughput configuration
# "${riva_ngc_org}/${riva_ngc_team}/rmir_asr_jasper_english_streaming_throughput_gpu_decoder:${riva_ngc_model_version}"
### Jasper Offline w/ GPU decoder
# "${riva_ngc_org}/${riva_ngc_team}/rmir_asr_jasper_english_offline_gpu_decoder:${riva_ngc_model_version}"
)
########## NLP MODELS ##########
models_nlp=(
### Bert base Punctuation model
"${riva_ngc_org}/${riva_ngc_team}/rmir_nlp_punctuation_bert_base:${riva_ngc_model_version}"
### BERT base Named Entity Recognition model fine-tuned on GMB dataset with class labels LOC, PER, ORG etc.
"${riva_ngc_org}/${riva_ngc_team}/rmir_nlp_named_entity_recognition_bert_base:${riva_ngc_model_version}"
### BERT Base Intent Slot model fine-tuned on weather dataset.
"${riva_ngc_org}/${riva_ngc_team}/rmir_nlp_intent_slot_bert_base:${riva_ngc_model_version}"
### BERT Base Question Answering model fine-tuned on Squad v2.
"${riva_ngc_org}/${riva_ngc_team}/rmir_nlp_question_answering_bert_base:${riva_ngc_model_version}"
### Megatron345M Question Answering model fine-tuned on Squad v2.
# "${riva_ngc_org}/${riva_ngc_team}/rmir_nlp_question_answering_megatron:${riva_ngc_model_version}"
### Bert base Text Classification model fine-tuned on 4class (weather, meteorology, personality, nomatch) domain model.
"${riva_ngc_org}/${riva_ngc_team}/rmir_nlp_text_classification_bert_base:${riva_ngc_model_version}"
)
########## TTS MODELS ##########
models_tts=(
"${riva_ngc_org}/${riva_ngc_team}/rmir_tts_fastpitch_hifigan_ljspeech:${riva_ngc_model_version}"
# "${riva_ngc_org}/${riva_ngc_team}/rmir_tts_tacotron_waveglow_ljspeech:${riva_ngc_model_version}"
)
NGC_TARGET=${riva_ngc_org}
if [[ ! -z ${riva_ngc_team} ]]; then
NGC_TARGET="${NGC_TARGET}/${riva_ngc_team}"
else
team="\"\""
fi
# define docker images required to run Riva
image_client="nvcr.io/${NGC_TARGET}/riva-speech-client:${riva_ngc_image_version}"
image_speech_api="nvcr.io/${NGC_TARGET}/riva-speech:${riva_ngc_image_version}-server"
# define docker images required to setup Riva
image_init_speech="nvcr.io/${NGC_TARGET}/riva-speech:${riva_ngc_image_version}-servicemaker"
# daemon names
riva_daemon_speech="riva-speech"
riva_daemon_client="riva-client"
riva_init.sh
#!/bin/bash
# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
#
# NVIDIA CORPORATION and its licensors retain all intellectual property
# and proprietary rights in and to this software, related documentation
# and any modifications thereto. Any use, reproduction, disclosure or
# distribution of this software and related documentation without an express
# license agreement from NVIDIA CORPORATION is strictly prohibited.
get_ngc_key_from_environment() {
# first check the global NGC_API_KEY environment variable
local ngc_key=$NGC_API_KEY
# if env variable was not set, and a ~/.ngc/config exists
# try to get it from there
if [ -z "$ngc_key" ] && [[ -f "$HOME/.ngc/config" ]]
then
ngc_key=$(cat $HOME/.ngc/config | grep -m 1 -G "^\s*apikey\s*=.*" | sed 's/^\s*apikey\s*=\s*//g')
fi
echo $ngc_key
}
docker_pull() {
image_exists=$(docker images --filter=reference=$1 -q | wc -l)
if [[ $image_exists -eq 1 ]]; then
echo " > Image $1 exists. Skipping."
return
fi
attempts=3
echo " > Pulling $1. This may take some time..."
for ((i = 1 ; i <= $attempts ; i++)); do
docker pull -q $1 &> /dev/null
if [ $? -ne 0 ]; then
echo " > Attempt $i out of $attempts failed"
if [ $i -eq $attempts ]; then
echo "Error occurred pulling '$1'."
docker pull $1
echo "Exiting."
exit 1
else
echo " > Trying again..."
continue
fi
else
break
fi
done
}
check_docker_version() {
version_string=$(docker version --format '{{.Server.Version}}')
if [ $? -ne 0 ]; then
echo "Unable to run Docker. Please check that Docker is installed and functioning."
exit 1
fi
maj_ver=$(echo $version_string | awk -F. '{print $1}')
min_ver=$(echo $version_string | awk -F. '{print $2}')
if [ "$maj_ver" -lt "19" ] || ([ "$maj_ver" -eq "19" ] && [ "$min_ver" -lt "03" ]); then
echo "Docker version insufficient. Please use Docker 19.03 or later"
exit 1;
fi
}
# BEGIN SCRIPT
#check_docker_version
# load config file
script_path="$( cd "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )"
if [ -z "$1" ]; then
config_path="${script_path}/config.sh"
else
config_path=$(readlink -f $1)
fi
if [[ ! -f $config_path ]]; then
echo 'Unable to load configuration file. Override path to file with -c argument.'
exit 1
fi
source $config_path || exit 1
# automatically get NGC_API_KEY or request from user if necessary
#NGC_API_KEY="$(get_ngc_key_from_environment)"
#if [ -z "$NGC_API_KEY" ]; then
# read -sp 'Please enter API key for ngc.nvidia.com: ' NGC_API_KEY
# echo
#fi
# use the API key to run docker login for the NGC registry
# exit early if the key is invalid, because we won't be able to do anything
#echo "Logging into NGC docker registry if necessary..."
#echo $NGC_API_KEY | docker login -u '$oauthtoken' --password-stdin nvcr.io &> /dev/null
#if [ $? -ne 0 ]; then
# echo 'NGC API Key is invalid. Please check and try again.'
# exit 1
#fi
# pull all the requisite images we're going to need
echo "Pulling required docker images if necessary..."
echo "Note: This may take some time, depending on the speed of your Internet connection."
# pull the speech server if any of asr/nlp/tts services are requested
if [ "$service_enabled_asr" = true ] || [ "$service_enabled_nlp" = true ] || [ "$service_enabled_tts" = true ]; then
echo "> Pulling Riva Speech Server images."
docker_pull $image_speech_api
docker_pull $image_client
docker_pull $image_init_speech
fi
if [ "$use_existing_rmirs" = false ]; then
echo
echo "Downloading models (RMIRs) from NGC..."
echo "Note: this may take some time, depending on the speed of your Internet connection."
echo "To skip this process and use existing RMIRs set the location and corresponding flag in config.sh."
# build up commands to download from NGC
if [ "$service_enabled_asr" = true ] || [ "$service_enabled_nlp" = true ] || [ "$service_enabled_tts" = true ]; then
gmr_speech_models=""
if [ "$service_enabled_asr" = true ]; then
for model in ${models_asr[@]}; do
gmr_speech_models+=" $model"
done
fi
if [ "$service_enabled_nlp" = true ]; then
for model in ${models_nlp[@]}; do
gmr_speech_models+=" $model"
done
fi
if [ "$service_enabled_tts" = true ]; then
for model in ${models_tts[@]}; do
gmr_speech_models+=" $model"
done
fi
# download required images
docker run --init -it --rm --gpus '"'$gpus_to_use'"' \
-v $riva_model_loc:/data \
-e "NGC_CLI_API_KEY=$NGC_API_KEY" \
-e "NGC_CLI_ORG=nvidia" \
--name riva-service-maker \
$image_init_speech download_ngc_models $gmr_speech_models
if [ $? -ne 0 ]; then
echo "Error in downloading models."
exit 1
fi
fi
fi
# convert all rmirs
echo
echo "Converting RMIRs at $riva_model_loc/rmir to Riva Model repository."
set -x
docker run --init -it --rm --gpus '"'$gpus_to_use'"' \
-v $riva_model_loc:/data \
-e "MODEL_DEPLOY_KEY=${MODEL_DEPLOY_KEY}" \
--name riva-service-maker \
$image_init_speech deploy_all_models /data/rmir /data/models
echo
echo "Riva initialization complete. Run ./riva_start.sh to launch services."
