Failed to get riva started

AI & Data Science Deep Learning (Training & Inference) Riva
1

Please provide the following information when requesting support.

Hardware - GPU: Geforce 3090
Hardware - CPU: 10900K
Operating System: Ubuntu 20.04
Riva Version: Beta 1.6.0

so i try to run riva following the quick start guide, while trying to execute bash riva_start.sh, i get the following error

Starting Riva Speech Services. This may take several minutes depending on the number of models deployed.
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Health ready check failed.
Check Riva logs with: docker logs riva-speech

and docker logs riva-speech gives:

Untitled Document 1 (167.7 KB)

too many text in log, so i have to upload the file.

any suggestion?

2

Hi,

Our team is looking into this issue. The server doesn’t start because of this error:

E1201 23:46:44.631690 69 model_repository_manager.cc:1946] Poll failed for model directory ‘riva-trt-citrinet-1024’: failed to open text file for read /data/models/riva-trt-citrinet-1024/config.pbtxt: No such file or directory

Could you please share completed logs of bash riva_start.sh

Thank you.

3

Could you please share the logs from when you ran riva_init? It looks like something went wrong optimizing CitriNet. You could try running riva_clean.sh and riva_init again.

4

@rleary ive got the same issue , brand new agx orin dev kit, followed instructions, even ran the riva_clean sh. here is my full output from riva_start.sh

dumped output of later half of start script and the docker error output logs

Please advise followed online docs to this point :/

5

HI @philipmullis

Thanks for your interest in Riva,

Apologies, the Pastebin link is not working
Can you share the output as file in this thread

Also Request to share the following

  1. config.sh used
  2. Jetpack Version used

Thanks

6

Thanks @rvinobha

Jetpack is version 5.0.1-b118

here is the config.sh

riva_target_gpu_family=“tegra”

Name of tegra platform that is being used. Supported tegra platforms: orin, xavier

riva_tegra_platform=“orin”

Enable or Disable Riva Services

service_enabled_asr=true
service_enabled_nlp=true
service_enabled_tts=true

Enable Riva Enterprise

If enrolled in Enterprise, enable Riva Enterprise by setting configuration

here. You must explicitly acknowledge you have read and agree to the EULA.

RIVA_API_KEY=

RIVA_API_NGC_ORG=

RIVA_EULA=accept

Language code to fetch models of a specify language

Currently only ASR supports languages other than English

Supported language codes: en-US, en-GB, de-DE, es-US, ru-RU, zh-CN, hi-IN, fr-FR, ko-KR, pt-BR

for any language other than English, set service_enabled_nlp and service_enabled_tts to False

for multiple languages enter space separated language codes.

language_code=(“en-US”)

ASR acoustic model architecture

Supported values are: conformer, citrinet_1024, citrinet_256 (en-US + arm64 only), jasper (en-US + amd64 only), quartznet (en-US + amd64 only)

asr_acoustic_model=(“conformer”)

Specify one or more GPUs to use

specifying more than one GPU is currently an experimental feature, and may result in undefined behaviours.

gpus_to_use=“device=0”

Specify the encryption key to use to deploy models

MODEL_DEPLOY_KEY=“tlt_encode”

Locations to use for storing models artifacts

If an absolute path is specified, the data will be written to that location

Otherwise, a docker volume will be used (default).

riva_init.sh will create a rmir and models directory in the volume or

path specified.

RMIR ($riva_model_loc/rmir)

Riva uses an intermediate representation (RMIR) for models

that are ready to deploy but not yet fully optimized for deployment. Pretrained

versions can be obtained from NGC (by specifying NGC models below) and will be

downloaded to $riva_model_loc/rmir by riva_init.sh

Custom models produced by NeMo or TLT and prepared using riva-build

may also be copied manually to this location $(riva_model_loc/rmir).

Models ($riva_model_loc/models)

During the riva_init process, the RMIR files in $riva_model_loc/rmir

are inspected and optimized for deployment. The optimized versions are

stored in $riva_model_loc/models. The riva server exclusively uses these

optimized versions.

riva_model_loc=“riva-model-repo”

if [[ $riva_target_gpu_family == “tegra” ]]; then
riva_model_loc=“pwd/model_repository”
fi

The default RMIRs are downloaded from NGC by default in the above $riva_rmir_loc directory

If you’d like to skip the download from NGC and use the existing RMIRs in the $riva_rmir_loc

then set the below $use_existing_rmirs flag to true. You can also deploy your set of custom

RMIRs by keeping them in the riva_rmir_loc dir and use this quickstart script with the

below flag to deploy them all together.

use_existing_rmirs=false

Ports to expose for Riva services

riva_speech_api_port=“50051”

NGC orgs

riva_ngc_org=“nvidia”
riva_ngc_team=“riva”
riva_ngc_image_version=“2.7.0”
riva_ngc_model_version=“2.7.0”

Pre-built models listed below will be downloaded from NGC. If models already exist in $riva-rmir

then models can be commented out to skip download from NGC

########## ASR MODELS ##########

models_asr=()

for lang_code in ${language_code[@]}; do
modified_lang_code=“${lang_code/-/_}”
modified_lang_code=${modified_lang_code,}
if [[ $riva_target_gpu_family == “tegra” ]]; then

  if [[ ${asr_acoustic_model} == "jasper" || \
        ${asr_acoustic_model} == "quartznet" ]]; then
      echo "Jasper and Quartznet models are not available for arm64 architecture"
      exit 1
  fi

  if [[ ${asr_acoustic_model} == "citrinet_256" && ${lang_code} != "en-US" ]]; then
    echo "For arm64 architecture, citrinet_256 acoustic model is only available for language code en-US."
    exit 1
  fi
  models_asr+=(
      "${riva_ngc_org}/${riva_ngc_team}/models_asr_${asr_acoustic_model}_${modified_lang_code}_str:${riva_ngc_model_version}-${riva_target_gpu_family}-${riva_tegra_platform}"
  )
else

  if [[ ${asr_acoustic_model} != "conformer" && \
        ${asr_acoustic_model} != "citrinet_1024" && \
        ${asr_acoustic_model} != "jasper" && \
        ${asr_acoustic_model} != "quartznet" ]]; then
    echo "For amd64 architecture, valid acoustic models are conformer, citrinet_1024, jasper and quartznet."
    exit 1
  fi

  if [[ (${asr_acoustic_model} == "jasper" || \
        ${asr_acoustic_model} == "quartznet") && \
        ${lang_code} != "en-US" ]]; then
    echo "jasper and quartznet acoustic models are only available for language code en-US."
    exit 1
  fi
  #Setting default Conformer Mandarin models to greedy decoder due to high high latency in os2s.
  decoder=""
  if [[ ${asr_acoustic_model} == "conformer" && ${lang_code} == "zh-CN" ]]; then
    decoder="_gre"
  fi
  models_asr+=(
  ### Streaming w/ CPU decoder, best latency configuration
      "${riva_ngc_org}/${riva_ngc_team}/rmir_asr_${asr_acoustic_model}_${modified_lang_code}_str${decoder}:${riva_ngc_model_version}"

  ### Streaming w/ CPU decoder, best throughput configuration
  #    "${riva_ngc_org}/${riva_ngc_team}/rmir_asr_${asr_acoustic_model}_${modified_lang_code}_str_thr${decoder}:${riva_ngc_model_version}"

  ### Offline w/ CPU decoder,
      "${riva_ngc_org}/${riva_ngc_team}/rmir_asr_${asr_acoustic_model}_${modified_lang_code}_ofl${decoder}:${riva_ngc_model_version}"
  )
fi

### Punctuation model
if [[ "${lang_code}"  == "en-US" || "${lang_code}" == "de-DE" || "${lang_code}" == "hi-IN" || "${lang_code}" == "fr-FR" || "${lang_code}" == "es-US" || "${lang_code}" == "zh-CN" || "${lang_code}" == "en-GB" ]]; then
  if [[ $riva_target_gpu_family == "tegra" ]]; then
    models_asr+=(
        "${riva_ngc_org}/${riva_ngc_team}/models_nlp_punctuation_bert_base_${modified_lang_code}:${riva_ngc_model_version}-${riva_target_gpu_family}-${riva_tegra_platform}"
    )
  else
    models_asr+=(
        "${riva_ngc_org}/${riva_ngc_team}/rmir_nlp_punctuation_bert_base_${modified_lang_code}:${riva_ngc_model_version}"
    )
  fi
fi

done
########## NLP MODELS ##########

if [[ $riva_target_gpu_family == “tegra” ]]; then
models_nlp=(

BERT Base Intent Slot model for misty domain fine-tuned on weather, smalltalk/personality, poi/map datasets.

  "${riva_ngc_org}/${riva_ngc_team}/models_nlp_intent_slot_misty_bert_base:${riva_ngc_model_version}-${riva_target_gpu_family}-${riva_tegra_platform}"

DistilBERT Intent Slot model for misty domain fine-tuned on weather, smalltalk/personality, poi/map datasets.

“${riva_ngc_org}/${riva_ngc_team}/models_nlp_intent_slot_misty_distilbert:${riva_ngc_model_version}-${riva_target_gpu_family}-${riva_tegra_platform}”

)
else
models_nlp=(

Bert base Punctuation model

  "${riva_ngc_org}/${riva_ngc_team}/rmir_nlp_punctuation_bert_base_en_us:${riva_ngc_model_version}"

BERT base Named Entity Recognition model fine-tuned on GMB dataset with class labels LOC, PER, ORG etc.

  "${riva_ngc_org}/${riva_ngc_team}/rmir_nlp_named_entity_recognition_bert_base:${riva_ngc_model_version}"

BERT Base Intent Slot model fine-tuned on weather dataset.

  "${riva_ngc_org}/${riva_ngc_team}/rmir_nlp_intent_slot_bert_base:${riva_ngc_model_version}"

BERT Base Question Answering model fine-tuned on Squad v2.

  "${riva_ngc_org}/${riva_ngc_team}/rmir_nlp_question_answering_bert_base:${riva_ngc_model_version}"

Megatron345M Question Answering model fine-tuned on Squad v2.

“${riva_ngc_org}/${riva_ngc_team}/rmir_nlp_question_answering_megatron:${riva_ngc_model_version}”

Bert base Text Classification model fine-tuned on 4class (weather, meteorology, personality, nomatch) domain model.

  "${riva_ngc_org}/${riva_ngc_team}/rmir_nlp_text_classification_bert_base:${riva_ngc_model_version}"

)
fi

########## TTS MODELS ##########

if [[ $riva_target_gpu_family == “tegra” ]]; then
models_tts=(
“${riva_ngc_org}/${riva_ngc_team}/models_tts_fastpitch_hifigan_en_us:${riva_ngc_model_version}-${riva_target_gpu_family}-${riva_tegra_platform}”
)
else
models_tts=(
“${riva_ngc_org}/${riva_ngc_team}/rmir_tts_fastpitch_hifigan_en_us:${riva_ngc_model_version}”
)
fi

NGC_TARGET=${riva_ngc_org}
if [[ ! -z ${riva_ngc_team} ]]; then
NGC_TARGET=“${NGC_TARGET}/${riva_ngc_team}”
else
team=“""”
fi

Specify paths to SSL Key and Certificate files to use TLS/SSL Credentials for a secured connection.

If either are empty, an insecure connection will be used.

Stored within container at /ssl/servert.crt and /ssl/server.key

Optional, one can also specify a root certificate, stored within container at /ssl/root_server.crt

ssl_server_cert=“”
ssl_server_key=“”
ssl_root_cert=“”

define docker images required to run Riva

image_speech_api=“nvcr.io/${NGC_TARGET}/riva-speech:${riva_ngc_image_version}-l4t-aarch64”

define docker images required to setup Riva

image_init_speech=“nvcr.io/${NGC_TARGET}/riva-speech:${riva_ngc_image_version}-servicemaker-l4t-aarch64”

daemon names

riva_daemon_speech=“riva-speech”
if [[ $riva_target_gpu_family != "teg

Here is dock output from the failed start

~/ngc-cli/riva_quickstart_arm64_v2.7.0$ sudo docker logs riva-speech
I1202 03:50:14.868510 7 riva_server.cc:120] Using Insecure Server Credentials
E1202 03:50:14.870491 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:50:24.871191 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:50:34.871980 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:50:44.872862 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:50:54.873823 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:51:04.874648 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:51:14.875097 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:51:24.876180 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:51:34.877255 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:51:44.878162 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:51:54.879103 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:52:04.880172 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:52:14.880605 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:52:24.880968 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:52:34.881918 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:52:44.882876 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:52:54.883330 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:53:04.883716 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:53:14.884136 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:53:24.884516 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:53:34.884899 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:53:44.885828 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:53:54.886866 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:54:04.887876 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:54:14.888924 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:54:24.889405 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:54:34.889793 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:54:44.890640 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:54:54.891553 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:55:04.892001 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:55:14.892419 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:55:24.893033 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:55:34.893978 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:55:44.894459 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:55:54.894850 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:56:04.895241 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:56:14.896167 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:56:24.896628 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:56:34.897008 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:56:44.897925 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:56:54.898407 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:57:04.899315 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:57:14.900310 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:57:24.901358 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:57:34.901794 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:57:44.902652 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:57:54.903702 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:58:04.904745 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:58:14.905804 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:58:24.906292 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:58:34.907181 7 model_registry.cc:90] Cannot get model repository information failed to connect to all addresses

And here is the output of the start itself

…
},
{
“name”: “PARTIAL_TRANSCRIPTS”,
“data_type”: “TYPE_STRING”,
“dims”: [
-1
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “PARTIAL_TRANSCRIPTS_STABILITY”,
“data_type”: “TYPE_FP32”,
“dims”: [
-1
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “PARTIAL_WORDS_START_END”,
“data_type”: “TYPE_INT32”,
“dims”: [
-1,
2
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “FINAL_WORDS_SCORE”,
“data_type”: “TYPE_FP32”,
“dims”: [
-1
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “PARTIAL_WORDS_SCORE”,
“data_type”: “TYPE_FP32”,
“dims”: [
-1
],
“label_filename”: “”,
“is_shape_tensor”: false
}
],
“batch_input”: ,
“batch_output”: ,
“optimization”: {
“graph”: {
“level”: 0
},
“priority”: “PRIORITY_DEFAULT”,
“cuda”: {
“graphs”: false,
“busy_wait_events”: false,
“graph_spec”: ,
“output_copy_stream”: true
},
“input_pinned_memory”: {
“enable”: true
},
“output_pinned_memory”: {
“enable”: true
},
“gather_kernel_buffer_threshold”: 0,
“eager_batching”: false
},
“sequence_batching”: {
“oldest”: {
“max_candidate_sequences”: 1024,
“preferred_batch_size”: [
32,
64
],
“max_queue_delay_microseconds”: 1000
},
“max_sequence_idle_microseconds”: 60000000,
“control_input”: [
{
“name”: “START”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_START”,
“int32_false_true”: [
0,
1
],
“fp32_false_true”: ,
“bool_false_true”: ,
“data_type”: “TYPE_INVALID”
}
]
},
{
“name”: “READY”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_READY”,
“int32_false_true”: [
0,
1
],
“fp32_false_true”: ,
“bool_false_true”: ,
“data_type”: “TYPE_INVALID”
}
]
},
{
“name”: “END”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_END”,
“int32_false_true”: [
0,
1
],
“fp32_false_true”: ,
“bool_false_true”: ,
“data_type”: “TYPE_INVALID”
}
]
},
{
“name”: “CORRID”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_CORRID”,
“int32_false_true”: ,
“fp32_false_true”: ,
“bool_false_true”: ,
“data_type”: “TYPE_UINT64”
}
]
}
],
“state”:
},
“instance_group”: [
{
“name”: “conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming_0”,
“kind”: “KIND_CPU”,
“count”: 1,
“gpus”: ,
“secondary_devices”: ,
“profile”: ,
“passive”: false,
“host_policy”: “”
}
],
“default_model_filename”: “”,
“cc_model_filenames”: {},
“metric_tags”: {},
“parameters”: {
“profane_words_file”: {
“string_value”: “/data/models/conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/profane_words_file.txt”
},
“forerunner_beam_size_token”: {
“string_value”: “8”
},
“forerunner_beam_threshold”: {
“string_value”: “10.0”
},
“asr_model_delay”: {
“string_value”: “-1”
},
“decoder_num_worker_threads”: {
“string_value”: “-1”
},
“word_insertion_score”: {
“string_value”: “1.0”
},
“unk_token”: {
“string_value”: “”
},
“left_padding_size”: {
“string_value”: “1.92”
},
“set_default_index_to_unk_token”: {
“string_value”: “False”
},
“decoder_type”: {
“string_value”: “flashlight”
},
“forerunner_beam_size”: {
“string_value”: “8”
},
“unk_score”: {
“string_value”: “-inf”
},
“chunk_size”: {
“string_value”: “0.16”
},
“max_supported_transcripts”: {
“string_value”: “1”
},
“lexicon_file”: {
“string_value”: “/data/models/conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/lexicon.txt”
},
“smearing_mode”: {
“string_value”: “max”
},
“log_add”: {
“string_value”: “True”
},
“lm_weight”: {
“string_value”: “0.8”
},
“blank_token”: {
“string_value”: “#”
},
“vocab_file”: {
“string_value”: “/data/models/conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/riva_decoder_vocabulary.txt”
},
“ms_per_timestep”: {
“string_value”: “40”
},
“streaming”: {
“string_value”: “True”
},
“use_subword”: {
“string_value”: “True”
},
“beam_size”: {
“string_value”: “32”
},
“right_padding_size”: {
“string_value”: “1.92”
},
“beam_size_token”: {
“string_value”: “16”
},
“sil_token”: {
“string_value”: “▁”
},
“num_tokenization”: {
“string_value”: “1”
},
“beam_threshold”: {
“string_value”: “20.0”
},
“tokenizer_model”: {
“string_value”: “/data/models/conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/8b8f095152034e98b24ab33726708bd0_tokenizer.model”
},
“language_model_file”: {
“string_value”: “/data/models/conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/4gram-pruned-0_2_7_9-en-lm-set-2.0.bin”
},
“max_execution_batch_size”: {
“string_value”: “1”
},
“forerunner_use_lm”: {
“string_value”: “true”
}
},
“model_warmup”: ,
“model_transaction_policy”: {
“decoupled”: false
}
}
I1202 03:50:15.322618 24 ctc-decoder-library.cc:23] TRITONBACKEND_ModelInstanceInitialize: conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming_0 (device 0)
I1202 03:50:15.360650 24 model_repository_manager.cc:1077] loading: conformer-en-US-asr-streaming-feature-extractor-streaming:1
I1202 03:50:15.461090 24 model_repository_manager.cc:1077] loading: intent_slot_detokenizer:1
I1202 03:50:15.561469 24 model_repository_manager.cc:1077] loading: intent_slot_label_tokens_misty:1
I1202 03:50:15.661844 24 model_repository_manager.cc:1077] loading: intent_slot_tokenizer-en-US-misty:1
I1202 03:50:15.762207 24 model_repository_manager.cc:1077] loading: riva-onnx-fastpitch_encoder-English-US:1
I1202 03:50:15.862836 24 model_repository_manager.cc:1077] loading: riva-punctuation-en-US:1
I1202 03:50:15.963219 24 model_repository_manager.cc:1077] loading: riva-trt-conformer-en-US-asr-streaming-am-streaming:1
Waiting for Riva server to load all models…retrying in 10 seconds
I1202 03:50:16.063722 24 model_repository_manager.cc:1077] loading: riva-trt-hifigan-English-US:1
I1202 03:50:16.164087 24 model_repository_manager.cc:1077] loading: riva-trt-riva-punctuation-en-US-nn-bert-base-uncased:1
I1202 03:50:16.264459 24 model_repository_manager.cc:1077] loading: riva-trt-riva_intent_misty-nn-bert-base-uncased:1
I1202 03:50:16.364846 24 model_repository_manager.cc:1077] loading: spectrogram_chunker-English-US:1
I1202 03:50:16.465220 24 model_repository_manager.cc:1077] loading: tts_postprocessor-English-US:1
I1202 03:50:16.565620 24 model_repository_manager.cc:1077] loading: tts_preprocessor-English-US:1
I1202 03:50:18.082005 31 ctc-decoder.cc:174] Beam Decoder initialized successfully!
I1202 03:50:18.082991 24 endpointing_library.cc:18] TRITONBACKEND_ModelInitialize: conformer-en-US-asr-streaming-endpointing-streaming (version 1)
I1202 03:50:18.083802 24 model_repository_manager.cc:1231] successfully loaded ‘conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming’ version 1
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1202 03:50:18.083829 32 parameter_parser.cc:144] Parameter ‘chunk_size’ set but unused.
W1202 03:50:18.083846 32 parameter_parser.cc:144] Parameter ‘ms_per_timestep’ set but unused.
W1202 03:50:18.083849 32 parameter_parser.cc:144] Parameter ‘residue_blanks_at_end’ set but unused.
W1202 03:50:18.083853 32 parameter_parser.cc:144] Parameter ‘residue_blanks_at_start’ set but unused.
W1202 03:50:18.083855 32 parameter_parser.cc:144] Parameter ‘start_history’ set but unused.
W1202 03:50:18.083858 32 parameter_parser.cc:144] Parameter ‘start_th’ set but unused.
W1202 03:50:18.083860 32 parameter_parser.cc:144] Parameter ‘stop_history’ set but unused.
W1202 03:50:18.083863 32 parameter_parser.cc:144] Parameter ‘stop_th’ set but unused.
W1202 03:50:18.083866 32 parameter_parser.cc:144] Parameter ‘streaming’ set but unused.
W1202 03:50:18.083868 32 parameter_parser.cc:144] Parameter ‘use_subword’ set but unused.
W1202 03:50:18.083870 32 parameter_parser.cc:144] Parameter ‘vocab_file’ set but unused.
I1202 03:50:18.094513 24 backend_model.cc:303] model configuration:
{
“name”: “conformer-en-US-asr-streaming-endpointing-streaming”,
“platform”: “”,
“backend”: “riva_asr_endpointing”,
“version_policy”: {
“latest”: {
“num_versions”: 1
}
},
“max_batch_size”: 1,
“input”: [
{
“name”: “CLASS_LOGITS”,
“data_type”: “TYPE_FP32”,
“format”: “FORMAT_NONE”,
“dims”: [
-1,
257
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false,
“optional”: false
}
],
“output”: [
{
“name”: “SEGMENTS_START_END”,
“data_type”: “TYPE_FP32”,
“dims”: [
-1,
2
],
“label_filename”: “”,
“is_shape_tensor”: false
}
],
“batch_input”: ,
“batch_output”: ,
“optimization”: {
“graph”: {
“level”: 0
},
“priority”: “PRIORITY_DEFAULT”,
“cuda”: {
“graphs”: false,
“busy_wait_events”: false,
“graph_spec”: ,
“output_copy_stream”: true
},
“input_pinned_memory”: {
“enable”: true
},
“output_pinned_memory”: {
“enable”: true
},
“gather_kernel_buffer_threshold”: 0,
“eager_batching”: false
},
“sequence_batching”: {
“max_sequence_idle_microseconds”: 60000000,
“control_input”: [
{
“name”: “START”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_START”,
“int32_false_true”: [
0,
1
],
“fp32_false_true”: ,
“bool_false_true”: ,
“data_type”: “TYPE_INVALID”
}
]
},
{
“name”: “READY”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_READY”,
“int32_false_true”: [
0,
1
],
“fp32_false_true”: ,
“bool_false_true”: ,
“data_type”: “TYPE_INVALID”
}
]
}
],
“state”:
},
“instance_group”: [
{
“name”: “conformer-en-US-asr-streaming-endpointing-streaming_0”,
“kind”: “KIND_CPU”,
“count”: 1,
“gpus”: ,
“secondary_devices”: ,
“profile”: ,
“passive”: false,
“host_policy”: “”
}
],
“default_model_filename”: “”,
“cc_model_filenames”: {},
“metric_tags”: {},
“parameters”: {
“endpointing_type”: {
“string_value”: “greedy_ctc”
},
“stop_th”: {
“string_value”: “0.98”
},
“vocab_file”: {
“string_value”: “/data/models/conformer-en-US-asr-streaming-endpointing-streaming/1/riva_decoder_vocabulary.txt”
},
“start_history”: {
“string_value”: “200”
},
“residue_blanks_at_start”: {
“string_value”: “-2”
},
“ms_per_timestep”: {
“string_value”: “40”
},
“streaming”: {
“string_value”: “True”
},
“use_subword”: {
“string_value”: “True”
},
“stop_history”: {
“string_value”: “800”
},
“residue_blanks_at_end”: {
“string_value”: “0”
},
“start_th”: {
“string_value”: “0.2”
},
“chunk_size”: {
“string_value”: “0.16”
}
},
“model_warmup”: ,
“model_transaction_policy”: {
“decoupled”: false
}
}
I1202 03:50:18.095640 24 feature-extractor.cc:400] TRITONBACKEND_ModelInitialize: conformer-en-US-asr-streaming-feature-extractor-streaming (version 1)
I1202 03:50:18.108957 24 backend_model.cc:303] model configuration:
{
“name”: “conformer-en-US-asr-streaming-feature-extractor-streaming”,
“platform”: “”,
“backend”: “riva_asr_features”,
“version_policy”: {
“latest”: {
“num_versions”: 1
}
},
“max_batch_size”: 1024,
“input”: [
{
“name”: “AUDIO_SIGNAL”,
“data_type”: “TYPE_FP32”,
“format”: “FORMAT_NONE”,
“dims”: [
-1
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false,
“optional”: false
},
{
“name”: “SAMPLE_RATE”,
“data_type”: “TYPE_UINT32”,
“format”: “FORMAT_NONE”,
“dims”: [
1
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false,
“optional”: false
}
],
“output”: [
{
“name”: “AUDIO_FEATURES”,
“data_type”: “TYPE_FP32”,
“dims”: [
80,
-1
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “AUDIO_PROCESSED”,
“data_type”: “TYPE_FP32”,
“dims”: [
1
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “AUDIO_FEATURES_LENGTH”,
“data_type”: “TYPE_INT32”,
“dims”: [
1
],
“label_filename”: “”,
“is_shape_tensor”: false
}
],
“batch_input”: ,
“batch_output”: ,
“optimization”: {
“graph”: {
“level”: 0
},
“priority”: “PRIORITY_DEFAULT”,
“cuda”: {
“graphs”: false,
“busy_wait_events”: false,
“graph_spec”: ,
“output_copy_stream”: true
},
“input_pinned_memory”: {
“enable”: true
},
“output_pinned_memory”: {
“enable”: true
},
“gather_kernel_buffer_threshold”: 0,
“eager_batching”: false
},
“sequence_batching”: {
“oldest”: {
“max_candidate_sequences”: 1024,
“preferred_batch_size”: [
256,
512
],
“max_queue_delay_microseconds”: 1000
},
“max_sequence_idle_microseconds”: 60000000,
“control_input”: [
{
“name”: “START”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_START”,
“int32_false_true”: [
0,
1
],
“fp32_false_true”: ,
“bool_false_true”: ,
“data_type”: “TYPE_INVALID”
}
]
},
{
“name”: “READY”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_READY”,
“int32_false_true”: [
0,
1
],
“fp32_false_true”: ,
“bool_false_true”: ,
“data_type”: “TYPE_INVALID”
}
]
},
{
“name”: “END”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_END”,
“int32_false_true”: [
0,
1
],
“fp32_false_true”: ,
“bool_false_true”: ,
“data_type”: “TYPE_INVALID”
}
]
},
{
“name”: “CORRID”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_CORRID”,
“int32_false_true”: ,
“fp32_false_true”: ,
“bool_false_true”: ,
“data_type”: “TYPE_UINT64”
}
]
}
],
“state”:
},
“instance_group”: [
{
“name”: “conformer-en-US-asr-streaming-feature-extractor-streaming_0”,
“kind”: “KIND_GPU”,
“count”: 1,
“gpus”: [
0
],
“secondary_devices”: ,
“profile”: ,
“passive”: false,
“host_policy”: “”
}
],
“default_model_filename”: “”,
“cc_model_filenames”: {},
“metric_tags”: {},
“parameters”: {
“window_stride”: {
“string_value”: “0.01”
},
“window_size”: {
“string_value”: “0.025”
},
“num_features”: {
“string_value”: “80”
},
“streaming”: {
“string_value”: “True”
},
“left_padding_size”: {
“string_value”: “1.92”
},
“stddev_floor”: {
“string_value”: “1e-05”
},
“transpose”: {
“string_value”: “False”
},
“right_padding_size”: {
“string_value”: “1.92”
},
“gain”: {
“string_value”: “1.0”
},
“precalc_norm_time_steps”: {
“string_value”: “0”
},
“use_utterance_norm_params”: {
“string_value”: “False”
},
“dither”: {
“string_value”: “0.0”
},
“apply_normalization”: {
“string_value”: “True”
},
“precalc_norm_params”: {
“string_value”: “False”
},
“norm_per_feature”: {
“string_value”: “True”
},
“mean”: {
“string_value”: “-11.4412, -9.9334, -9.1292, -9.0365, -9.2804, -9.5643, -9.7342, -9.6925, -9.6333, -9.2808, -9.1887, -9.1422, -9.1397, -9.2028, -9.2749, -9.4776, -9.9185, -10.1557, -10.3800, -10.5067, -10.3190, -10.4728, -10.5529, -10.6402, -10.6440, -10.5113, -10.7395, -10.7870, -10.6074, -10.5033, -10.8278, -10.6384, -10.8481, -10.6875, -10.5454, -10.4747, -10.5165, -10.4930, -10.3413, -10.3472, -10.3735, -10.6830, -10.8813, -10.6338, -10.3856, -10.7727, -10.8957, -10.8068, -10.7373, -10.6108, -10.3405, -10.2889, -10.3922, -10.4946, -10.3367, -10.4164, -10.9949, -10.7196, -10.3971, -10.1734, -9.9257, -9.6557, -9.1761, -9.6653, -9.7876, -9.7230, -9.7792, -9.7056, -9.2702, -9.4650, -9.2755, -9.1369, -9.1174, -8.9197, -8.5394, -8.2614, -8.1353, -8.1422, -8.3430, -8.6655”
},
“stddev”: {
“string_value”: “2.2668, 3.1642, 3.7079, 3.7642, 3.5349, 3.5901, 3.7640, 3.8424, 4.0145, 4.1475, 4.0457, 3.9048, 3.7709, 3.6117, 3.3188, 3.1489, 3.0615, 3.0362, 2.9929, 3.0500, 3.0341, 3.0484, 3.0103, 2.9474, 2.9128, 2.8669, 2.8332, 2.9411, 3.0378, 3.0712, 3.0190, 2.9992, 3.0124, 3.0024, 3.0275, 3.0870, 3.0656, 3.0142, 3.0493, 3.1373, 3.1135, 3.0675, 2.8828, 2.7018, 2.6296, 2.8826, 2.9325, 2.9288, 2.9271, 2.9890, 3.0137, 2.9855, 3.0839, 2.9319, 2.3512, 2.3795, 2.6191, 2.7555, 2.9326, 2.9931, 3.1543, 3.0855, 2.6820, 3.0566, 3.1272, 3.1663, 3.1836, 3.0018, 2.9089, 3.1727, 3.1626, 3.1086, 2.9804, 3.1107, 3.2998, 3.3697, 3.3716, 3.2487, 3.1597, 3.1181”
},
“chunk_size”: {
“string_value”: “0.16”
},
“max_execution_batch_size”: {
“string_value”: “1”
},
“sample_rate”: {
“string_value”: “16000”
}
},
“model_warmup”: ,
“model_transaction_policy”: {
“decoupled”: false
}
}
I1202 03:50:18.109062 24 endpointing_library.cc:22] TRITONBACKEND_ModelInstanceInitialize: conformer-en-US-asr-streaming-endpointing-streaming_0 (device 0)
I1202 03:50:18.110422 24 model_repository_manager.cc:1231] successfully loaded ‘conformer-en-US-asr-streaming-endpointing-streaming’ version 1
I1202 03:50:18.110888 24 detokenizer_cbe.cc:145] TRITONBACKEND_ModelInitialize: intent_slot_detokenizer (version 1)
I1202 03:50:18.111762 24 backend_model.cc:303] model configuration:
{
“name”: “intent_slot_detokenizer”,
“platform”: “”,
“backend”: “riva_nlp_detokenizer”,
“version_policy”: {
“latest”: {
“num_versions”: 1
}
},
“max_batch_size”: 1,
“input”: [
{
“name”: “IN_TOKEN_LABELS__0”,
“data_type”: “TYPE_STRING”,
“format”: “FORMAT_NONE”,
“dims”: [
-1
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false,
“optional”: false
},
{
“name”: “IN_TOKEN_SCORES__1”,
“data_type”: “TYPE_FP32”,
“format”: “FORMAT_NONE”,
“dims”: [
-1
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false,
“optional”: false
},
{
“name”: “IN_SEQ_LEN__2”,
“data_type”: “TYPE_INT64”,
“format”: “FORMAT_NONE”,
“dims”: [
1
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false,
“optional”: false
},
{
“name”: “IN_TOK_STR__3”,
“data_type”: “TYPE_STRING”,
“format”: “FORMAT_NONE”,
“dims”: [
-1
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false,
“optional”: false
}
],
“output”: [
{
“name”: “OUT_TOKEN_LABELS__0”,
“data_type”: “TYPE_STRING”,
“dims”: [
-1
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “OUT_TOKEN_SCORES__1”,
“data_type”: “TYPE_FP32”,
“dims”: [
-1
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “OUT_SEQ_LEN__2”,
“data_type”: “TYPE_INT64”,
“dims”: [
1
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “OUT_TOK_STR__3”,
“data_type”: “TYPE_STRING”,
“dims”: [
-1
],
“label_filename”: “”,
“is_shape_tensor”: false
}
],
“batch_input”: ,
“batch_output”: ,
“optimization”: {
“graph”: {
“level”: 0
},
“priority”: “PRIORITY_DEFAULT”,
“cuda”: {
“graphs”: false,
“busy_wait_events”: false,
“graph_spec”: ,
“output_copy_stream”: true
},
“input_pinned_memory”: {
“enable”: true
},
“output_pinned_memory”: {
“enable”: true
},
“gather_kernel_buffer_threshold”: 0,
“eager_batching”: false
},
“instance_group”: [
{
“name”: “intent_slot_detokenizer_0”,
“kind”: “KIND_CPU”,
“count”: 1,
“gpus”: ,
“secondary_devices”: ,
“profile”: ,
“passive”: false,
“host_policy”: “”
}
],
“default_model_filename”: “”,
“cc_model_filenames”: {},
“metric_tags”: {},
“parameters”: {},
“model_warmup”: ,
“model_transaction_policy”: {
“decoupled”: false
}
}
I1202 03:50:18.111829 24 feature-extractor.cc:402] TRITONBACKEND_ModelInstanceInitialize: conformer-en-US-asr-streaming-feature-extractor-streaming_0 (device 0)
I1202 03:50:18.983574 24 model_repository_manager.cc:1231] successfully loaded ‘conformer-en-US-asr-streaming-feature-extractor-streaming’ version 1
I1202 03:50:18.992186 24 sequence_label_cbe.cc:137] TRITONBACKEND_ModelInitialize: intent_slot_label_tokens_misty (version 1)
I1202 03:50:18.992936 24 backend_model.cc:303] model configuration:
{
“name”: “intent_slot_label_tokens_misty”,
“platform”: “”,
“backend”: “riva_nlp_seqlabel”,
“version_policy”: {
“latest”: {
“num_versions”: 1
}
},
“max_batch_size”: 1,
“input”: [
{
“name”: “TOKEN_LOGIT__1”,
“data_type”: “TYPE_FP32”,
“format”: “FORMAT_NONE”,
“dims”: [
-1,
31
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false,
“optional”: false
}
],
“output”: [
{
“name”: “TOKEN_LABELS__0”,
“data_type”: “TYPE_STRING”,
“dims”: [
-1
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “TOKEN_SCORES__1”,
“data_type”: “TYPE_FP32”,
“dims”: [
-1
],
“label_filename”: “”,
“is_shape_tensor”: false
}
],
“batch_input”: ,
“batch_output”: ,
“optimization”: {
“priority”: “PRIORITY_DEFAULT”,
“input_pinned_memory”: {
“enable”: true
},
“output_pinned_memory”: {
“enable”: true
},
“gather_kernel_buffer_threshold”: 0,
“eager_batching”: false
},
“instance_group”: [
{
“name”: “intent_slot_label_tokens_misty_0”,
“kind”: “KIND_CPU”,
“count”: 1,
“gpus”: ,
“secondary_devices”: ,
“profile”: ,
“passive”: false,
“host_policy”: “”
}
],
“default_model_filename”: “”,
“cc_model_filenames”: {},
“metric_tags”: {},
“parameters”: {
“classes”: {
“string_value”: “/data/models/intent_slot_label_tokens_misty/1/slot_labels.csv”
}
},
“model_warmup”:
}
I1202 03:50:18.993003 24 detokenizer_cbe.cc:147] TRITONBACKEND_ModelInstanceInitialize: intent_slot_detokenizer_0 (device 0)
I1202 03:50:18.993114 24 sequence_label_cbe.cc:139] TRITONBACKEND_ModelInstanceInitialize: intent_slot_label_tokens_misty_0 (device 0)
I1202 03:50:18.993153 24 onnxruntime.cc:2481] TRITONBACKEND_ModelInitialize: riva-onnx-fastpitch_encoder-English-US (version 1)
I1202 03:50:18.993230 24 model_repository_manager.cc:1231] successfully loaded ‘intent_slot_detokenizer’ version 1
I1202 03:50:18.993287 24 model_repository_manager.cc:1231] successfully loaded ‘intent_slot_label_tokens_misty’ version 1
I1202 03:50:19.002393 24 tokenizer_library.cc:18] TRITONBACKEND_ModelInitialize: intent_slot_tokenizer-en-US-misty (version 1)
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1202 03:50:19.002995 36 parameter_parser.cc:144] Parameter ‘unk_token’ set but unused.
W1202 03:50:19.003010 36 parameter_parser.cc:144] Parameter ‘vocab’ set but unused.
I1202 03:50:19.003080 24 backend_model.cc:303] model configuration:
{
“name”: “intent_slot_tokenizer-en-US-misty”,
“platform”: “”,
“backend”: “riva_nlp_tokenizer”,
“version_policy”: {
“latest”: {
“num_versions”: 1
}
},
“max_batch_size”: 1,
“input”: [
{
“name”: “INPUT_STR__0”,
“data_type”: “TYPE_STRING”,
“format”: “FORMAT_NONE”,
“dims”: [
1
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false,
“optional”: false
}
],
“output”: [
{
“name”: “SEQ__0”,
“data_type”: “TYPE_INT32”,
“dims”: [
128
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “MASK__1”,
“data_type”: “TYPE_INT32”,
“dims”: [
128
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “SEGMENT__4”,
“data_type”: “TYPE_INT32”,
“dims”: [
128
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “SEQ_LEN__2”,
“data_type”: “TYPE_INT64”,
“dims”: [
1
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “TOK_STR__3”,
“data_type”: “TYPE_STRING”,
“dims”: [
128
],
“label_filename”: “”,
“is_shape_tensor”: false
}
],
“batch_input”: ,
“batch_output”: ,
“optimization”: {
“priority”: “PRIORITY_DEFAULT”,
“input_pinned_memory”: {
“enable”: true
},
“output_pinned_memory”: {
“enable”: true
},
“gather_kernel_buffer_threshold”: 0,
“eager_batching”: false
},
“instance_group”: [
{
“name”: “intent_slot_tokenizer-en-US-misty_0”,
“kind”: “KIND_CPU”,
“count”: 1,
“gpus”: ,
“secondary_devices”: ,
“profile”: ,
“passive”: false,
“host_policy”: “”
}
],
“default_model_filename”: “”,
“cc_model_filenames”: {},
“metric_tags”: {},
“parameters”: {
“pad_chars_with_spaces”: {
“string_value”: “False”
},
“task”: {
“string_value”: “single_input”
},
“unk_token”: {
“string_value”: “[UNK]”
},
“vocab”: {
“string_value”: “/data/models/intent_slot_tokenizer-en-US-misty/1/tokenizer.vocab_file”
},
“tokenizer”: {
“string_value”: “wordpiece”
},
“bos_token”: {
“string_value”: “[CLS]”
},
“eos_token”: {
“string_value”: “[SEP]”
},
“to_lower”: {
“string_value”: “true”
}
},
“model_warmup”:
}
I1202 03:50:19.003161 24 onnxruntime.cc:2524] TRITONBACKEND_ModelInstanceInitialize: riva-onnx-fastpitch_encoder-English-US_0 (GPU device 0)
2022-12-02 03:50:19.370558187 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘418’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370601484 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘490’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370609420 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘375’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370616396 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘346’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370621676 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘354’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370626188 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘307’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370630796 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘379’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370635404 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘373’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370643213 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘301’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370647565 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘286’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370653261 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘447’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370657645 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘358’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370663373 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘281’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370670509 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘274’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370674989 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘374’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370679181 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘181’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370684237 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘303’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370688525 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘302’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370693357 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘426’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370698893 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘425’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370704877 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘430’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370710702 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘282’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370722542 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘497’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370727726 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘445’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370731726 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘451’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370736558 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘498’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370742158 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘502’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370746734 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘446’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370751278 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘353’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370757006 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘518’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370762190 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘519’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370766862 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘517’. It is not used by any node and should be removed from the model.
2022-12-02 03:50:19.370770830 [W:onnxruntime:, graph.cc:3526 CleanUnusedInitializersAndNodeArgs] Removing initializer ‘523’. It is not used by any node and should be removed from the model.
I1202 03:50:20.136208 24 model_repository_manager.cc:1231] successfully loaded ‘riva-onnx-fastpitch_encoder-English-US’ version 1
E1202 03:50:20.157348 24 model_repository_manager.cc:1234] failed to load ‘riva-trt-conformer-en-US-asr-streaming-am-streaming’ version 1: Not found: unable to load shared library: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8: undefined symbol: _ZN5nvdla8IProfile17setUseSoftMaxOptzEb
I1202 03:50:20.164667 24 tokenizer_library.cc:21] TRITONBACKEND_ModelInstanceInitialize: intent_slot_tokenizer-en-US-misty_0 (device 0)
E1202 03:50:20.164690 24 model_repository_manager.cc:1234] failed to load ‘riva-trt-hifigan-English-US’ version 1: Not found: unable to load shared library: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8: undefined symbol: _ZN5nvdla8IProfile17setUseSoftMaxOptzEb
I1202 03:50:20.186607 24 pipeline_library.cc:22] TRITONBACKEND_ModelInitialize: riva-punctuation-en-US (version 1)
I1202 03:50:20.186827 24 model_repository_manager.cc:1231] successfully loaded ‘intent_slot_tokenizer-en-US-misty’ version 1
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1202 03:50:20.188380 39 parameter_parser.cc:144] Parameter ‘attn_mask_tensor_name’ set but unused.
W1202 03:50:20.188414 39 parameter_parser.cc:144] Parameter ‘bos_token’ set but unused.
W1202 03:50:20.188422 39 parameter_parser.cc:144] Parameter ‘capit_logits_tensor_name’ set but unused.
W1202 03:50:20.188427 39 parameter_parser.cc:144] Parameter ‘capitalization_mapping_path’ set but unused.
W1202 03:50:20.188432 39 parameter_parser.cc:144] Parameter ‘delimiter’ set but unused.
W1202 03:50:20.188436 39 parameter_parser.cc:144] Parameter ‘eos_token’ set but unused.
W1202 03:50:20.188441 39 parameter_parser.cc:144] Parameter ‘input_ids_tensor_name’ set but unused.
W1202 03:50:20.188444 39 parameter_parser.cc:144] Parameter ‘language_code’ set but unused.
W1202 03:50:20.188448 39 parameter_parser.cc:144] Parameter ‘model_api’ set but unused.
W1202 03:50:20.188452 39 parameter_parser.cc:144] Parameter ‘model_family’ set but unused.
W1202 03:50:20.188457 39 parameter_parser.cc:144] Parameter ‘pad_chars_with_spaces’ set but unused.
W1202 03:50:20.188462 39 parameter_parser.cc:144] Parameter ‘punct_logits_tensor_name’ set but unused.
W1202 03:50:20.188465 39 parameter_parser.cc:144] Parameter ‘punctuation_mapping_path’ set but unused.
W1202 03:50:20.188470 39 parameter_parser.cc:144] Parameter ‘remove_spaces’ set but unused.
W1202 03:50:20.188474 39 parameter_parser.cc:144] Parameter ‘to_lower’ set but unused.
W1202 03:50:20.188478 39 parameter_parser.cc:144] Parameter ‘token_type_tensor_name’ set but unused.
W1202 03:50:20.188483 39 parameter_parser.cc:144] Parameter ‘tokenizer_to_lower’ set but unused.
W1202 03:50:20.188488 39 parameter_parser.cc:144] Parameter ‘unk_token’ set but unused.
W1202 03:50:20.188491 39 parameter_parser.cc:144] Parameter ‘use_int64_nn_inputs’ set but unused.
W1202 03:50:20.188495 39 parameter_parser.cc:144] Parameter ‘vocab’ set but unused.
W1202 03:50:20.188594 39 parameter_parser.cc:144] Parameter ‘model_api’ set but unused.
W1202 03:50:20.188606 39 parameter_parser.cc:144] Parameter ‘model_family’ set but unused.
I1202 03:50:20.188696 24 backend_model.cc:303] model configuration:
{
“name”: “riva-punctuation-en-US”,
“platform”: “”,
“backend”: “riva_nlp_pipeline”,
“version_policy”: {
“latest”: {
“num_versions”: 1
}
},
“max_batch_size”: 1,
“input”: [
{
“name”: “PIPELINE_INPUT”,
“data_type”: “TYPE_STRING”,
“format”: “FORMAT_NONE”,
“dims”: [
1
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false,
“optional”: false
}
],
“output”: [
{
“name”: “PIPELINE_OUTPUT”,
“data_type”: “TYPE_STRING”,
“dims”: [
1
],
“label_filename”: “”,
“is_shape_tensor”: false
}
],
“batch_input”: ,
“batch_output”: ,
“optimization”: {
“priority”: “PRIORITY_DEFAULT”,
“input_pinned_memory”: {
“enable”: true
},
“output_pinned_memory”: {
“enable”: true
},
“gather_kernel_buffer_threshold”: 0,
“eager_batching”: false
},
“instance_group”: [
{
“name”: “riva-punctuation-en-US_0”,
“kind”: “KIND_CPU”,
“count”: 1,
“gpus”: ,
“secondary_devices”: ,
“profile”: ,
“passive”: false,
“host_policy”: “”
}
],
“default_model_filename”: “”,
“cc_model_filenames”: {},
“metric_tags”: {},
“parameters”: {
“token_type_tensor_name”: {
“string_value”: “token_type_ids”
},
“tokenizer”: {
“string_value”: “wordpiece”
},
“delimiter”: {
“string_value”: " "
},
“pad_chars_with_spaces”: {
“string_value”: “False”
},
“remove_spaces”: {
“string_value”: “False”
},
“use_int64_nn_inputs”: {
“string_value”: “False”
},
“model_family”: {
“string_value”: “riva”
},
“unk_token”: {
“string_value”: “[UNK]”
},
“bos_token”: {
“string_value”: “[CLS]”
},
“punctuation_mapping_path”: {
“string_value”: “/data/models/riva-punctuation-en-US/1/15eace99434b4c87ba28cbd294b48f43_punct_label_ids.csv”
},
“model_api”: {
“string_value”: “/nvidia.riva.nlp.RivaLanguageUnderstanding/PunctuateText”
},
“to_lower”: {
“string_value”: “true”
},
“load_model”: {
“string_value”: “false”
},
“attn_mask_tensor_name”: {
“string_value”: “attention_mask”
},
“punct_logits_tensor_name”: {
“string_value”: “punct_logits”
},
“language_code”: {
“string_value”: “en-US”
},
“input_ids_tensor_name”: {
“string_value”: “input_ids”
},
“model_name”: {
“string_value”: “riva-trt-riva-punctuation-en-US-nn-bert-base-uncased”
},
“tokenizer_to_lower”: {
“string_value”: “true”
},
“vocab”: {
“string_value”: “/data/models/riva-punctuation-en-US/1/e222f352288a423da453a79b96cc7b75_vocab.txt”
},
“capit_logits_tensor_name”: {
“string_value”: “capit_logits”
},
“capitalization_mapping_path”: {
“string_value”: “/data/models/riva-punctuation-en-US/1/fb06800834e74de1bdc32db51da9619c_capit_label_ids.csv”
},
“pipeline_type”: {
“string_value”: “punctuation”
},
“eos_token”: {
“string_value”: “[SEP]”
}
},
“model_warmup”:
}
I1202 03:50:20.195766 24 pipeline_library.cc:25] TRITONBACKEND_ModelInstanceInitialize: riva-punctuation-en-US_0 (device 0)E1202 03:50:20.195773 24 model_repository_manager.cc:1234] failed to load ‘riva-trt-riva-punctuation-en-US-nn-bert-base-uncased’ version 1: Not found: unable to load shared library: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8: undefined symbol: _ZN5nvdla8IProfile17setUseSoftMaxOptzEb

I1202 03:50:20.234110 24 model_repository_manager.cc:1231] successfully loaded ‘riva-punctuation-en-US’ version 1
E1202 03:50:20.240516 24 model_repository_manager.cc:1234] failed to load ‘riva-trt-riva_intent_misty-nn-bert-base-uncased’ version 1: Not found: unable to load shared library: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8: undefined symbol: _ZN5nvdla8IProfile17setUseSoftMaxOptzEb
I1202 03:50:20.241234 24 spectrogram-chunker.cc:274] TRITONBACKEND_ModelInitialize: spectrogram_chunker-English-US (version 1)
… (putting this here since post limit size truncating this a bit)
I1202 03:50:20.505140 24 tts-preprocessor.cc:282] TRITONBACKEND_ModelInstanceInitialize: tts_preprocessor-English-US_0 (device 0)
I1202 03:50:20.505717 24 model_repository_manager.cc:1231] successfully loaded ‘tts_preprocessor-English-US’ version 1
E1202 03:50:20.505863 24 model_repository_manager.cc:1420] Invalid argument: ensemble ‘conformer-en-US-asr-streaming’ depends on ‘riva-trt-conformer-en-US-asr-streaming-am-streaming’ which has no loaded version
E1202 03:50:20.505890 24 model_repository_manager.cc:1420] Invalid argument: ensemble ‘fastpitch_hifigan_ensemble-English-US’ depends on ‘tts_postprocessor-English-US’ which has no loaded version
E1202 03:50:20.505899 24 model_repository_manager.cc:1420] Invalid argument: ensemble ‘riva_intent_misty’ depends on ‘riva-trt-riva_intent_misty-nn-bert-base-uncased’ which has no loaded version
I1202 03:50:20.506028 24 server.cc:549]
±-----------------±-----+
| Repository Agent | Path |
±-----------------±-----+
±-----------------±-----+

I1202 03:50:20.506330 24 server.cc:576]
±----------------------±------------------------------------------------------------------------------------±-------+
| Backend | Path | Config |
±----------------------±------------------------------------------------------------------------------------±-------+
| onnxruntime | /opt/tritonserver/backends/onnxruntime/libtriton_onnxruntime.so | {} |
| riva_asr_decoder | /opt/tritonserver/backends/riva_asr_decoder/libtriton_riva_asr_decoder.so | {} |
| riva_nlp_detokenizer | /opt/tritonserver/backends/riva_nlp_detokenizer/libtriton_riva_nlp_detokenizer.so | {} |
| riva_asr_endpointing | /opt/tritonserver/backends/riva_asr_endpointing/libtriton_riva_asr_endpointing.so | {} |
| riva_nlp_tokenizer | /opt/tritonserver/backends/riva_nlp_tokenizer/libtriton_riva_nlp_tokenizer.so | {} |
| riva_asr_features | /opt/tritonserver/backends/riva_asr_features/libtriton_riva_asr_features.so | {} |
| riva_tts_preprocessor | /opt/tritonserver/backends/riva_tts_preprocessor/libtriton_riva_tts_preprocessor.so | {} |
| riva_nlp_pipeline | /opt/tritonserver/backends/riva_nlp_pipeline/libtriton_riva_nlp_pipeline.so | {} |
| riva_nlp_seqlabel | /opt/tritonserver/backends/riva_nlp_seqlabel/libtriton_riva_nlp_seqlabel.so | {} |
| riva_tts_chunker | /opt/tritonserver/backends/riva_tts_chunker/libtriton_riva_tts_chunker.so | {} |
±----------------------±------------------------------------------------------------------------------------±-------+

I1202 03:50:20.506750 24 server.cc:619]
±----------------------------------------------------------±--------±-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Model | Version | Status |
±----------------------------------------------------------±--------±-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming | 1 | READY |
| conformer-en-US-asr-streaming-endpointing-streaming | 1 | READY |
| conformer-en-US-asr-streaming-feature-extractor-streaming | 1 | READY |
| intent_slot_detokenizer | 1 | READY |
| intent_slot_label_tokens_misty | 1 | READY |
| intent_slot_tokenizer-en-US-misty | 1 | READY |
| riva-onnx-fastpitch_encoder-English-US | 1 | READY |
| riva-punctuation-en-US | 1 | READY |
| riva-trt-conformer-en-US-asr-streaming-am-streaming | 1 | UNAVAILABLE: Not found: unable to load shared library: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8: undefined symbol: _ZN5nvdla8IProfile17setUseSoftMaxOptzEb |
| riva-trt-hifigan-English-US | 1 | UNAVAILABLE: Not found: unable to load shared library: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8: undefined symbol: _ZN5nvdla8IProfile17setUseSoftMaxOptzEb |
| riva-trt-riva-punctuation-en-US-nn-bert-base-uncased | 1 | UNAVAILABLE: Not found: unable to load shared library: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8: undefined symbol: _ZN5nvdla8IProfile17setUseSoftMaxOptzEb |
| riva-trt-riva_intent_misty-nn-bert-base-uncased | 1 | UNAVAILABLE: Not found: unable to load shared library: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8: undefined symbol: _ZN5nvdla8IProfile17setUseSoftMaxOptzEb |
| spectrogram_chunker-English-US | 1 | READY |
| tts_postprocessor-English-US | 1 | UNAVAILABLE: Not found: unable to load shared library: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8: undefined symbol: _ZN5nvdla8IProfile17setUseSoftMaxOptzEb |
| tts_preprocessor-English-US | 1 | READY |
±----------------------------------------------------------±--------±-------------------------------------------------------------------------------------------------------------------------------------------------------------+

W1202 03:50:20.506812 24 metrics.cc:324] Neither cache metrics nor gpu metrics are enabled. Not polling for them.
I1202 03:50:20.507117 24 tritonserver.cc:2123]
±---------------------------------±---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Option | Value |
±---------------------------------±---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.21.0 |
| server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |
| model_repository_path[0] | /data/models |
| model_control_mode | MODE_POLL |
| strict_model_config | 1 |
| rate_limit | OFF |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 67108864 |
| response_cache_byte_size | 0 |
| min_supported_compute_capability | 5.3 |
| strict_readiness | 1 |
| exit_timeout | 30 |
±---------------------------------±---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I1202 03:50:20.507146 24 server.cc:250] Waiting for in-flight requests to complete.
I1202 03:50:20.507181 24 server.cc:266] Timeout 30: Found 0 model versions that have in-flight inferences
I1202 03:50:20.507213 24 model_repository_manager.cc:1109] unloading: riva-punctuation-en-US:1
I1202 03:50:20.507294 24 model_repository_manager.cc:1109] unloading: spectrogram_chunker-English-US:1
I1202 03:50:20.507662 24 pipeline_library.cc:28] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1202 03:50:20.507697 24 model_repository_manager.cc:1109] unloading: conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming:1
I1202 03:50:20.507848 24 model_repository_manager.cc:1109] unloading: conformer-en-US-asr-streaming-endpointing-streaming:1
I1202 03:50:20.507977 24 spectrogram-chunker.cc:279] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1202 03:50:20.508197 24 spectrogram-chunker.cc:275] TRITONBACKEND_ModelFinalize: delete model state
I1202 03:50:20.508279 24 ctc-decoder-library.cc:25] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1202 03:50:20.508389 24 model_repository_manager.cc:1109] unloading: tts_preprocessor-English-US:1
I1202 03:50:20.508682 24 endpointing_library.cc:26] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1202 03:50:20.508765 24 model_repository_manager.cc:1109] unloading: conformer-en-US-asr-streaming-feature-extractor-streaming:1
I1202 03:50:20.508877 24 model_repository_manager.cc:1214] successfully unloaded ‘spectrogram_chunker-English-US’ version 1
I1202 03:50:20.509063 24 tts-preprocessor.cc:284] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1202 03:50:20.509123 24 model_repository_manager.cc:1109] unloading: intent_slot_label_tokens_misty:1
I1202 03:50:20.509149 24 tts-preprocessor.cc:281] TRITONBACKEND_ModelFinalize: delete model state
I1202 03:50:20.509202 24 feature-extractor.cc:404] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1202 03:50:20.509412 24 endpointing_library.cc:21] TRITONBACKEND_ModelFinalize: delete model state
I1202 03:50:20.509416 24 model_repository_manager.cc:1109] unloading: riva-onnx-fastpitch_encoder-English-US:1
I1202 03:50:20.509615 24 sequence_label_cbe.cc:141] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1202 03:50:20.509674 24 model_repository_manager.cc:1109] unloading: intent_slot_detokenizer:1
I1202 03:50:20.509690 24 sequence_label_cbe.cc:138] TRITONBACKEND_ModelFinalize: delete model state
I1202 03:50:20.509892 24 model_repository_manager.cc:1109] unloading: intent_slot_tokenizer-en-US-misty:1
I1202 03:50:20.510222 24 model_repository_manager.cc:1214] successfully unloaded ‘intent_slot_label_tokens_misty’ version 1
I1202 03:50:20.510295 24 detokenizer_cbe.cc:149] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1202 03:50:20.510366 24 server.cc:281] All models are stopped, unloading models
I1202 03:50:20.510491 24 model_repository_manager.cc:1214] successfully unloaded ‘conformer-en-US-asr-streaming-endpointing-streaming’ version 1
I1202 03:50:20.510453 24 detokenizer_cbe.cc:146] TRITONBACKEND_ModelFinalize: delete model state
I1202 03:50:20.510434 24 tokenizer_library.cc:25] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1202 03:50:20.510534 24 server.cc:288] Timeout 30: Found 8 live models and 0 in-flight non-inference requests
I1202 03:50:20.510354 24 onnxruntime.cc:2558] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1202 03:50:20.510798 24 model_repository_manager.cc:1214] successfully unloaded ‘intent_slot_detokenizer’ version 1
I1202 03:50:20.515305 24 pipeline_library.cc:24] TRITONBACKEND_ModelFinalize: delete model state
I1202 03:50:20.516593 24 feature-extractor.cc:401] TRITONBACKEND_ModelFinalize: delete model state
I1202 03:50:20.520094 24 model_repository_manager.cc:1214] successfully unloaded ‘riva-punctuation-en-US’ version 1
I1202 03:50:20.520676 24 tokenizer_library.cc:20] TRITONBACKEND_ModelFinalize: delete model state
I1202 03:50:20.527464 24 model_repository_manager.cc:1214] successfully unloaded ‘intent_slot_tokenizer-en-US-misty’ version 1
I1202 03:50:20.529599 24 model_repository_manager.cc:1214] successfully unloaded ‘conformer-en-US-asr-streaming-feature-extractor-streaming’ version 1
I1202 03:50:20.550461 24 onnxruntime.cc:2504] TRITONBACKEND_ModelFinalize: delete model state
I1202 03:50:20.550822 24 model_repository_manager.cc:1214] successfully unloaded ‘riva-onnx-fastpitch_encoder-English-US’ version 1
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Health ready check failed.
Check Riva logs with: docker logs riva-speech
philip@orin-agx-ai-board:~/ngc-cli/riva_quickstart_arm64_v2.7.0$ sudo docker logs riva-speech
I1202 03:50:14.868510 7 riva_server.cc:120] Using Insecure Server Credentials
E1202 03:50:14.870491 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:50:24.871191 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:50:34.871980 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:50:44.872862 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:50:54.873823 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:51:04.874648 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:51:14.875097 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:51:24.876180 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:51:34.877255 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:51:44.878162 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:51:54.879103 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:52:04.880172 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:52:14.880605 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:52:24.880968 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:52:34.881918 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:52:44.882876 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:52:54.883330 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:53:04.883716 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:53:14.884136 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:53:24.884516 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:53:34.884899 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:53:44.885828 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:53:54.886866 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:54:04.887876 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:54:14.888924 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:54:24.889405 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:54:34.889793 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:54:44.890640 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:54:54.891553 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:55:04.892001 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:55:14.892419 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:55:24.893033 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:55:34.893978 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:55:44.894459 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:55:54.894850 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:56:04.895241 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:56:14.896167 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:56:24.896628 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:56:34.897008 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:56:44.897925 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:56:54.898407 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:57:04.899315 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:57:14.900310 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:57:24.901358 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:57:34.901794 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:57:44.902652 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:57:54.903702 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:58:04.904745 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:58:14.905804 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:58:24.906292 7 model_registry.cc:236] error: unable to get server status: failed to connect to all addresses
E1202 03:58:34.907181 7 model_registry.cc:90] Cannot get model repository information failed to connect to all addresses
philip@orin-agx-ai-board:~/ngc-cli/riva_quickstart_arm64_v2.7.0$ cat output.txt
Starting Riva Speech Services. This may take several minutes depending on the number of models deployed.
a90de6235530391bb701d51e897c9a6c422f685f55561488700fcdaeab92ae55
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Health ready check failed.
Check Riva logs with: docker logs riva-speech
philip@orin-agx-board:~/ngc-cli/riva_quickstart_arm64_v2.7.0$

7

HI @philipmullis

Thanks for sharing the information

Your current Jetpack version is 5.0.1-b118
The Jetpacks needs to be updated
NVIDIA JetPack version 5.0.2 will be required on the Jetson platform
Reference
https://docs.nvidia.com/deeplearning/riva/user-guide/docs/support-matrix.html#embedded
https://docs.nvidia.com/deeplearning/riva/user-guide/docs/support-matrix.html

Thanks

8

@rvinobha thanks that did the trick :)