sudo docker logs ae04bb8a44a6 ========================== === Riva Speech Skills === ========================== NVIDIA Release (build 46434648) Copyright (c) 2018-2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved. Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES. All rights reserved. This container image and its contents are governed by the NVIDIA Deep Learning Container License. By pulling and using the container, you accept the terms and conditions of this license: https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license WARNING: CUDA Minor Version Compatibility mode ENABLED. Using driver version 471.68 which has support for CUDA 11.4. This container was built with CUDA 11.6 and will be run in Minor Version Compatibility mode. CUDA Forward Compatibility is preferred over Minor Version Compatibility for use with this container but was unavailable: [[]] See https://docs.nvidia.com/deploy/cuda-compatibility/ for details. > Riva waiting for Triton server to load all models...retrying in 1 second I1103 12:06:37.232119 110 libtorch.cc:1381] TRITONBACKEND_Initialize: pytorch I1103 12:06:37.232201 110 libtorch.cc:1391] Triton TRITONBACKEND API version: 1.9 I1103 12:06:37.232205 110 libtorch.cc:1397] 'pytorch' TRITONBACKEND API version: 1.9 I1103 12:06:37.233603 110 onnxruntime.cc:2400] TRITONBACKEND_Initialize: onnxruntime I1103 12:06:37.233636 110 onnxruntime.cc:2410] Triton TRITONBACKEND API version: 1.9 I1103 12:06:37.233640 110 onnxruntime.cc:2416] 'onnxruntime' TRITONBACKEND API version: 1.9 I1103 12:06:37.233642 110 onnxruntime.cc:2446] backend configuration: {} > Riva waiting for Triton server to load all models...retrying in 1 second I1103 12:06:38.473865 110 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x303b50000' with size 268435456 I1103 12:06:38.473969 110 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 1000000000 > Riva waiting for Triton server to load all models...retrying in 1 second I1103 12:06:39.248225 110 model_repository_manager.cc:1077] loading: conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline:1 I1103 12:06:39.352515 110 model_repository_manager.cc:1077] loading: conformer-en-US-asr-offline-endpointing-streaming-offline:1 I1103 12:06:39.354958 110 ctc-decoder-library.cc:20] TRITONBACKEND_ModelInitialize: conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline (version 1) WARNING: Logging before InitGoogleLogging() is written to STDERR W1103 12:06:39.356168 123 parameter_parser.cc:144] Parameter 'beam_size' set but unused. W1103 12:06:39.356195 123 parameter_parser.cc:144] Parameter 'beam_size_token' set but unused. W1103 12:06:39.356199 123 parameter_parser.cc:144] Parameter 'beam_threshold' set but unused. W1103 12:06:39.356200 123 parameter_parser.cc:144] Parameter 'blank_token' set but unused. W1103 12:06:39.356201 123 parameter_parser.cc:144] Parameter 'decoder_num_worker_threads' set but unused. W1103 12:06:39.356204 123 parameter_parser.cc:144] Parameter 'forerunner_beam_size' set but unused. W1103 12:06:39.356204 123 parameter_parser.cc:144] Parameter 'forerunner_beam_size_token' set but unused. W1103 12:06:39.356205 123 parameter_parser.cc:144] Parameter 'forerunner_beam_threshold' set but unused. W1103 12:06:39.356206 123 parameter_parser.cc:144] Parameter 'forerunner_use_lm' set but unused. W1103 12:06:39.356209 123 parameter_parser.cc:144] Parameter 'language_model_file' set but unused. W1103 12:06:39.356209 123 parameter_parser.cc:144] Parameter 'lexicon_file' set but unused. W1103 12:06:39.356210 123 parameter_parser.cc:144] Parameter 'lm_weight' set but unused. W1103 12:06:39.356211 123 parameter_parser.cc:144] Parameter 'log_add' set but unused. W1103 12:06:39.356213 123 parameter_parser.cc:144] Parameter 'max_execution_batch_size' set but unused. W1103 12:06:39.356215 123 parameter_parser.cc:144] Parameter 'max_supported_transcripts' set but unused. W1103 12:06:39.356215 123 parameter_parser.cc:144] Parameter 'num_tokenization' set but unused. W1103 12:06:39.356217 123 parameter_parser.cc:144] Parameter 'profane_words_file' set but unused. W1103 12:06:39.356218 123 parameter_parser.cc:144] Parameter 'set_default_index_to_unk_token' set but unused. W1103 12:06:39.356220 123 parameter_parser.cc:144] Parameter 'sil_token' set but unused. W1103 12:06:39.356220 123 parameter_parser.cc:144] Parameter 'smearing_mode' set but unused. W1103 12:06:39.356222 123 parameter_parser.cc:144] Parameter 'tokenizer_model' set but unused. W1103 12:06:39.356223 123 parameter_parser.cc:144] Parameter 'unk_score' set but unused. W1103 12:06:39.356225 123 parameter_parser.cc:144] Parameter 'unk_token' set but unused. W1103 12:06:39.356225 123 parameter_parser.cc:144] Parameter 'vocab_file' set but unused. W1103 12:06:39.356227 123 parameter_parser.cc:144] Parameter 'word_insertion_score' set but unused. I1103 12:06:39.363670 110 backend_model.cc:303] model configuration: { "name": "conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline", "platform": "", "backend": "riva_asr_decoder", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 1024, "input": [ { "name": "CLASS_LOGITS", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1, 257 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "END_FLAG", "data_type": "TYPE_UINT32", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "SEGMENTS_START_END", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1, 2 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "CUSTOM_CONFIGURATION", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ -1, 2 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "FINAL_TRANSCRIPTS", "data_type": "TYPE_STRING", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "FINAL_TRANSCRIPTS_SCORE", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "FINAL_WORDS_START_END", "data_type": "TYPE_INT32", "dims": [ -1, 2 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PARTIAL_TRANSCRIPTS", "data_type": "TYPE_STRING", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PARTIAL_TRANSCRIPTS_STABILITY", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PARTIAL_WORDS_START_END", "data_type": "TYPE_INT32", "dims": [ -1, 2 ], "label_filename": "", "is_shape_tensor": false }, { "name": "FINAL_WORDS_SCORE", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PARTIAL_WORDS_SCORE", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "graph": { "level": 0 }, "priority": "PRIORITY_DEFAULT", "cuda": { "graphs": false, "busy_wait_events": false, "graph_spec": [], "output_copy_stream": true }, "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "oldest": { "max_candidate_sequences": 1024, "preferred_batch_size": [ 32, 64 ], "max_queue_delay_microseconds": 1000 }, "max_sequence_idle_microseconds": 60000000, "control_input": [ { "name": "START", "control": [ { "kind": "CONTROL_SEQUENCE_START", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "READY", "control": [ { "kind": "CONTROL_SEQUENCE_READY", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "END", "control": [ { "kind": "CONTROL_SEQUENCE_END", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "CORRID", "control": [ { "kind": "CONTROL_SEQUENCE_CORRID", "int32_false_true": [], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_UINT64" } ] } ], "state": [] }, "instance_group": [ { "name": "conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "forerunner_beam_size": { "string_value": "8" }, "unk_score": { "string_value": "-inf" }, "chunk_size": { "string_value": "4.8" }, "max_supported_transcripts": { "string_value": "1" }, "lexicon_file": { "string_value": "/data/models/conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline/1/lexicon.txt" }, "smearing_mode": { "string_value": "max" }, "log_add": { "string_value": "True" }, "lm_weight": { "string_value": "0.8" }, "blank_token": { "string_value": "#" }, "vocab_file": { "string_value": "/data/models/conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline/1/riva_decoder_vocabulary.txt" }, "ms_per_timestep": { "string_value": "40" }, "streaming": { "string_value": "True" }, "use_subword": { "string_value": "True" }, "beam_size": { "string_value": "32" }, "right_padding_size": { "string_value": "1.6" }, "beam_size_token": { "string_value": "16" }, "sil_token": { "string_value": "▁" }, "num_tokenization": { "string_value": "1" }, "beam_threshold": { "string_value": "20.0" }, "tokenizer_model": { "string_value": "/data/models/conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline/1/8b8f095152034e98b24ab33726708bd0_tokenizer.model" }, "language_model_file": { "string_value": "/data/models/conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline/1/4gram-pruned-0_2_7_9-en-lm-set-2.0.bin" }, "max_execution_batch_size": { "string_value": "1024" }, "forerunner_use_lm": { "string_value": "true" }, "forerunner_beam_size_token": { "string_value": "8" }, "profane_words_file": { "string_value": "/data/models/conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline/1/profane_words_file.txt" }, "forerunner_beam_threshold": { "string_value": "10.0" }, "asr_model_delay": { "string_value": "-1" }, "decoder_num_worker_threads": { "string_value": "-1" }, "word_insertion_score": { "string_value": "1.0" }, "unk_token": { "string_value": "" }, "left_padding_size": { "string_value": "1.6" }, "set_default_index_to_unk_token": { "string_value": "False" }, "decoder_type": { "string_value": "flashlight" } }, "model_warmup": [], "model_transaction_policy": { "decoupled": false } } I1103 12:06:39.363758 110 ctc-decoder-library.cc:23] TRITONBACKEND_ModelInstanceInitialize: conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline_0 (device 0) I1103 12:06:39.456025 110 model_repository_manager.cc:1077] loading: conformer-en-US-asr-offline-feature-extractor-streaming-offline:1 I1103 12:06:39.560007 110 model_repository_manager.cc:1077] loading: conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming:1 I1103 12:06:39.663979 110 model_repository_manager.cc:1077] loading: conformer-en-US-asr-streaming-endpointing-streaming:1 I1103 12:06:39.767725 110 model_repository_manager.cc:1077] loading: conformer-en-US-asr-streaming-feature-extractor-streaming:1 I1103 12:06:39.872147 110 model_repository_manager.cc:1077] loading: intent_slot_detokenizer:1 I1103 12:06:39.977098 110 model_repository_manager.cc:1077] loading: intent_slot_label_tokens_weather:1 > Riva waiting for Triton server to load all models...retrying in 1 second I1103 12:06:40.081051 110 model_repository_manager.cc:1077] loading: intent_slot_tokenizer-en-US-weather:1 I1103 12:06:40.185436 110 model_repository_manager.cc:1077] loading: qa_qa_postprocessor:1 I1103 12:06:40.289421 110 model_repository_manager.cc:1077] loading: qa_tokenizer-en-US:1 I1103 12:06:40.393407 110 model_repository_manager.cc:1077] loading: riva-onnx-fastpitch_encoder-English-US:1 I1103 12:06:40.497356 110 model_repository_manager.cc:1077] loading: riva-punctuation-en-US:1 I1103 12:06:40.601581 110 model_repository_manager.cc:1077] loading: riva-trt-conformer-en-US-asr-offline-am-streaming-offline:1 I1103 12:06:40.705554 110 model_repository_manager.cc:1077] loading: riva-trt-conformer-en-US-asr-streaming-am-streaming:1 I1103 12:06:40.809571 110 model_repository_manager.cc:1077] loading: riva-trt-hifigan-English-US:1 I1103 12:06:40.913666 110 model_repository_manager.cc:1077] loading: riva-trt-riva-punctuation-en-US-nn-bert-base-uncased:1 I1103 12:06:41.017939 110 model_repository_manager.cc:1077] loading: riva-trt-riva_intent_weather-nn-bert-base-uncased:1 > Riva waiting for Triton server to load all models...retrying in 1 second I1103 12:06:41.122029 110 model_repository_manager.cc:1077] loading: riva-trt-riva_ner-nn-bert-base-uncased:1 I1103 12:06:41.226091 110 model_repository_manager.cc:1077] loading: riva-trt-riva_qa-nn-bert-base-uncased:1 I1103 12:06:41.330219 110 model_repository_manager.cc:1077] loading: riva-trt-riva_text_classification_domain-nn-bert-base-uncased:1 I1103 12:06:41.434468 110 model_repository_manager.cc:1077] loading: spectrogram_chunker-English-US:1 I1103 12:06:41.538608 110 model_repository_manager.cc:1077] loading: text_classification_tokenizer-en-US-domain:1 I1103 12:06:41.642619 110 model_repository_manager.cc:1077] loading: token_classification_detokenizer:1 I1103 12:06:41.746857 110 model_repository_manager.cc:1077] loading: token_classification_label_tokens:1 I1103 12:06:41.851028 110 model_repository_manager.cc:1077] loading: token_classification_tokenizer-en-US:1 I1103 12:06:41.954984 110 model_repository_manager.cc:1077] loading: tts_postprocessor-English-US:1 > Riva waiting for Triton server to load all models...retrying in 1 second I1103 12:06:42.059444 110 model_repository_manager.cc:1077] loading: tts_preprocessor-English-US:1 > Riva waiting for Triton server to load all models...retrying in 1 second > Riva waiting for Triton server to load all models...retrying in 1 second I1103 12:06:44.885905 123 ctc-decoder.cc:174] Beam Decoder initialized successfully! I1103 12:06:44.886919 110 endpointing_library.cc:18] TRITONBACKEND_ModelInitialize: conformer-en-US-asr-offline-endpointing-streaming-offline (version 1) I1103 12:06:44.886970 110 model_repository_manager.cc:1231] successfully loaded 'conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline' version 1 WARNING: Logging before InitGoogleLogging() is written to STDERR W1103 12:06:44.887362 124 parameter_parser.cc:144] Parameter 'chunk_size' set but unused. W1103 12:06:44.887389 124 parameter_parser.cc:144] Parameter 'ms_per_timestep' set but unused. W1103 12:06:44.887392 124 parameter_parser.cc:144] Parameter 'residue_blanks_at_end' set but unused. W1103 12:06:44.887394 124 parameter_parser.cc:144] Parameter 'residue_blanks_at_start' set but unused. W1103 12:06:44.887395 124 parameter_parser.cc:144] Parameter 'start_history' set but unused. W1103 12:06:44.887396 124 parameter_parser.cc:144] Parameter 'start_th' set but unused. W1103 12:06:44.887398 124 parameter_parser.cc:144] Parameter 'stop_history' set but unused. W1103 12:06:44.887398 124 parameter_parser.cc:144] Parameter 'stop_th' set but unused. W1103 12:06:44.887400 124 parameter_parser.cc:144] Parameter 'streaming' set but unused. W1103 12:06:44.887401 124 parameter_parser.cc:144] Parameter 'use_subword' set but unused. W1103 12:06:44.887403 124 parameter_parser.cc:144] Parameter 'vocab_file' set but unused. I1103 12:06:44.891046 110 backend_model.cc:303] model configuration: { "name": "conformer-en-US-asr-offline-endpointing-streaming-offline", "platform": "", "backend": "riva_asr_endpointing", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 2048, "input": [ { "name": "CLASS_LOGITS", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1, 257 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "SEGMENTS_START_END", "data_type": "TYPE_FP32", "dims": [ -1, 2 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "graph": { "level": 0 }, "priority": "PRIORITY_DEFAULT", "cuda": { "graphs": false, "busy_wait_events": false, "graph_spec": [], "output_copy_stream": true }, "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "max_sequence_idle_microseconds": 60000000, "control_input": [ { "name": "START", "control": [ { "kind": "CONTROL_SEQUENCE_START", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "READY", "control": [ { "kind": "CONTROL_SEQUENCE_READY", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] } ], "state": [] }, "instance_group": [ { "name": "conformer-en-US-asr-offline-endpointing-streaming-offline_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "streaming": { "string_value": "True" }, "use_subword": { "string_value": "True" }, "stop_history": { "string_value": "800" }, "residue_blanks_at_end": { "string_value": "0" }, "start_th": { "string_value": "0.2" }, "chunk_size": { "string_value": "4.8" }, "endpointing_type": { "string_value": "greedy_ctc" }, "stop_th": { "string_value": "0.98" }, "vocab_file": { "string_value": "/data/models/conformer-en-US-asr-offline-endpointing-streaming-offline/1/riva_decoder_vocabulary.txt" }, "start_history": { "string_value": "200" }, "ms_per_timestep": { "string_value": "40" }, "residue_blanks_at_start": { "string_value": "-2" } }, "model_warmup": [], "model_transaction_policy": { "decoupled": false } } I1103 12:06:44.891555 110 endpointing_library.cc:22] TRITONBACKEND_ModelInstanceInitialize: conformer-en-US-asr-offline-endpointing-streaming-offline_0 (device 0) I1103 12:06:44.914642 110 feature-extractor.cc:400] TRITONBACKEND_ModelInitialize: conformer-en-US-asr-offline-feature-extractor-streaming-offline (version 1) I1103 12:06:44.915588 110 model_repository_manager.cc:1231] successfully loaded 'conformer-en-US-asr-offline-endpointing-streaming-offline' version 1 I1103 12:06:44.925985 110 backend_model.cc:303] model configuration: { "name": "conformer-en-US-asr-offline-feature-extractor-streaming-offline", "platform": "", "backend": "riva_asr_features", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 512, "input": [ { "name": "AUDIO_SIGNAL", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "SAMPLE_RATE", "data_type": "TYPE_UINT32", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "AUDIO_FEATURES", "data_type": "TYPE_FP32", "dims": [ 80, -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "AUDIO_PROCESSED", "data_type": "TYPE_FP32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "AUDIO_FEATURES_LENGTH", "data_type": "TYPE_INT32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "graph": { "level": 0 }, "priority": "PRIORITY_DEFAULT", "cuda": { "graphs": false, "busy_wait_events": false, "graph_spec": [], "output_copy_stream": true }, "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "oldest": { "max_candidate_sequences": 512, "preferred_batch_size": [ 256, 512 ], "max_queue_delay_microseconds": 1000 }, "max_sequence_idle_microseconds": 60000000, "control_input": [ { "name": "START", "control": [ { "kind": "CONTROL_SEQUENCE_START", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "READY", "control": [ { "kind": "CONTROL_SEQUENCE_READY", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "END", "control": [ { "kind": "CONTROL_SEQUENCE_END", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "CORRID", "control": [ { "kind": "CONTROL_SEQUENCE_CORRID", "int32_false_true": [], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_UINT64" } ] } ], "state": [] }, "instance_group": [ { "name": "conformer-en-US-asr-offline-feature-extractor-streaming-offline_0", "kind": "KIND_GPU", "count": 1, "gpus": [ 0 ], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "precalc_norm_time_steps": { "string_value": "0" }, "use_utterance_norm_params": { "string_value": "False" }, "dither": { "string_value": "0.0" }, "apply_normalization": { "string_value": "True" }, "precalc_norm_params": { "string_value": "False" }, "norm_per_feature": { "string_value": "True" }, "mean": { "string_value": "-11.4412, -9.9334, -9.1292, -9.0365, -9.2804, -9.5643, -9.7342, -9.6925, -9.6333, -9.2808, -9.1887, -9.1422, -9.1397, -9.2028, -9.2749, -9.4776, -9.9185, -10.1557, -10.3800, -10.5067, -10.3190, -10.4728, -10.5529, -10.6402, -10.6440, -10.5113, -10.7395, -10.7870, -10.6074, -10.5033, -10.8278, -10.6384, -10.8481, -10.6875, -10.5454, -10.4747, -10.5165, -10.4930, -10.3413, -10.3472, -10.3735, -10.6830, -10.8813, -10.6338, -10.3856, -10.7727, -10.8957, -10.8068, -10.7373, -10.6108, -10.3405, -10.2889, -10.3922, -10.4946, -10.3367, -10.4164, -10.9949, -10.7196, -10.3971, -10.1734, -9.9257, -9.6557, -9.1761, -9.6653, -9.7876, -9.7230, -9.7792, -9.7056, -9.2702, -9.4650, -9.2755, -9.1369, -9.1174, -8.9197, -8.5394, -8.2614, -8.1353, -8.1422, -8.3430, -8.6655" }, "stddev": { "string_value": "2.2668, 3.1642, 3.7079, 3.7642, 3.5349, 3.5901, 3.7640, 3.8424, 4.0145, 4.1475, 4.0457, 3.9048, 3.7709, 3.6117, 3.3188, 3.1489, 3.0615, 3.0362, 2.9929, 3.0500, 3.0341, 3.0484, 3.0103, 2.9474, 2.9128, 2.8669, 2.8332, 2.9411, 3.0378, 3.0712, 3.0190, 2.9992, 3.0124, 3.0024, 3.0275, 3.0870, 3.0656, 3.0142, 3.0493, 3.1373, 3.1135, 3.0675, 2.8828, 2.7018, 2.6296, 2.8826, 2.9325, 2.9288, 2.9271, 2.9890, 3.0137, 2.9855, 3.0839, 2.9319, 2.3512, 2.3795, 2.6191, 2.7555, 2.9326, 2.9931, 3.1543, 3.0855, 2.6820, 3.0566, 3.1272, 3.1663, 3.1836, 3.0018, 2.9089, 3.1727, 3.1626, 3.1086, 2.9804, 3.1107, 3.2998, 3.3697, 3.3716, 3.2487, 3.1597, 3.1181" }, "chunk_size": { "string_value": "4.8" }, "max_execution_batch_size": { "string_value": "512" }, "sample_rate": { "string_value": "16000" }, "window_stride": { "string_value": "0.01" }, "num_features": { "string_value": "80" }, "window_size": { "string_value": "0.025" }, "streaming": { "string_value": "True" }, "left_padding_size": { "string_value": "1.6" }, "transpose": { "string_value": "False" }, "stddev_floor": { "string_value": "1e-05" }, "right_padding_size": { "string_value": "1.6" }, "gain": { "string_value": "1.0" } }, "model_warmup": [], "model_transaction_policy": { "decoupled": false } } I1103 12:06:44.926074 110 ctc-decoder-library.cc:20] TRITONBACKEND_ModelInitialize: conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming (version 1) W1103 12:06:44.926651 126 parameter_parser.cc:144] Parameter 'beam_size' set but unused. W1103 12:06:44.926678 126 parameter_parser.cc:144] Parameter 'beam_size_token' set but unused. W1103 12:06:44.926682 126 parameter_parser.cc:144] Parameter 'beam_threshold' set but unused. W1103 12:06:44.926684 126 parameter_parser.cc:144] Parameter 'blank_token' set but unused. W1103 12:06:44.926685 126 parameter_parser.cc:144] Parameter 'decoder_num_worker_threads' set but unused. W1103 12:06:44.926687 126 parameter_parser.cc:144] Parameter 'forerunner_beam_size' set but unused. W1103 12:06:44.926688 126 parameter_parser.cc:144] Parameter 'forerunner_beam_size_token' set but unused. W1103 12:06:44.926689 126 parameter_parser.cc:144] Parameter 'forerunner_beam_threshold' set but unused. W1103 12:06:44.926690 126 parameter_parser.cc:144] Parameter 'forerunner_use_lm' set but unused. W1103 12:06:44.926692 126 parameter_parser.cc:144] Parameter 'language_model_file' set but unused. W1103 12:06:44.926693 126 parameter_parser.cc:144] Parameter 'lexicon_file' set but unused. W1103 12:06:44.926694 126 parameter_parser.cc:144] Parameter 'lm_weight' set but unused. W1103 12:06:44.926695 126 parameter_parser.cc:144] Parameter 'log_add' set but unused. W1103 12:06:44.926697 126 parameter_parser.cc:144] Parameter 'max_execution_batch_size' set but unused. W1103 12:06:44.926698 126 parameter_parser.cc:144] Parameter 'max_supported_transcripts' set but unused. W1103 12:06:44.926699 126 parameter_parser.cc:144] Parameter 'num_tokenization' set but unused. W1103 12:06:44.926700 126 parameter_parser.cc:144] Parameter 'profane_words_file' set but unused. W1103 12:06:44.926702 126 parameter_parser.cc:144] Parameter 'set_default_index_to_unk_token' set but unused. W1103 12:06:44.926703 126 parameter_parser.cc:144] Parameter 'sil_token' set but unused. W1103 12:06:44.926704 126 parameter_parser.cc:144] Parameter 'smearing_mode' set but unused. W1103 12:06:44.926705 126 parameter_parser.cc:144] Parameter 'tokenizer_model' set but unused. W1103 12:06:44.926707 126 parameter_parser.cc:144] Parameter 'unk_score' set but unused. W1103 12:06:44.926708 126 parameter_parser.cc:144] Parameter 'unk_token' set but unused. W1103 12:06:44.926709 126 parameter_parser.cc:144] Parameter 'vocab_file' set but unused. W1103 12:06:44.926710 126 parameter_parser.cc:144] Parameter 'word_insertion_score' set but unused. I1103 12:06:44.933293 110 backend_model.cc:303] model configuration: { "name": "conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming", "platform": "", "backend": "riva_asr_decoder", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 1024, "input": [ { "name": "CLASS_LOGITS", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1, 257 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "END_FLAG", "data_type": "TYPE_UINT32", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "SEGMENTS_START_END", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1, 2 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "CUSTOM_CONFIGURATION", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ -1, 2 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "FINAL_TRANSCRIPTS", "data_type": "TYPE_STRING", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "FINAL_TRANSCRIPTS_SCORE", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "FINAL_WORDS_START_END", "data_type": "TYPE_INT32", "dims": [ -1, 2 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PARTIAL_TRANSCRIPTS", "data_type": "TYPE_STRING", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PARTIAL_TRANSCRIPTS_STABILITY", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PARTIAL_WORDS_START_END", "data_type": "TYPE_INT32", "dims": [ -1, 2 ], "label_filename": "", "is_shape_tensor": false }, { "name": "FINAL_WORDS_SCORE", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PARTIAL_WORDS_SCORE", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "graph": { "level": 0 }, "priority": "PRIORITY_DEFAULT", "cuda": { "graphs": false, "busy_wait_events": false, "graph_spec": [], "output_copy_stream": true }, "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "oldest": { "max_candidate_sequences": 1024, "preferred_batch_size": [ 32, 64 ], "max_queue_delay_microseconds": 1000 }, "max_sequence_idle_microseconds": 60000000, "control_input": [ { "name": "START", "control": [ { "kind": "CONTROL_SEQUENCE_START", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "READY", "control": [ { "kind": "CONTROL_SEQUENCE_READY", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "END", "control": [ { "kind": "CONTROL_SEQUENCE_END", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "CORRID", "control": [ { "kind": "CONTROL_SEQUENCE_CORRID", "int32_false_true": [], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_UINT64" } ] } ], "state": [] }, "instance_group": [ { "name": "conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "beam_threshold": { "string_value": "20.0" }, "tokenizer_model": { "string_value": "/data/models/conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/8b8f095152034e98b24ab33726708bd0_tokenizer.model" }, "language_model_file": { "string_value": "/data/models/conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/4gram-pruned-0_2_7_9-en-lm-set-2.0.bin" }, "max_execution_batch_size": { "string_value": "1024" }, "forerunner_use_lm": { "string_value": "true" }, "profane_words_file": { "string_value": "/data/models/conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/profane_words_file.txt" }, "forerunner_beam_size_token": { "string_value": "8" }, "forerunner_beam_threshold": { "string_value": "10.0" }, "decoder_num_worker_threads": { "string_value": "-1" }, "asr_model_delay": { "string_value": "-1" }, "word_insertion_score": { "string_value": "1.0" }, "unk_token": { "string_value": "" }, "left_padding_size": { "string_value": "1.92" }, "set_default_index_to_unk_token": { "string_value": "False" }, "decoder_type": { "string_value": "flashlight" }, "forerunner_beam_size": { "string_value": "8" }, "unk_score": { "string_value": "-inf" }, "chunk_size": { "string_value": "0.16" }, "max_supported_transcripts": { "string_value": "1" }, "lexicon_file": { "string_value": "/data/models/conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/lexicon.txt" }, "smearing_mode": { "string_value": "max" }, "log_add": { "string_value": "True" }, "blank_token": { "string_value": "#" }, "lm_weight": { "string_value": "0.8" }, "vocab_file": { "string_value": "/data/models/conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/riva_decoder_vocabulary.txt" }, "ms_per_timestep": { "string_value": "40" }, "use_subword": { "string_value": "True" }, "streaming": { "string_value": "True" }, "beam_size": { "string_value": "32" }, "right_padding_size": { "string_value": "1.92" }, "beam_size_token": { "string_value": "16" }, "sil_token": { "string_value": "▁" }, "num_tokenization": { "string_value": "1" } }, "model_warmup": [], "model_transaction_policy": { "decoupled": false } } I1103 12:06:44.933384 110 endpointing_library.cc:18] TRITONBACKEND_ModelInitialize: conformer-en-US-asr-streaming-endpointing-streaming (version 1) W1103 12:06:44.933751 127 parameter_parser.cc:144] Parameter 'chunk_size' set but unused. W1103 12:06:44.933777 127 parameter_parser.cc:144] Parameter 'ms_per_timestep' set but unused. W1103 12:06:44.933781 127 parameter_parser.cc:144] Parameter 'residue_blanks_at_end' set but unused. W1103 12:06:44.933782 127 parameter_parser.cc:144] Parameter 'residue_blanks_at_start' set but unused. W1103 12:06:44.933784 127 parameter_parser.cc:144] Parameter 'start_history' set but unused. W1103 12:06:44.933785 127 parameter_parser.cc:144] Parameter 'start_th' set but unused. W1103 12:06:44.933786 127 parameter_parser.cc:144] Parameter 'stop_history' set but unused. W1103 12:06:44.933787 127 parameter_parser.cc:144] Parameter 'stop_th' set but unused. W1103 12:06:44.933789 127 parameter_parser.cc:144] Parameter 'streaming' set but unused. W1103 12:06:44.933790 127 parameter_parser.cc:144] Parameter 'use_subword' set but unused. W1103 12:06:44.933791 127 parameter_parser.cc:144] Parameter 'vocab_file' set but unused. I1103 12:06:44.936856 110 backend_model.cc:303] model configuration: { "name": "conformer-en-US-asr-streaming-endpointing-streaming", "platform": "", "backend": "riva_asr_endpointing", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 2048, "input": [ { "name": "CLASS_LOGITS", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1, 257 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "SEGMENTS_START_END", "data_type": "TYPE_FP32", "dims": [ -1, 2 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "graph": { "level": 0 }, "priority": "PRIORITY_DEFAULT", "cuda": { "graphs": false, "busy_wait_events": false, "graph_spec": [], "output_copy_stream": true }, "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "max_sequence_idle_microseconds": 60000000, "control_input": [ { "name": "START", "control": [ { "kind": "CONTROL_SEQUENCE_START", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "READY", "control": [ { "kind": "CONTROL_SEQUENCE_READY", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] } ], "state": [] }, "instance_group": [ { "name": "conformer-en-US-asr-streaming-endpointing-streaming_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "endpointing_type": { "string_value": "greedy_ctc" }, "stop_th": { "string_value": "0.98" }, "vocab_file": { "string_value": "/data/models/conformer-en-US-asr-streaming-endpointing-streaming/1/riva_decoder_vocabulary.txt" }, "start_history": { "string_value": "200" }, "residue_blanks_at_start": { "string_value": "-2" }, "ms_per_timestep": { "string_value": "40" }, "use_subword": { "string_value": "True" }, "streaming": { "string_value": "True" }, "stop_history": { "string_value": "800" }, "residue_blanks_at_end": { "string_value": "0" }, "start_th": { "string_value": "0.2" }, "chunk_size": { "string_value": "0.16" } }, "model_warmup": [], "model_transaction_policy": { "decoupled": false } } I1103 12:06:44.936927 110 feature-extractor.cc:400] TRITONBACKEND_ModelInitialize: conformer-en-US-asr-streaming-feature-extractor-streaming (version 1) I1103 12:06:44.937462 110 backend_model.cc:303] model configuration: { "name": "conformer-en-US-asr-streaming-feature-extractor-streaming", "platform": "", "backend": "riva_asr_features", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 1024, "input": [ { "name": "AUDIO_SIGNAL", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "SAMPLE_RATE", "data_type": "TYPE_UINT32", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "AUDIO_FEATURES", "data_type": "TYPE_FP32", "dims": [ 80, -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "AUDIO_PROCESSED", "data_type": "TYPE_FP32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "AUDIO_FEATURES_LENGTH", "data_type": "TYPE_INT32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "graph": { "level": 0 }, "priority": "PRIORITY_DEFAULT", "cuda": { "graphs": false, "busy_wait_events": false, "graph_spec": [], "output_copy_stream": true }, "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "oldest": { "max_candidate_sequences": 1024, "preferred_batch_size": [ 256, 512 ], "max_queue_delay_microseconds": 1000 }, "max_sequence_idle_microseconds": 60000000, "control_input": [ { "name": "START", "control": [ { "kind": "CONTROL_SEQUENCE_START", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "READY", "control": [ { "kind": "CONTROL_SEQUENCE_READY", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "END", "control": [ { "kind": "CONTROL_SEQUENCE_END", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "CORRID", "control": [ { "kind": "CONTROL_SEQUENCE_CORRID", "int32_false_true": [], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_UINT64" } ] } ], "state": [] }, "instance_group": [ { "name": "conformer-en-US-asr-streaming-feature-extractor-streaming_0", "kind": "KIND_GPU", "count": 1, "gpus": [ 0 ], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "window_stride": { "string_value": "0.01" }, "window_size": { "string_value": "0.025" }, "num_features": { "string_value": "80" }, "streaming": { "string_value": "True" }, "transpose": { "string_value": "False" }, "stddev_floor": { "string_value": "1e-05" }, "left_padding_size": { "string_value": "1.92" }, "right_padding_size": { "string_value": "1.92" }, "gain": { "string_value": "1.0" }, "precalc_norm_time_steps": { "string_value": "0" }, "use_utterance_norm_params": { "string_value": "False" }, "dither": { "string_value": "0.0" }, "apply_normalization": { "string_value": "True" }, "precalc_norm_params": { "string_value": "False" }, "norm_per_feature": { "string_value": "True" }, "mean": { "string_value": "-11.4412, -9.9334, -9.1292, -9.0365, -9.2804, -9.5643, -9.7342, -9.6925, -9.6333, -9.2808, -9.1887, -9.1422, -9.1397, -9.2028, -9.2749, -9.4776, -9.9185, -10.1557, -10.3800, -10.5067, -10.3190, -10.4728, -10.5529, -10.6402, -10.6440, -10.5113, -10.7395, -10.7870, -10.6074, -10.5033, -10.8278, -10.6384, -10.8481, -10.6875, -10.5454, -10.4747, -10.5165, -10.4930, -10.3413, -10.3472, -10.3735, -10.6830, -10.8813, -10.6338, -10.3856, -10.7727, -10.8957, -10.8068, -10.7373, -10.6108, -10.3405, -10.2889, -10.3922, -10.4946, -10.3367, -10.4164, -10.9949, -10.7196, -10.3971, -10.1734, -9.9257, -9.6557, -9.1761, -9.6653, -9.7876, -9.7230, -9.7792, -9.7056, -9.2702, -9.4650, -9.2755, -9.1369, -9.1174, -8.9197, -8.5394, -8.2614, -8.1353, -8.1422, -8.3430, -8.6655" }, "stddev": { "string_value": "2.2668, 3.1642, 3.7079, 3.7642, 3.5349, 3.5901, 3.7640, 3.8424, 4.0145, 4.1475, 4.0457, 3.9048, 3.7709, 3.6117, 3.3188, 3.1489, 3.0615, 3.0362, 2.9929, 3.0500, 3.0341, 3.0484, 3.0103, 2.9474, 2.9128, 2.8669, 2.8332, 2.9411, 3.0378, 3.0712, 3.0190, 2.9992, 3.0124, 3.0024, 3.0275, 3.0870, 3.0656, 3.0142, 3.0493, 3.1373, 3.1135, 3.0675, 2.8828, 2.7018, 2.6296, 2.8826, 2.9325, 2.9288, 2.9271, 2.9890, 3.0137, 2.9855, 3.0839, 2.9319, 2.3512, 2.3795, 2.6191, 2.7555, 2.9326, 2.9931, 3.1543, 3.0855, 2.6820, 3.0566, 3.1272, 3.1663, 3.1836, 3.0018, 2.9089, 3.1727, 3.1626, 3.1086, 2.9804, 3.1107, 3.2998, 3.3697, 3.3716, 3.2487, 3.1597, 3.1181" }, "chunk_size": { "string_value": "0.16" }, "max_execution_batch_size": { "string_value": "1024" }, "sample_rate": { "string_value": "16000" } }, "model_warmup": [], "model_transaction_policy": { "decoupled": false } } I1103 12:06:44.937938 110 feature-extractor.cc:402] TRITONBACKEND_ModelInstanceInitialize: conformer-en-US-asr-offline-feature-extractor-streaming-offline_0 (device 0) > Riva waiting for Triton server to load all models...retrying in 1 second > Riva waiting for Triton server to load all models...retrying in 1 second I1103 12:06:46.232622 110 ctc-decoder-library.cc:23] TRITONBACKEND_ModelInstanceInitialize: conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming_0 (device 0) I1103 12:06:46.233537 110 model_repository_manager.cc:1231] successfully loaded 'conformer-en-US-asr-offline-feature-extractor-streaming-offline' version 1 > Riva waiting for Triton server to load all models...retrying in 1 second > Riva waiting for Triton server to load all models...retrying in 1 second > Riva waiting for Triton server to load all models...retrying in 1 second > Riva waiting for Triton server to load all models...retrying in 1 second > Riva waiting for Triton server to load all models...retrying in 1 second I1103 12:06:51.648084 126 ctc-decoder.cc:174] Beam Decoder initialized successfully! I1103 12:06:51.648235 110 endpointing_library.cc:22] TRITONBACKEND_ModelInstanceInitialize: conformer-en-US-asr-streaming-endpointing-streaming_0 (device 0) I1103 12:06:51.648952 110 model_repository_manager.cc:1231] successfully loaded 'conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming' version 1 I1103 12:06:51.671204 110 feature-extractor.cc:402] TRITONBACKEND_ModelInstanceInitialize: conformer-en-US-asr-streaming-feature-extractor-streaming_0 (device 0) I1103 12:06:51.671928 110 model_repository_manager.cc:1231] successfully loaded 'conformer-en-US-asr-streaming-endpointing-streaming' version 1 I1103 12:06:51.698091 110 detokenizer_cbe.cc:145] TRITONBACKEND_ModelInitialize: intent_slot_detokenizer (version 1) I1103 12:06:51.698508 110 backend_model.cc:303] model configuration: { "name": "intent_slot_detokenizer", "platform": "", "backend": "riva_nlp_detokenizer", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 8, "input": [ { "name": "IN_TOKEN_LABELS__0", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "IN_TOKEN_SCORES__1", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "IN_SEQ_LEN__2", "data_type": "TYPE_INT64", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "IN_TOK_STR__3", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "OUT_TOKEN_LABELS__0", "data_type": "TYPE_STRING", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "OUT_TOKEN_SCORES__1", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "OUT_SEQ_LEN__2", "data_type": "TYPE_INT64", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "OUT_TOK_STR__3", "data_type": "TYPE_STRING", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "graph": { "level": 0 }, "priority": "PRIORITY_DEFAULT", "cuda": { "graphs": false, "busy_wait_events": false, "graph_spec": [], "output_copy_stream": true }, "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "instance_group": [ { "name": "intent_slot_detokenizer_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": {}, "model_warmup": [], "model_transaction_policy": { "decoupled": false } } I1103 12:06:51.698698 110 model_repository_manager.cc:1231] successfully loaded 'conformer-en-US-asr-streaming-feature-extractor-streaming' version 1 I1103 12:06:51.699114 110 detokenizer_cbe.cc:147] TRITONBACKEND_ModelInstanceInitialize: intent_slot_detokenizer_0 (device 0) I1103 12:06:51.699204 110 sequence_label_cbe.cc:137] TRITONBACKEND_ModelInitialize: intent_slot_label_tokens_weather (version 1) I1103 12:06:51.699393 110 model_repository_manager.cc:1231] successfully loaded 'intent_slot_detokenizer' version 1 I1103 12:06:51.701832 110 backend_model.cc:303] model configuration: { "name": "intent_slot_label_tokens_weather", "platform": "", "backend": "riva_nlp_seqlabel", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 8, "input": [ { "name": "TOKEN_LOGIT__1", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1, 65 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "TOKEN_LABELS__0", "data_type": "TYPE_STRING", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "TOKEN_SCORES__1", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "priority": "PRIORITY_DEFAULT", "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "instance_group": [ { "name": "intent_slot_label_tokens_weather_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "classes": { "string_value": "/data/models/intent_slot_label_tokens_weather/1/slot_labels.csv" } }, "model_warmup": [] } I1103 12:06:51.706687 110 sequence_label_cbe.cc:139] TRITONBACKEND_ModelInstanceInitialize: intent_slot_label_tokens_weather_0 (device 0) I1103 12:06:51.706792 110 tokenizer_library.cc:18] TRITONBACKEND_ModelInitialize: intent_slot_tokenizer-en-US-weather (version 1) I1103 12:06:51.707012 110 model_repository_manager.cc:1231] successfully loaded 'intent_slot_label_tokens_weather' version 1 WARNING: Logging before InitGoogleLogging() is written to STDERR W1103 12:06:51.707163 134 parameter_parser.cc:144] Parameter 'unk_token' set but unused. W1103 12:06:51.707192 134 parameter_parser.cc:144] Parameter 'vocab' set but unused. I1103 12:06:51.707247 110 backend_model.cc:303] model configuration: { "name": "intent_slot_tokenizer-en-US-weather", "platform": "", "backend": "riva_nlp_tokenizer", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 8, "input": [ { "name": "INPUT_STR__0", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "SEQ__0", "data_type": "TYPE_INT32", "dims": [ 128 ], "label_filename": "", "is_shape_tensor": false }, { "name": "MASK__1", "data_type": "TYPE_INT32", "dims": [ 128 ], "label_filename": "", "is_shape_tensor": false }, { "name": "SEGMENT__4", "data_type": "TYPE_INT32", "dims": [ 128 ], "label_filename": "", "is_shape_tensor": false }, { "name": "SEQ_LEN__2", "data_type": "TYPE_INT64", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "TOK_STR__3", "data_type": "TYPE_STRING", "dims": [ 128 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "priority": "PRIORITY_DEFAULT", "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "instance_group": [ { "name": "intent_slot_tokenizer-en-US-weather_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "task": { "string_value": "single_input" }, "unk_token": { "string_value": "[UNK]" }, "vocab": { "string_value": "/data/models/intent_slot_tokenizer-en-US-weather/1/tokenizer.vocab_file" }, "tokenizer": { "string_value": "wordpiece" }, "bos_token": { "string_value": "[CLS]" }, "eos_token": { "string_value": "[SEP]" }, "to_lower": { "string_value": "true" }, "pad_chars_with_spaces": { "string_value": "False" } }, "model_warmup": [] } I1103 12:06:51.712236 110 tokenizer_library.cc:21] TRITONBACKEND_ModelInstanceInitialize: intent_slot_tokenizer-en-US-weather_0 (device 0) I1103 12:06:51.728944 110 qa_postprocessor_cbe.cc:124] TRITONBACKEND_ModelInitialize: qa_qa_postprocessor (version 1) I1103 12:06:51.729163 110 model_repository_manager.cc:1231] successfully loaded 'intent_slot_tokenizer-en-US-weather' version 1 I1103 12:06:51.729439 110 backend_model.cc:303] model configuration: { "name": "qa_qa_postprocessor", "platform": "", "backend": "riva_nlp_qa", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 8, "input": [ { "name": "QA_LOGITS__0", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ 384, 2 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "SEQ_LEN__1", "data_type": "TYPE_INT64", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "TOK_STR__2", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ 384 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "TOK_TO_ORIG__3", "data_type": "TYPE_UINT16", "format": "FORMAT_NONE", "dims": [ 384 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "IN_PASSAGE_STR__4", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "ANSWER_SPANS__0", "data_type": "TYPE_STRING", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "ANSWER_SCORES__1", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "priority": "PRIORITY_DEFAULT", "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "instance_group": [ { "name": "qa_qa_postprocessor_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "bert_model_seq_length": { "string_value": "384" }, "version_2_with_negative": { "string_value": "True" }, "n_best_size": { "string_value": "20" }, "max_answer_length": { "string_value": "30" } }, "model_warmup": [] } I1103 12:06:51.729511 110 tokenizer_library.cc:18] TRITONBACKEND_ModelInitialize: qa_tokenizer-en-US (version 1) W1103 12:06:51.729813 136 parameter_parser.cc:144] Parameter 'unk_token' set but unused. W1103 12:06:51.729838 136 parameter_parser.cc:144] Parameter 'vocab' set but unused. I1103 12:06:51.729891 110 backend_model.cc:303] model configuration: { "name": "qa_tokenizer-en-US", "platform": "", "backend": "riva_nlp_tokenizer", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 8, "input": [ { "name": "IN_QUERY_STR__0", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "IN_PASSAGE_STR__1", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "SEQ__0", "data_type": "TYPE_INT32", "dims": [ 384 ], "label_filename": "", "is_shape_tensor": false }, { "name": "MASK__1", "data_type": "TYPE_INT32", "dims": [ 384 ], "label_filename": "", "is_shape_tensor": false }, { "name": "SEQ_LEN__2", "data_type": "TYPE_INT64", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "TOK_STR__3", "data_type": "TYPE_STRING", "dims": [ 384 ], "label_filename": "", "is_shape_tensor": false }, { "name": "SEGMENT__4", "data_type": "TYPE_INT32", "dims": [ 384 ], "label_filename": "", "is_shape_tensor": false }, { "name": "TOK_TO_ORIG__5", "data_type": "TYPE_UINT16", "dims": [ 384 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "priority": "PRIORITY_DEFAULT", "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "instance_group": [ { "name": "qa_tokenizer-en-US_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "pad_chars_with_spaces": { "string_value": "False" }, "task": { "string_value": "qa" }, "doc_stride": { "string_value": "128" }, "unk_token": { "string_value": "[UNK]" }, "vocab": { "string_value": "/data/models/qa_tokenizer-en-US/1/tokenizer.vocab_file" }, "tokenizer": { "string_value": "wordpiece" }, "bos_token": { "string_value": "[CLS]" }, "max_query_length": { "string_value": "64" }, "eos_token": { "string_value": "[SEP]" }, "to_lower": { "string_value": "true" } }, "model_warmup": [] } I1103 12:06:51.735503 110 onnxruntime.cc:2481] TRITONBACKEND_ModelInitialize: riva-onnx-fastpitch_encoder-English-US (version 1) I1103 12:06:51.735932 110 qa_postprocessor_cbe.cc:126] TRITONBACKEND_ModelInstanceInitialize: qa_qa_postprocessor_0 (device 0) I1103 12:06:51.735994 110 tokenizer_library.cc:21] TRITONBACKEND_ModelInstanceInitialize: qa_tokenizer-en-US_0 (device 0) I1103 12:06:51.736124 110 model_repository_manager.cc:1231] successfully loaded 'qa_qa_postprocessor' version 1 I1103 12:06:51.752414 110 model_repository_manager.cc:1231] successfully loaded 'qa_tokenizer-en-US' version 1 I1103 12:06:51.752794 110 tensorrt.cc:5294] TRITONBACKEND_Initialize: tensorrt I1103 12:06:51.752827 110 tensorrt.cc:5304] Triton TRITONBACKEND API version: 1.9 I1103 12:06:51.752834 110 tensorrt.cc:5310] 'tensorrt' TRITONBACKEND API version: 1.9 I1103 12:06:51.752904 110 tensorrt.cc:5353] backend configuration: {} I1103 12:06:51.752964 110 onnxruntime.cc:2524] TRITONBACKEND_ModelInstanceInitialize: riva-onnx-fastpitch_encoder-English-US_0 (GPU device 0) I1103 12:06:51.915112 110 onnxruntime.cc:2558] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:51.915131 110 pipeline_library.cc:22] TRITONBACKEND_ModelInitialize: riva-punctuation-en-US (version 1) I1103 12:06:51.915180 110 onnxruntime.cc:2504] TRITONBACKEND_ModelFinalize: delete model state E1103 12:06:51.915199 110 model_repository_manager.cc:1234] failed to load 'riva-onnx-fastpitch_encoder-English-US' version 1: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. WARNING: Logging before InitGoogleLogging() is written to STDERR W1103 12:06:51.915737 138 parameter_parser.cc:144] Parameter 'attn_mask_tensor_name' set but unused. W1103 12:06:51.915766 138 parameter_parser.cc:144] Parameter 'bos_token' set but unused. W1103 12:06:51.915769 138 parameter_parser.cc:144] Parameter 'capit_logits_tensor_name' set but unused. W1103 12:06:51.915771 138 parameter_parser.cc:144] Parameter 'capitalization_mapping_path' set but unused. W1103 12:06:51.915772 138 parameter_parser.cc:144] Parameter 'delimiter' set but unused. W1103 12:06:51.915773 138 parameter_parser.cc:144] Parameter 'eos_token' set but unused. W1103 12:06:51.915774 138 parameter_parser.cc:144] Parameter 'input_ids_tensor_name' set but unused. W1103 12:06:51.915776 138 parameter_parser.cc:144] Parameter 'language_code' set but unused. W1103 12:06:51.915777 138 parameter_parser.cc:144] Parameter 'model_api' set but unused. W1103 12:06:51.915778 138 parameter_parser.cc:144] Parameter 'model_family' set but unused. W1103 12:06:51.915779 138 parameter_parser.cc:144] Parameter 'pad_chars_with_spaces' set but unused. W1103 12:06:51.915781 138 parameter_parser.cc:144] Parameter 'preserve_accents' set but unused. W1103 12:06:51.915782 138 parameter_parser.cc:144] Parameter 'punct_logits_tensor_name' set but unused. W1103 12:06:51.915783 138 parameter_parser.cc:144] Parameter 'punctuation_mapping_path' set but unused. W1103 12:06:51.915784 138 parameter_parser.cc:144] Parameter 'remove_spaces' set but unused. W1103 12:06:51.915786 138 parameter_parser.cc:144] Parameter 'to_lower' set but unused. W1103 12:06:51.915787 138 parameter_parser.cc:144] Parameter 'token_type_tensor_name' set but unused. W1103 12:06:51.915788 138 parameter_parser.cc:144] Parameter 'tokenizer_to_lower' set but unused. W1103 12:06:51.915789 138 parameter_parser.cc:144] Parameter 'unk_token' set but unused. W1103 12:06:51.915791 138 parameter_parser.cc:144] Parameter 'use_int64_nn_inputs' set but unused. W1103 12:06:51.915792 138 parameter_parser.cc:144] Parameter 'vocab' set but unused. W1103 12:06:51.915834 138 parameter_parser.cc:144] Parameter 'model_api' set but unused. W1103 12:06:51.915858 138 parameter_parser.cc:144] Parameter 'model_family' set but unused. I1103 12:06:51.915906 110 backend_model.cc:303] model configuration: { "name": "riva-punctuation-en-US", "platform": "", "backend": "riva_nlp_pipeline", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 8, "input": [ { "name": "PIPELINE_INPUT", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "PIPELINE_OUTPUT", "data_type": "TYPE_STRING", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "priority": "PRIORITY_DEFAULT", "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "instance_group": [ { "name": "riva-punctuation-en-US_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "input_ids_tensor_name": { "string_value": "input_ids" }, "model_name": { "string_value": "riva-trt-riva-punctuation-en-US-nn-bert-base-uncased" }, "tokenizer_to_lower": { "string_value": "true" }, "vocab": { "string_value": "/data/models/riva-punctuation-en-US/1/e222f352288a423da453a79b96cc7b75_vocab.txt" }, "capit_logits_tensor_name": { "string_value": "capit_logits" }, "pipeline_type": { "string_value": "punctuation" }, "eos_token": { "string_value": "[SEP]" }, "capitalization_mapping_path": { "string_value": "/data/models/riva-punctuation-en-US/1/fb06800834e74de1bdc32db51da9619c_capit_label_ids.csv" }, "token_type_tensor_name": { "string_value": "token_type_ids" }, "tokenizer": { "string_value": "wordpiece" }, "delimiter": { "string_value": " " }, "pad_chars_with_spaces": { "string_value": "False" }, "remove_spaces": { "string_value": "False" }, "use_int64_nn_inputs": { "string_value": "False" }, "preserve_accents": { "string_value": "false" }, "unk_token": { "string_value": "[UNK]" }, "model_family": { "string_value": "riva" }, "bos_token": { "string_value": "[CLS]" }, "punctuation_mapping_path": { "string_value": "/data/models/riva-punctuation-en-US/1/15eace99434b4c87ba28cbd294b48f43_punct_label_ids.csv" }, "model_api": { "string_value": "/nvidia.riva.nlp.RivaLanguageUnderstanding/PunctuateText" }, "to_lower": { "string_value": "true" }, "load_model": { "string_value": "false" }, "attn_mask_tensor_name": { "string_value": "attention_mask" }, "punct_logits_tensor_name": { "string_value": "punct_logits" }, "language_code": { "string_value": "en-US" } }, "model_warmup": [] } I1103 12:06:51.915990 110 tensorrt.cc:5405] TRITONBACKEND_ModelInitialize: riva-trt-conformer-en-US-asr-offline-am-streaming-offline (version 1) I1103 12:06:51.916534 110 tensorrt.cc:5405] TRITONBACKEND_ModelInitialize: riva-trt-conformer-en-US-asr-streaming-am-streaming (version 1) I1103 12:06:51.916887 110 tensorrt.cc:5405] TRITONBACKEND_ModelInitialize: riva-trt-hifigan-English-US (version 1) I1103 12:06:51.917221 110 backend_model.cc:181] Overriding execution policy to "TRITONBACKEND_EXECUTION_BLOCKING" for sequence model "riva-trt-hifigan-English-US" I1103 12:06:51.917228 110 tensorrt.cc:5405] TRITONBACKEND_ModelInitialize: riva-trt-riva-punctuation-en-US-nn-bert-base-uncased (version 1) I1103 12:06:51.917623 110 tensorrt.cc:5405] TRITONBACKEND_ModelInitialize: riva-trt-riva_intent_weather-nn-bert-base-uncased (version 1) I1103 12:06:51.917965 110 tensorrt.cc:5405] TRITONBACKEND_ModelInitialize: riva-trt-riva_ner-nn-bert-base-uncased (version 1) I1103 12:06:51.918297 110 tensorrt.cc:5405] TRITONBACKEND_ModelInitialize: riva-trt-riva_qa-nn-bert-base-uncased (version 1) I1103 12:06:51.918627 110 tensorrt.cc:5405] TRITONBACKEND_ModelInitialize: riva-trt-riva_text_classification_domain-nn-bert-base-uncased (version 1) I1103 12:06:51.919370 110 pipeline_library.cc:25] TRITONBACKEND_ModelInstanceInitialize: riva-punctuation-en-US_0 (device 0) I1103 12:06:51.939514 110 tensorrt.cc:5454] TRITONBACKEND_ModelInstanceInitialize: riva-trt-conformer-en-US-asr-offline-am-streaming-offline_0 (GPU device 0) I1103 12:06:51.939739 110 model_repository_manager.cc:1231] successfully loaded 'riva-punctuation-en-US' version 1 I1103 12:06:51.962775 110 tensorrt.cc:5492] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:51.962792 110 tensorrt.cc:5454] TRITONBACKEND_ModelInstanceInitialize: riva-trt-conformer-en-US-asr-streaming-am-streaming_0 (GPU device 0) I1103 12:06:51.962829 110 tensorrt.cc:5431] TRITONBACKEND_ModelFinalize: delete model state E1103 12:06:51.962844 110 model_repository_manager.cc:1234] failed to load 'riva-trt-conformer-en-US-asr-offline-am-streaming-offline' version 1: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. I1103 12:06:51.985933 110 tensorrt.cc:5492] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:51.985945 110 tensorrt.cc:5454] TRITONBACKEND_ModelInstanceInitialize: riva-trt-hifigan-English-US_0 (GPU device 0) I1103 12:06:51.986002 110 tensorrt.cc:5431] TRITONBACKEND_ModelFinalize: delete model state E1103 12:06:51.986037 110 model_repository_manager.cc:1234] failed to load 'riva-trt-conformer-en-US-asr-streaming-am-streaming' version 1: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. I1103 12:06:52.008673 110 tensorrt.cc:5492] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.008691 110 tensorrt.cc:5454] TRITONBACKEND_ModelInstanceInitialize: riva-trt-riva-punctuation-en-US-nn-bert-base-uncased_0 (GPU device 0) I1103 12:06:52.008733 110 tensorrt.cc:5431] TRITONBACKEND_ModelFinalize: delete model state E1103 12:06:52.008746 110 model_repository_manager.cc:1234] failed to load 'riva-trt-hifigan-English-US' version 1: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. I1103 12:06:52.031567 110 tensorrt.cc:5492] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.031583 110 tensorrt.cc:5454] TRITONBACKEND_ModelInstanceInitialize: riva-trt-riva_intent_weather-nn-bert-base-uncased_0 (GPU device 0) I1103 12:06:52.031617 110 tensorrt.cc:5431] TRITONBACKEND_ModelFinalize: delete model state E1103 12:06:52.031629 110 model_repository_manager.cc:1234] failed to load 'riva-trt-riva-punctuation-en-US-nn-bert-base-uncased' version 1: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. I1103 12:06:52.055005 110 tensorrt.cc:5492] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.055022 110 tensorrt.cc:5454] TRITONBACKEND_ModelInstanceInitialize: riva-trt-riva_ner-nn-bert-base-uncased_0 (GPU device 0) I1103 12:06:52.055060 110 tensorrt.cc:5431] TRITONBACKEND_ModelFinalize: delete model state E1103 12:06:52.055073 110 model_repository_manager.cc:1234] failed to load 'riva-trt-riva_intent_weather-nn-bert-base-uncased' version 1: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. I1103 12:06:52.078318 110 tensorrt.cc:5492] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.078336 110 tensorrt.cc:5454] TRITONBACKEND_ModelInstanceInitialize: riva-trt-riva_qa-nn-bert-base-uncased_0 (GPU device 0) I1103 12:06:52.078373 110 tensorrt.cc:5431] TRITONBACKEND_ModelFinalize: delete model state E1103 12:06:52.078384 110 model_repository_manager.cc:1234] failed to load 'riva-trt-riva_ner-nn-bert-base-uncased' version 1: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. > Riva waiting for Triton server to load all models...retrying in 1 second I1103 12:06:52.101384 110 tensorrt.cc:5492] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.101417 110 tensorrt.cc:5454] TRITONBACKEND_ModelInstanceInitialize: riva-trt-riva_text_classification_domain-nn-bert-base-uncased_0 (GPU device 0) I1103 12:06:52.101455 110 tensorrt.cc:5431] TRITONBACKEND_ModelFinalize: delete model state E1103 12:06:52.101474 110 model_repository_manager.cc:1234] failed to load 'riva-trt-riva_qa-nn-bert-base-uncased' version 1: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. I1103 12:06:52.128577 110 tensorrt.cc:5492] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.128600 110 spectrogram-chunker.cc:274] TRITONBACKEND_ModelInitialize: spectrogram_chunker-English-US (version 1) I1103 12:06:52.128642 110 tensorrt.cc:5431] TRITONBACKEND_ModelFinalize: delete model state I1103 12:06:52.129295 110 backend_model.cc:303] model configuration: { "name": "spectrogram_chunker-English-US", "platform": "", "backend": "riva_tts_chunker", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 8, "input": [ { "name": "SPECTROGRAM", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ 80, -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "IS_LAST_SENTENCE", "data_type": "TYPE_INT32", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "NUM_VALID_FRAMES_IN", "data_type": "TYPE_INT64", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "SENTENCE_NUM", "data_type": "TYPE_INT32", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "DURATIONS", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "PROCESSED_TEXT", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "VOLUME", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "SPECTROGRAM_CHUNK", "data_type": "TYPE_FP32", "dims": [ 80, -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "END_FLAG", "data_type": "TYPE_INT32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "NUM_VALID_SAMPLES_OUT", "data_type": "TYPE_INT32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "SENTENCE_NUM", "data_type": "TYPE_INT32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "DURATIONS", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PROCESSED_TEXT", "data_type": "TYPE_STRING", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "VOLUME", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "priority": "PRIORITY_DEFAULT", "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "oldest": { "max_candidate_sequences": 8, "preferred_batch_size": [ 8 ], "max_queue_delay_microseconds": 1000 }, "max_sequence_idle_microseconds": 60000000, "control_input": [ { "name": "START", "control": [ { "kind": "CONTROL_SEQUENCE_START", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "READY", "control": [ { "kind": "CONTROL_SEQUENCE_READY", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "END", "control": [ { "kind": "CONTROL_SEQUENCE_END", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "CORRID", "control": [ { "kind": "CONTROL_SEQUENCE_CORRID", "int32_false_true": [], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_UINT64" } ] } ], "state": [] }, "instance_group": [ { "name": "spectrogram_chunker-English-US_0", "kind": "KIND_GPU", "count": 1, "gpus": [ 0 ], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "num_mels": { "string_value": "80" }, "supports_volume": { "string_value": "True" }, "num_samples_per_frame": { "string_value": "512" }, "max_execution_batch_size": { "string_value": "8" }, "chunk_length": { "string_value": "80" } }, "model_warmup": [], "model_transaction_policy": { "decoupled": true } } I1103 12:06:52.129394 110 tokenizer_library.cc:18] TRITONBACKEND_ModelInitialize: text_classification_tokenizer-en-US-domain (version 1) W1103 12:06:52.129734 151 parameter_parser.cc:144] Parameter 'unk_token' set but unused. W1103 12:06:52.129765 151 parameter_parser.cc:144] Parameter 'vocab' set but unused. I1103 12:06:52.129818 110 backend_model.cc:303] model configuration: { "name": "text_classification_tokenizer-en-US-domain", "platform": "", "backend": "riva_nlp_tokenizer", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 8, "input": [ { "name": "INPUT_STR__0", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "SEQ__0", "data_type": "TYPE_INT32", "dims": [ 128 ], "label_filename": "", "is_shape_tensor": false }, { "name": "MASK__1", "data_type": "TYPE_INT32", "dims": [ 128 ], "label_filename": "", "is_shape_tensor": false }, { "name": "SEGMENT__4", "data_type": "TYPE_INT32", "dims": [ 128 ], "label_filename": "", "is_shape_tensor": false }, { "name": "SEQ_LEN__2", "data_type": "TYPE_INT64", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "TOK_STR__3", "data_type": "TYPE_STRING", "dims": [ 128 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "priority": "PRIORITY_DEFAULT", "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "instance_group": [ { "name": "text_classification_tokenizer-en-US-domain_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "unk_token": { "string_value": "[UNK]" }, "vocab": { "string_value": "/data/models/text_classification_tokenizer-en-US-domain/1/tokenizer.vocab_file" }, "tokenizer": { "string_value": "wordpiece" }, "bos_token": { "string_value": "[CLS]" }, "to_lower": { "string_value": "true" }, "eos_token": { "string_value": "[SEP]" }, "pad_chars_with_spaces": { "string_value": "False" }, "task": { "string_value": "single_input" } }, "model_warmup": [] } I1103 12:06:52.129903 110 detokenizer_cbe.cc:145] TRITONBACKEND_ModelInitialize: token_classification_detokenizer (version 1) I1103 12:06:52.130230 110 backend_model.cc:303] model configuration: { "name": "token_classification_detokenizer", "platform": "", "backend": "riva_nlp_detokenizer", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 8, "input": [ { "name": "IN_TOKEN_LABELS__0", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "IN_TOKEN_SCORES__1", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "IN_SEQ_LEN__2", "data_type": "TYPE_INT64", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "IN_TOK_STR__3", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "OUT_TOKEN_LABELS__0", "data_type": "TYPE_STRING", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "OUT_TOKEN_SCORES__1", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "OUT_SEQ_LEN__2", "data_type": "TYPE_INT64", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "OUT_TOK_STR__3", "data_type": "TYPE_STRING", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "graph": { "level": 0 }, "priority": "PRIORITY_DEFAULT", "cuda": { "graphs": false, "busy_wait_events": false, "graph_spec": [], "output_copy_stream": true }, "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "instance_group": [ { "name": "token_classification_detokenizer_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": {}, "model_warmup": [], "model_transaction_policy": { "decoupled": false } } I1103 12:06:52.130309 110 sequence_label_cbe.cc:137] TRITONBACKEND_ModelInitialize: token_classification_label_tokens (version 1) I1103 12:06:52.132976 110 backend_model.cc:303] model configuration: { "name": "token_classification_label_tokens", "platform": "", "backend": "riva_nlp_seqlabel", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 8, "input": [ { "name": "TOKEN_LOGIT__1", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1, 13 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "TOKEN_LABELS__0", "data_type": "TYPE_STRING", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "TOKEN_SCORES__1", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "priority": "PRIORITY_DEFAULT", "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "instance_group": [ { "name": "token_classification_label_tokens_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "classes": { "string_value": "/data/models/token_classification_label_tokens/1/label_ids.csv" } }, "model_warmup": [] } I1103 12:06:52.133048 110 tokenizer_library.cc:18] TRITONBACKEND_ModelInitialize: token_classification_tokenizer-en-US (version 1) W1103 12:06:52.133531 154 parameter_parser.cc:144] Parameter 'unk_token' set but unused. W1103 12:06:52.133558 154 parameter_parser.cc:144] Parameter 'vocab' set but unused. I1103 12:06:52.133611 110 backend_model.cc:303] model configuration: { "name": "token_classification_tokenizer-en-US", "platform": "", "backend": "riva_nlp_tokenizer", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 8, "input": [ { "name": "INPUT_STR__0", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "SEQ__0", "data_type": "TYPE_INT32", "dims": [ 128 ], "label_filename": "", "is_shape_tensor": false }, { "name": "MASK__1", "data_type": "TYPE_INT32", "dims": [ 128 ], "label_filename": "", "is_shape_tensor": false }, { "name": "SEGMENT__4", "data_type": "TYPE_INT32", "dims": [ 128 ], "label_filename": "", "is_shape_tensor": false }, { "name": "SEQ_LEN__2", "data_type": "TYPE_INT64", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "TOK_STR__3", "data_type": "TYPE_STRING", "dims": [ 128 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "priority": "PRIORITY_DEFAULT", "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "instance_group": [ { "name": "token_classification_tokenizer-en-US_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "to_lower": { "string_value": "true" }, "eos_token": { "string_value": "[SEP]" }, "pad_chars_with_spaces": { "string_value": "False" }, "task": { "string_value": "single_input" }, "unk_token": { "string_value": "[UNK]" }, "tokenizer": { "string_value": "wordpiece" }, "vocab": { "string_value": "/data/models/token_classification_tokenizer-en-US/1/tokenizer.vocab_file" }, "bos_token": { "string_value": "[CLS]" } }, "model_warmup": [] } I1103 12:06:52.136572 110 spectrogram-chunker.cc:276] TRITONBACKEND_ModelInstanceInitialize: spectrogram_chunker-English-US_0 (device 0) I1103 12:06:52.136643 110 tokenizer_library.cc:21] TRITONBACKEND_ModelInstanceInitialize: text_classification_tokenizer-en-US-domain_0 (device 0) I1103 12:06:52.137642 110 model_repository_manager.cc:1231] successfully loaded 'spectrogram_chunker-English-US' version 1 I1103 12:06:52.152629 110 detokenizer_cbe.cc:147] TRITONBACKEND_ModelInstanceInitialize: token_classification_detokenizer_0 (device 0) I1103 12:06:52.152705 110 sequence_label_cbe.cc:139] TRITONBACKEND_ModelInstanceInitialize: token_classification_label_tokens_0 (device 0) I1103 12:06:52.152758 110 model_repository_manager.cc:1231] successfully loaded 'text_classification_tokenizer-en-US-domain' version 1 I1103 12:06:52.152785 110 tokenizer_library.cc:21] TRITONBACKEND_ModelInstanceInitialize: token_classification_tokenizer-en-US_0 (device 0) I1103 12:06:52.153313 110 model_repository_manager.cc:1231] successfully loaded 'token_classification_detokenizer' version 1 I1103 12:06:52.153811 110 model_repository_manager.cc:1231] successfully loaded 'token_classification_label_tokens' version 1 I1103 12:06:52.167590 110 tts-postprocessor.cc:300] TRITONBACKEND_ModelInitialize: tts_postprocessor-English-US (version 1) I1103 12:06:52.167782 110 model_repository_manager.cc:1231] successfully loaded 'token_classification_tokenizer-en-US' version 1 I1103 12:06:52.168202 110 backend_model.cc:303] model configuration: { "name": "tts_postprocessor-English-US", "platform": "", "backend": "riva_tts_postprocessor", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 8, "input": [ { "name": "INPUT", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ 1, -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "NUM_VALID_SAMPLES", "data_type": "TYPE_INT32", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "Prosody_volume", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "OUTPUT", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "priority": "PRIORITY_DEFAULT", "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "oldest": { "max_candidate_sequences": 8, "preferred_batch_size": [ 8 ], "max_queue_delay_microseconds": 100 }, "max_sequence_idle_microseconds": 60000000, "control_input": [ { "name": "START", "control": [ { "kind": "CONTROL_SEQUENCE_START", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "READY", "control": [ { "kind": "CONTROL_SEQUENCE_READY", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "END", "control": [ { "kind": "CONTROL_SEQUENCE_END", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "CORRID", "control": [ { "kind": "CONTROL_SEQUENCE_CORRID", "int32_false_true": [], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_UINT64" } ] } ], "state": [] }, "instance_group": [ { "name": "tts_postprocessor-English-US_0", "kind": "KIND_GPU", "count": 1, "gpus": [ 0 ], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "hop_length": { "string_value": "256" }, "filter_length": { "string_value": "1024" }, "supports_volume": { "string_value": "True" }, "num_samples_per_frame": { "string_value": "512" }, "use_denoiser": { "string_value": "False" }, "chunk_num_samples": { "string_value": "40960" }, "fade_length": { "string_value": "256" }, "max_chunk_size": { "string_value": "131072" }, "max_execution_batch_size": { "string_value": "8" } }, "model_warmup": [], "model_transaction_policy": { "decoupled": false } } E1103 12:06:52.168814 110 model_repository_manager.cc:1234] failed to load 'riva-trt-riva_text_classification_domain-nn-bert-base-uncased' version 1: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain.I1103 12:06:52.168833 110 tts-preprocessor.cc:280] TRITONBACKEND_ModelInitialize: tts_preprocessor-English-US (version 1) W1103 12:06:52.169417 110 tts-preprocessor.cc:241] Parameter abbreviation_path is deprecated WARNING: Logging before InitGoogleLogging() is written to STDERR I1103 12:06:52.171298 159 preprocessor.cc:232] TTS character mapping loaded from /data/models/tts_preprocessor-English-US/1/mapping.txt I1103 12:06:52.319345 159 preprocessor.cc:269] TTS phonetic mapping loaded from /data/models/tts_preprocessor-English-US/1/18d619cc5aaf458bbda98fe02588e6a1_cmudict-0.7b_nv22.01 I1103 12:06:52.321259 159 preprocessor.cc:282] Abbreviation mapping loaded from /data/models/tts_preprocessor-English-US/1/abbr.txt I1103 12:06:52.517040 159 preprocessor.cc:292] TTS normalizer loaded from /data/models/tts_preprocessor-English-US/1/ I1103 12:06:52.517161 110 backend_model.cc:303] model configuration: { "name": "tts_preprocessor-English-US", "platform": "", "backend": "riva_tts_preprocessor", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 8, "input": [ { "name": "input_string", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "speaker", "data_type": "TYPE_INT64", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "output", "data_type": "TYPE_INT64", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "output_mask", "data_type": "TYPE_FP32", "dims": [ 1, 400, 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "output_length", "data_type": "TYPE_INT32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "is_last_sentence", "data_type": "TYPE_INT32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "output_string", "data_type": "TYPE_STRING", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "sentence_num", "data_type": "TYPE_INT32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "pitch", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "duration", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "volume", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "speaker", "data_type": "TYPE_INT64", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "graph": { "level": 0 }, "priority": "PRIORITY_DEFAULT", "cuda": { "graphs": false, "busy_wait_events": false, "graph_spec": [], "output_copy_stream": true }, "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "oldest": { "max_candidate_sequences": 8, "preferred_batch_size": [ 8 ], "max_queue_delay_microseconds": 100 }, "max_sequence_idle_microseconds": 60000000, "control_input": [ { "name": "START", "control": [ { "kind": "CONTROL_SEQUENCE_START", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "READY", "control": [ { "kind": "CONTROL_SEQUENCE_READY", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "END", "control": [ { "kind": "CONTROL_SEQUENCE_END", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "CORRID", "control": [ { "kind": "CONTROL_SEQUENCE_CORRID", "int32_false_true": [], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_UINT64" } ] } ], "state": [] }, "instance_group": [ { "name": "tts_preprocessor-English-US_0", "kind": "KIND_GPU", "count": 1, "gpus": [ 0 ], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "max_sequence_length": { "string_value": "400" }, "enable_emphasis_tag": { "string_value": "True" }, "pad_with_space": { "string_value": "True" }, "start_of_emphasis_token": { "string_value": "[" }, "dictionary_path": { "string_value": "/data/models/tts_preprocessor-English-US/1/18d619cc5aaf458bbda98fe02588e6a1_cmudict-0.7b_nv22.01" }, "supports_ragged_batches": { "string_value": "True" }, "upper_case_chars": { "string_value": "False" }, "g2p_ignore_ambiguous": { "string_value": "True" }, "mapping_path": { "string_value": "/data/models/tts_preprocessor-English-US/1/mapping.txt" }, "max_input_length": { "string_value": "2000" }, "end_of_emphasis_token": { "string_value": "]" }, "abbreviations_path": { "string_value": "/data/models/tts_preprocessor-English-US/1/abbr.txt" }, "norm_proto_path": { "string_value": "/data/models/tts_preprocessor-English-US/1/" }, "language": { "string_value": "en-US" }, "phone_set": { "string_value": "arpabet" } }, "model_warmup": [], "model_transaction_policy": { "decoupled": true } } I1103 12:06:52.517247 110 tts-postprocessor.cc:302] TRITONBACKEND_ModelInstanceInitialize: tts_postprocessor-English-US_0 (device 0) I1103 12:06:52.602757 110 tts-preprocessor.cc:282] TRITONBACKEND_ModelInstanceInitialize: tts_preprocessor-English-US_0 (device 0) I1103 12:06:52.604322 110 model_repository_manager.cc:1231] successfully loaded 'tts_postprocessor-English-US' version 1 I1103 12:06:52.604664 110 model_repository_manager.cc:1231] successfully loaded 'tts_preprocessor-English-US' version 1 E1103 12:06:52.604788 110 model_repository_manager.cc:1420] Invalid argument: ensemble 'conformer-en-US-asr-offline' depends on 'riva-trt-conformer-en-US-asr-offline-am-streaming-offline' which has no loaded version E1103 12:06:52.604828 110 model_repository_manager.cc:1420] Invalid argument: ensemble 'conformer-en-US-asr-streaming' depends on 'riva-trt-conformer-en-US-asr-streaming-am-streaming' which has no loaded version E1103 12:06:52.604835 110 model_repository_manager.cc:1420] Invalid argument: ensemble 'fastpitch_hifigan_ensemble-English-US' depends on 'riva-trt-hifigan-English-US' which has no loaded version E1103 12:06:52.604839 110 model_repository_manager.cc:1420] Invalid argument: ensemble 'riva_intent_weather' depends on 'riva-trt-riva_intent_weather-nn-bert-base-uncased' which has no loaded version E1103 12:06:52.604842 110 model_repository_manager.cc:1420] Invalid argument: ensemble 'riva_ner' depends on 'riva-trt-riva_ner-nn-bert-base-uncased' which has no loaded version E1103 12:06:52.604846 110 model_repository_manager.cc:1420] Invalid argument: ensemble 'riva_qa' depends on 'riva-trt-riva_qa-nn-bert-base-uncased' which has no loaded version E1103 12:06:52.604849 110 model_repository_manager.cc:1420] Invalid argument: ensemble 'riva_text_classification_domain' depends on 'riva-trt-riva_text_classification_domain-nn-bert-base-uncased' which has no loaded version I1103 12:06:52.604906 110 server.cc:549] +------------------+------+ | Repository Agent | Path | +------------------+------+ +------------------+------+ I1103 12:06:52.605023 110 server.cc:576] +------------------------+---------------------------------------------------------------------------------------+--------+ | Backend | Path | Config | +------------------------+---------------------------------------------------------------------------------------+--------+ | riva_nlp_qa | /opt/tritonserver/backends/riva_nlp_qa/libtriton_riva_nlp_qa.so | {} | | riva_nlp_tokenizer | /opt/tritonserver/backends/riva_nlp_tokenizer/libtriton_riva_nlp_tokenizer.so | {} | | riva_tts_chunker | /opt/tritonserver/backends/riva_tts_chunker/libtriton_riva_tts_chunker.so | {} | | riva_nlp_seqlabel | /opt/tritonserver/backends/riva_nlp_seqlabel/libtriton_riva_nlp_seqlabel.so | {} | | riva_asr_endpointing | /opt/tritonserver/backends/riva_asr_endpointing/libtriton_riva_asr_endpointing.so | {} | | riva_nlp_detokenizer | /opt/tritonserver/backends/riva_nlp_detokenizer/libtriton_riva_nlp_detokenizer.so | {} | | riva_tts_postprocessor | /opt/tritonserver/backends/riva_tts_postprocessor/libtriton_riva_tts_postprocessor.so | {} | | riva_asr_decoder | /opt/tritonserver/backends/riva_asr_decoder/libtriton_riva_asr_decoder.so | {} | | pytorch | /opt/tritonserver/backends/pytorch/libtriton_pytorch.so | {} | | onnxruntime | /opt/tritonserver/backends/onnxruntime/libtriton_onnxruntime.so | {} | | riva_nlp_pipeline | /opt/tritonserver/backends/riva_nlp_pipeline/libtriton_riva_nlp_pipeline.so | {} | | riva_asr_features | /opt/tritonserver/backends/riva_asr_features/libtriton_riva_asr_features.so | {} | | riva_tts_preprocessor | /opt/tritonserver/backends/riva_tts_preprocessor/libtriton_riva_tts_preprocessor.so | {} | +------------------------+---------------------------------------------------------------------------------------+--------+ I1103 12:06:52.605202 110 server.cc:619] +-----------------------------------------------------------------+---------+--------------------------------------------------------------------------------------------------------------+ | Model | Version | Status | +-----------------------------------------------------------------+---------+--------------------------------------------------------------------------------------------------------------+ | conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline | 1 | READY | | conformer-en-US-asr-offline-endpointing-streaming-offline | 1 | READY | | conformer-en-US-asr-offline-feature-extractor-streaming-offline | 1 | READY | | conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming | 1 | READY | | conformer-en-US-asr-streaming-endpointing-streaming | 1 | READY | | conformer-en-US-asr-streaming-feature-extractor-streaming | 1 | READY | | intent_slot_detokenizer | 1 | READY | | intent_slot_label_tokens_weather | 1 | READY | | intent_slot_tokenizer-en-US-weather | 1 | READY | | qa_qa_postprocessor | 1 | READY | | qa_tokenizer-en-US | 1 | READY | | riva-onnx-fastpitch_encoder-English-US | 1 | UNAVAILABLE: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. | | riva-punctuation-en-US | 1 | READY | | riva-trt-conformer-en-US-asr-offline-am-streaming-offline | 1 | UNAVAILABLE: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. | | riva-trt-conformer-en-US-asr-streaming-am-streaming | 1 | UNAVAILABLE: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. | | riva-trt-hifigan-English-US | 1 | UNAVAILABLE: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. | | riva-trt-riva-punctuation-en-US-nn-bert-base-uncased | 1 | UNAVAILABLE: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. | | riva-trt-riva_intent_weather-nn-bert-base-uncased | 1 | UNAVAILABLE: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. | | riva-trt-riva_ner-nn-bert-base-uncased | 1 | UNAVAILABLE: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. | | riva-trt-riva_qa-nn-bert-base-uncased | 1 | UNAVAILABLE: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. | | riva-trt-riva_text_classification_domain-nn-bert-base-uncased | 1 | UNAVAILABLE: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. | | spectrogram_chunker-English-US | 1 | READY | | text_classification_tokenizer-en-US-domain | 1 | READY | | token_classification_detokenizer | 1 | READY | | token_classification_label_tokens | 1 | READY | | token_classification_tokenizer-en-US | 1 | READY | | tts_postprocessor-English-US | 1 | READY | | tts_preprocessor-English-US | 1 | READY | +-----------------------------------------------------------------+---------+--------------------------------------------------------------------------------------------------------------+ I1103 12:06:52.638111 110 metrics.cc:650] Collecting metrics for GPU 0: NVIDIA GeForce RTX 3090 I1103 12:06:52.638823 110 tritonserver.cc:2123] +----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Option | Value | +----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | server_id | triton | | server_version | 2.21.0 | | server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace | | model_repository_path[0] | /data/models | | model_control_mode | MODE_NONE | | strict_model_config | 1 | | rate_limit | OFF | | pinned_memory_pool_byte_size | 268435456 | | cuda_memory_pool_byte_size{0} | 1000000000 | | response_cache_byte_size | 0 | | min_supported_compute_capability | 6.0 | | strict_readiness | 1 | | exit_timeout | 30 | +----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ I1103 12:06:52.638853 110 server.cc:250] Waiting for in-flight requests to complete. I1103 12:06:52.638880 110 server.cc:266] Timeout 30: Found 0 model versions that have in-flight inferences I1103 12:06:52.638885 110 model_repository_manager.cc:1109] unloading: tts_preprocessor-English-US:1 I1103 12:06:52.638948 110 model_repository_manager.cc:1109] unloading: tts_postprocessor-English-US:1 I1103 12:06:52.639046 110 model_repository_manager.cc:1109] unloading: token_classification_detokenizer:1 I1103 12:06:52.639146 110 model_repository_manager.cc:1109] unloading: text_classification_tokenizer-en-US-domain:1 I1103 12:06:52.639191 110 tts-preprocessor.cc:284] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.639259 110 tts-preprocessor.cc:281] TRITONBACKEND_ModelFinalize: delete model state I1103 12:06:52.639447 110 tts-postprocessor.cc:304] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.639495 110 model_repository_manager.cc:1109] unloading: qa_qa_postprocessor:1 I1103 12:06:52.639516 110 detokenizer_cbe.cc:149] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.639565 110 detokenizer_cbe.cc:146] TRITONBACKEND_ModelFinalize: delete model state I1103 12:06:52.639640 110 model_repository_manager.cc:1214] successfully unloaded 'token_classification_detokenizer' version 1 I1103 12:06:52.639758 110 tokenizer_library.cc:25] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.639879 110 model_repository_manager.cc:1109] unloading: intent_slot_detokenizer:1 I1103 12:06:52.639971 110 model_repository_manager.cc:1109] unloading: conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming:1 I1103 12:06:52.640059 110 qa_postprocessor_cbe.cc:128] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.640137 110 qa_postprocessor_cbe.cc:125] TRITONBACKEND_ModelFinalize: delete model state I1103 12:06:52.640318 110 detokenizer_cbe.cc:149] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.640387 110 detokenizer_cbe.cc:146] TRITONBACKEND_ModelFinalize: delete model state I1103 12:06:52.640401 110 model_repository_manager.cc:1109] unloading: spectrogram_chunker-English-US:1 I1103 12:06:52.640705 110 ctc-decoder-library.cc:25] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.640846 110 model_repository_manager.cc:1109] unloading: conformer-en-US-asr-streaming-feature-extractor-streaming:1 I1103 12:06:52.641214 110 model_repository_manager.cc:1214] successfully unloaded 'qa_qa_postprocessor' version 1 I1103 12:06:52.641508 110 spectrogram-chunker.cc:279] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.641617 110 spectrogram-chunker.cc:275] TRITONBACKEND_ModelFinalize: delete model state I1103 12:06:52.641738 110 model_repository_manager.cc:1109] unloading: riva-punctuation-en-US:1 I1103 12:06:52.641829 110 model_repository_manager.cc:1214] successfully unloaded 'intent_slot_detokenizer' version 1 I1103 12:06:52.641901 110 feature-extractor.cc:404] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.641833 110 model_repository_manager.cc:1109] unloading: conformer-en-US-asr-offline-feature-extractor-streaming-offline:1 I1103 12:06:52.642022 110 pipeline_library.cc:28] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.642139 110 model_repository_manager.cc:1109] unloading: conformer-en-US-asr-offline-endpointing-streaming-offline:1 I1103 12:06:52.642447 110 feature-extractor.cc:404] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.642610 110 model_repository_manager.cc:1109] unloading: intent_slot_label_tokens_weather:1 I1103 12:06:52.642699 110 model_repository_manager.cc:1109] unloading: conformer-en-US-asr-streaming-endpointing-streaming:1 I1103 12:06:52.642877 110 endpointing_library.cc:26] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.643008 110 sequence_label_cbe.cc:141] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.643086 110 sequence_label_cbe.cc:138] TRITONBACKEND_ModelFinalize: delete model state I1103 12:06:52.643160 110 model_repository_manager.cc:1214] successfully unloaded 'intent_slot_label_tokens_weather' version 1 I1103 12:06:52.643619 110 model_repository_manager.cc:1109] unloading: conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline:1 I1103 12:06:52.643681 110 tokenizer_library.cc:20] TRITONBACKEND_ModelFinalize: delete model state I1103 12:06:52.643841 110 endpointing_library.cc:26] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.643847 110 model_repository_manager.cc:1214] successfully unloaded 'spectrogram_chunker-English-US' version 1 I1103 12:06:52.644683 110 model_repository_manager.cc:1214] successfully unloaded 'text_classification_tokenizer-en-US-domain' version 1 I1103 12:06:52.645927 110 model_repository_manager.cc:1109] unloading: qa_tokenizer-en-US:1 I1103 12:06:52.646042 110 model_repository_manager.cc:1109] unloading: intent_slot_tokenizer-en-US-weather:1 I1103 12:06:52.646153 110 ctc-decoder-library.cc:25] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.646616 110 tokenizer_library.cc:25] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.646669 110 model_repository_manager.cc:1109] unloading: token_classification_label_tokens:1 I1103 12:06:52.646788 110 pipeline_library.cc:24] TRITONBACKEND_ModelFinalize: delete model stateI1103 12:06:52.646851 110 tokenizer_library.cc:25] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.647268 110 model_repository_manager.cc:1109] unloading: token_classification_tokenizer-en-US:1 I1103 12:06:52.647832 110 sequence_label_cbe.cc:141] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.647915 110 server.cc:281] All models are stopped, unloading models I1103 12:06:52.647938 110 sequence_label_cbe.cc:138] TRITONBACKEND_ModelFinalize: delete model state I1103 12:06:52.647972 110 server.cc:288] Timeout 30: Found 13 live models and 0 in-flight non-inference requests I1103 12:06:52.648087 110 tokenizer_library.cc:25] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1103 12:06:52.648849 110 model_repository_manager.cc:1214] successfully unloaded 'token_classification_label_tokens' version 1 I1103 12:06:52.650750 110 tokenizer_library.cc:20] TRITONBACKEND_ModelFinalize: delete model state I1103 12:06:52.651354 110 endpointing_library.cc:21] TRITONBACKEND_ModelFinalize: delete model state I1103 12:06:52.651515 110 model_repository_manager.cc:1214] successfully unloaded 'intent_slot_tokenizer-en-US-weather' version 1 I1103 12:06:52.652168 110 model_repository_manager.cc:1214] successfully unloaded 'conformer-en-US-asr-offline-endpointing-streaming-offline' version 1 I1103 12:06:52.652537 110 endpointing_library.cc:21] TRITONBACKEND_ModelFinalize: delete model state I1103 12:06:52.652526 110 tokenizer_library.cc:20] TRITONBACKEND_ModelFinalize: delete model state I1103 12:06:52.653564 110 tokenizer_library.cc:20] TRITONBACKEND_ModelFinalize: delete model state I1103 12:06:52.653844 110 model_repository_manager.cc:1214] successfully unloaded 'riva-punctuation-en-US' version 1 I1103 12:06:52.654806 110 model_repository_manager.cc:1214] successfully unloaded 'qa_tokenizer-en-US' version 1 I1103 12:06:52.654921 110 tts-postprocessor.cc:301] TRITONBACKEND_ModelFinalize: delete model state I1103 12:06:52.656702 110 model_repository_manager.cc:1214] successfully unloaded 'conformer-en-US-asr-streaming-endpointing-streaming' version 1 I1103 12:06:52.657775 110 model_repository_manager.cc:1214] successfully unloaded 'token_classification_tokenizer-en-US' version 1 I1103 12:06:52.657853 110 model_repository_manager.cc:1214] successfully unloaded 'tts_postprocessor-English-US' version 1 I1103 12:06:52.718361 110 feature-extractor.cc:401] TRITONBACKEND_ModelFinalize: delete model state /opt/riva/bin/start-riva: line 4: 110 Segmentation fault ${CUSTOM_TRITON_ENV} tritonserver --log-verbose=0 --strict-model-config=true $model_repos --cuda-memory-pool-byte-size=0:1000000000 > Triton server died before reaching ready state. Terminating Riva startup.