/opt/riva/bin/start-riva: line 10: curl: command not found /opt/riva/bin/start-riva: line 11: [: -ne: unary operator expected > Triton server is ready... I0312 06:02:18.154769 10 riva_server.cc:122] Using SSL Credentials E0312 06:02:18.197974 10 model_registry.cc:246] error: unable to get server status: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8001: Failed to connect to remote host: Connection refused Warning: '--strict-model-config' has been deprecated! Please use '--disable-auto-complete-config' instead. I0312 06:02:18.427252 8 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x205227000' with size 268435456 I0312 06:02:18.427563 8 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 1000000000 I0312 06:02:18.682314 8 model_lifecycle.cc:459] loading: conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming:1 I0312 06:02:18.682397 8 model_lifecycle.cc:459] loading: conformer-en-US-asr-streaming-endpointing-streaming:1 I0312 06:02:18.682445 8 model_lifecycle.cc:459] loading: conformer-en-US-asr-streaming-feature-extractor-streaming:1 I0312 06:02:18.682638 8 model_lifecycle.cc:459] loading: intent_slot_detokenizer:1 I0312 06:02:18.682675 8 model_lifecycle.cc:459] loading: intent_slot_label_tokens_misty:1 I0312 06:02:18.682724 8 model_lifecycle.cc:459] loading: intent_slot_tokenizer-en-US-misty:1 I0312 06:02:18.682788 8 model_lifecycle.cc:459] loading: riva-onnx-fastpitch_encoder-English-US:1 I0312 06:02:18.682838 8 model_lifecycle.cc:459] loading: riva-punctuation-en-US:1 I0312 06:02:18.682875 8 model_lifecycle.cc:459] loading: riva-trt-conformer-en-US-asr-streaming-am-streaming:1 I0312 06:02:18.683028 8 model_lifecycle.cc:459] loading: riva-trt-hifigan-English-US:1 I0312 06:02:18.683068 8 model_lifecycle.cc:459] loading: riva-trt-riva-punctuation-en-US-nn-bert-base-uncased:1 I0312 06:02:18.683099 8 model_lifecycle.cc:459] loading: riva-trt-riva_intent_misty-nn-bert-base-uncased:1 I0312 06:02:18.683135 8 model_lifecycle.cc:459] loading: spectrogram_chunker-English-US:1 I0312 06:02:18.683177 8 model_lifecycle.cc:459] loading: tts_postprocessor-English-US:1 I0312 06:02:18.683224 8 model_lifecycle.cc:459] loading: tts_preprocessor-English-US:1 E0312 06:02:18.798242 8 model_lifecycle.cc:596] failed to load 'riva-trt-conformer-en-US-asr-streaming-am-streaming' version 1: Not found: unable to load shared library: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8: undefined symbol: _ZN5nvdla8IProfile17setUseSoftMaxOptzEb E0312 06:02:18.812434 8 model_lifecycle.cc:596] failed to load 'riva-trt-riva_intent_misty-nn-bert-base-uncased' version 1: Not found: unable to load shared library: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8: undefined symbol: _ZN5nvdla8IProfile17setUseSoftMaxOptzEb E0312 06:02:18.826404 8 model_lifecycle.cc:596] failed to load 'riva-trt-riva-punctuation-en-US-nn-bert-base-uncased' version 1: Not found: unable to load shared library: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8: undefined symbol: _ZN5nvdla8IProfile17setUseSoftMaxOptzEb E0312 06:02:18.843128 8 model_lifecycle.cc:596] failed to load 'riva-trt-hifigan-English-US' version 1: Not found: unable to load shared library: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8: undefined symbol: _ZN5nvdla8IProfile17setUseSoftMaxOptzEb I0312 06:02:20.256778 8 feature-extractor.cc:400] TRITONBACKEND_ModelInitialize: conformer-en-US-asr-streaming-feature-extractor-streaming (version 1) E0312 06:02:28.198421 10 model_registry.cc:246] error: unable to get server status: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8001: Failed to connect to remote host: Connection refused E0312 06:02:38.198807 10 model_registry.cc:246] error: unable to get server status: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8001: Failed to connect to remote host: Connection refused E0312 06:02:48.199131 10 model_registry.cc:246] error: unable to get server status: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8001: Failed to connect to remote host: Connection refused E0312 06:02:58.199476 10 model_registry.cc:246] error: unable to get server status: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8001: Failed to connect to remote host: Connection refused I0312 06:03:05.862127 8 backend_model.cc:303] model configuration: { "name": "conformer-en-US-asr-streaming-feature-extractor-streaming", "platform": "", "backend": "riva_asr_features", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 1024, "input": [ { "name": "AUDIO_SIGNAL", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "SAMPLE_RATE", "data_type": "TYPE_UINT32", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "AUDIO_FEATURES", "data_type": "TYPE_FP32", "dims": [ 80, -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "AUDIO_PROCESSED", "data_type": "TYPE_FP32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "AUDIO_FEATURES_LENGTH", "data_type": "TYPE_INT32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "graph": { "level": 0 }, "priority": "PRIORITY_DEFAULT", "cuda": { "graphs": false, "busy_wait_events": false, "graph_spec": [], "output_copy_stream": true }, "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "oldest": { "max_candidate_sequences": 1024, "preferred_batch_size": [ 256, 512 ], "max_queue_delay_microseconds": 1000 }, "max_sequence_idle_microseconds": 60000000, "control_input": [ { "name": "START", "control": [ { "kind": "CONTROL_SEQUENCE_START", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "READY", "control": [ { "kind": "CONTROL_SEQUENCE_READY", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "END", "control": [ { "kind": "CONTROL_SEQUENCE_END", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "CORRID", "control": [ { "kind": "CONTROL_SEQUENCE_CORRID", "int32_false_true": [], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_UINT64" } ] } ], "state": [] }, "instance_group": [ { "name": "conformer-en-US-asr-streaming-feature-extractor-streaming_0", "kind": "KIND_GPU", "count": 1, "gpus": [ 0 ], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "precalc_norm_params": { "string_value": "False" }, "stddev_floor": { "string_value": "1e-05" }, "right_padding_size": { "string_value": "1.92" }, "chunk_size": { "string_value": "0.16" }, "num_features": { "string_value": "80" }, "gain": { "string_value": "1.0" }, "streaming": { "string_value": "True" }, "norm_per_feature": { "string_value": "True" }, "apply_normalization": { "string_value": "True" }, "mean": { "string_value": "-11.4412, -9.9334, -9.1292, -9.0365, -9.2804, -9.5643, -9.7342, -9.6925, -9.6333, -9.2808, -9.1887, -9.1422, -9.1397, -9.2028, -9.2749, -9.4776, -9.9185, -10.1557, -10.3800, -10.5067, -10.3190, -10.4728, -10.5529, -10.6402, -10.6440, -10.5113, -10.7395, -10.7870, -10.6074, -10.5033, -10.8278, -10.6384, -10.8481, -10.6875, -10.5454, -10.4747, -10.5165, -10.4930, -10.3413, -10.3472, -10.3735, -10.6830, -10.8813, -10.6338, -10.3856, -10.7727, -10.8957, -10.8068, -10.7373, -10.6108, -10.3405, -10.2889, -10.3922, -10.4946, -10.3367, -10.4164, -10.9949, -10.7196, -10.3971, -10.1734, -9.9257, -9.6557, -9.1761, -9.6653, -9.7876, -9.7230, -9.7792, -9.7056, -9.2702, -9.4650, -9.2755, -9.1369, -9.1174, -8.9197, -8.5394, -8.2614, -8.1353, -8.1422, -8.3430, -8.6655" }, "window_size": { "string_value": "0.025" }, "sample_rate": { "string_value": "16000" }, "window_stride": { "string_value": "0.01" }, "transpose": { "string_value": "False" }, "max_execution_batch_size": { "string_value": "1" }, "left_padding_size": { "string_value": "1.92" }, "use_utterance_norm_params": { "string_value": "False" }, "stddev": { "string_value": "2.2668, 3.1642, 3.7079, 3.7642, 3.5349, 3.5901, 3.7640, 3.8424, 4.0145, 4.1475, 4.0457, 3.9048, 3.7709, 3.6117, 3.3188, 3.1489, 3.0615, 3.0362, 2.9929, 3.0500, 3.0341, 3.0484, 3.0103, 2.9474, 2.9128, 2.8669, 2.8332, 2.9411, 3.0378, 3.0712, 3.0190, 2.9992, 3.0124, 3.0024, 3.0275, 3.0870, 3.0656, 3.0142, 3.0493, 3.1373, 3.1135, 3.0675, 2.8828, 2.7018, 2.6296, 2.8826, 2.9325, 2.9288, 2.9271, 2.9890, 3.0137, 2.9855, 3.0839, 2.9319, 2.3512, 2.3795, 2.6191, 2.7555, 2.9326, 2.9931, 3.1543, 3.0855, 2.6820, 3.0566, 3.1272, 3.1663, 3.1836, 3.0018, 2.9089, 3.1727, 3.1626, 3.1086, 2.9804, 3.1107, 3.2998, 3.3697, 3.3716, 3.2487, 3.1597, 3.1181" }, "precalc_norm_time_steps": { "string_value": "0" }, "dither": { "string_value": "0.0" } }, "model_warmup": [], "model_transaction_policy": { "decoupled": false } } I0312 06:03:05.875113 8 pipeline_library.cc:22] TRITONBACKEND_ModelInitialize: riva-punctuation-en-US (version 1) WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 06:03:05.875913 39 parameter_parser.cc:144] Parameter 'attn_mask_tensor_name' set but unused. W0312 06:03:05.875933 39 parameter_parser.cc:144] Parameter 'bos_token' set but unused. W0312 06:03:05.875936 39 parameter_parser.cc:144] Parameter 'capit_logits_tensor_name' set but unused. W0312 06:03:05.875938 39 parameter_parser.cc:144] Parameter 'capitalization_mapping_path' set but unused. W0312 06:03:05.875941 39 parameter_parser.cc:144] Parameter 'delimiter' set but unused. W0312 06:03:05.875942 39 parameter_parser.cc:144] Parameter 'eos_token' set but unused. W0312 06:03:05.875945 39 parameter_parser.cc:144] Parameter 'input_ids_tensor_name' set but unused. W0312 06:03:05.875947 39 parameter_parser.cc:144] Parameter 'language_code' set but unused. W0312 06:03:05.875949 39 parameter_parser.cc:144] Parameter 'model_api' set but unused. W0312 06:03:05.875952 39 parameter_parser.cc:144] Parameter 'model_family' set but unused. W0312 06:03:05.875953 39 parameter_parser.cc:144] Parameter 'pad_chars_with_spaces' set but unused. W0312 06:03:05.875957 39 parameter_parser.cc:144] Parameter 'preserve_accents' set but unused. W0312 06:03:05.875958 39 parameter_parser.cc:144] Parameter 'punct_logits_tensor_name' set but unused. W0312 06:03:05.875962 39 parameter_parser.cc:144] Parameter 'punctuation_mapping_path' set but unused. W0312 06:03:05.875963 39 parameter_parser.cc:144] Parameter 'remove_spaces' set but unused. W0312 06:03:05.875965 39 parameter_parser.cc:144] Parameter 'to_lower' set but unused. W0312 06:03:05.875967 39 parameter_parser.cc:144] Parameter 'token_type_tensor_name' set but unused. W0312 06:03:05.875970 39 parameter_parser.cc:144] Parameter 'tokenizer_to_lower' set but unused. W0312 06:03:05.875972 39 parameter_parser.cc:144] Parameter 'unicode_normalize' set but unused. W0312 06:03:05.875974 39 parameter_parser.cc:144] Parameter 'unk_token' set but unused. W0312 06:03:05.875977 39 parameter_parser.cc:144] Parameter 'use_int64_nn_inputs' set but unused. W0312 06:03:05.875979 39 parameter_parser.cc:144] Parameter 'vocab' set but unused. W0312 06:03:05.876041 39 parameter_parser.cc:144] Parameter 'model_api' set but unused. W0312 06:03:05.876049 39 parameter_parser.cc:144] Parameter 'model_family' set but unused. I0312 06:03:05.876128 8 backend_model.cc:303] model configuration: { "name": "riva-punctuation-en-US", "platform": "", "backend": "riva_nlp_pipeline", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 1, "input": [ { "name": "PIPELINE_INPUT", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "PIPELINE_OUTPUT", "data_type": "TYPE_STRING", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "priority": "PRIORITY_DEFAULT", "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "instance_group": [ { "name": "riva-punctuation-en-US_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "eos_token": { "string_value": "[SEP]" }, "punctuation_mapping_path": { "string_value": "/data/models/riva-punctuation-en-US/1/15eace99434b4c87ba28cbd294b48f43_punct_label_ids.csv" }, "delimiter": { "string_value": " " }, "to_lower": { "string_value": "true" }, "model_api": { "string_value": "/nvidia.riva.nlp.RivaLanguageUnderstanding/PunctuateText" }, "unk_token": { "string_value": "[UNK]" }, "capitalization_mapping_path": { "string_value": "/data/models/riva-punctuation-en-US/1/fb06800834e74de1bdc32db51da9619c_capit_label_ids.csv" }, "capit_logits_tensor_name": { "string_value": "capit_logits" }, "input_ids_tensor_name": { "string_value": "input_ids" }, "model_family": { "string_value": "riva" }, "punct_logits_tensor_name": { "string_value": "punct_logits" }, "model_name": { "string_value": "riva-trt-riva-punctuation-en-US-nn-bert-base-uncased" }, "bos_token": { "string_value": "[CLS]" }, "load_model": { "string_value": "false" }, "unicode_normalize": { "string_value": "False" }, "preserve_accents": { "string_value": "false" }, "attn_mask_tensor_name": { "string_value": "attention_mask" }, "language_code": { "string_value": "en-US" }, "pipeline_type": { "string_value": "punctuation" }, "token_type_tensor_name": { "string_value": "token_type_ids" }, "vocab": { "string_value": "/data/models/riva-punctuation-en-US/1/e222f352288a423da453a79b96cc7b75_vocab.txt" }, "use_int64_nn_inputs": { "string_value": "False" }, "remove_spaces": { "string_value": "False" }, "tokenizer": { "string_value": "wordpiece" }, "tokenizer_to_lower": { "string_value": "true" }, "pad_chars_with_spaces": { "string_value": "False" } }, "model_warmup": [] } I0312 06:03:05.876174 8 feature-extractor.cc:402] TRITONBACKEND_ModelInstanceInitialize: conformer-en-US-asr-streaming-feature-extractor-streaming_0 (device 0) E0312 06:03:08.199925 10 model_registry.cc:246] error: unable to get server status: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8001: Failed to connect to remote host: Connection refused I0312 06:03:09.605073 8 model_lifecycle.cc:693] successfully loaded 'conformer-en-US-asr-streaming-feature-extractor-streaming' version 1 I0312 06:03:09.615079 8 detokenizer_cbe.cc:145] TRITONBACKEND_ModelInitialize: intent_slot_detokenizer (version 1) I0312 06:03:09.615709 8 backend_model.cc:303] model configuration: { "name": "intent_slot_detokenizer", "platform": "", "backend": "riva_nlp_detokenizer", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 1, "input": [ { "name": "IN_TOKEN_LABELS__0", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "IN_TOKEN_SCORES__1", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "IN_SEQ_LEN__2", "data_type": "TYPE_INT64", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "IN_TOK_STR__3", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "OUT_TOKEN_LABELS__0", "data_type": "TYPE_STRING", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "OUT_TOKEN_SCORES__1", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "OUT_SEQ_LEN__2", "data_type": "TYPE_INT64", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "OUT_TOK_STR__3", "data_type": "TYPE_STRING", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "graph": { "level": 0 }, "priority": "PRIORITY_DEFAULT", "cuda": { "graphs": false, "busy_wait_events": false, "graph_spec": [], "output_copy_stream": true }, "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "instance_group": [ { "name": "intent_slot_detokenizer_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": {}, "model_warmup": [], "model_transaction_policy": { "decoupled": false } } I0312 06:03:09.615773 8 pipeline_library.cc:25] TRITONBACKEND_ModelInstanceInitialize: riva-punctuation-en-US_0 (device 0) I0312 06:03:09.631247 8 model_lifecycle.cc:693] successfully loaded 'riva-punctuation-en-US' version 1 I0312 06:03:09.640799 8 endpointing_library.cc:18] TRITONBACKEND_ModelInitialize: conformer-en-US-asr-streaming-endpointing-streaming (version 1) WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 06:03:09.641707 33 parameter_parser.cc:144] Parameter 'chunk_size' set but unused. W0312 06:03:09.641731 33 parameter_parser.cc:144] Parameter 'ms_per_timestep' set but unused. W0312 06:03:09.641736 33 parameter_parser.cc:144] Parameter 'residue_blanks_at_end' set but unused. W0312 06:03:09.641739 33 parameter_parser.cc:144] Parameter 'residue_blanks_at_start' set but unused. W0312 06:03:09.641743 33 parameter_parser.cc:144] Parameter 'start_history' set but unused. W0312 06:03:09.641746 33 parameter_parser.cc:144] Parameter 'start_th' set but unused. W0312 06:03:09.641750 33 parameter_parser.cc:144] Parameter 'stop_history' set but unused. W0312 06:03:09.641755 33 parameter_parser.cc:144] Parameter 'stop_th' set but unused. W0312 06:03:09.641758 33 parameter_parser.cc:144] Parameter 'streaming' set but unused. W0312 06:03:09.641764 33 parameter_parser.cc:144] Parameter 'use_subword' set but unused. W0312 06:03:09.641769 33 parameter_parser.cc:144] Parameter 'vocab_file' set but unused. I0312 06:03:09.642133 8 backend_model.cc:303] model configuration: { "name": "conformer-en-US-asr-streaming-endpointing-streaming", "platform": "", "backend": "riva_asr_endpointing", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 1, "input": [ { "name": "CLASS_LOGITS", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1, 257 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "SEGMENTS_START_END", "data_type": "TYPE_FP32", "dims": [ -1, 2 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "graph": { "level": 0 }, "priority": "PRIORITY_DEFAULT", "cuda": { "graphs": false, "busy_wait_events": false, "graph_spec": [], "output_copy_stream": true }, "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "max_sequence_idle_microseconds": 60000000, "control_input": [ { "name": "START", "control": [ { "kind": "CONTROL_SEQUENCE_START", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "READY", "control": [ { "kind": "CONTROL_SEQUENCE_READY", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] } ], "state": [] }, "instance_group": [ { "name": "conformer-en-US-asr-streaming-endpointing-streaming_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "residue_blanks_at_start": { "string_value": "-2" }, "streaming": { "string_value": "True" }, "vocab_file": { "string_value": "/data/models/conformer-en-US-asr-streaming-endpointing-streaming/1/riva_decoder_vocabulary.txt" }, "stop_th": { "string_value": "0.98" }, "chunk_size": { "string_value": "0.16" }, "start_th": { "string_value": "0.2" }, "start_history": { "string_value": "200" }, "endpointing_type": { "string_value": "greedy_ctc" }, "ms_per_timestep": { "string_value": "40" }, "residue_blanks_at_end": { "string_value": "0" }, "use_subword": { "string_value": "True" }, "stop_history": { "string_value": "800" } }, "model_warmup": [], "model_transaction_policy": { "decoupled": false } } I0312 06:03:09.642190 8 detokenizer_cbe.cc:147] TRITONBACKEND_ModelInstanceInitialize: intent_slot_detokenizer_0 (device 0) I0312 06:03:09.642268 8 sequence_label_cbe.cc:137] TRITONBACKEND_ModelInitialize: intent_slot_label_tokens_misty (version 1) I0312 06:03:09.642429 8 model_lifecycle.cc:693] successfully loaded 'intent_slot_detokenizer' version 1 I0312 06:03:09.642915 8 backend_model.cc:303] model configuration: { "name": "intent_slot_label_tokens_misty", "platform": "", "backend": "riva_nlp_seqlabel", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 1, "input": [ { "name": "TOKEN_LOGIT__1", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1, 65 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "TOKEN_LABELS__0", "data_type": "TYPE_STRING", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "TOKEN_SCORES__1", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "priority": "PRIORITY_DEFAULT", "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "instance_group": [ { "name": "intent_slot_label_tokens_misty_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "classes": { "string_value": "/data/models/intent_slot_label_tokens_misty/1/slot_labels.csv" } }, "model_warmup": [] } I0312 06:03:11.963498 8 sequence_label_cbe.cc:139] TRITONBACKEND_ModelInstanceInitialize: intent_slot_label_tokens_misty_0 (device 0) I0312 06:03:11.963626 8 ctc-decoder-library.cc:20] TRITONBACKEND_ModelInitialize: conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming (version 1) I0312 06:03:11.963877 8 model_lifecycle.cc:693] successfully loaded 'intent_slot_label_tokens_misty' version 1 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 06:03:11.964607 32 parameter_parser.cc:144] Parameter 'append_space_to_transcripts' set but unused. W0312 06:03:11.964624 32 parameter_parser.cc:144] Parameter 'beam_size' set but unused. W0312 06:03:11.964627 32 parameter_parser.cc:144] Parameter 'beam_size_token' set but unused. W0312 06:03:11.964629 32 parameter_parser.cc:144] Parameter 'beam_threshold' set but unused. W0312 06:03:11.964632 32 parameter_parser.cc:144] Parameter 'blank_token' set but unused. W0312 06:03:11.964634 32 parameter_parser.cc:144] Parameter 'decoder_num_worker_threads' set but unused. W0312 06:03:11.964637 32 parameter_parser.cc:144] Parameter 'forerunner_beam_size' set but unused. W0312 06:03:11.964638 32 parameter_parser.cc:144] Parameter 'forerunner_beam_size_token' set but unused. W0312 06:03:11.964640 32 parameter_parser.cc:144] Parameter 'forerunner_beam_threshold' set but unused. W0312 06:03:11.964643 32 parameter_parser.cc:144] Parameter 'forerunner_use_lm' set but unused. W0312 06:03:11.964644 32 parameter_parser.cc:144] Parameter 'language_model_file' set but unused. W0312 06:03:11.964646 32 parameter_parser.cc:144] Parameter 'lexicon_file' set but unused. W0312 06:03:11.964648 32 parameter_parser.cc:144] Parameter 'lm_weight' set but unused. W0312 06:03:11.964650 32 parameter_parser.cc:144] Parameter 'log_add' set but unused. W0312 06:03:11.964653 32 parameter_parser.cc:144] Parameter 'max_execution_batch_size' set but unused. W0312 06:03:11.964654 32 parameter_parser.cc:144] Parameter 'max_supported_transcripts' set but unused. W0312 06:03:11.964656 32 parameter_parser.cc:144] Parameter 'num_tokenization' set but unused. W0312 06:03:11.964658 32 parameter_parser.cc:144] Parameter 'profane_words_file' set but unused. W0312 06:03:11.964660 32 parameter_parser.cc:144] Parameter 'set_default_index_to_unk_token' set but unused. W0312 06:03:11.964661 32 parameter_parser.cc:144] Parameter 'sil_token' set but unused. W0312 06:03:11.964663 32 parameter_parser.cc:144] Parameter 'smearing_mode' set but unused. W0312 06:03:11.964665 32 parameter_parser.cc:144] Parameter 'tokenizer_model' set but unused. W0312 06:03:11.964668 32 parameter_parser.cc:144] Parameter 'unk_score' set but unused. W0312 06:03:11.964669 32 parameter_parser.cc:144] Parameter 'unk_token' set but unused. W0312 06:03:11.964671 32 parameter_parser.cc:144] Parameter 'use_lexicon_free_decoding' set but unused. W0312 06:03:11.964673 32 parameter_parser.cc:144] Parameter 'vocab_file' set but unused. W0312 06:03:11.964675 32 parameter_parser.cc:144] Parameter 'word_insertion_score' set but unused. I0312 06:03:11.965239 8 backend_model.cc:303] model configuration: { "name": "conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming", "platform": "", "backend": "riva_asr_decoder", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 1024, "input": [ { "name": "CLASS_LOGITS", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1, 257 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "END_FLAG", "data_type": "TYPE_UINT32", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "SEGMENTS_START_END", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1, 2 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "CUSTOM_CONFIGURATION", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ -1, 2 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "FINAL_TRANSCRIPTS", "data_type": "TYPE_STRING", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "FINAL_TRANSCRIPTS_SCORE", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "FINAL_WORDS_START_END", "data_type": "TYPE_INT32", "dims": [ -1, 2 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PARTIAL_TRANSCRIPTS", "data_type": "TYPE_STRING", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PARTIAL_TRANSCRIPTS_STABILITY", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PARTIAL_WORDS_START_END", "data_type": "TYPE_INT32", "dims": [ -1, 2 ], "label_filename": "", "is_shape_tensor": false }, { "name": "FINAL_WORDS_SCORE", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PARTIAL_WORDS_SCORE", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "graph": { "level": 0 }, "priority": "PRIORITY_DEFAULT", "cuda": { "graphs": false, "busy_wait_events": false, "graph_spec": [], "output_copy_stream": true }, "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "oldest": { "max_candidate_sequences": 1024, "preferred_batch_size": [ 32, 64 ], "max_queue_delay_microseconds": 1000 }, "max_sequence_idle_microseconds": 60000000, "control_input": [ { "name": "START", "control": [ { "kind": "CONTROL_SEQUENCE_START", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "READY", "control": [ { "kind": "CONTROL_SEQUENCE_READY", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "END", "control": [ { "kind": "CONTROL_SEQUENCE_END", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "CORRID", "control": [ { "kind": "CONTROL_SEQUENCE_CORRID", "int32_false_true": [], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_UINT64" } ] } ], "state": [] }, "instance_group": [ { "name": "conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "left_padding_size": { "string_value": "1.92" }, "use_lexicon_free_decoding": { "string_value": "False" }, "word_insertion_score": { "string_value": "1.0" }, "unk_token": { "string_value": "" }, "forerunner_beam_threshold": { "string_value": "10.0" }, "streaming": { "string_value": "True" }, "beam_size": { "string_value": "32" }, "language_model_file": { "string_value": "/data/models/conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/4gram-pruned-0_2_7_9-en-lm-set-1.0.bin" }, "append_space_to_transcripts": { "string_value": "True" }, "num_tokenization": { "string_value": "1" }, "max_supported_transcripts": { "string_value": "1" }, "ms_per_timestep": { "string_value": "40" }, "forerunner_beam_size": { "string_value": "8" }, "blank_token": { "string_value": "#" }, "decoder_num_worker_threads": { "string_value": "-1" }, "right_padding_size": { "string_value": "1.92" }, "beam_threshold": { "string_value": "20.0" }, "lm_weight": { "string_value": "0.8" }, "sil_token": { "string_value": "▁" }, "set_default_index_to_unk_token": { "string_value": "False" }, "use_subword": { "string_value": "True" }, "max_execution_batch_size": { "string_value": "1" }, "chunk_size": { "string_value": "0.16" }, "lexicon_file": { "string_value": "/data/models/conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/lexicon.txt" }, "decoder_type": { "string_value": "flashlight" }, "smearing_mode": { "string_value": "max" }, "forerunner_use_lm": { "string_value": "true" }, "log_add": { "string_value": "True" }, "asr_model_delay": { "string_value": "-1" }, "tokenizer_model": { "string_value": "/data/models/conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/8b8f095152034e98b24ab33726708bd0_tokenizer.model" }, "profane_words_file": { "string_value": "/data/models/conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/profane_words_file.txt" }, "vocab_file": { "string_value": "/data/models/conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/riva_decoder_vocabulary.txt" }, "forerunner_beam_size_token": { "string_value": "8" }, "beam_size_token": { "string_value": "16" }, "unk_score": { "string_value": "-inf" } }, "model_warmup": [], "model_transaction_policy": { "decoupled": false } } I0312 06:03:11.965315 8 tokenizer_library.cc:18] TRITONBACKEND_ModelInitialize: intent_slot_tokenizer-en-US-misty (version 1) WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 06:03:11.965831 37 parameter_parser.cc:144] Parameter 'unk_token' set but unused. W0312 06:03:11.965847 37 parameter_parser.cc:144] Parameter 'vocab' set but unused. I0312 06:03:11.965927 8 backend_model.cc:303] model configuration: { "name": "intent_slot_tokenizer-en-US-misty", "platform": "", "backend": "riva_nlp_tokenizer", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 1, "input": [ { "name": "INPUT_STR__0", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "SEQ__0", "data_type": "TYPE_INT32", "dims": [ 128 ], "label_filename": "", "is_shape_tensor": false }, { "name": "MASK__1", "data_type": "TYPE_INT32", "dims": [ 128 ], "label_filename": "", "is_shape_tensor": false }, { "name": "SEGMENT__4", "data_type": "TYPE_INT32", "dims": [ 128 ], "label_filename": "", "is_shape_tensor": false }, { "name": "SEQ_LEN__2", "data_type": "TYPE_INT64", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "TOK_STR__3", "data_type": "TYPE_STRING", "dims": [ 128 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "priority": "PRIORITY_DEFAULT", "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "instance_group": [ { "name": "intent_slot_tokenizer-en-US-misty_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "bos_token": { "string_value": "[CLS]" }, "unk_token": { "string_value": "[UNK]" }, "task": { "string_value": "single_input" }, "pad_chars_with_spaces": { "string_value": "False" }, "vocab": { "string_value": "/data/models/intent_slot_tokenizer-en-US-misty/1/tokenizer.vocab_file" }, "eos_token": { "string_value": "[SEP]" }, "tokenizer": { "string_value": "wordpiece" }, "to_lower": { "string_value": "true" } }, "model_warmup": [] } I0312 06:03:11.974124 8 endpointing_library.cc:22] TRITONBACKEND_ModelInstanceInitialize: conformer-en-US-asr-streaming-endpointing-streaming_0 (device 0) E0312 06:03:11.974130 8 model_lifecycle.cc:596] failed to load 'tts_postprocessor-English-US' version 1: Not found: unable to load shared library: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8: undefined symbol: _ZN5nvdla8IProfile17setUseSoftMaxOptzEb I0312 06:03:11.974890 8 ctc-decoder-library.cc:23] TRITONBACKEND_ModelInstanceInitialize: conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming_0 (device 0) I0312 06:03:11.975091 8 model_lifecycle.cc:693] successfully loaded 'conformer-en-US-asr-streaming-endpointing-streaming' version 1 I0312 06:03:14.674629 32 ctc-decoder.cc:174] Beam Decoder initialized successfully! I0312 06:03:14.675018 8 tokenizer_library.cc:21] TRITONBACKEND_ModelInstanceInitialize: intent_slot_tokenizer-en-US-misty_0 (device 0) I0312 06:03:14.693859 8 model_lifecycle.cc:693] successfully loaded 'conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming' version 1 I0312 06:03:14.695611 8 model_lifecycle.cc:693] successfully loaded 'intent_slot_tokenizer-en-US-misty' version 1 I0312 06:03:14.699872 8 spectrogram-chunker.cc:274] TRITONBACKEND_ModelInitialize: spectrogram_chunker-English-US (version 1) I0312 06:03:14.700899 8 backend_model.cc:303] model configuration: { "name": "spectrogram_chunker-English-US", "platform": "", "backend": "riva_tts_chunker", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 1, "input": [ { "name": "SPECTROGRAM", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ 80, -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "IS_LAST_SENTENCE", "data_type": "TYPE_INT32", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "NUM_VALID_FRAMES_IN", "data_type": "TYPE_INT64", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "SENTENCE_NUM", "data_type": "TYPE_INT32", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "DURATIONS", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "PROCESSED_TEXT", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "VOLUME", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "SPECTROGRAM_CHUNK", "data_type": "TYPE_FP32", "dims": [ 80, -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "END_FLAG", "data_type": "TYPE_INT32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "NUM_VALID_SAMPLES_OUT", "data_type": "TYPE_INT32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "SENTENCE_NUM", "data_type": "TYPE_INT32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "DURATIONS", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PROCESSED_TEXT", "data_type": "TYPE_STRING", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "VOLUME", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "priority": "PRIORITY_DEFAULT", "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "oldest": { "max_candidate_sequences": 1, "preferred_batch_size": [ 1 ], "max_queue_delay_microseconds": 1000 }, "max_sequence_idle_microseconds": 60000000, "control_input": [ { "name": "START", "control": [ { "kind": "CONTROL_SEQUENCE_START", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "READY", "control": [ { "kind": "CONTROL_SEQUENCE_READY", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "END", "control": [ { "kind": "CONTROL_SEQUENCE_END", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "CORRID", "control": [ { "kind": "CONTROL_SEQUENCE_CORRID", "int32_false_true": [], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_UINT64" } ] } ], "state": [] }, "instance_group": [ { "name": "spectrogram_chunker-English-US_0", "kind": "KIND_GPU", "count": 1, "gpus": [ 0 ], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "supports_volume": { "string_value": "True" }, "chunk_length": { "string_value": "80" }, "num_samples_per_frame": { "string_value": "512" }, "num_mels": { "string_value": "80" }, "max_execution_batch_size": { "string_value": "8" } }, "model_warmup": [], "model_transaction_policy": { "decoupled": true } } I0312 06:03:14.701098 8 spectrogram-chunker.cc:276] TRITONBACKEND_ModelInstanceInitialize: spectrogram_chunker-English-US_0 (device 0) I0312 06:03:14.721143 8 model_lifecycle.cc:693] successfully loaded 'spectrogram_chunker-English-US' version 1 I0312 06:03:14.792095 8 onnxruntime.cc:2459] TRITONBACKEND_Initialize: onnxruntime I0312 06:03:14.792437 8 onnxruntime.cc:2469] Triton TRITONBACKEND API version: 1.10 I0312 06:03:14.792595 8 onnxruntime.cc:2475] 'onnxruntime' TRITONBACKEND API version: 1.10 I0312 06:03:14.792748 8 onnxruntime.cc:2505] backend configuration: {"cmdline":{"auto-complete-config":"false","min-compute-capability":"5.300000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} I0312 06:03:14.810963 8 tts-preprocessor.cc:280] TRITONBACKEND_ModelInitialize: tts_preprocessor-English-US (version 1) W0312 06:03:14.812094 8 tts-preprocessor.cc:241] Parameter abbreviation_path is deprecated WARNING: Logging before InitGoogleLogging() is written to STDERR I0312 06:03:14.812383 46 preprocessor.cc:206] TTS character mapping loaded from /data/models/tts_preprocessor-English-US/1/mapping.txt I0312 06:03:15.015141 46 preprocessor.cc:243] TTS phonetic mapping loaded from /data/models/tts_preprocessor-English-US/1/ipa_cmudict-0.7b_nv22.08.txt I0312 06:03:15.015602 46 preprocessor.cc:256] Abbreviation mapping loaded from /data/models/tts_preprocessor-English-US/1/abbr.txt W0312 06:03:15.015799 46 normalize.cc:52] Speech Class far file missing:/data/models/tts_preprocessor-English-US/1/speech_class.far I0312 06:03:15.109019 46 preprocessor.cc:266] TTS normalizer loaded from /data/models/tts_preprocessor-English-US/1/ I0312 06:03:15.109183 8 backend_model.cc:303] model configuration: { "name": "tts_preprocessor-English-US", "platform": "", "backend": "riva_tts_preprocessor", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 1, "input": [ { "name": "input_string", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "speaker", "data_type": "TYPE_INT64", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "output", "data_type": "TYPE_INT64", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "output_mask", "data_type": "TYPE_FP32", "dims": [ 1, 400, 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "output_length", "data_type": "TYPE_INT32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "is_last_sentence", "data_type": "TYPE_INT32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "output_string", "data_type": "TYPE_STRING", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "sentence_num", "data_type": "TYPE_INT32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "pitch", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "duration", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "volume", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "speaker", "data_type": "TYPE_INT64", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "graph": { "level": 0 }, "priority": "PRIORITY_DEFAULT", "cuda": { "graphs": false, "busy_wait_events": false, "graph_spec": [], "output_copy_stream": true }, "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "oldest": { "max_candidate_sequences": 1, "preferred_batch_size": [ 1 ], "max_queue_delay_microseconds": 100 }, "max_sequence_idle_microseconds": 60000000, "control_input": [ { "name": "START", "control": [ { "kind": "CONTROL_SEQUENCE_START", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "READY", "control": [ { "kind": "CONTROL_SEQUENCE_READY", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "END", "control": [ { "kind": "CONTROL_SEQUENCE_END", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "CORRID", "control": [ { "kind": "CONTROL_SEQUENCE_CORRID", "int32_false_true": [], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_UINT64" } ] } ], "state": [] }, "instance_group": [ { "name": "tts_preprocessor-English-US_0", "kind": "KIND_GPU", "count": 1, "gpus": [ 0 ], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "dictionary_path": { "string_value": "/data/models/tts_preprocessor-English-US/1/ipa_cmudict-0.7b_nv22.08.txt" }, "upper_case_chars": { "string_value": "True" }, "language": { "string_value": "en-US" }, "max_sequence_length": { "string_value": "400" }, "max_input_length": { "string_value": "2000" }, "end_of_emphasis_token": { "string_value": "]" }, "mapping_path": { "string_value": "/data/models/tts_preprocessor-English-US/1/mapping.txt" }, "abbreviations_path": { "string_value": "/data/models/tts_preprocessor-English-US/1/abbr.txt" }, "norm_proto_path": { "string_value": "/data/models/tts_preprocessor-English-US/1/" }, "supports_ragged_batches": { "string_value": "True" }, "pad_with_space": { "string_value": "True" }, "g2p_ignore_ambiguous": { "string_value": "True" }, "phone_set": { "string_value": "ipa" }, "start_of_emphasis_token": { "string_value": "[" }, "enable_emphasis_tag": { "string_value": "True" } }, "model_warmup": [], "model_transaction_policy": { "decoupled": true } } I0312 06:03:15.109262 8 onnxruntime.cc:2563] TRITONBACKEND_ModelInitialize: riva-onnx-fastpitch_encoder-English-US (version 1) I0312 06:03:15.110165 8 onnxruntime.cc:2606] TRITONBACKEND_ModelInstanceInitialize: riva-onnx-fastpitch_encoder-English-US_0 (GPU device 0) 2023-03-12 06:03:16.658841411 [W:onnxruntime:, session_state.cc:1030 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf. 2023-03-12 06:03:16.658890531 [W:onnxruntime:, session_state.cc:1032 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments. I0312 06:03:17.386352 8 tts-preprocessor.cc:282] TRITONBACKEND_ModelInstanceInitialize: tts_preprocessor-English-US_0 (device 0) I0312 06:03:17.387416 8 model_lifecycle.cc:693] successfully loaded 'riva-onnx-fastpitch_encoder-English-US' version 1 I0312 06:03:17.388055 8 model_lifecycle.cc:693] successfully loaded 'tts_preprocessor-English-US' version 1 E0312 06:03:17.388273 8 model_repository_manager.cc:481] Invalid argument: ensemble 'conformer-en-US-asr-streaming' depends on 'riva-trt-conformer-en-US-asr-streaming-am-streaming' which has no loaded version E0312 06:03:17.388294 8 model_repository_manager.cc:481] Invalid argument: ensemble 'fastpitch_hifigan_ensemble-English-US' depends on 'tts_postprocessor-English-US' which has no loaded version E0312 06:03:17.388302 8 model_repository_manager.cc:481] Invalid argument: ensemble 'riva_intent_misty' depends on 'riva-trt-riva_intent_misty-nn-bert-base-uncased' which has no loaded version I0312 06:03:17.388380 8 server.cc:563] +------------------+------+ | Repository Agent | Path | +------------------+------+ +------------------+------+ I0312 06:03:17.388501 8 server.cc:590] +-----------------------+-------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Backend | Path | Config | +-----------------------+-------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+ | onnxruntime | /opt/tritonserver/backends/onnxruntime/libtriton_onnxruntime.so | {"cmdline":{"auto-complete-config":"false","min-compute-capability":"5.300000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} | | riva_asr_features | /opt/tritonserver/backends/riva_asr_features/libtriton_riva_asr_features.so | {"cmdline":{"auto-complete-config":"false","min-compute-capability":"5.300000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} | | riva_tts_preprocessor | /opt/tritonserver/backends/riva_tts_preprocessor/libtriton_riva_tts_preprocessor.so | {"cmdline":{"auto-complete-config":"false","min-compute-capability":"5.300000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} | | riva_nlp_pipeline | /opt/tritonserver/backends/riva_nlp_pipeline/libtriton_riva_nlp_pipeline.so | {"cmdline":{"auto-complete-config":"false","min-compute-capability":"5.300000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} | | riva_asr_decoder | /opt/tritonserver/backends/riva_asr_decoder/libtriton_riva_asr_decoder.so | {"cmdline":{"auto-complete-config":"false","min-compute-capability":"5.300000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} | | riva_nlp_detokenizer | /opt/tritonserver/backends/riva_nlp_detokenizer/libtriton_riva_nlp_detokenizer.so | {"cmdline":{"auto-complete-config":"false","min-compute-capability":"5.300000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} | | riva_asr_endpointing | /opt/tritonserver/backends/riva_asr_endpointing/libtriton_riva_asr_endpointing.so | {"cmdline":{"auto-complete-config":"false","min-compute-capability":"5.300000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} | | riva_nlp_tokenizer | /opt/tritonserver/backends/riva_nlp_tokenizer/libtriton_riva_nlp_tokenizer.so | {"cmdline":{"auto-complete-config":"false","min-compute-capability":"5.300000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} | | riva_nlp_seqlabel | /opt/tritonserver/backends/riva_nlp_seqlabel/libtriton_riva_nlp_seqlabel.so | {"cmdline":{"auto-complete-config":"false","min-compute-capability":"5.300000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} | | riva_tts_chunker | /opt/tritonserver/backends/riva_tts_chunker/libtriton_riva_tts_chunker.so | {"cmdline":{"auto-complete-config":"false","min-compute-capability":"5.300000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} | +-----------------------+-------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+ I0312 06:03:17.388646 8 server.cc:633] +-----------------------------------------------------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Model | Version | Status | +-----------------------------------------------------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ | conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming | 1 | READY | | conformer-en-US-asr-streaming-endpointing-streaming | 1 | READY | | conformer-en-US-asr-streaming-feature-extractor-streaming | 1 | READY | | intent_slot_detokenizer | 1 | READY | | intent_slot_label_tokens_misty | 1 | READY | | intent_slot_tokenizer-en-US-misty | 1 | READY | | riva-onnx-fastpitch_encoder-English-US | 1 | READY | | riva-punctuation-en-US | 1 | READY | | riva-trt-conformer-en-US-asr-streaming-am-streaming | 1 | UNAVAILABLE: Not found: unable to load shared library: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8: undefined symbol: _ZN5nvdla8IProfile17setUseSoftMaxOptzEb | | riva-trt-hifigan-English-US | 1 | UNAVAILABLE: Not found: unable to load shared library: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8: undefined symbol: _ZN5nvdla8IProfile17setUseSoftMaxOptzEb | | riva-trt-riva-punctuation-en-US-nn-bert-base-uncased | 1 | UNAVAILABLE: Not found: unable to load shared library: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8: undefined symbol: _ZN5nvdla8IProfile17setUseSoftMaxOptzEb | | riva-trt-riva_intent_misty-nn-bert-base-uncased | 1 | UNAVAILABLE: Not found: unable to load shared library: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8: undefined symbol: _ZN5nvdla8IProfile17setUseSoftMaxOptzEb | | spectrogram_chunker-English-US | 1 | READY | | tts_postprocessor-English-US | 1 | UNAVAILABLE: Not found: unable to load shared library: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8: undefined symbol: _ZN5nvdla8IProfile17setUseSoftMaxOptzEb | | tts_preprocessor-English-US | 1 | READY | +-----------------------------------------------------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+ W0312 06:03:17.388670 8 metrics.cc:354] No polling metrics (CPU, GPU, Cache) are enabled. Will not poll for them. I0312 06:03:17.388803 8 tritonserver.cc:2264] +----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Option | Value | +----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | server_id | triton | | server_version | 2.27.0 | | server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace logging | | model_repository_path[0] | /data/models | | model_control_mode | MODE_NONE | | strict_model_config | 1 | | rate_limit | OFF | | pinned_memory_pool_byte_size | 268435456 | | cuda_memory_pool_byte_size{0} | 1000000000 | | response_cache_byte_size | 0 | | min_supported_compute_capability | 5.3 | | strict_readiness | 1 | | exit_timeout | 30 | +----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ I0312 06:03:17.388811 8 server.cc:264] Waiting for in-flight requests to complete. I0312 06:03:17.388826 8 server.cc:280] Timeout 30: Found 0 model versions that have in-flight inferences I0312 06:03:17.389360 8 pipeline_library.cc:28] TRITONBACKEND_ModelInstanceFinalize: delete instance state I0312 06:03:17.389678 8 ctc-decoder-library.cc:25] TRITONBACKEND_ModelInstanceFinalize: delete instance state I0312 06:03:17.390111 8 server.cc:295] All models are stopped, unloading models I0312 06:03:17.390310 8 endpointing_library.cc:26] TRITONBACKEND_ModelInstanceFinalize: delete instance state I0312 06:03:17.390367 8 server.cc:302] Timeout 30: Found 10 live models and 0 in-flight non-inference requests I0312 06:03:17.390809 8 tts-preprocessor.cc:284] TRITONBACKEND_ModelInstanceFinalize: delete instance state I0312 06:03:17.390900 8 tts-preprocessor.cc:281] TRITONBACKEND_ModelFinalize: delete model state I0312 06:03:17.391070 8 detokenizer_cbe.cc:149] TRITONBACKEND_ModelInstanceFinalize: delete instance state I0312 06:03:17.391124 8 sequence_label_cbe.cc:141] TRITONBACKEND_ModelInstanceFinalize: delete instance state I0312 06:03:17.391249 8 detokenizer_cbe.cc:146] TRITONBACKEND_ModelFinalize: delete model state I0312 06:03:17.391275 8 sequence_label_cbe.cc:138] TRITONBACKEND_ModelFinalize: delete model state I0312 06:03:17.391504 8 model_lifecycle.cc:578] successfully unloaded 'intent_slot_detokenizer' version 1 I0312 06:03:17.391541 8 model_lifecycle.cc:578] successfully unloaded 'intent_slot_label_tokens_misty' version 1 I0312 06:03:17.391987 8 endpointing_library.cc:21] TRITONBACKEND_ModelFinalize: delete model state I0312 06:03:17.392136 8 feature-extractor.cc:404] TRITONBACKEND_ModelInstanceFinalize: delete instance state I0312 06:03:17.392553 8 model_lifecycle.cc:578] successfully unloaded 'conformer-en-US-asr-streaming-endpointing-streaming' version 1 I0312 06:03:17.393197 8 spectrogram-chunker.cc:279] TRITONBACKEND_ModelInstanceFinalize: delete instance state I0312 06:03:17.393580 8 spectrogram-chunker.cc:275] TRITONBACKEND_ModelFinalize: delete model state I0312 06:03:17.394212 8 model_lifecycle.cc:578] successfully unloaded 'spectrogram_chunker-English-US' version 1 I0312 06:03:17.397952 8 tokenizer_library.cc:25] TRITONBACKEND_ModelInstanceFinalize: delete instance state I0312 06:03:17.401500 8 onnxruntime.cc:2640] TRITONBACKEND_ModelInstanceFinalize: delete instance state I0312 06:03:17.404777 8 feature-extractor.cc:401] TRITONBACKEND_ModelFinalize: delete model state I0312 06:03:17.405145 8 pipeline_library.cc:24] TRITONBACKEND_ModelFinalize: delete model state I0312 06:03:17.409244 8 tokenizer_library.cc:20] TRITONBACKEND_ModelFinalize: delete model state I0312 06:03:17.411879 8 model_lifecycle.cc:578] successfully unloaded 'riva-punctuation-en-US' version 1 I0312 06:03:17.423850 8 model_lifecycle.cc:578] successfully unloaded 'intent_slot_tokenizer-en-US-misty' version 1 I0312 06:03:17.429706 8 model_lifecycle.cc:578] successfully unloaded 'conformer-en-US-asr-streaming-feature-extractor-streaming' version 1 I0312 06:03:17.456033 8 onnxruntime.cc:2586] TRITONBACKEND_ModelFinalize: delete model state I0312 06:03:17.456573 8 model_lifecycle.cc:578] successfully unloaded 'riva-onnx-fastpitch_encoder-English-US' version 1 I0312 06:03:17.527630 8 model_lifecycle.cc:578] successfully unloaded 'tts_preprocessor-English-US' version 1 I0312 06:03:18.185733 8 ctc-decoder-library.cc:22] TRITONBACKEND_ModelFinalize: delete model state I0312 06:03:18.186557 8 model_lifecycle.cc:578] successfully unloaded 'conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming' version 1 E0312 06:03:18.200240 10 model_registry.cc:246] error: unable to get server status: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8001: Failed to connect to remote host: Connection refused I0312 06:03:18.390682 8 server.cc:302] Timeout 29: Found 0 live models and 0 in-flight non-inference requests error: creating server: Internal - failed to load all models One of the processes has exited unexpectedly. Stopping container. W0312 06:03:23.207298 10 riva_server.cc:195] Signal: 15