========================== === Riva Speech Skills === ========================== NVIDIA Release (build 49131132) Copyright (c) 2018-2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved. Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES. All rights reserved. This container image and its contents are governed by the NVIDIA Deep Learning Container License. By pulling and using the container, you accept the terms and conditions of this license: https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license WARNING: CUDA Minor Version Compatibility mode ENABLED. Using driver version 510.47.03 which has support for CUDA 11.6. This container was built with CUDA 11.8 and will be run in Minor Version Compatibility mode. CUDA Forward Compatibility is preferred over Minor Version Compatibility for use with this container but was unavailable: [[Forward compatibility was attempted on non supported HW (CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE) cuInit()=804]] See https://docs.nvidia.com/deploy/cuda-compatibility/ for details. > Riva waiting for Triton server to load all models...retrying in 1 second Warning: '--strict-model-config' has been deprecated! Please use '--disable-auto-complete-config' instead. I1216 03:05:15.369343 106 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f94e8000000' with size 268435456 I1216 03:05:15.370039 106 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 1000000000 I1216 03:05:15.375776 106 model_lifecycle.cc:459] loading: conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline:1 I1216 03:05:15.375822 106 model_lifecycle.cc:459] loading: conformer-en-US-asr-offline-endpointing-streaming-offline:1 I1216 03:05:15.375858 106 model_lifecycle.cc:459] loading: conformer-en-US-asr-offline-feature-extractor-streaming-offline:1 I1216 03:05:15.375898 106 model_lifecycle.cc:459] loading: conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming:1 I1216 03:05:15.375942 106 model_lifecycle.cc:459] loading: conformer-en-US-asr-streaming-endpointing-streaming:1 I1216 03:05:15.376032 106 model_lifecycle.cc:459] loading: conformer-en-US-asr-streaming-feature-extractor-streaming:1 I1216 03:05:15.376075 106 model_lifecycle.cc:459] loading: riva-punctuation-en-US:1 I1216 03:05:15.376097 106 model_lifecycle.cc:459] loading: riva-trt-conformer-en-US-asr-offline-am-streaming-offline:1 I1216 03:05:15.376115 106 model_lifecycle.cc:459] loading: riva-trt-conformer-en-US-asr-streaming-am-streaming:1 I1216 03:05:15.376294 106 model_lifecycle.cc:459] loading: riva-trt-riva-punctuation-en-US-nn-bert-base-uncased:1 I1216 03:05:15.407051 106 tensorrt.cc:5444] TRITONBACKEND_Initialize: tensorrt I1216 03:05:15.407073 106 tensorrt.cc:5454] Triton TRITONBACKEND API version: 1.10 I1216 03:05:15.407292 106 tensorrt.cc:5460] 'tensorrt' TRITONBACKEND API version: 1.10 I1216 03:05:15.407297 106 tensorrt.cc:5488] backend configuration: {"cmdline":{"auto-complete-config":"false","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} I1216 03:05:15.448375 106 tensorrt.cc:5578] TRITONBACKEND_ModelInitialize: riva-trt-riva-punctuation-en-US-nn-bert-base-uncased (version 1) I1216 03:05:15.450038 106 tensorrt.cc:5578] TRITONBACKEND_ModelInitialize: riva-trt-conformer-en-US-asr-streaming-am-streaming (version 1) I1216 03:05:15.451150 106 tensorrt.cc:5578] TRITONBACKEND_ModelInitialize: riva-trt-conformer-en-US-asr-offline-am-streaming-offline (version 1) I1216 03:05:15.453180 106 pipeline_library.cc:22] TRITONBACKEND_ModelInitialize: riva-punctuation-en-US (version 1) WARNING: Logging before InitGoogleLogging() is written to STDERR W1216 03:05:15.453977 119 parameter_parser.cc:144] Parameter 'attn_mask_tensor_name' set but unused. W1216 03:05:15.453984 119 parameter_parser.cc:144] Parameter 'bos_token' set but unused. W1216 03:05:15.453986 119 parameter_parser.cc:144] Parameter 'capit_logits_tensor_name' set but unused. W1216 03:05:15.453987 119 parameter_parser.cc:144] Parameter 'capitalization_mapping_path' set but unused. W1216 03:05:15.453989 119 parameter_parser.cc:144] Parameter 'delimiter' set but unused. W1216 03:05:15.453989 119 parameter_parser.cc:144] Parameter 'eos_token' set but unused. W1216 03:05:15.453990 119 parameter_parser.cc:144] Parameter 'input_ids_tensor_name' set but unused. W1216 03:05:15.453991 119 parameter_parser.cc:144] Parameter 'language_code' set but unused. W1216 03:05:15.453992 119 parameter_parser.cc:144] Parameter 'model_api' set but unused. W1216 03:05:15.453994 119 parameter_parser.cc:144] Parameter 'model_family' set but unused. W1216 03:05:15.453995 119 parameter_parser.cc:144] Parameter 'pad_chars_with_spaces' set but unused. W1216 03:05:15.453996 119 parameter_parser.cc:144] Parameter 'preserve_accents' set but unused. W1216 03:05:15.453997 119 parameter_parser.cc:144] Parameter 'punct_logits_tensor_name' set but unused. W1216 03:05:15.453999 119 parameter_parser.cc:144] Parameter 'punctuation_mapping_path' set but unused. W1216 03:05:15.453999 119 parameter_parser.cc:144] Parameter 'remove_spaces' set but unused. W1216 03:05:15.454000 119 parameter_parser.cc:144] Parameter 'to_lower' set but unused. W1216 03:05:15.454001 119 parameter_parser.cc:144] Parameter 'token_type_tensor_name' set but unused. W1216 03:05:15.454002 119 parameter_parser.cc:144] Parameter 'tokenizer_to_lower' set but unused. W1216 03:05:15.454003 119 parameter_parser.cc:144] Parameter 'unicode_normalize' set but unused. W1216 03:05:15.454005 119 parameter_parser.cc:144] Parameter 'unk_token' set but unused. W1216 03:05:15.454005 119 parameter_parser.cc:144] Parameter 'use_int64_nn_inputs' set but unused. W1216 03:05:15.454006 119 parameter_parser.cc:144] Parameter 'vocab' set but unused. W1216 03:05:15.454330 119 parameter_parser.cc:144] Parameter 'model_api' set but unused. W1216 03:05:15.454334 119 parameter_parser.cc:144] Parameter 'model_family' set but unused. I1216 03:05:15.454376 106 backend_model.cc:303] model configuration: { "name": "riva-punctuation-en-US", "platform": "", "backend": "riva_nlp_pipeline", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 8, "input": [ { "name": "PIPELINE_INPUT", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "PIPELINE_OUTPUT", "data_type": "TYPE_STRING", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "priority": "PRIORITY_DEFAULT", "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "instance_group": [ { "name": "riva-punctuation-en-US_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "model_family": { "string_value": "riva" }, "tokenizer_to_lower": { "string_value": "true" }, "delimiter": { "string_value": " " }, "use_int64_nn_inputs": { "string_value": "False" }, "tokenizer": { "string_value": "wordpiece" }, "input_ids_tensor_name": { "string_value": "input_ids" }, "model_api": { "string_value": "/nvidia.riva.nlp.RivaLanguageUnderstanding/PunctuateText" }, "attn_mask_tensor_name": { "string_value": "attention_mask" }, "preserve_accents": { "string_value": "false" }, "language_code": { "string_value": "en-US" }, "pipeline_type": { "string_value": "punctuation" }, "eos_token": { "string_value": "[SEP]" }, "vocab": { "string_value": "/data/models/riva-punctuation-en-US/1/e222f352288a423da453a79b96cc7b75_vocab.txt" }, "pad_chars_with_spaces": { "string_value": "False" }, "capitalization_mapping_path": { "string_value": "/data/models/riva-punctuation-en-US/1/fb06800834e74de1bdc32db51da9619c_capit_label_ids.csv" }, "load_model": { "string_value": "false" }, "to_lower": { "string_value": "true" }, "unicode_normalize": { "string_value": "False" }, "punctuation_mapping_path": { "string_value": "/data/models/riva-punctuation-en-US/1/15eace99434b4c87ba28cbd294b48f43_punct_label_ids.csv" }, "bos_token": { "string_value": "[CLS]" }, "capit_logits_tensor_name": { "string_value": "capit_logits" }, "model_name": { "string_value": "riva-trt-riva-punctuation-en-US-nn-bert-base-uncased" }, "unk_token": { "string_value": "[UNK]" }, "punct_logits_tensor_name": { "string_value": "punct_logits" }, "token_type_tensor_name": { "string_value": "token_type_ids" }, "remove_spaces": { "string_value": "False" } }, "model_warmup": [] } I1216 03:05:15.454396 106 tensorrt.cc:5627] TRITONBACKEND_ModelInstanceInitialize: riva-trt-riva-punctuation-en-US-nn-bert-base-uncased_0 (GPU device 0) I1216 03:05:15.481541 106 tensorrt.cc:5665] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1216 03:05:15.481596 106 tensorrt.cc:5604] TRITONBACKEND_ModelFinalize: delete model state E1216 03:05:15.481612 106 model_lifecycle.cc:596] failed to load 'riva-trt-riva-punctuation-en-US-nn-bert-base-uncased' version 1: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. I1216 03:05:15.481612 106 tensorrt.cc:5627] TRITONBACKEND_ModelInstanceInitialize: riva-trt-conformer-en-US-asr-streaming-am-streaming_0 (GPU device 0) I1216 03:05:15.524468 106 tensorrt.cc:5665] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1216 03:05:15.524504 106 tensorrt.cc:5604] TRITONBACKEND_ModelFinalize: delete model state E1216 03:05:15.524521 106 model_lifecycle.cc:596] failed to load 'riva-trt-conformer-en-US-asr-streaming-am-streaming' version 1: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. I1216 03:05:15.524533 106 tensorrt.cc:5627] TRITONBACKEND_ModelInstanceInitialize: riva-trt-conformer-en-US-asr-offline-am-streaming-offline_0 (GPU device 0) I1216 03:05:15.565738 106 tensorrt.cc:5665] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1216 03:05:15.565768 106 tensorrt.cc:5604] TRITONBACKEND_ModelFinalize: delete model state E1216 03:05:15.565782 106 model_lifecycle.cc:596] failed to load 'riva-trt-conformer-en-US-asr-offline-am-streaming-offline' version 1: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. I1216 03:05:15.583021 106 endpointing_library.cc:18] TRITONBACKEND_ModelInitialize: conformer-en-US-asr-streaming-endpointing-streaming (version 1) WARNING: Logging before InitGoogleLogging() is written to STDERR W1216 03:05:15.584312 118 parameter_parser.cc:144] Parameter 'chunk_size' set but unused. W1216 03:05:15.584331 118 parameter_parser.cc:144] Parameter 'ms_per_timestep' set but unused. W1216 03:05:15.584336 118 parameter_parser.cc:144] Parameter 'residue_blanks_at_end' set but unused. W1216 03:05:15.584339 118 parameter_parser.cc:144] Parameter 'residue_blanks_at_start' set but unused. W1216 03:05:15.584343 118 parameter_parser.cc:144] Parameter 'start_history' set but unused. W1216 03:05:15.584347 118 parameter_parser.cc:144] Parameter 'start_th' set but unused. W1216 03:05:15.584352 118 parameter_parser.cc:144] Parameter 'stop_history' set but unused. W1216 03:05:15.584355 118 parameter_parser.cc:144] Parameter 'stop_th' set but unused. W1216 03:05:15.584358 118 parameter_parser.cc:144] Parameter 'streaming' set but unused. W1216 03:05:15.584367 118 parameter_parser.cc:144] Parameter 'use_subword' set but unused. W1216 03:05:15.584371 118 parameter_parser.cc:144] Parameter 'vocab_file' set but unused. I1216 03:05:15.584943 106 backend_model.cc:303] model configuration: { "name": "conformer-en-US-asr-streaming-endpointing-streaming", "platform": "", "backend": "riva_asr_endpointing", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 2048, "input": [ { "name": "CLASS_LOGITS", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1, 257 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "SEGMENTS_START_END", "data_type": "TYPE_FP32", "dims": [ -1, 2 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "graph": { "level": 0 }, "priority": "PRIORITY_DEFAULT", "cuda": { "graphs": false, "busy_wait_events": false, "graph_spec": [], "output_copy_stream": true }, "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "max_sequence_idle_microseconds": 60000000, "control_input": [ { "name": "START", "control": [ { "kind": "CONTROL_SEQUENCE_START", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "READY", "control": [ { "kind": "CONTROL_SEQUENCE_READY", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] } ], "state": [] }, "instance_group": [ { "name": "conformer-en-US-asr-streaming-endpointing-streaming_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "streaming": { "string_value": "True" }, "stop_th": { "string_value": "0.98" }, "start_history": { "string_value": "200" }, "start_th": { "string_value": "0.2" }, "use_subword": { "string_value": "True" }, "residue_blanks_at_end": { "string_value": "0" }, "stop_history": { "string_value": "800" }, "endpointing_type": { "string_value": "greedy_ctc" }, "ms_per_timestep": { "string_value": "40" }, "residue_blanks_at_start": { "string_value": "-2" }, "vocab_file": { "string_value": "/data/models/conformer-en-US-asr-streaming-endpointing-streaming/1/riva_decoder_vocabulary.txt" }, "chunk_size": { "string_value": "0.16" } }, "model_warmup": [], "model_transaction_policy": { "decoupled": false } } I1216 03:05:15.585010 106 pipeline_library.cc:25] TRITONBACKEND_ModelInstanceInitialize: riva-punctuation-en-US_0 (device 0) I1216 03:05:15.610857 106 model_lifecycle.cc:693] successfully loaded 'riva-punctuation-en-US' version 1 I1216 03:05:15.629632 106 feature-extractor.cc:400] TRITONBACKEND_ModelInitialize: conformer-en-US-asr-offline-feature-extractor-streaming-offline (version 1) I1216 03:05:15.720123 106 backend_model.cc:303] model configuration: { "name": "conformer-en-US-asr-offline-feature-extractor-streaming-offline", "platform": "", "backend": "riva_asr_features", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 512, "input": [ { "name": "AUDIO_SIGNAL", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "SAMPLE_RATE", "data_type": "TYPE_UINT32", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "AUDIO_FEATURES", "data_type": "TYPE_FP32", "dims": [ 80, -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "AUDIO_PROCESSED", "data_type": "TYPE_FP32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "AUDIO_FEATURES_LENGTH", "data_type": "TYPE_INT32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "graph": { "level": 0 }, "priority": "PRIORITY_DEFAULT", "cuda": { "graphs": false, "busy_wait_events": false, "graph_spec": [], "output_copy_stream": true }, "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "oldest": { "max_candidate_sequences": 512, "preferred_batch_size": [ 256, 512 ], "max_queue_delay_microseconds": 1000 }, "max_sequence_idle_microseconds": 60000000, "control_input": [ { "name": "START", "control": [ { "kind": "CONTROL_SEQUENCE_START", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "READY", "control": [ { "kind": "CONTROL_SEQUENCE_READY", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "END", "control": [ { "kind": "CONTROL_SEQUENCE_END", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "CORRID", "control": [ { "kind": "CONTROL_SEQUENCE_CORRID", "int32_false_true": [], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_UINT64" } ] } ], "state": [] }, "instance_group": [ { "name": "conformer-en-US-asr-offline-feature-extractor-streaming-offline_0", "kind": "KIND_GPU", "count": 1, "gpus": [ 0 ], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "transpose": { "string_value": "False" }, "mean": { "string_value": "-11.4412, -9.9334, -9.1292, -9.0365, -9.2804, -9.5643, -9.7342, -9.6925, -9.6333, -9.2808, -9.1887, -9.1422, -9.1397, -9.2028, -9.2749, -9.4776, -9.9185, -10.1557, -10.3800, -10.5067, -10.3190, -10.4728, -10.5529, -10.6402, -10.6440, -10.5113, -10.7395, -10.7870, -10.6074, -10.5033, -10.8278, -10.6384, -10.8481, -10.6875, -10.5454, -10.4747, -10.5165, -10.4930, -10.3413, -10.3472, -10.3735, -10.6830, -10.8813, -10.6338, -10.3856, -10.7727, -10.8957, -10.8068, -10.7373, -10.6108, -10.3405, -10.2889, -10.3922, -10.4946, -10.3367, -10.4164, -10.9949, -10.7196, -10.3971, -10.1734, -9.9257, -9.6557, -9.1761, -9.6653, -9.7876, -9.7230, -9.7792, -9.7056, -9.2702, -9.4650, -9.2755, -9.1369, -9.1174, -8.9197, -8.5394, -8.2614, -8.1353, -8.1422, -8.3430, -8.6655" }, "right_padding_size": { "string_value": "1.6" }, "precalc_norm_params": { "string_value": "False" }, "use_utterance_norm_params": { "string_value": "False" }, "window_size": { "string_value": "0.025" }, "norm_per_feature": { "string_value": "True" }, "sample_rate": { "string_value": "16000" }, "stddev": { "string_value": "2.2668, 3.1642, 3.7079, 3.7642, 3.5349, 3.5901, 3.7640, 3.8424, 4.0145, 4.1475, 4.0457, 3.9048, 3.7709, 3.6117, 3.3188, 3.1489, 3.0615, 3.0362, 2.9929, 3.0500, 3.0341, 3.0484, 3.0103, 2.9474, 2.9128, 2.8669, 2.8332, 2.9411, 3.0378, 3.0712, 3.0190, 2.9992, 3.0124, 3.0024, 3.0275, 3.0870, 3.0656, 3.0142, 3.0493, 3.1373, 3.1135, 3.0675, 2.8828, 2.7018, 2.6296, 2.8826, 2.9325, 2.9288, 2.9271, 2.9890, 3.0137, 2.9855, 3.0839, 2.9319, 2.3512, 2.3795, 2.6191, 2.7555, 2.9326, 2.9931, 3.1543, 3.0855, 2.6820, 3.0566, 3.1272, 3.1663, 3.1836, 3.0018, 2.9089, 3.1727, 3.1626, 3.1086, 2.9804, 3.1107, 3.2998, 3.3697, 3.3716, 3.2487, 3.1597, 3.1181" }, "precalc_norm_time_steps": { "string_value": "0" }, "left_padding_size": { "string_value": "1.6" }, "streaming": { "string_value": "True" }, "num_features": { "string_value": "80" }, "apply_normalization": { "string_value": "True" }, "dither": { "string_value": "0.0" }, "window_stride": { "string_value": "0.01" }, "chunk_size": { "string_value": "4.8" }, "max_execution_batch_size": { "string_value": "512" }, "stddev_floor": { "string_value": "1e-05" }, "gain": { "string_value": "1.0" } }, "model_warmup": [], "model_transaction_policy": { "decoupled": false } } I1216 03:05:15.720238 106 endpointing_library.cc:22] TRITONBACKEND_ModelInstanceInitialize: conformer-en-US-asr-streaming-endpointing-streaming_0 (device 0) I1216 03:05:15.764025 106 feature-extractor.cc:400] TRITONBACKEND_ModelInitialize: conformer-en-US-asr-streaming-feature-extractor-streaming (version 1) I1216 03:05:15.765227 106 model_lifecycle.cc:693] successfully loaded 'conformer-en-US-asr-streaming-endpointing-streaming' version 1 I1216 03:05:15.766110 106 backend_model.cc:303] model configuration: { "name": "conformer-en-US-asr-streaming-feature-extractor-streaming", "platform": "", "backend": "riva_asr_features", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 1024, "input": [ { "name": "AUDIO_SIGNAL", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "SAMPLE_RATE", "data_type": "TYPE_UINT32", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "AUDIO_FEATURES", "data_type": "TYPE_FP32", "dims": [ 80, -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "AUDIO_PROCESSED", "data_type": "TYPE_FP32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "AUDIO_FEATURES_LENGTH", "data_type": "TYPE_INT32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "graph": { "level": 0 }, "priority": "PRIORITY_DEFAULT", "cuda": { "graphs": false, "busy_wait_events": false, "graph_spec": [], "output_copy_stream": true }, "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "oldest": { "max_candidate_sequences": 1024, "preferred_batch_size": [ 256, 512 ], "max_queue_delay_microseconds": 1000 }, "max_sequence_idle_microseconds": 60000000, "control_input": [ { "name": "START", "control": [ { "kind": "CONTROL_SEQUENCE_START", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "READY", "control": [ { "kind": "CONTROL_SEQUENCE_READY", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "END", "control": [ { "kind": "CONTROL_SEQUENCE_END", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "CORRID", "control": [ { "kind": "CONTROL_SEQUENCE_CORRID", "int32_false_true": [], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_UINT64" } ] } ], "state": [] }, "instance_group": [ { "name": "conformer-en-US-asr-streaming-feature-extractor-streaming_0", "kind": "KIND_GPU", "count": 1, "gpus": [ 0 ], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "right_padding_size": { "string_value": "1.92" }, "use_utterance_norm_params": { "string_value": "False" }, "dither": { "string_value": "0.0" }, "stddev": { "string_value": "2.2668, 3.1642, 3.7079, 3.7642, 3.5349, 3.5901, 3.7640, 3.8424, 4.0145, 4.1475, 4.0457, 3.9048, 3.7709, 3.6117, 3.3188, 3.1489, 3.0615, 3.0362, 2.9929, 3.0500, 3.0341, 3.0484, 3.0103, 2.9474, 2.9128, 2.8669, 2.8332, 2.9411, 3.0378, 3.0712, 3.0190, 2.9992, 3.0124, 3.0024, 3.0275, 3.0870, 3.0656, 3.0142, 3.0493, 3.1373, 3.1135, 3.0675, 2.8828, 2.7018, 2.6296, 2.8826, 2.9325, 2.9288, 2.9271, 2.9890, 3.0137, 2.9855, 3.0839, 2.9319, 2.3512, 2.3795, 2.6191, 2.7555, 2.9326, 2.9931, 3.1543, 3.0855, 2.6820, 3.0566, 3.1272, 3.1663, 3.1836, 3.0018, 2.9089, 3.1727, 3.1626, 3.1086, 2.9804, 3.1107, 3.2998, 3.3697, 3.3716, 3.2487, 3.1597, 3.1181" }, "transpose": { "string_value": "False" }, "left_padding_size": { "string_value": "1.92" }, "streaming": { "string_value": "True" }, "mean": { "string_value": "-11.4412, -9.9334, -9.1292, -9.0365, -9.2804, -9.5643, -9.7342, -9.6925, -9.6333, -9.2808, -9.1887, -9.1422, -9.1397, -9.2028, -9.2749, -9.4776, -9.9185, -10.1557, -10.3800, -10.5067, -10.3190, -10.4728, -10.5529, -10.6402, -10.6440, -10.5113, -10.7395, -10.7870, -10.6074, -10.5033, -10.8278, -10.6384, -10.8481, -10.6875, -10.5454, -10.4747, -10.5165, -10.4930, -10.3413, -10.3472, -10.3735, -10.6830, -10.8813, -10.6338, -10.3856, -10.7727, -10.8957, -10.8068, -10.7373, -10.6108, -10.3405, -10.2889, -10.3922, -10.4946, -10.3367, -10.4164, -10.9949, -10.7196, -10.3971, -10.1734, -9.9257, -9.6557, -9.1761, -9.6653, -9.7876, -9.7230, -9.7792, -9.7056, -9.2702, -9.4650, -9.2755, -9.1369, -9.1174, -8.9197, -8.5394, -8.2614, -8.1353, -8.1422, -8.3430, -8.6655" }, "gain": { "string_value": "1.0" }, "chunk_size": { "string_value": "0.16" }, "norm_per_feature": { "string_value": "True" }, "num_features": { "string_value": "80" }, "precalc_norm_time_steps": { "string_value": "0" }, "window_size": { "string_value": "0.025" }, "precalc_norm_params": { "string_value": "False" }, "window_stride": { "string_value": "0.01" }, "apply_normalization": { "string_value": "True" }, "stddev_floor": { "string_value": "1e-05" }, "sample_rate": { "string_value": "16000" }, "max_execution_batch_size": { "string_value": "1024" } }, "model_warmup": [], "model_transaction_policy": { "decoupled": false } } I1216 03:05:15.766208 106 ctc-decoder-library.cc:20] TRITONBACKEND_ModelInitialize: conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming (version 1) WARNING: Logging before InitGoogleLogging() is written to STDERR W1216 03:05:15.768709 117 parameter_parser.cc:144] Parameter 'append_space_to_transcripts' set but unused. W1216 03:05:15.768728 117 parameter_parser.cc:144] Parameter 'beam_size' set but unused. W1216 03:05:15.768733 117 parameter_parser.cc:144] Parameter 'beam_size_token' set but unused. W1216 03:05:15.768736 117 parameter_parser.cc:144] Parameter 'beam_threshold' set but unused. W1216 03:05:15.768740 117 parameter_parser.cc:144] Parameter 'blank_token' set but unused. W1216 03:05:15.768745 117 parameter_parser.cc:144] Parameter 'decoder_num_worker_threads' set but unused. W1216 03:05:15.768749 117 parameter_parser.cc:144] Parameter 'forerunner_beam_size' set but unused. W1216 03:05:15.768754 117 parameter_parser.cc:144] Parameter 'forerunner_beam_size_token' set but unused. W1216 03:05:15.768757 117 parameter_parser.cc:144] Parameter 'forerunner_beam_threshold' set but unused. W1216 03:05:15.768761 117 parameter_parser.cc:144] Parameter 'forerunner_use_lm' set but unused. W1216 03:05:15.768765 117 parameter_parser.cc:144] Parameter 'language_model_file' set but unused. W1216 03:05:15.768770 117 parameter_parser.cc:144] Parameter 'lexicon_file' set but unused. W1216 03:05:15.768774 117 parameter_parser.cc:144] Parameter 'lm_weight' set but unused. W1216 03:05:15.768779 117 parameter_parser.cc:144] Parameter 'log_add' set but unused. W1216 03:05:15.768783 117 parameter_parser.cc:144] Parameter 'max_execution_batch_size' set but unused. W1216 03:05:15.768790 117 parameter_parser.cc:144] Parameter 'max_supported_transcripts' set but unused. W1216 03:05:15.768793 117 parameter_parser.cc:144] Parameter 'num_tokenization' set but unused. W1216 03:05:15.768797 117 parameter_parser.cc:144] Parameter 'profane_words_file' set but unused. W1216 03:05:15.768802 117 parameter_parser.cc:144] Parameter 'set_default_index_to_unk_token' set but unused. W1216 03:05:15.768806 117 parameter_parser.cc:144] Parameter 'sil_token' set but unused. W1216 03:05:15.768812 117 parameter_parser.cc:144] Parameter 'smearing_mode' set but unused. W1216 03:05:15.768817 117 parameter_parser.cc:144] Parameter 'tokenizer_model' set but unused. W1216 03:05:15.768821 117 parameter_parser.cc:144] Parameter 'unk_score' set but unused. W1216 03:05:15.768826 117 parameter_parser.cc:144] Parameter 'unk_token' set but unused. W1216 03:05:15.768831 117 parameter_parser.cc:144] Parameter 'vocab_file' set but unused. W1216 03:05:15.768836 117 parameter_parser.cc:144] Parameter 'word_insertion_score' set but unused. I1216 03:05:15.769769 106 backend_model.cc:303] model configuration: { "name": "conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming", "platform": "", "backend": "riva_asr_decoder", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 1024, "input": [ { "name": "CLASS_LOGITS", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1, 257 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "END_FLAG", "data_type": "TYPE_UINT32", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "SEGMENTS_START_END", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1, 2 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "CUSTOM_CONFIGURATION", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ -1, 2 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "FINAL_TRANSCRIPTS", "data_type": "TYPE_STRING", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "FINAL_TRANSCRIPTS_SCORE", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "FINAL_WORDS_START_END", "data_type": "TYPE_INT32", "dims": [ -1, 2 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PARTIAL_TRANSCRIPTS", "data_type": "TYPE_STRING", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PARTIAL_TRANSCRIPTS_STABILITY", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PARTIAL_WORDS_START_END", "data_type": "TYPE_INT32", "dims": [ -1, 2 ], "label_filename": "", "is_shape_tensor": false }, { "name": "FINAL_WORDS_SCORE", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PARTIAL_WORDS_SCORE", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "graph": { "level": 0 }, "priority": "PRIORITY_DEFAULT", "cuda": { "graphs": false, "busy_wait_events": false, "graph_spec": [], "output_copy_stream": true }, "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "oldest": { "max_candidate_sequences": 1024, "preferred_batch_size": [ 32, 64 ], "max_queue_delay_microseconds": 1000 }, "max_sequence_idle_microseconds": 60000000, "control_input": [ { "name": "START", "control": [ { "kind": "CONTROL_SEQUENCE_START", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "READY", "control": [ { "kind": "CONTROL_SEQUENCE_READY", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "END", "control": [ { "kind": "CONTROL_SEQUENCE_END", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "CORRID", "control": [ { "kind": "CONTROL_SEQUENCE_CORRID", "int32_false_true": [], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_UINT64" } ] } ], "state": [] }, "instance_group": [ { "name": "conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "streaming": { "string_value": "True" }, "ms_per_timestep": { "string_value": "40" }, "log_add": { "string_value": "True" }, "smearing_mode": { "string_value": "max" }, "forerunner_beam_size_token": { "string_value": "8" }, "beam_threshold": { "string_value": "20.0" }, "max_execution_batch_size": { "string_value": "1024" }, "set_default_index_to_unk_token": { "string_value": "False" }, "sil_token": { "string_value": "▁" }, "forerunner_beam_size": { "string_value": "8" }, "left_padding_size": { "string_value": "1.92" }, "forerunner_beam_threshold": { "string_value": "10.0" }, "profane_words_file": { "string_value": "/data/models/conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/profane_words_file.txt" }, "decoder_type": { "string_value": "flashlight" }, "word_insertion_score": { "string_value": "1.0" }, "right_padding_size": { "string_value": "1.92" }, "beam_size": { "string_value": "32" }, "forerunner_use_lm": { "string_value": "true" }, "chunk_size": { "string_value": "0.16" }, "unk_token": { "string_value": "" }, "vocab_file": { "string_value": "/data/models/conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/riva_decoder_vocabulary.txt" }, "decoder_num_worker_threads": { "string_value": "-1" }, "tokenizer_model": { "string_value": "/data/models/conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/8b8f095152034e98b24ab33726708bd0_tokenizer.model" }, "lexicon_file": { "string_value": "/data/models/conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/lexicon.txt" }, "use_subword": { "string_value": "True" }, "num_tokenization": { "string_value": "1" }, "beam_size_token": { "string_value": "16" }, "lm_weight": { "string_value": "0.8" }, "append_space_to_transcripts": { "string_value": "True" }, "language_model_file": { "string_value": "/data/models/conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/4gram-pruned-0_2_7_9-en-lm-set-2.0.bin" }, "asr_model_delay": { "string_value": "-1" }, "max_supported_transcripts": { "string_value": "1" }, "blank_token": { "string_value": "#" }, "unk_score": { "string_value": "-inf" } }, "model_warmup": [], "model_transaction_policy": { "decoupled": false } } I1216 03:05:15.769840 106 endpointing_library.cc:18] TRITONBACKEND_ModelInitialize: conformer-en-US-asr-offline-endpointing-streaming-offline (version 1) W1216 03:05:15.771062 114 parameter_parser.cc:144] Parameter 'chunk_size' set but unused. W1216 03:05:15.771078 114 parameter_parser.cc:144] Parameter 'ms_per_timestep' set but unused. W1216 03:05:15.771083 114 parameter_parser.cc:144] Parameter 'residue_blanks_at_end' set but unused. W1216 03:05:15.771087 114 parameter_parser.cc:144] Parameter 'residue_blanks_at_start' set but unused. W1216 03:05:15.771091 114 parameter_parser.cc:144] Parameter 'start_history' set but unused. W1216 03:05:15.771095 114 parameter_parser.cc:144] Parameter 'start_th' set but unused. W1216 03:05:15.771099 114 parameter_parser.cc:144] Parameter 'stop_history' set but unused. W1216 03:05:15.771103 114 parameter_parser.cc:144] Parameter 'stop_th' set but unused. W1216 03:05:15.771107 114 parameter_parser.cc:144] Parameter 'streaming' set but unused. W1216 03:05:15.771133 114 parameter_parser.cc:144] Parameter 'use_subword' set but unused. W1216 03:05:15.771137 114 parameter_parser.cc:144] Parameter 'vocab_file' set but unused. I1216 03:05:15.771625 106 backend_model.cc:303] model configuration: { "name": "conformer-en-US-asr-offline-endpointing-streaming-offline", "platform": "", "backend": "riva_asr_endpointing", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 2048, "input": [ { "name": "CLASS_LOGITS", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1, 257 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "SEGMENTS_START_END", "data_type": "TYPE_FP32", "dims": [ -1, 2 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "graph": { "level": 0 }, "priority": "PRIORITY_DEFAULT", "cuda": { "graphs": false, "busy_wait_events": false, "graph_spec": [], "output_copy_stream": true }, "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "max_sequence_idle_microseconds": 60000000, "control_input": [ { "name": "START", "control": [ { "kind": "CONTROL_SEQUENCE_START", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "READY", "control": [ { "kind": "CONTROL_SEQUENCE_READY", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] } ], "state": [] }, "instance_group": [ { "name": "conformer-en-US-asr-offline-endpointing-streaming-offline_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "residue_blanks_at_start": { "string_value": "-2" }, "chunk_size": { "string_value": "4.8" }, "stop_history": { "string_value": "800" }, "use_subword": { "string_value": "True" }, "vocab_file": { "string_value": "/data/models/conformer-en-US-asr-offline-endpointing-streaming-offline/1/riva_decoder_vocabulary.txt" }, "residue_blanks_at_end": { "string_value": "0" }, "streaming": { "string_value": "True" }, "start_th": { "string_value": "0.2" }, "start_history": { "string_value": "200" }, "ms_per_timestep": { "string_value": "40" }, "stop_th": { "string_value": "0.98" }, "endpointing_type": { "string_value": "greedy_ctc" } }, "model_warmup": [], "model_transaction_policy": { "decoupled": false } } I1216 03:05:15.771700 106 ctc-decoder-library.cc:20] TRITONBACKEND_ModelInitialize: conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline (version 1) W1216 03:05:15.773428 113 parameter_parser.cc:144] Parameter 'append_space_to_transcripts' set but unused. W1216 03:05:15.773442 113 parameter_parser.cc:144] Parameter 'beam_size' set but unused. W1216 03:05:15.773447 113 parameter_parser.cc:144] Parameter 'beam_size_token' set but unused. W1216 03:05:15.773452 113 parameter_parser.cc:144] Parameter 'beam_threshold' set but unused. W1216 03:05:15.773456 113 parameter_parser.cc:144] Parameter 'blank_token' set but unused. W1216 03:05:15.773460 113 parameter_parser.cc:144] Parameter 'decoder_num_worker_threads' set but unused. W1216 03:05:15.773465 113 parameter_parser.cc:144] Parameter 'forerunner_beam_size' set but unused. W1216 03:05:15.773469 113 parameter_parser.cc:144] Parameter 'forerunner_beam_size_token' set but unused. W1216 03:05:15.773473 113 parameter_parser.cc:144] Parameter 'forerunner_beam_threshold' set but unused. W1216 03:05:15.773478 113 parameter_parser.cc:144] Parameter 'forerunner_use_lm' set but unused. W1216 03:05:15.773481 113 parameter_parser.cc:144] Parameter 'language_model_file' set but unused. W1216 03:05:15.773486 113 parameter_parser.cc:144] Parameter 'lexicon_file' set but unused. W1216 03:05:15.773490 113 parameter_parser.cc:144] Parameter 'lm_weight' set but unused. W1216 03:05:15.773499 113 parameter_parser.cc:144] Parameter 'log_add' set but unused. W1216 03:05:15.773504 113 parameter_parser.cc:144] Parameter 'max_execution_batch_size' set but unused. W1216 03:05:15.773510 113 parameter_parser.cc:144] Parameter 'max_supported_transcripts' set but unused. W1216 03:05:15.773514 113 parameter_parser.cc:144] Parameter 'num_tokenization' set but unused. W1216 03:05:15.773520 113 parameter_parser.cc:144] Parameter 'profane_words_file' set but unused. W1216 03:05:15.773525 113 parameter_parser.cc:144] Parameter 'set_default_index_to_unk_token' set but unused. W1216 03:05:15.773531 113 parameter_parser.cc:144] Parameter 'sil_token' set but unused. W1216 03:05:15.773535 113 parameter_parser.cc:144] Parameter 'smearing_mode' set but unused. W1216 03:05:15.773541 113 parameter_parser.cc:144] Parameter 'tokenizer_model' set but unused. W1216 03:05:15.773546 113 parameter_parser.cc:144] Parameter 'unk_score' set but unused. W1216 03:05:15.773551 113 parameter_parser.cc:144] Parameter 'unk_token' set but unused. W1216 03:05:15.773556 113 parameter_parser.cc:144] Parameter 'vocab_file' set but unused. W1216 03:05:15.773561 113 parameter_parser.cc:144] Parameter 'word_insertion_score' set but unused. I1216 03:05:15.774469 106 backend_model.cc:303] model configuration: { "name": "conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline", "platform": "", "backend": "riva_asr_decoder", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 1024, "input": [ { "name": "CLASS_LOGITS", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1, 257 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "END_FLAG", "data_type": "TYPE_UINT32", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "SEGMENTS_START_END", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ -1, 2 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "CUSTOM_CONFIGURATION", "data_type": "TYPE_STRING", "format": "FORMAT_NONE", "dims": [ -1, 2 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "FINAL_TRANSCRIPTS", "data_type": "TYPE_STRING", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "FINAL_TRANSCRIPTS_SCORE", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "FINAL_WORDS_START_END", "data_type": "TYPE_INT32", "dims": [ -1, 2 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PARTIAL_TRANSCRIPTS", "data_type": "TYPE_STRING", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PARTIAL_TRANSCRIPTS_STABILITY", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PARTIAL_WORDS_START_END", "data_type": "TYPE_INT32", "dims": [ -1, 2 ], "label_filename": "", "is_shape_tensor": false }, { "name": "FINAL_WORDS_SCORE", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "PARTIAL_WORDS_SCORE", "data_type": "TYPE_FP32", "dims": [ -1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "graph": { "level": 0 }, "priority": "PRIORITY_DEFAULT", "cuda": { "graphs": false, "busy_wait_events": false, "graph_spec": [], "output_copy_stream": true }, "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "oldest": { "max_candidate_sequences": 1024, "preferred_batch_size": [ 32, 64 ], "max_queue_delay_microseconds": 1000 }, "max_sequence_idle_microseconds": 60000000, "control_input": [ { "name": "START", "control": [ { "kind": "CONTROL_SEQUENCE_START", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "READY", "control": [ { "kind": "CONTROL_SEQUENCE_READY", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "END", "control": [ { "kind": "CONTROL_SEQUENCE_END", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "CORRID", "control": [ { "kind": "CONTROL_SEQUENCE_CORRID", "int32_false_true": [], "fp32_false_true": [], "bool_false_true": [], "data_type": "TYPE_UINT64" } ] } ], "state": [] }, "instance_group": [ { "name": "conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline_0", "kind": "KIND_CPU", "count": 1, "gpus": [], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "smearing_mode": { "string_value": "max" }, "use_subword": { "string_value": "True" }, "ms_per_timestep": { "string_value": "40" }, "decoder_type": { "string_value": "flashlight" }, "num_tokenization": { "string_value": "1" }, "set_default_index_to_unk_token": { "string_value": "False" }, "streaming": { "string_value": "True" }, "tokenizer_model": { "string_value": "/data/models/conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline/1/8b8f095152034e98b24ab33726708bd0_tokenizer.model" }, "unk_score": { "string_value": "-inf" }, "max_execution_batch_size": { "string_value": "1024" }, "chunk_size": { "string_value": "4.8" }, "forerunner_beam_size": { "string_value": "8" }, "left_padding_size": { "string_value": "1.6" }, "language_model_file": { "string_value": "/data/models/conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline/1/4gram-pruned-0_2_7_9-en-lm-set-2.0.bin" }, "append_space_to_transcripts": { "string_value": "True" }, "sil_token": { "string_value": "▁" }, "lm_weight": { "string_value": "0.8" }, "vocab_file": { "string_value": "/data/models/conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline/1/riva_decoder_vocabulary.txt" }, "unk_token": { "string_value": "" }, "beam_size_token": { "string_value": "16" }, "forerunner_beam_size_token": { "string_value": "8" }, "forerunner_beam_threshold": { "string_value": "10.0" }, "forerunner_use_lm": { "string_value": "true" }, "decoder_num_worker_threads": { "string_value": "-1" }, "max_supported_transcripts": { "string_value": "1" }, "beam_size": { "string_value": "32" }, "blank_token": { "string_value": "#" }, "lexicon_file": { "string_value": "/data/models/conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline/1/lexicon.txt" }, "log_add": { "string_value": "True" }, "asr_model_delay": { "string_value": "-1" }, "profane_words_file": { "string_value": "/data/models/conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline/1/profane_words_file.txt" }, "beam_threshold": { "string_value": "20.0" }, "word_insertion_score": { "string_value": "1.0" }, "right_padding_size": { "string_value": "1.6" } }, "model_warmup": [], "model_transaction_policy": { "decoupled": false } } I1216 03:05:15.774547 106 feature-extractor.cc:402] TRITONBACKEND_ModelInstanceInitialize: conformer-en-US-asr-offline-feature-extractor-streaming-offline_0 (device 0) > Riva waiting for Triton server to load all models...retrying in 1 second I1216 03:05:16.546288 106 feature-extractor.cc:402] TRITONBACKEND_ModelInstanceInitialize: conformer-en-US-asr-streaming-feature-extractor-streaming_0 (device 0) I1216 03:05:16.546552 106 model_lifecycle.cc:693] successfully loaded 'conformer-en-US-asr-offline-feature-extractor-streaming-offline' version 1 I1216 03:05:16.554198 106 ctc-decoder-library.cc:23] TRITONBACKEND_ModelInstanceInitialize: conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming_0 (device 0) I1216 03:05:16.555876 106 model_lifecycle.cc:693] successfully loaded 'conformer-en-US-asr-streaming-feature-extractor-streaming' version 1 > Riva waiting for Triton server to load all models...retrying in 1 second I1216 03:05:18.039964 117 ctc-decoder.cc:174] Beam Decoder initialized successfully! I1216 03:05:18.040026 106 endpointing_library.cc:22] TRITONBACKEND_ModelInstanceInitialize: conformer-en-US-asr-offline-endpointing-streaming-offline_0 (device 0) I1216 03:05:18.040257 106 model_lifecycle.cc:693] successfully loaded 'conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming' version 1 I1216 03:05:18.082392 106 ctc-decoder-library.cc:23] TRITONBACKEND_ModelInstanceInitialize: conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline_0 (device 0) I1216 03:05:18.083337 106 model_lifecycle.cc:693] successfully loaded 'conformer-en-US-asr-offline-endpointing-streaming-offline' version 1 > Riva waiting for Triton server to load all models...retrying in 1 second > Riva waiting for Triton server to load all models...retrying in 1 second I1216 03:05:19.532558 113 ctc-decoder.cc:174] Beam Decoder initialized successfully! I1216 03:05:19.532900 106 model_lifecycle.cc:693] successfully loaded 'conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline' version 1 E1216 03:05:19.532944 106 model_repository_manager.cc:481] Invalid argument: ensemble 'conformer-en-US-asr-offline' depends on 'riva-trt-conformer-en-US-asr-offline-am-streaming-offline' which has no loaded version E1216 03:05:19.532951 106 model_repository_manager.cc:481] Invalid argument: ensemble 'conformer-en-US-asr-streaming' depends on 'riva-trt-conformer-en-US-asr-streaming-am-streaming' which has no loaded version I1216 03:05:19.532979 106 server.cc:563] +------------------+------+ | Repository Agent | Path | +------------------+------+ +------------------+------+ I1216 03:05:19.533017 106 server.cc:590] +----------------------+-----------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Backend | Path | Config | +----------------------+-----------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+ | tensorrt | /opt/tritonserver/backends/tensorrt/libtriton_tensorrt.so | {"cmdline":{"auto-complete-config":"false","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} | | riva_asr_decoder | /opt/tritonserver/backends/riva_asr_decoder/libtriton_riva_asr_decoder.so | {"cmdline":{"auto-complete-config":"false","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} | | riva_asr_endpointing | /opt/tritonserver/backends/riva_asr_endpointing/libtriton_riva_asr_endpointing.so | {"cmdline":{"auto-complete-config":"false","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} | | riva_nlp_pipeline | /opt/tritonserver/backends/riva_nlp_pipeline/libtriton_riva_nlp_pipeline.so | {"cmdline":{"auto-complete-config":"false","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} | | riva_asr_features | /opt/tritonserver/backends/riva_asr_features/libtriton_riva_asr_features.so | {"cmdline":{"auto-complete-config":"false","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} | +----------------------+-----------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+ I1216 03:05:19.533069 106 server.cc:633] +-----------------------------------------------------------------+---------+--------------------------------------------------------------------------------------------------------------+ | Model | Version | Status | +-----------------------------------------------------------------+---------+--------------------------------------------------------------------------------------------------------------+ | conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline | 1 | READY | | conformer-en-US-asr-offline-endpointing-streaming-offline | 1 | READY | | conformer-en-US-asr-offline-feature-extractor-streaming-offline | 1 | READY | | conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming | 1 | READY | | conformer-en-US-asr-streaming-endpointing-streaming | 1 | READY | | conformer-en-US-asr-streaming-feature-extractor-streaming | 1 | READY | | riva-punctuation-en-US | 1 | READY | | riva-trt-conformer-en-US-asr-offline-am-streaming-offline | 1 | UNAVAILABLE: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. | | riva-trt-conformer-en-US-asr-streaming-am-streaming | 1 | UNAVAILABLE: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. | | riva-trt-riva-punctuation-en-US-nn-bert-base-uncased | 1 | UNAVAILABLE: Internal: unable to create stream: the provided PTX was compiled with an unsupported toolchain. | +-----------------------------------------------------------------+---------+--------------------------------------------------------------------------------------------------------------+ I1216 03:05:19.582841 106 metrics.cc:864] Collecting metrics for GPU 0: NVIDIA GeForce RTX 3080 I1216 03:05:19.583179 106 metrics.cc:757] Collecting CPU metrics I1216 03:05:19.583746 106 tritonserver.cc:2264] +----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Option | Value | +----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | server_id | triton | | server_version | 2.27.0 | | server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace logging | | model_repository_path[0] | /data/models | | model_control_mode | MODE_NONE | | strict_model_config | 1 | | rate_limit | OFF | | pinned_memory_pool_byte_size | 268435456 | | cuda_memory_pool_byte_size{0} | 1000000000 | | response_cache_byte_size | 0 | | min_supported_compute_capability | 6.0 | | strict_readiness | 1 | | exit_timeout | 30 | +----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ I1216 03:05:19.583764 106 server.cc:264] Waiting for in-flight requests to complete. I1216 03:05:19.583789 106 server.cc:280] Timeout 30: Found 0 model versions that have in-flight inferences I1216 03:05:19.584256 106 endpointing_library.cc:26] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1216 03:05:19.584443 106 feature-extractor.cc:404] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1216 03:05:19.584488 106 pipeline_library.cc:28] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1216 03:05:19.584820 106 feature-extractor.cc:404] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1216 03:05:19.584941 106 ctc-decoder-library.cc:25] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1216 03:05:19.584996 106 server.cc:295] All models are stopped, unloading models I1216 03:05:19.585104 106 server.cc:302] Timeout 30: Found 7 live models and 0 in-flight non-inference requests I1216 03:05:19.587159 106 ctc-decoder-library.cc:25] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1216 03:05:19.587194 106 endpointing_library.cc:26] TRITONBACKEND_ModelInstanceFinalize: delete instance state I1216 03:05:19.590167 106 endpointing_library.cc:21] TRITONBACKEND_ModelFinalize: delete model state I1216 03:05:19.591618 106 model_lifecycle.cc:578] successfully unloaded 'conformer-en-US-asr-streaming-endpointing-streaming' version 1 I1216 03:05:19.591644 106 pipeline_library.cc:24] TRITONBACKEND_ModelFinalize: delete model state I1216 03:05:19.593998 106 endpointing_library.cc:21] TRITONBACKEND_ModelFinalize: delete model state I1216 03:05:19.594732 106 model_lifecycle.cc:578] successfully unloaded 'conformer-en-US-asr-offline-endpointing-streaming-offline' version 1 I1216 03:05:19.595233 106 model_lifecycle.cc:578] successfully unloaded 'riva-punctuation-en-US' version 1 I1216 03:05:19.650214 106 feature-extractor.cc:401] TRITONBACKEND_ModelFinalize: delete model state I1216 03:05:19.652414 106 feature-extractor.cc:401] TRITONBACKEND_ModelFinalize: delete model state I1216 03:05:19.652680 106 model_lifecycle.cc:578] successfully unloaded 'conformer-en-US-asr-offline-feature-extractor-streaming-offline' version 1 I1216 03:05:19.654732 106 model_lifecycle.cc:578] successfully unloaded 'conformer-en-US-asr-streaming-feature-extractor-streaming' version 1 I1216 03:05:20.139884 106 ctc-decoder-library.cc:22] TRITONBACKEND_ModelFinalize: delete model state I1216 03:05:20.140267 106 model_lifecycle.cc:578] successfully unloaded 'conformer-en-US-asr-offline-ctc-decoder-cpu-streaming-offline' version 1 I1216 03:05:20.152423 106 ctc-decoder-library.cc:22] TRITONBACKEND_ModelFinalize: delete model state I1216 03:05:20.152733 106 model_lifecycle.cc:578] successfully unloaded 'conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming' version 1 > Riva waiting for Triton server to load all models...retrying in 1 second I1216 03:05:20.585236 106 server.cc:302] Timeout 29: Found 0 live models and 0 in-flight non-inference requests error: creating server: Internal - failed to load all models > Riva waiting for Triton server to load all models...retrying in 1 second > Riva waiting for Triton server to load all models...retrying in 1 second > Triton server died before reaching ready state. Terminating Riva startup. Check Triton logs with: docker logs /opt/riva/bin/start-riva: line 1: kill: (106) - No such process