========================== === Riva Speech Skills === ========================== NVIDIA Release 21.12 (build 30304767) Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved. Various files include modifications (c) NVIDIA CORPORATION. All rights reserved. NVIDIA modifications are covered by the license terms that apply to the underlying project or file. NOTE: Legacy NVIDIA Driver detected. Compatibility mode ENABLED. > Riva waiting for Triton server to load all models...retrying in 1 second I0104 13:46:40.491953 73 metrics.cc:290] Collecting metrics for GPU 0: Tesla V100-DGXS-32GB I0104 13:46:40.495267 73 onnxruntime.cc:1970] TRITONBACKEND_Initialize: onnxruntime I0104 13:46:40.495298 73 onnxruntime.cc:1980] Triton TRITONBACKEND API version: 1.4 I0104 13:46:40.495305 73 onnxruntime.cc:1986] 'onnxruntime' TRITONBACKEND API version: 1.4 I0104 13:46:40.775613 73 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7fee8c000000' with size 268435456 I0104 13:46:40.776524 73 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 1000000000 E0104 13:46:40.781316 73 model_repository_manager.cc:276] Failed to determine modification time for '/data/models/citrinet-1024-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming/1/libtriton_riva_asr_decoder_cpu.so': Internal: failed to stat file /data/models/citrinet-1024-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming/1/libtriton_riva_asr_decoder_cpu.so E0104 13:46:40.784711 73 model_repository_manager.cc:1919] Poll failed for model directory 'citrinet-1024-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming': unexpected platform type 'custom' for citrinet-1024-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming E0104 13:46:40.785484 73 model_repository_manager.cc:1919] Poll failed for model directory 'citrinet-1024-asr-trt-ensemble-vad-streaming-feature-extractor-streaming': unexpected platform type 'custom' for citrinet-1024-asr-trt-ensemble-vad-streaming-feature-extractor-streaming E0104 13:46:40.785924 73 model_repository_manager.cc:276] Failed to determine modification time for '/data/models/citrinet-1024-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline/1/libtriton_riva_asr_decoder_cpu.so': Internal: failed to stat file /data/models/citrinet-1024-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline/1/libtriton_riva_asr_decoder_cpu.so E0104 13:46:40.786628 73 model_repository_manager.cc:1919] Poll failed for model directory 'citrinet-1024-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline': unexpected platform type 'custom' for citrinet-1024-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline E0104 13:46:40.787357 73 model_repository_manager.cc:1919] Poll failed for model directory 'citrinet-1024-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline': unexpected platform type 'custom' for citrinet-1024-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline E0104 13:46:40.788018 73 model_repository_manager.cc:1919] Poll failed for model directory 'citrinet-1024-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline': unexpected platform type 'custom' for citrinet-1024-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline E0104 13:46:40.788683 73 model_repository_manager.cc:1919] Poll failed for model directory 'citrinet-1024-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming': unexpected platform type 'custom' for citrinet-1024-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming E0104 13:46:40.788761 73 model_repository_manager.cc:276] Failed to determine modification time for '/data/models/denoiser/1/libtriton_riva_tts_denoiser.so': Internal: failed to stat file /data/models/denoiser/1/libtriton_riva_tts_denoiser.so E0104 13:46:40.789371 73 model_repository_manager.cc:1919] Poll failed for model directory 'denoiser': unexpected platform type 'custom' for denoiser E0104 13:46:40.797997 73 model_repository_manager.cc:1919] Poll failed for model directory 'riva_detokenize': unexpected platform type 'custom' for riva_detokenize E0104 13:46:40.798927 73 model_repository_manager.cc:1919] Poll failed for model directory 'riva_label_tokens_weather': unexpected platform type 'custom' for riva_label_tokens_weather E0104 13:46:40.799803 73 model_repository_manager.cc:1919] Poll failed for model directory 'riva_ner_label_tokens': unexpected platform type 'custom' for riva_ner_label_tokens E0104 13:46:40.800219 73 model_repository_manager.cc:276] Failed to determine modification time for '/data/models/riva_punctuation_gen_output/1/libtriton_riva_nlp_punctuation.so': Internal: failed to stat file /data/models/riva_punctuation_gen_output/1/libtriton_riva_nlp_punctuation.so E0104 13:46:40.800770 73 model_repository_manager.cc:1919] Poll failed for model directory 'riva_punctuation_gen_output': unexpected platform type 'custom' for riva_punctuation_gen_output E0104 13:46:40.801349 73 model_repository_manager.cc:1919] Poll failed for model directory 'riva_punctuation_label_tokens_cap': unexpected platform type 'custom' for riva_punctuation_label_tokens_cap E0104 13:46:40.802226 73 model_repository_manager.cc:1919] Poll failed for model directory 'riva_punctuation_label_tokens_punct': unexpected platform type 'custom' for riva_punctuation_label_tokens_punct E0104 13:46:40.802304 73 model_repository_manager.cc:276] Failed to determine modification time for '/data/models/riva_punctuation_merge_labels/1/libtriton_riva_nlp_labels.so': Internal: failed to stat file /data/models/riva_punctuation_merge_labels/1/libtriton_riva_nlp_labels.so E0104 13:46:40.803321 73 model_repository_manager.cc:1919] Poll failed for model directory 'riva_punctuation_merge_labels': unexpected platform type 'custom' for riva_punctuation_merge_labels E0104 13:46:40.804228 73 model_repository_manager.cc:1919] Poll failed for model directory 'riva_qa_postprocessor': unexpected platform type 'custom' for riva_qa_postprocessor E0104 13:46:40.804973 73 model_repository_manager.cc:1919] Poll failed for model directory 'riva_qa_preprocessor': unexpected platform type 'custom' for riva_qa_preprocessor E0104 13:46:40.806222 73 model_repository_manager.cc:1919] Poll failed for model directory 'riva_tokenizer': unexpected platform type 'custom' for riva_tokenizer E0104 13:46:40.810234 73 model_repository_manager.cc:1404] Invalid argument: ensemble citrinet-1024-asr-trt-ensemble-vad-streaming-offline contains models that are not available: citrinet-1024-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline, citrinet-1024-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline, citrinet-1024-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline E0104 13:46:40.810252 73 model_repository_manager.cc:1404] Invalid argument: ensemble citrinet-1024-asr-trt-ensemble-vad-streaming contains models that are not available: citrinet-1024-asr-trt-ensemble-vad-streaming-feature-extractor-streaming, citrinet-1024-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming, citrinet-1024-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming E0104 13:46:40.810257 73 model_repository_manager.cc:1404] Invalid argument: ensemble fastpitch_hifigan_ensemble contains models that are not available: denoiser E0104 13:46:40.810265 73 model_repository_manager.cc:1404] Invalid argument: ensemble riva_intent_weather contains models that are not available: riva_tokenizer, riva_label_tokens_weather, riva_detokenize E0104 13:46:40.810271 73 model_repository_manager.cc:1404] Invalid argument: ensemble riva_ner contains models that are not available: riva_ner_label_tokens E0104 13:46:40.810277 73 model_repository_manager.cc:1404] Invalid argument: ensemble riva_punctuation contains models that are not available: riva_punctuation_label_tokens_punct, riva_punctuation_label_tokens_cap, riva_punctuation_merge_labels, riva_punctuation_gen_output E0104 13:46:40.810282 73 model_repository_manager.cc:1404] Invalid argument: ensemble riva_qa contains models that are not available: riva_qa_preprocessor, riva_qa_postprocessor I0104 13:46:40.810335 73 model_repository_manager.cc:1045] loading: riva-onnx-fastpitch_encoder:1 I0104 13:46:40.910651 73 model_repository_manager.cc:1045] loading: riva-trt-citrinet-1024:1 I0104 13:46:40.911530 73 onnxruntime.cc:2029] TRITONBACKEND_ModelInitialize: riva-onnx-fastpitch_encoder (version 1) I0104 13:46:40.918151 73 onnxruntime.cc:2072] TRITONBACKEND_ModelInstanceInitialize: riva-onnx-fastpitch_encoder_0 (GPU device 0) I0104 13:46:41.011019 73 model_repository_manager.cc:1045] loading: riva-trt-hifigan:1 I0104 13:46:41.111400 73 model_repository_manager.cc:1045] loading: riva-trt-riva_intent_weather-nn-bert-base-uncased:1 I0104 13:46:41.211794 73 model_repository_manager.cc:1045] loading: riva-trt-riva_ner-nn-bert-base-uncased:1 I0104 13:46:41.312231 73 model_repository_manager.cc:1045] loading: riva-trt-riva_punctuation-nn-bert-base-uncased:1 I0104 13:46:41.412541 73 model_repository_manager.cc:1045] loading: riva-trt-riva_qa-nn-bert-base-uncased:1 > Riva waiting for Triton server to load all models...retrying in 1 second I0104 13:46:41.512956 73 model_repository_manager.cc:1045] loading: riva-trt-riva_text_classification_domain-nn-bert-base-uncased:1 I0104 13:46:41.613367 73 model_repository_manager.cc:1045] loading: spectrogram_chunker:1 I0104 13:46:41.717221 73 model_repository_manager.cc:1045] loading: tts_preprocessor:1 I0104 13:46:42.332382 73 logging.cc:49] [MemUsageChange] Init CUDA: CPU +1351, GPU +0, now: CPU 1928, GPU 1492 (MiB) I0104 13:46:42.334231 73 logging.cc:49] Loaded engine size: 31 MB I0104 13:46:42.334337 73 logging.cc:49] [MemUsageSnapshot] deserializeCudaEngine begin: CPU 1928 MiB, GPU 1492 MiB E0104 13:46:42.348404 73 logging.cc:43] 1: [stdArchiveReader.cpp::StdArchiveReader::34] Error Code 1: Serialization (Serialization assertion safeVersionRead == safeSerializationVersion failed.Version tag does not match. Note: Current Version: 43, Serialized Engine Version: 96) E0104 13:46:42.348453 73 logging.cc:43] 4: [runtime.cpp::deserializeCudaEngine::75] Error Code 4: Internal Error (Engine deserialization failed.) I0104 13:46:42.349644 73 logging.cc:49] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 1900, GPU 1492 (MiB) I0104 13:46:42.349694 73 logging.cc:49] Loaded engine size: 271 MB I0104 13:46:42.349875 73 logging.cc:49] [MemUsageSnapshot] deserializeCudaEngine begin: CPU 1900 MiB, GPU 1492 MiB E0104 13:46:42.350625 73 model_repository_manager.cc:1215] failed to load 'riva-trt-hifigan' version 1: Internal: unable to create TensorRT engine E0104 13:46:42.353134 73 logging.cc:43] 1: [stdArchiveReader.cpp::StdArchiveReader::34] Error Code 1: Serialization (Serialization assertion safeVersionRead == safeSerializationVersion failed.Version tag does not match. Note: Current Version: 43, Serialized Engine Version: 96) E0104 13:46:42.353183 73 logging.cc:43] 4: [runtime.cpp::deserializeCudaEngine::75] Error Code 4: Internal Error (Engine deserialization failed.) I0104 13:46:42.354200 73 logging.cc:49] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 1629, GPU 1494 (MiB) I0104 13:46:42.354252 73 logging.cc:49] Loaded engine size: 210 MB I0104 13:46:42.354426 73 logging.cc:49] [MemUsageSnapshot] deserializeCudaEngine begin: CPU 1629 MiB, GPU 1494 MiB E0104 13:46:42.372378 73 model_repository_manager.cc:1215] failed to load 'riva-trt-citrinet-1024' version 1: Internal: unable to create TensorRT engine E0104 13:46:42.376906 73 logging.cc:43] 1: [stdArchiveReader.cpp::StdArchiveReader::34] Error Code 1: Serialization (Serialization assertion safeVersionRead == safeSerializationVersion failed.Version tag does not match. Note: Current Version: 43, Serialized Engine Version: 96) E0104 13:46:42.376967 73 logging.cc:43] 4: [runtime.cpp::deserializeCudaEngine::75] Error Code 4: Internal Error (Engine deserialization failed.) I0104 13:46:42.378191 73 logging.cc:49] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 1421, GPU 1494 (MiB) I0104 13:46:42.378248 73 logging.cc:49] Loaded engine size: 209 MB I0104 13:46:42.378449 73 logging.cc:49] [MemUsageSnapshot] deserializeCudaEngine begin: CPU 1421 MiB, GPU 1494 MiB E0104 13:46:42.380949 73 logging.cc:43] 1: [stdArchiveReader.cpp::StdArchiveReader::34] Error Code 1: Serialization (Serialization assertion safeVersionRead == safeSerializationVersion failed.Version tag does not match. Note: Current Version: 43, Serialized Engine Version: 96) E0104 13:46:42.380996 73 logging.cc:43] 4: [runtime.cpp::deserializeCudaEngine::75] Error Code 4: Internal Error (Engine deserialization failed.) E0104 13:46:42.396412 73 model_repository_manager.cc:1215] failed to load 'riva-trt-riva_intent_weather-nn-bert-base-uncased' version 1: Internal: unable to create TensorRT engine E0104 13:46:42.412277 73 model_repository_manager.cc:1215] failed to load 'riva-trt-riva_ner-nn-bert-base-uncased' version 1: Internal: unable to create TensorRT engine I0104 13:46:42.412591 73 logging.cc:49] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 1213, GPU 1494 (MiB) I0104 13:46:42.412633 73 logging.cc:49] Loaded engine size: 208 MB I0104 13:46:42.412788 73 logging.cc:49] [MemUsageSnapshot] deserializeCudaEngine begin: CPU 1213 MiB, GPU 1494 MiB > Riva waiting for Triton server to load all models...retrying in 1 second E0104 13:46:42.435032 73 logging.cc:43] 1: [stdArchiveReader.cpp::StdArchiveReader::34] Error Code 1: Serialization (Serialization assertion safeVersionRead == safeSerializationVersion failed.Version tag does not match. Note: Current Version: 43, Serialized Engine Version: 96) E0104 13:46:42.435074 73 logging.cc:43] 4: [runtime.cpp::deserializeCudaEngine::75] Error Code 4: Internal Error (Engine deserialization failed.) I0104 13:46:42.436613 73 logging.cc:49] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 1015, GPU 1500 (MiB) I0104 13:46:42.436675 73 logging.cc:49] Loaded engine size: 208 MB I0104 13:46:42.436939 73 logging.cc:49] [MemUsageSnapshot] deserializeCudaEngine begin: CPU 1015 MiB, GPU 1500 MiB E0104 13:46:42.453269 73 model_repository_manager.cc:1215] failed to load 'riva-trt-riva_punctuation-nn-bert-base-uncased' version 1: Internal: unable to create TensorRT engine E0104 13:46:42.453669 73 logging.cc:43] 1: [stdArchiveReader.cpp::StdArchiveReader::34] Error Code 1: Serialization (Serialization assertion safeVersionRead == safeSerializationVersion failed.Version tag does not match. Note: Current Version: 43, Serialized Engine Version: 96) E0104 13:46:42.453694 73 logging.cc:43] 4: [runtime.cpp::deserializeCudaEngine::75] Error Code 4: Internal Error (Engine deserialization failed.) I0104 13:46:42.454789 73 logging.cc:49] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 810, GPU 1500 (MiB) I0104 13:46:42.454831 73 logging.cc:49] Loaded engine size: 209 MB I0104 13:46:42.454989 73 logging.cc:49] [MemUsageSnapshot] deserializeCudaEngine begin: CPU 810 MiB, GPU 1500 MiB E0104 13:46:42.466058 73 model_repository_manager.cc:1215] failed to load 'riva-trt-riva_qa-nn-bert-base-uncased' version 1: Internal: unable to create TensorRT engine E0104 13:46:42.466685 73 logging.cc:43] 1: [stdArchiveReader.cpp::StdArchiveReader::34] Error Code 1: Serialization (Serialization assertion safeVersionRead == safeSerializationVersion failed.Version tag does not match. Note: Current Version: 43, Serialized Engine Version: 96) E0104 13:46:42.466734 73 logging.cc:43] 4: [runtime.cpp::deserializeCudaEngine::75] Error Code 4: Internal Error (Engine deserialization failed.) E0104 13:46:42.484980 73 model_repository_manager.cc:1215] failed to load 'riva-trt-riva_text_classification_domain-nn-bert-base-uncased' version 1: Internal: unable to create TensorRT engine > Riva waiting for Triton server to load all models...retrying in 1 second > Riva waiting for Triton server to load all models...retrying in 1 second I0104 13:46:44.685931 73 model_repository_manager.cc:1212] successfully loaded 'riva-onnx-fastpitch_encoder' version 1 I0104 13:46:44.688714 73 spectrogram-chunker.cc:240] TRITONBACKEND_ModelInitialize: spectrogram_chunker (version 1) I0104 13:46:44.689892 73 backend_model.cc:255] model configuration: { "name": "spectrogram_chunker", "platform": "", "backend": "riva_tts_chunker", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 8, "input": [ { "name": "SPECTROGRAM", "data_type": "TYPE_FP32", "format": "FORMAT_NONE", "dims": [ 80, -1 ], "is_shape_tensor": false, "allow_ragged_batch": false }, { "name": "IS_LAST_SENTENCE", "data_type": "TYPE_INT32", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false }, { "name": "NUM_VALID_FRAMES_IN", "data_type": "TYPE_INT64", "format": "FORMAT_NONE", "dims": [ 1 ], "is_shape_tensor": false, "allow_ragged_batch": false } ], "output": [ { "name": "SPECTROGRAM_CHUNK", "data_type": "TYPE_FP32", "dims": [ 80, -1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "END_FLAG", "data_type": "TYPE_INT32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false }, { "name": "NUM_VALID_SAMPLES_OUT", "data_type": "TYPE_INT32", "dims": [ 1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "priority": "PRIORITY_DEFAULT", "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "oldest": { "max_candidate_sequences": 8, "preferred_batch_size": [ 8 ], "max_queue_delay_microseconds": 1000 }, "max_sequence_idle_microseconds": 60000000, "control_input": [ { "name": "START", "control": [ { "kind": "CONTROL_SEQUENCE_START", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "READY", "control": [ { "kind": "CONTROL_SEQUENCE_READY", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "END", "control": [ { "kind": "CONTROL_SEQUENCE_END", "int32_false_true": [ 0, 1 ], "fp32_false_true": [], "data_type": "TYPE_INVALID" } ] }, { "name": "CORRID", "control": [ { "kind": "CONTROL_SEQUENCE_CORRID", "int32_false_true": [], "fp32_false_true": [], "data_type": "TYPE_UINT64" } ] } ] }, "instance_group": [ { "name": "spectrogram_chunker_0", "kind": "KIND_GPU", "count": 1, "gpus": [ 0 ], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "", "cc_model_filenames": {}, "metric_tags": {}, "parameters": { "num_mels": { "string_value": "80" }, "num_samples_per_frame": { "string_value": "256" }, "max_execution_batch_size": { "string_value": "8" }, "chunk_length": { "string_value": "80" } }, "model_warmup": [], "model_transaction_policy": { "decoupled": true } } I0104 13:46:44.689982 73 tts-preprocessor.cc:243] TRITONBACKEND_ModelInitialize: tts_preprocessor (version 1) I0104 13:46:44.691298 73 tts-preprocessor.cc:244] TRITONBACKEND_ModelFinalize: delete model state E0104 13:46:44.691758 73 model_repository_manager.cc:1215] failed to load 'tts_preprocessor' version 1: Internal: attempt to access non-existing object member 'max_sequence_length' I0104 13:46:44.693536 73 spectrogram-chunker.cc:242] TRITONBACKEND_ModelInstanceInitialize: spectrogram_chunker_0 (device 0) I0104 13:46:44.693837 73 model_repository_manager.cc:1212] successfully loaded 'spectrogram_chunker' version 1 E0104 13:46:44.693902 73 model_repository_manager.cc:1404] Invalid argument: ensemble 'riva_text_classification_domain' depends on 'riva-trt-riva_text_classification_domain-nn-bert-base-uncased' which has no loaded version I0104 13:46:44.693982 73 server.cc:504] +------------------+------+ | Repository Agent | Path | +------------------+------+ +------------------+------+ I0104 13:46:44.694051 73 server.cc:543] +------------------+---------------------------------------------------------------------------+--------+ | Backend | Path | Config | +------------------+---------------------------------------------------------------------------+--------+ | tensorrt | | {} | | onnxruntime | /opt/tritonserver/backends/onnxruntime/libtriton_onnxruntime.so | {} | | riva_tts_chunker | /opt/tritonserver/backends/riva_tts_chunker/libtriton_riva_tts_chunker.so | {} | +------------------+---------------------------------------------------------------------------+--------+ I0104 13:46:44.694207 73 server.cc:586] +---------------------------------------------------------------+---------+-------------------------------------------------------------------------------------------+ | Model | Version | Status | +---------------------------------------------------------------+---------+-------------------------------------------------------------------------------------------+ | riva-onnx-fastpitch_encoder | 1 | READY | | riva-trt-citrinet-1024 | 1 | UNAVAILABLE: Internal: unable to create TensorRT engine | | riva-trt-hifigan | 1 | UNAVAILABLE: Internal: unable to create TensorRT engine | | riva-trt-riva_intent_weather-nn-bert-base-uncased | 1 | UNAVAILABLE: Internal: unable to create TensorRT engine | | riva-trt-riva_ner-nn-bert-base-uncased | 1 | UNAVAILABLE: Internal: unable to create TensorRT engine | | riva-trt-riva_punctuation-nn-bert-base-uncased | 1 | UNAVAILABLE: Internal: unable to create TensorRT engine | | riva-trt-riva_qa-nn-bert-base-uncased | 1 | UNAVAILABLE: Internal: unable to create TensorRT engine | | riva-trt-riva_text_classification_domain-nn-bert-base-uncased | 1 | UNAVAILABLE: Internal: unable to create TensorRT engine | | spectrogram_chunker | 1 | READY | | tts_preprocessor | 1 | UNAVAILABLE: Internal: attempt to access non-existing object member 'max_sequence_length' | +---------------------------------------------------------------+---------+-------------------------------------------------------------------------------------------+ I0104 13:46:44.694372 73 tritonserver.cc:1718] +----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Option | Value | +----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | server_id | triton | | server_version | 2.13.0 | | server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics | | model_repository_path[0] | /data/models | | model_control_mode | MODE_NONE | | strict_model_config | 1 | | pinned_memory_pool_byte_size | 268435456 | | cuda_memory_pool_byte_size{0} | 1000000000 | | min_supported_compute_capability | 6.0 | | strict_readiness | 1 | | exit_timeout | 30 | +----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ I0104 13:46:44.694392 73 server.cc:234] Waiting for in-flight requests to complete. I0104 13:46:44.694406 73 model_repository_manager.cc:1078] unloading: spectrogram_chunker:1 I0104 13:46:44.694467 73 model_repository_manager.cc:1078] unloading: riva-onnx-fastpitch_encoder:1 I0104 13:46:44.694639 73 server.cc:249] Timeout 30: Found 2 live models and 0 in-flight non-inference requests I0104 13:46:44.694684 73 spectrogram-chunker.cc:245] TRITONBACKEND_ModelInstanceFinalize: delete instance state I0104 13:46:44.694746 73 spectrogram-chunker.cc:241] TRITONBACKEND_ModelFinalize: delete model state I0104 13:46:44.694917 73 onnxruntime.cc:2106] TRITONBACKEND_ModelInstanceFinalize: delete instance state I0104 13:46:44.695029 73 model_repository_manager.cc:1195] successfully unloaded 'spectrogram_chunker' version 1 I0104 13:46:44.715024 73 onnxruntime.cc:2052] TRITONBACKEND_ModelFinalize: delete model state I0104 13:46:44.715291 73 model_repository_manager.cc:1195] successfully unloaded 'riva-onnx-fastpitch_encoder' version 1 > Riva waiting for Triton server to load all models...retrying in 1 second I0104 13:46:45.694747 73 server.cc:249] Timeout 29: Found 0 live models and 0 in-flight non-inference requests error: creating server: Internal - failed to load all models > Riva waiting for Triton server to load all models...retrying in 1 second > Riva waiting for Triton server to load all models...retrying in 1 second > Triton server died before reaching ready state. Terminating Riva startup. Check Triton logs with: docker logs /opt/riva/bin/start-riva: line 1: kill: (73) - No such process