Build_waveglow failed to generate waveglow.eng. (in jarvis_init.sh)

AI & Data Science Deep Learning (Training & Inference) Riva
1

Hi, I’m having trouble when execute the jarvis_init.sh, I have this log in the end:

2021-04-20 15:27:06,494 [INFO] Extract_binaries for decoder → /data/models/tacotron2_decoder_postnet/1
Available devices:
Device: 0 : ‘GeForce RTX 3090’, 82 SMs, support Co-op Launch ← [ ACTIVE ]
Conv selected stable alg 115
Conv selected stable alg 115
Conv selected stable alg 115
Selected stable alg 0
FC selected stable alg 56
2021-04-20 15:28:41,341 [INFO] Extract_binaries for waveglow → /data/models/jarvis-trt-waveglow/1
Available devices:
Device: 0 : ‘GeForce RTX 3090’, 82 SMs, support Co-op Launch ← [ ACTIVE ]
Tensor ‘spect’ ={1 1 80 80 }
Tensor ‘spect’ ={8 1 80 80 }
Tensor ‘z’ ={1 8 2656 1 }
Tensor ‘z’ ={8 8 2656 1 }
2021-04-20 15:36:15,838 [ERROR] Traceback (most recent call last):
File “/opt/conda/lib/python3.8/site-packages/servicemaker/cli/deploy.py”, line 87, in deploy_from_jmir
generator.serialize_to_disk(
File “/opt/conda/lib/python3.8/site-packages/servicemaker/triton/triton.py”, line 323, in serialize_to_disk
module.serialize_to_disk(repo_dir, jmir, config_only, verbose, overwrite)
File “/opt/conda/lib/python3.8/site-packages/servicemaker/triton/triton.py”, line 214, in serialize_to_disk
self.update_binary(version_dir, jmir, verbose)
File “/opt/conda/lib/python3.8/site-packages/servicemaker/triton/tts.py”, line 294, in update_binary
raise Exception(‘build_waveglow failed to generate waveglow.eng.’)
Exception: build_waveglow failed to generate waveglow.eng.

  • echo

  • echo ‘Jarvis initialization complete. Run ./jarvis_start.sh to launch services.’
    Jarvis initialization complete. Run ./jarvis_start.sh to launch services.

If later I start with bash jarvis_start.sh I have the time out problem…

Let me know if you need more information about this issue.
My Driver version is: 460.67 on an RTX 3090

Regards,
Iago

2

Hi @iagogb,

Could you please share the complete error log/command output so we can help better? Also please share docker logs jarvis-speech output as well.

Thanks

1 Like
3

Hi,
The logs:
Logging into NGC docker registry if necessary…
Pulling required docker images if necessary…
Note: This may take some time, depending on the speed of your Internet connection.

Pulling Jarvis Speech Server images.
Image nvcr.io/nvidia/jarvis/jarvis-speech:1.0.0-b.3-server exists. Skipping.
Image nvcr.io/nvidia/jarvis/jarvis-speech-client:1.0.0-b.3 exists. Skipping.
Image nvcr.io/nvidia/jarvis/jarvis-speech:1.0.0-b.3-servicemaker exists. Skipping.

Downloading models (JMIRs) from NGC…
Note: this may take some time, depending on the speed of your Internet connection.
To skip this process and use existing JMIRs set the location and corresponding flag in config.sh.

==========================
== Jarvis Speech Skills ==

NVIDIA Release (build 20138038)

Copyright (c) 2018-2020, NVIDIA CORPORATION. All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying
project or file.
find: File system loop detected; ‘/usr/bin/X11’ is part of the same file system loop as ‘/usr/bin’.

NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be
insufficient for the inference server. NVIDIA recommends the use of the following flags:
nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 …

/data/artifacts /opt/jarvis

Downloading nvidia/jarvis/jmir_punctuation:1.0.0-b.1…
Downloaded 418.11 MB in 1m 27s, Download speed: 4.8 MB/s

Transfer id: jmir_punctuation_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_punctuation_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 418.11 MB
Started at: 2021-04-20 14:58:08.739418
Completed at: 2021-04-20 14:59:35.853712
Duration taken: 1m 27s

Downloading nvidia/jarvis/jmir_jarvis_asr_jasper_english_streaming:1.0.0-b.1…
Downloaded 1.98 GB in 6m 19s, Download speed: 5.35 MB/s

Transfer id: jmir_jarvis_asr_jasper_english_streaming_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_jarvis_asr_jasper_english_streaming_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 1.98 GB
Started at: 2021-04-20 14:59:41.891182
Completed at: 2021-04-20 15:06:01.387731
Duration taken: 6m 19s

Downloading nvidia/jarvis/jmir_jarvis_asr_jasper_english_offline:1.0.0-b.1…
Downloaded 1.98 GB in 6m 10s, Download speed: 5.48 MB/s

Transfer id: jmir_jarvis_asr_jasper_english_offline_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_jarvis_asr_jasper_english_offline_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 1.98 GB
Started at: 2021-04-20 15:06:08.415254
Completed at: 2021-04-20 15:12:18.902045
Duration taken: 6m 10s

Directory jmir_punctuation_v1.0.0-b.1 already exists, skipping. Use ‘–force’ option to override.

Downloading nvidia/jarvis/jmir_named_entity_recognition:1.0.0-b.1…
Downloaded 420.38 MB in 1m 22s, Download speed: 5.12 MB/s

Transfer id: jmir_named_entity_recognition_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_named_entity_recognition_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 420.38 MB
Started at: 2021-04-20 15:12:25.543467
Completed at: 2021-04-20 15:13:47.648550
Duration taken: 1m 22s

Downloading nvidia/jarvis/jmir_intent_slot:1.0.0-b.1…
Downloaded 422.71 MB in 1m 24s, Download speed: 5.03 MB/s

Transfer id: jmir_intent_slot_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_intent_slot_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 422.71 MB
Started at: 2021-04-20 15:13:53.540459
Completed at: 2021-04-20 15:15:17.652818
Duration taken: 1m 24s

Downloading nvidia/jarvis/jmir_question_answering:1.0.0-b.1…
Downloaded 418.06 MB in 1m 22s, Download speed: 5.09 MB/s

Transfer id: jmir_question_answering_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_question_answering_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 418.06 MB
Started at: 2021-04-20 15:15:23.564428
Completed at: 2021-04-20 15:16:45.664686
Duration taken: 1m 22s

Downloading nvidia/jarvis/jmir_text_classification:1.0.0-b.1…
Downloaded 420.27 MB in 1m 29s, Download speed: 4.72 MB/s

Transfer id: jmir_text_classification_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_text_classification_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 420.27 MB
Started at: 2021-04-20 15:16:51.502626
Completed at: 2021-04-20 15:18:20.624115
Duration taken: 1m 29s

Downloading nvidia/jarvis/jmir_jarvis_tts_ljspeech:1.0.0-b.1…
Downloaded 527.36 MB in 1m 49s, Download speed: 4.83 MB/s

Transfer id: jmir_jarvis_tts_ljspeech_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_jarvis_tts_ljspeech_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 527.36 MB
Started at: 2021-04-20 15:18:26.892437
Completed at: 2021-04-20 15:20:16.029246
Duration taken: 1m 49s

/opt/jarvis

Converting JMIRs at jarvis-model-repo/jmir to Jarvis Model repository.

==========================
== Jarvis Speech Skills ==

NVIDIA Release (build 20138038)

Copyright (c) 2018-2020, NVIDIA CORPORATION. All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying
project or file.
find: File system loop detected; ‘/usr/bin/X11’ is part of the same file system loop as ‘/usr/bin’.

NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be
insufficient for the inference server. NVIDIA recommends the use of the following flags:
nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 …

2021-04-20 15:20:21,078 [INFO] Writing Jarvis model repository to ‘/data/models’…
2021-04-20 15:20:21,078 [INFO] The jarvis model repo target directory is /data/models
2021-04-20 15:20:21,935 [INFO] Extract_binaries for tokenizer → /data/models/jarvis_tokenizer/1
2021-04-20 15:20:22,804 [INFO] Extract_binaries for language_model → /data/models/jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased/1
2021-04-20 15:20:26,567 [INFO] Building TRT engine from PyTorch Checkpoint
2021-04-20 15:21:03,070 [INFO] Text Classification classes:4
2021-04-20 15:21:03,071 [INFO] Extract_binaries for self → /data/models/jarvis_text_classification_domain/1
2021-04-20 15:21:11,842 [INFO] Extract_binaries for featurizer → /data/models/jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline/1
2021-04-20 15:21:11,844 [INFO] Extract_binaries for nn → /data/models/jarvis-trt-jasper/1
2021-04-20 15:21:28,467 [INFO] Building TRT engine from ONNX file
2021-04-20 15:21:29,128 [INFO] Optimizing for FP16
[libprotobuf WARNING /home/jenkins/workspace/OSS/L0_MergeRequest/oss/build/third_party.protobuf/src/third_party.protobuf/src/google/protobuf/io/coded_stream.cc:604] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING /home/jenkins/workspace/OSS/L0_MergeRequest/oss/build/third_party.protobuf/src/third_party.protobuf/src/google/protobuf/io/coded_stream.cc:81] The total number of bytes read was 1331055768
[TensorRT] WARNING: /home/jenkins/workspace/OSS/L0_MergeRequest/oss/parsers/onnx/onnx2trt_utils.cpp:226: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
2021-04-20 15:23:30,025 [INFO] Extract_binaries for vad → /data/models/jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline/1
2021-04-20 15:23:30,026 [INFO] Extract_binaries for lm_decoder → /data/models/jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline/1
2021-04-20 15:23:38,987 [INFO] Extract_binaries for self → /data/models/jasper-asr-trt-ensemble-vad-streaming-offline/1
2021-04-20 15:23:39,838 [INFO] Extract_binaries for tokenizer → /data/models/jarvis_qa_preprocessor/1
2021-04-20 15:23:40,690 [INFO] Extract_binaries for language_model → /data/models/jarvis-trt-jarvis_qa-nn-bert-base-uncased/1
2021-04-20 15:23:43,761 [INFO] Building TRT engine from PyTorch Checkpoint
2021-04-20 15:24:25,543 [INFO] QA dimensions:(-1, 384, 2, 1, 1)
2021-04-20 15:24:25,544 [INFO] Extract_binaries for token_classifier → /data/models/jarvis_qa_postprocessor/1
2021-04-20 15:24:25,544 [INFO] Extract_binaries for self → /data/models/jarvis_qa/1
2021-04-20 15:24:26,459 [WARNING] /data/models/jarvis_tokenizer already exists, skipping deployment. To force deployment rerun with -f or remove the /data/models/jarvis_tokenizer
2021-04-20 15:24:26,459 [INFO] Extract_binaries for language_model → /data/models/jarvis-trt-jarvis_punctuation-nn-bert-base-uncased/1
2021-04-20 15:24:29,509 [INFO] Building TRT engine from PyTorch Checkpoint
2021-04-20 15:24:53,375 [INFO] Capit dimensions:(-1, 128, 2)
2021-04-20 15:24:53,376 [INFO] Punct dimensions:(-1, 128, 4)
2021-04-20 15:24:53,376 [INFO] Extract_binaries for label_tokens_punct → /data/models/jarvis_punctuation_label_tokens_punct/1
2021-04-20 15:24:54,215 [INFO] Extract_binaries for label_tokens_cap → /data/models/jarvis_punctuation_label_tokens_cap/1
2021-04-20 15:24:55,056 [INFO] Extract_binaries for self → /data/models/jarvis_punctuation/1
2021-04-20 15:24:55,900 [WARNING] /data/models/jarvis_tokenizer already exists, skipping deployment. To force deployment rerun with -f or remove the /data/models/jarvis_tokenizer
2021-04-20 15:24:55,900 [INFO] Extract_binaries for language_model → /data/models/jarvis-trt-jarvis_ner-nn-bert-base-uncased/1
2021-04-20 15:25:00,205 [INFO] Building TRT engine from PyTorch Checkpoint
2021-04-20 15:25:23,280 [INFO] NER classes: 13
2021-04-20 15:25:23,281 [INFO] Extract_binaries for label_tokens → /data/models/jarvis_ner_label_tokens/1
2021-04-20 15:25:24,105 [WARNING] /data/models/jarvis_detokenize already exists, skipping deployment. To force deployment rerun with -f or remove the /data/models/jarvis_detokenize
2021-04-20 15:25:24,105 [INFO] Extract_binaries for self → /data/models/jarvis_ner/1
2021-04-20 15:25:24,965 [WARNING] /data/models/jarvis_tokenizer already exists, skipping deployment. To force deployment rerun with -f or remove the /data/models/jarvis_tokenizer
2021-04-20 15:25:24,965 [INFO] Extract_binaries for language_model → /data/models/jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased/1
2021-04-20 15:25:28,644 [INFO] Building TRT engine from PyTorch Checkpoint
2021-04-20 15:25:55,493 [INFO] Intent classes: 18
2021-04-20 15:25:55,493 [INFO] Entity classes: 31
2021-04-20 15:25:55,494 [INFO] Extract_binaries for label_tokens → /data/models/jarvis_label_tokens_weather/1
2021-04-20 15:25:56,339 [WARNING] /data/models/jarvis_detokenize already exists, skipping deployment. To force deployment rerun with -f or remove the /data/models/jarvis_detokenize
2021-04-20 15:25:56,339 [INFO] Extract_binaries for self → /data/models/jarvis_intent_weather/1
2021-04-20 15:26:00,893 [INFO] Extract_binaries for preprocessor → /data/models/tts_preprocessor/1
2021-04-20 15:26:04,209 [INFO] Extract_binaries for encoder → /data/models/jarvis-trt-tacotron2_encoder/1
Available devices:
Device: 0 : ‘GeForce RTX 3090’, 82 SMs, support Co-op Launch ← [ ACTIVE ]
Conv selected stable alg 115
Conv selected stable alg 115
Conv selected stable alg 115
Selected stable alg 0
FC selected stable alg 56
2021-04-20 15:27:06,494 [INFO] Extract_binaries for decoder → /data/models/tacotron2_decoder_postnet/1
Available devices:
Device: 0 : ‘GeForce RTX 3090’, 82 SMs, support Co-op Launch ← [ ACTIVE ]
Conv selected stable alg 115
Conv selected stable alg 115
Conv selected stable alg 115
Selected stable alg 0
FC selected stable alg 56
2021-04-20 15:28:41,341 [INFO] Extract_binaries for waveglow → /data/models/jarvis-trt-waveglow/1
Available devices:
Device: 0 : ‘GeForce RTX 3090’, 82 SMs, support Co-op Launch ← [ ACTIVE ]
Tensor ‘spect’ ={1 1 80 80 }
Tensor ‘spect’ ={8 1 80 80 }
Tensor ‘z’ ={1 8 2656 1 }
Tensor ‘z’ ={8 8 2656 1 }
2021-04-20 15:36:15,838 [ERROR] Traceback (most recent call last):
File “/opt/conda/lib/python3.8/site-packages/servicemaker/cli/deploy.py”, line 87, in deploy_from_jmir
generator.serialize_to_disk(
File “/opt/conda/lib/python3.8/site-packages/servicemaker/triton/triton.py”, line 323, in serialize_to_disk
module.serialize_to_disk(repo_dir, jmir, config_only, verbose, overwrite)
File “/opt/conda/lib/python3.8/site-packages/servicemaker/triton/triton.py”, line 214, in serialize_to_disk
self.update_binary(version_dir, jmir, verbose)
File “/opt/conda/lib/python3.8/site-packages/servicemaker/triton/tts.py”, line 294, in update_binary
raise Exception(‘build_waveglow failed to generate waveglow.eng.’)
Exception: build_waveglow failed to generate waveglow.eng.

  • echo

  • echo ‘Jarvis initialization complete. Run ./jarvis_start.sh to launch services.’
    Jarvis initialization complete. Run ./jarvis_start.sh to launch services.

bash jarvis_start.sh:
Starting Jarvis Speech Services. This may take several minutes depending on the number of models deployed.
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Waiting for Jarvis server to load all models…retrying in 10 seconds
Health ready check failed.
Check Jarvis logs with: docker logs jarvis-speech

docker logs jarvis-speech:

== Jarvis Speech Skills ==

NVIDIA Release 21.03 (build 21236204)

Copyright (c) 2018-2020, NVIDIA CORPORATION. All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying
project or file.

NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be
insufficient for the inference server. NVIDIA recommends the use of the following flags:
nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 …

Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:16.439197 70 metrics.cc:221] Collecting metrics for GPU 0: GeForce RTX 3090
I0421 06:43:16.452213 70 onnxruntime.cc:1728] TRITONBACKEND_Initialize: onnxruntime
I0421 06:43:16.452317 70 onnxruntime.cc:1738] Triton TRITONBACKEND API version: 1.0
I0421 06:43:16.452325 70 onnxruntime.cc:1744] ‘onnxruntime’ TRITONBACKEND API version: 1.0
I0421 06:43:16.594375 70 pinned_memory_manager.cc:205] Pinned memory pool is created at ‘0x7ff8ee000000’ with size 268435456
I0421 06:43:16.594791 70 cuda_memory_manager.cc:103] CUDA memory pool is created on device 0 with size 1000000000
E0421 06:43:16.599351 70 model_repository_manager.cc:1682] failed to open text file for read /data/models/jarvis-trt-waveglow/config.pbtxt: No such file or directory
I0421 06:43:16.605063 70 model_repository_manager.cc:787] loading: jarvis-trt-jarvis_ner-nn-bert-base-uncased:1
I0421 06:43:16.705235 70 model_repository_manager.cc:787] loading: jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased:1
I0421 06:43:16.805421 70 model_repository_manager.cc:787] loading: jarvis-trt-jarvis_punctuation-nn-bert-base-uncased:1
I0421 06:43:16.905605 70 model_repository_manager.cc:787] loading: jarvis-trt-jarvis_qa-nn-bert-base-uncased:1
I0421 06:43:17.007252 70 model_repository_manager.cc:787] loading: jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased:1
I0421 06:43:17.107481 70 model_repository_manager.cc:787] loading: jarvis-trt-jasper:1
I0421 06:43:17.207699 70 model_repository_manager.cc:787] loading: jarvis-trt-tacotron2_encoder:1
I0421 06:43:17.307906 70 model_repository_manager.cc:787] loading: jarvis_detokenize:1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:17.408127 70 model_repository_manager.cc:787] loading: jarvis_label_tokens_weather:1
I0421 06:43:17.408261 70 custom_backend.cc:198] Creating instance jarvis_detokenize_0_0_cpu on CPU using libtriton_jarvis_nlp_detokenizer.so
I0421 06:43:17.448811 70 model_repository_manager.cc:960] successfully loaded ‘jarvis_detokenize’ version 1
I0421 06:43:17.508351 70 model_repository_manager.cc:787] loading: jarvis_ner_label_tokens:1
I0421 06:43:17.508543 70 custom_backend.cc:198] Creating instance jarvis_label_tokens_weather_0_0_cpu on CPU using libtriton_jarvis_nlp_seqlabel.so
I0421 06:43:17.550021 70 model_repository_manager.cc:960] successfully loaded ‘jarvis_label_tokens_weather’ version 1
I0421 06:43:17.608559 70 model_repository_manager.cc:787] loading: jarvis_punctuation_gen_output:1
I0421 06:43:17.608708 70 custom_backend.cc:198] Creating instance jarvis_ner_label_tokens_0_0_cpu on CPU using libtriton_jarvis_nlp_seqlabel.so
I0421 06:43:17.611517 70 model_repository_manager.cc:960] successfully loaded ‘jarvis_ner_label_tokens’ version 1
I0421 06:43:17.708801 70 model_repository_manager.cc:787] loading: jarvis_punctuation_label_tokens_cap:1
I0421 06:43:17.708916 70 custom_backend.cc:198] Creating instance jarvis_punctuation_gen_output_0_0_cpu on CPU using libtriton_jarvis_nlp_punctuation.so
I0421 06:43:17.743992 70 model_repository_manager.cc:960] successfully loaded ‘jarvis_punctuation_gen_output’ version 1
I0421 06:43:17.808994 70 model_repository_manager.cc:787] loading: jarvis_punctuation_label_tokens_punct:1
I0421 06:43:17.809124 70 custom_backend.cc:198] Creating instance jarvis_punctuation_label_tokens_cap_0_0_cpu on CPU using libtriton_jarvis_nlp_seqlabel.so
I0421 06:43:17.811180 70 model_repository_manager.cc:960] successfully loaded ‘jarvis_punctuation_label_tokens_cap’ version 1
I0421 06:43:17.909205 70 model_repository_manager.cc:787] loading: jarvis_punctuation_merge_labels:1
I0421 06:43:17.909373 70 custom_backend.cc:198] Creating instance jarvis_punctuation_label_tokens_punct_0_0_cpu on CPU using libtriton_jarvis_nlp_seqlabel.so
I0421 06:43:17.911355 70 model_repository_manager.cc:960] successfully loaded ‘jarvis_punctuation_label_tokens_punct’ version 1
I0421 06:43:18.009443 70 model_repository_manager.cc:787] loading: jarvis_qa_postprocessor:1
I0421 06:43:18.009579 70 custom_backend.cc:198] Creating instance jarvis_punctuation_merge_labels_0_0_cpu on CPU using libtriton_jarvis_nlp_labels.so
I0421 06:43:18.015727 70 model_repository_manager.cc:960] successfully loaded ‘jarvis_punctuation_merge_labels’ version 1
I0421 06:43:18.109657 70 model_repository_manager.cc:787] loading: jarvis_qa_preprocessor:1
I0421 06:43:18.109778 70 custom_backend.cc:198] Creating instance jarvis_qa_postprocessor_0_0_cpu on CPU using libtriton_jarvis_nlp_qa.so
I0421 06:43:18.118146 70 model_repository_manager.cc:960] successfully loaded ‘jarvis_qa_postprocessor’ version 1
I0421 06:43:18.209859 70 model_repository_manager.cc:787] loading: jarvis_tokenizer:1
I0421 06:43:18.209996 70 custom_backend.cc:198] Creating instance jarvis_qa_preprocessor_0_0_cpu on CPU using libtriton_jarvis_nlp_tokenizer.so
I0421 06:43:18.226903 70 model_repository_manager.cc:960] successfully loaded ‘jarvis_qa_preprocessor’ version 1
I0421 06:43:18.310093 70 model_repository_manager.cc:787] loading: jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline:1
I0421 06:43:18.310207 70 custom_backend.cc:198] Creating instance jarvis_tokenizer_0_0_cpu on CPU using libtriton_jarvis_nlp_tokenizer.so
I0421 06:43:18.319904 70 model_repository_manager.cc:960] successfully loaded ‘jarvis_tokenizer’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:18.410298 70 model_repository_manager.cc:787] loading: jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline:1
I0421 06:43:18.410578 70 custom_backend.cc:198] Creating instance jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline_0_0_cpu on CPU using libtriton_jarvis_asr_decoder_cpu.so
I0421 06:43:18.510521 70 model_repository_manager.cc:787] loading: jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline:1
I0421 06:43:18.510700 70 custom_backend.cc:201] Creating instance jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline_0_0_gpu0 on GPU 0 (8.6) using libtriton_jarvis_asr_features.so
I0421 06:43:18.610730 70 model_repository_manager.cc:787] loading: tacotron2_decoder_postnet:1
I0421 06:43:18.610828 70 custom_backend.cc:198] Creating instance jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline_0_0_cpu on CPU using libtriton_jarvis_asr_vad.so
I0421 06:43:18.636010 70 model_repository_manager.cc:960] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline’ version 1
I0421 06:43:18.710904 70 model_repository_manager.cc:787] loading: tts_preprocessor:1
I0421 06:43:18.811261 70 custom_backend.cc:201] Creating instance tts_preprocessor_0_0_gpu0 on GPU 0 (8.6) using libtriton_jarvis_tts_preprocessor.so
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:22.008299 70 model_repository_manager.cc:960] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:22.614577 70 tacotron-decoder-postnet.cc:873] TRITONBACKEND_ModelInitialize: tacotron2_decoder_postnet (version 1)
I0421 06:43:22.615859 70 tacotron-decoder-postnet.cc:767] model configuration:
{
“name”: “tacotron2_decoder_postnet”,
“platform”: “”,
“backend”: “jarvis_tts_taco_postnet”,
“version_policy”: {
“latest”: {
“num_versions”: 1
}
},
“max_batch_size”: 8,
“input”: [
{
“name”: “input_decoder”,
“data_type”: “TYPE_FP32”,
“format”: “FORMAT_NONE”,
“dims”: [
1,
400,
512
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false
},
{
“name”: “input_processed_decoder”,
“data_type”: “TYPE_FP32”,
“format”: “FORMAT_NONE”,
“dims”: [
400,
128,
1,
1
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false
},
{
“name”: “input_num_characters”,
“data_type”: “TYPE_INT32”,
“format”: “FORMAT_NONE”,
“dims”: [
1
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false
}
],
“output”: [
{
“name”: “spectrogram_chunk”,
“data_type”: “TYPE_FP32”,
“dims”: [
1,
80,
80
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “z”,
“data_type”: “TYPE_FP32”,
“dims”: [
8,
2656,
1
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “num_valid_samples”,
“data_type”: “TYPE_INT32”,
“dims”: [
1
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “end_flag”,
“data_type”: “TYPE_INT32”,
“dims”: [
1
],
“label_filename”: “”,
“is_shape_tensor”: false
}
],
“batch_input”: ,
“batch_output”: ,
“optimization”: {
“priority”: “PRIORITY_DEFAULT”,
“input_pinned_memory”: {
“enable”: true
},
“output_pinned_memory”: {
“enable”: true
}
},
“sequence_batching”: {
“oldest”: {
“max_candidate_sequences”: 8,
“preferred_batch_size”: [
8
],
“max_queue_delay_microseconds”: 100
},
“max_sequence_idle_microseconds”: 60000000,
“control_input”: [
{
“name”: “START”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_START”,
“int32_false_true”: [
0,
1
],
“fp32_false_true”: ,
“data_type”: “TYPE_INVALID”
}
]
},
{
“name”: “READY”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_READY”,
“int32_false_true”: [
0,
1
],
“fp32_false_true”: ,
“data_type”: “TYPE_INVALID”
}
]
},
{
“name”: “END”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_END”,
“int32_false_true”: [
0,
1
],
“fp32_false_true”: ,
“data_type”: “TYPE_INVALID”
}
]
},
{
“name”: “CORRID”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_CORRID”,
“int32_false_true”: ,
“fp32_false_true”: ,
“data_type”: “TYPE_UINT64”
}
]
}
]
},
“instance_group”: [
{
“name”: “tacotron2_decoder_postnet_0”,
“kind”: “KIND_GPU”,
“count”: 1,
“gpus”: [
0
],
“profile”:
}
],
“default_model_filename”: “”,
“cc_model_filenames”: {},
“metric_tags”: {},
“parameters”: {
“num_samples_per_frame”: {
“string_value”: “256”
},
“z_dim0”: {
“string_value”: “8”
},
“tacotron_decoder_engine”: {
“string_value”: “/data/models/tacotron2_decoder_postnet/1/model.plan”
},
“num_mels”: {
“string_value”: “80”
},
“encoding_dimension”: {
“string_value”: “512”
},
“z_dim1”: {
“string_value”: “2656”
},
“max_execution_batch_size”: {
“string_value”: “8”
},
“chunk_length”: {
“string_value”: “80”
},
“max_input_length”: {
“string_value”: “400”
},
“attention_dimension”: {
“string_value”: “128”
}
},
“model_warmup”: ,
“model_transaction_policy”: {
“decoupled”: true
}
}
I0421 06:43:22.615926 70 tacotron-decoder-postnet.cc:927] TRITONBACKEND_ModelInstanceInitialize: tacotron2_decoder_postnet_0 (device 0)
I0421 06:43:22.618317 70 model_repository_manager.cc:960] successfully loaded ‘tts_preprocessor’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:29.258405 70 model_repository_manager.cc:960] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:29.449209 70 plan_backend.cc:338] Creating instance jarvis-trt-jarvis_ner-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (8.6) using model.plan
I0421 06:43:30.317450 70 plan_backend.cc:671] Created instance jarvis-trt-jarvis_ner-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0421 06:43:30.338259 70 model_repository_manager.cc:960] successfully loaded ‘jarvis-trt-jarvis_ner-nn-bert-base-uncased’ version 1
I0421 06:43:30.376158 70 plan_backend.cc:338] Creating instance jarvis-trt-tacotron2_encoder_0_0_gpu0 on GPU 0 (8.6) using model.plan
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:30.493101 70 plan_backend.cc:675] Created instance jarvis-trt-tacotron2_encoder_0_0_gpu0 on GPU 0 with stream priority 0
I0421 06:43:30.497282 70 model_repository_manager.cc:960] successfully loaded ‘jarvis-trt-tacotron2_encoder’ version 1
I0421 06:43:31.183184 70 model_repository_manager.cc:960] successfully loaded ‘tacotron2_decoder_postnet’ version 1
I0421 06:43:31.330576 70 plan_backend.cc:338] Creating instance jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (8.6) using model.plan
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:32.182260 70 plan_backend.cc:671] Created instance jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0421 06:43:32.201035 70 model_repository_manager.cc:960] successfully loaded ‘jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:33.013031 70 plan_backend.cc:338] Creating instance jarvis-trt-jarvis_punctuation-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (8.6) using model.plan
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:33.874952 70 plan_backend.cc:671] Created instance jarvis-trt-jarvis_punctuation-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0421 06:43:33.895244 70 model_repository_manager.cc:960] successfully loaded ‘jarvis-trt-jarvis_punctuation-nn-bert-base-uncased’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:34.731114 70 plan_backend.cc:338] Creating instance jarvis-trt-jarvis_qa-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (8.6) using model.plan
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:35.600870 70 plan_backend.cc:671] Created instance jarvis-trt-jarvis_qa-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0421 06:43:35.621716 70 model_repository_manager.cc:960] successfully loaded ‘jarvis-trt-jarvis_qa-nn-bert-base-uncased’ version 1
I0421 06:43:36.437681 70 plan_backend.cc:338] Creating instance jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (8.6) using model.plan
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:37.293315 70 plan_backend.cc:671] Created instance jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0421 06:43:37.313236 70 model_repository_manager.cc:960] successfully loaded ‘jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:39.728401 70 plan_backend.cc:338] Creating instance jarvis-trt-jasper_0_0_gpu0 on GPU 0 (8.6) using model.plan
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:42.157626 70 plan_backend.cc:671] Created instance jarvis-trt-jasper_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0421 06:43:42.177546 70 model_repository_manager.cc:960] successfully loaded ‘jarvis-trt-jasper’ version 1
I0421 06:43:42.178355 70 model_repository_manager.cc:787] loading: jarvis_intent_weather:1
I0421 06:43:42.278832 70 model_repository_manager.cc:787] loading: jarvis_ner:1
I0421 06:43:42.379296 70 model_repository_manager.cc:960] successfully loaded ‘jarvis_intent_weather’ version 1
I0421 06:43:42.379552 70 model_repository_manager.cc:787] loading: jarvis_punctuation:1
I0421 06:43:42.480183 70 model_repository_manager.cc:787] loading: jarvis_qa:1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:42.580577 70 model_repository_manager.cc:960] successfully loaded ‘jarvis_ner’ version 1
I0421 06:43:42.580590 70 model_repository_manager.cc:960] successfully loaded ‘jarvis_punctuation’ version 1
I0421 06:43:42.580681 70 model_repository_manager.cc:787] loading: jarvis_text_classification_domain:1
I0421 06:43:42.681316 70 model_repository_manager.cc:787] loading: jasper-asr-trt-ensemble-vad-streaming-offline:1
I0421 06:43:42.684061 70 model_repository_manager.cc:960] successfully loaded ‘jarvis_text_classification_domain’ version 1
I0421 06:43:42.781745 70 model_repository_manager.cc:960] successfully loaded ‘jarvis_qa’ version 1
I0421 06:43:42.782135 70 model_repository_manager.cc:960] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-offline’ version 1
I0421 06:43:42.782385 70 server.cc:495]
±------------------------±----------------------------------------------------------------------------------------±-----+
| Backend | Config | Path |
±------------------------±----------------------------------------------------------------------------------------±-----+
| onnxruntime | /opt/tritonserver/backends/onnxruntime/libtriton_onnxruntime.so | {} |
| jarvis_tts_taco_postnet | /opt/tritonserver/backends/jarvis_tts_taco_postnet/libtriton_jarvis_tts_taco_postnet.so | {} |
±------------------------±----------------------------------------------------------------------------------------±-----+

I0421 06:43:42.782892 70 server.cc:538]
±--------------------------------------------------------------------------------------------±--------±-------+
| Model | Version | Status |
±--------------------------------------------------------------------------------------------±--------±-------+
| jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jarvis_ner-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jarvis_punctuation-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jarvis_qa-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jasper | 1 | READY |
| jarvis-trt-tacotron2_encoder | 1 | READY |
| jarvis_detokenize | 1 | READY |
| jarvis_intent_weather | 1 | READY |
| jarvis_label_tokens_weather | 1 | READY |
| jarvis_ner | 1 | READY |
| jarvis_ner_label_tokens | 1 | READY |
| jarvis_punctuation | 1 | READY |
| jarvis_punctuation_gen_output | 1 | READY |
| jarvis_punctuation_label_tokens_cap | 1 | READY |
| jarvis_punctuation_label_tokens_punct | 1 | READY |
| jarvis_punctuation_merge_labels | 1 | READY |
| jarvis_qa | 1 | READY |
| jarvis_qa_postprocessor | 1 | READY |
| jarvis_qa_preprocessor | 1 | READY |
| jarvis_text_classification_domain | 1 | READY |
| jarvis_tokenizer | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline | 1 | READY |
| tacotron2_decoder_postnet | 1 | READY |
| tts_preprocessor | 1 | READY |
±--------------------------------------------------------------------------------------------±--------±-------+

I0421 06:43:42.783204 70 tritonserver.cc:1642]
±---------------------------------±---------------------------------------------------------------------------------------------------------------------------------------------------+
| Option | Value |
±---------------------------------±---------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.7.0 |
| server_extensions | classification sequence model_repository schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics |
| model_repository_path[0] | /data/models |
| model_control_mode | MODE_NONE |
| strict_model_config | 1 |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 1000000000 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
±---------------------------------±---------------------------------------------------------------------------------------------------------------------------------------------------+

I0421 06:43:42.783228 70 server.cc:220] Waiting for in-flight requests to complete.
I0421 06:43:42.783246 70 model_repository_manager.cc:820] unloading: tts_preprocessor:1
I0421 06:43:42.783352 70 model_repository_manager.cc:820] unloading: jasper-asr-trt-ensemble-vad-streaming-offline:1
I0421 06:43:42.783527 70 model_repository_manager.cc:820] unloading: jarvis_tokenizer:1
I0421 06:43:42.783673 70 model_repository_manager.cc:943] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-offline’ version 1I0421 06:43:42.783699 70 model_repository_manager.cc:820] unloading: jarvis_text_classification_domain:1

I0421 06:43:42.783975 70 model_repository_manager.cc:820] unloading: jarvis_qa_preprocessor:1
I0421 06:43:42.784153 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis_text_classification_domain’ version 1
I0421 06:43:42.784268 70 model_repository_manager.cc:820] unloading: jarvis_qa_postprocessor:1
I0421 06:43:42.784376 70 model_repository_manager.cc:820] unloading: jarvis_punctuation_merge_labels:1
I0421 06:43:42.784553 70 model_repository_manager.cc:943] successfully unloaded ‘tts_preprocessor’ version 1
I0421 06:43:42.784769 70 model_repository_manager.cc:820] unloading: jarvis_punctuation_label_tokens_cap:1
I0421 06:43:42.784877 70 model_repository_manager.cc:820] unloading: jarvis_punctuation_gen_output:1
I0421 06:43:42.785783 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis_qa_postprocessor’ version 1
I0421 06:43:42.785795 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis_punctuation_label_tokens_cap’ version 1
I0421 06:43:42.786047 70 model_repository_manager.cc:820] unloading: jarvis_punctuation:1
I0421 06:43:42.786140 70 model_repository_manager.cc:820] unloading: jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline:1
I0421 06:43:42.786219 70 model_repository_manager.cc:820] unloading: jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased:1
I0421 06:43:42.786294 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis_punctuation’ version 1
I0421 06:43:42.786440 70 model_repository_manager.cc:820] unloading: jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline:1
I0421 06:43:42.786535 70 model_repository_manager.cc:820] unloading: jarvis_qa:1
I0421 06:43:42.786681 70 model_repository_manager.cc:820] unloading: jarvis_ner_label_tokens:1
I0421 06:43:42.786807 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis_qa’ version 1
I0421 06:43:42.786826 70 model_repository_manager.cc:820] unloading: tacotron2_decoder_postnet:1
I0421 06:43:42.787024 70 model_repository_manager.cc:820] unloading: jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline:1
I0421 06:43:42.787104 70 model_repository_manager.cc:820] unloading: jarvis_punctuation_label_tokens_punct:1
I0421 06:43:42.787284 70 model_repository_manager.cc:820] unloading: jarvis-trt-jarvis_punctuation-nn-bert-base-uncased:1
I0421 06:43:42.787453 70 model_repository_manager.cc:820] unloading: jarvis-trt-jasper:1
I0421 06:43:42.787641 70 model_repository_manager.cc:820] unloading: jarvis-trt-jarvis_ner-nn-bert-base-uncased:1
I0421 06:43:42.787800 70 tacotron-decoder-postnet.cc:1000] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0421 06:43:42.788036 70 model_repository_manager.cc:820] unloading: jarvis_intent_weather:1
I0421 06:43:42.795161 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis_punctuation_merge_labels’ version 1
I0421 06:43:42.795698 70 model_repository_manager.cc:820] unloading: jarvis-trt-jarvis_qa-nn-bert-base-uncased:1
I0421 06:43:42.796163 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis_punctuation_gen_output’ version 1
I0421 06:43:42.796215 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis_ner_label_tokens’ version 1
I0421 06:43:42.796852 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis_tokenizer’ version 1
I0421 06:43:42.799145 70 model_repository_manager.cc:820] unloading: jarvis-trt-tacotron2_encoder:1
I0421 06:43:42.863404 70 model_repository_manager.cc:820] unloading: jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased:1
I0421 06:43:42.863496 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis_punctuation_label_tokens_punct’ version 1
I0421 06:43:42.863532 70 model_repository_manager.cc:820] unloading: jarvis_detokenize:1
I0421 06:43:42.863545 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis_qa_preprocessor’ version 1
I0421 06:43:42.863551 70 model_repository_manager.cc:820] unloading: jarvis_label_tokens_weather:1
I0421 06:43:42.863586 70 model_repository_manager.cc:820] unloading: jarvis_ner:1
I0421 06:43:42.863614 70 server.cc:235] Timeout 30: Found 15 live models and 0 in-flight non-inference requests
I0421 06:43:42.863996 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis_intent_weather’ version 1
I0421 06:43:42.867447 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis_ner’ version 1
I0421 06:43:42.881126 70 model_repository_manager.cc:943] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline’ version 1
I0421 06:43:42.881263 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis_detokenize’ version 1
I0421 06:43:42.881353 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis_label_tokens_weather’ version 1
I0421 06:43:42.881532 70 model_repository_manager.cc:943] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline’ version 1
I0421 06:43:42.887837 70 model_repository_manager.cc:943] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline’ version 1
I0421 06:43:42.904347 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis-trt-tacotron2_encoder’ version 1
I0421 06:43:42.921186 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased’ version 1
I0421 06:43:42.932924 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis-trt-jasper’ version 1
I0421 06:43:42.937129 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased’ version 1
I0421 06:43:42.979314 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis-trt-jarvis_qa-nn-bert-base-uncased’ version 1
I0421 06:43:42.980430 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis-trt-jarvis_ner-nn-bert-base-uncased’ version 1
I0421 06:43:42.991443 70 model_repository_manager.cc:943] successfully unloaded ‘jarvis-trt-jarvis_punctuation-nn-bert-base-uncased’ version 1

Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:43.863690 70 server.cc:235] Timeout 29: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:44.863809 70 server.cc:235] Timeout 28: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:45.863925 70 server.cc:235] Timeout 27: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:46.864034 70 server.cc:235] Timeout 26: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:47.864151 70 server.cc:235] Timeout 25: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:48.864263 70 server.cc:235] Timeout 24: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:49.864379 70 server.cc:235] Timeout 23: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:50.864496 70 server.cc:235] Timeout 22: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:51.864656 70 server.cc:235] Timeout 21: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:52.864803 70 server.cc:235] Timeout 20: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:53.864939 70 server.cc:235] Timeout 19: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:54.865092 70 server.cc:235] Timeout 18: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:55.865208 70 server.cc:235] Timeout 17: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:56.865323 70 server.cc:235] Timeout 16: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:57.865443 70 server.cc:235] Timeout 15: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:58.865488 70 server.cc:235] Timeout 14: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:43:59.865647 70 server.cc:235] Timeout 13: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:44:00.865858 70 server.cc:235] Timeout 12: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:44:01.865986 70 server.cc:235] Timeout 11: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:44:02.866117 70 server.cc:235] Timeout 10: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:44:03.866267 70 server.cc:235] Timeout 9: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:44:04.866350 70 server.cc:235] Timeout 8: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:44:05.866434 70 server.cc:235] Timeout 7: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:44:06.866523 70 server.cc:235] Timeout 6: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:44:07.866606 70 server.cc:235] Timeout 5: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:44:08.866692 70 server.cc:235] Timeout 4: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:44:09.866777 70 server.cc:235] Timeout 3: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:44:10.866864 70 server.cc:235] Timeout 2: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:44:11.866947 70 server.cc:235] Timeout 1: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0421 06:44:12.867083 70 server.cc:235] Timeout 0: Found 1 live models and 0 in-flight non-inference requests
error: creating server: Internal - failed to load all models
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
Triton server died before reaching ready state. Terminating Jarvis startup.
Check Triton logs with: docker logs
/opt/jarvis/bin/start-jarvis: line 1: kill: (70) - No such process

4

Hi @iagogb,

We are investigating this issue, will let you know in case if any updates.

Thanks

5

I too am getting the same issue.
GPU 0: NVIDIA GeForce RTX 3060 Ti
Driver Version: 465.27
OS: Arch Linux

6

Hi,
I have exactly the same issue with this config:
My Driver version is: 460.80 on an RTX 3090.

when execute the jarvis_init.sh, I have this log in the end:
##########################################################
2021-05-28 08:45:27,924 [INFO] Extract_binaries for waveglow → /data/models/jarvis-trt-waveglow/1
Available devices:
Device: 0 : ‘GeForce RTX 3090’, 82 SMs, support Co-op Launch ← [ ACTIVE ]
Tensor ‘spect’ ={1 1 80 80 }
Tensor ‘spect’ ={8 1 80 80 }
Tensor ‘z’ ={1 8 2656 1 }
Tensor ‘z’ ={8 8 2656 1 }
2021-05-28 08:52:01,319 [ERROR] Traceback (most recent call last):
File “/opt/conda/lib/python3.8/site-packages/servicemaker/cli/deploy.py”, line 88, in deploy_from_jmir
generator.serialize_to_disk(
File “/opt/conda/lib/python3.8/site-packages/servicemaker/triton/triton.py”, line 341, in serialize_to_disk
module.serialize_to_disk(repo_dir, jmir, config_only, verbose, overwrite)
File “/opt/conda/lib/python3.8/site-packages/servicemaker/triton/triton.py”, line 232, in serialize_to_disk
self.update_binary(version_dir, jmir, verbose)
File “/opt/conda/lib/python3.8/site-packages/servicemaker/triton/tts.py”, line 294, in update_binary
raise Exception(‘build_waveglow failed to generate waveglow.eng.’)
Exception: build_waveglow failed to generate waveglow.eng.

  • echo

  • echo ‘Jarvis initialization complete. Run ./jarvis_start.sh to launch services.’
    Jarvis initialization complete. Run ./jarvis_start.sh to launch services.
    ###########################################################

If later I start with bash jarvis_start.sh I have the time out problem…
below the full logs:

==========================
== Jarvis Speech Skills ==

NVIDIA Release 21.04 (build 22933806)

Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying
project or file.

NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be
insufficient for the inference server. NVIDIA recommends the use of the following flags:
nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 …

Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:16:54.791271 71 metrics.cc:221] Collecting metrics for GPU 0: GeForce RTX 3090
I0528 13:16:54.793195 71 onnxruntime.cc:1728] TRITONBACKEND_Initialize: onnxruntime
I0528 13:16:54.793207 71 onnxruntime.cc:1738] Triton TRITONBACKEND API version: 1.0
I0528 13:16:54.793210 71 onnxruntime.cc:1744] ‘onnxruntime’ TRITONBACKEND API version: 1.0
I0528 13:16:54.960363 71 pinned_memory_manager.cc:205] Pinned memory pool is created at ‘0x7f66d4000000’ with size 268435456
I0528 13:16:54.960643 71 cuda_memory_manager.cc:103] CUDA memory pool is created on device 0 with size 1000000000
E0528 13:16:54.963165 71 model_repository_manager.cc:1937] Poll failed for model directory ‘jarvis-trt-waveglow’: failed to open text file for read /data/models/jarvis-trt-waveglow/config.pbtxt: No such file or directory
I0528 13:16:54.966284 71 model_repository_manager.cc:1065] loading: jarvis-trt-jasper:1
I0528 13:16:55.066500 71 model_repository_manager.cc:1065] loading: jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased:1
I0528 13:16:55.166843 71 model_repository_manager.cc:1065] loading: jarvis-trt-jarvis_qa-nn-bert-base-uncased:1
I0528 13:16:55.267258 71 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline:1
I0528 13:16:55.367609 71 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline:1
I0528 13:16:55.367792 71 custom_backend.cc:198] Creating instance jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline_0_0_cpu on CPU using libtriton_jarvis_asr_decoder_cpu.so
E:decoder_context.cc:696: vocabulary parameter will be deprecated. Please use vocab_file instead.
I0528 13:16:55.468117 71 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline:1
I0528 13:16:55.468678 71 custom_backend.cc:201] Creating instance jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline_0_0_gpu0 on GPU 0 (8.6) using libtriton_jarvis_asr_features.so
I0528 13:16:55.568551 71 model_repository_manager.cc:1065] loading: tacotron2_decoder_postnet:1
I0528 13:16:55.570164 71 custom_backend.cc:198] Creating instance jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline_0_0_cpu on CPU using libtriton_jarvis_asr_vad.so
E:voice-activity-detector.cc:414: vocabulary parameter will be deprecated. Please use vocab_file instead.
I0528 13:16:55.590279 71 model_repository_manager.cc:1239] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline’ version 1
I0528 13:16:55.669065 71 model_repository_manager.cc:1065] loading: tts_preprocessor:1
I0528 13:16:55.730415 71 tacotron-decoder-postnet.cc:873] TRITONBACKEND_ModelInitialize: tacotron2_decoder_postnet (version 1)
I0528 13:16:55.732756 71 tacotron-decoder-postnet.cc:767] model configuration:
{
“name”: “tacotron2_decoder_postnet”,
“platform”: “”,
“backend”: “jarvis_tts_taco_postnet”,
“version_policy”: {
“latest”: {
“num_versions”: 1
}
},
“max_batch_size”: 8,
“input”: [
{
“name”: “input_decoder”,
“data_type”: “TYPE_FP32”,
“format”: “FORMAT_NONE”,
“dims”: [
1,
400,
512
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false
},
{
“name”: “input_processed_decoder”,
“data_type”: “TYPE_FP32”,
“format”: “FORMAT_NONE”,
“dims”: [
400,
128,
1,
1
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false
},
{
“name”: “input_num_characters”,
“data_type”: “TYPE_INT32”,
“format”: “FORMAT_NONE”,
“dims”: [
1
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false
}
],
“output”: [
{
“name”: “spectrogram_chunk”,
“data_type”: “TYPE_FP32”,
“dims”: [
1,
80,
80
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “z”,
“data_type”: “TYPE_FP32”,
“dims”: [
8,
2656,
1
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “num_valid_samples”,
“data_type”: “TYPE_INT32”,
“dims”: [
1
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “end_flag”,
“data_type”: “TYPE_INT32”,
“dims”: [
1
],
“label_filename”: “”,
“is_shape_tensor”: false
}
],
“batch_input”: ,
“batch_output”: ,
“optimization”: {
“priority”: “PRIORITY_DEFAULT”,
“input_pinned_memory”: {
“enable”: true
},
“output_pinned_memory”: {
“enable”: true
},
“gather_kernel_buffer_threshold”: 0,
“eager_batching”: false
},
“sequence_batching”: {
“oldest”: {
“max_candidate_sequences”: 8,
“preferred_batch_size”: [
8
],
“max_queue_delay_microseconds”: 100
},
“max_sequence_idle_microseconds”: 60000000,
“control_input”: [
{
“name”: “START”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_START”,
“int32_false_true”: [
0,
1
],
“fp32_false_true”: ,
“data_type”: “TYPE_INVALID”
}
]
},
{
“name”: “READY”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_READY”,
“int32_false_true”: [
0,
1
],
“fp32_false_true”: ,
“data_type”: “TYPE_INVALID”
}
]
},
{
“name”: “END”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_END”,
“int32_false_true”: [
0,
1
],
“fp32_false_true”: ,
“data_type”: “TYPE_INVALID”
}
]
},
{
“name”: “CORRID”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_CORRID”,
“int32_false_true”: ,
“fp32_false_true”: ,
“data_type”: “TYPE_UINT64”
}
]
}
]
},
“instance_group”: [
{
“name”: “tacotron2_decoder_postnet_0”,
“kind”: “KIND_GPU”,
“count”: 1,
“gpus”: [
0
],
“profile”:
}
],
“default_model_filename”: “”,
“cc_model_filenames”: {},
“metric_tags”: {},
“parameters”: {
“z_dim1”: {
“string_value”: “2656”
},
“encoding_dimension”: {
“string_value”: “512”
},
“num_mels”: {
“string_value”: “80”
},
“tacotron_decoder_engine”: {
“string_value”: “/data/models/tacotron2_decoder_postnet/1/model.plan”
},
“max_execution_batch_size”: {
“string_value”: “8”
},
“max_input_length”: {
“string_value”: “400”
},
“chunk_length”: {
“string_value”: “80”
},
“attention_dimension”: {
“string_value”: “128”
},
“num_samples_per_frame”: {
“string_value”: “256”
},
“z_dim0”: {
“string_value”: “8”
}
},
“model_warmup”: ,
“model_transaction_policy”: {
“decoupled”: true
}
}
I0528 13:16:55.733052 71 tacotron-decoder-postnet.cc:927] TRITONBACKEND_ModelInstanceInitialize: tacotron2_decoder_postnet_0 (device 0)
I0528 13:16:55.769766 71 model_repository_manager.cc:1065] loading: jarvis-trt-tacotron2_encoder:1
I0528 13:16:55.771067 71 custom_backend.cc:201] Creating instance tts_preprocessor_0_0_gpu0 on GPU 0 (8.6) using libtriton_jarvis_tts_preprocessor.so
I0528 13:16:55.777500 71 model_repository_manager.cc:1239] successfully loaded ‘tts_preprocessor’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:16:55.871531 71 model_repository_manager.cc:1065] loading: jarvis_qa_postprocessor:1
I0528 13:16:55.972467 71 model_repository_manager.cc:1065] loading: jarvis_qa_preprocessor:1
I0528 13:16:55.974072 71 custom_backend.cc:198] Creating instance jarvis_qa_postprocessor_0_0_cpu on CPU using libtriton_jarvis_nlp_qa.so
I0528 13:16:56.024728 71 model_repository_manager.cc:1239] successfully loaded ‘jarvis_qa_postprocessor’ version 1
I0528 13:16:56.073131 71 model_repository_manager.cc:1065] loading: jarvis_tokenizer:1
I0528 13:16:56.075312 71 custom_backend.cc:198] Creating instance jarvis_qa_preprocessor_0_0_cpu on CPU using libtriton_jarvis_nlp_tokenizer.so
I0528 13:16:56.124808 71 model_repository_manager.cc:1239] successfully loaded ‘jarvis_qa_preprocessor’ version 1
I0528 13:16:56.174309 71 custom_backend.cc:198] Creating instance jarvis_tokenizer_0_0_cpu on CPU using libtriton_jarvis_nlp_tokenizer.so
I0528 13:16:56.205127 71 model_repository_manager.cc:1239] successfully loaded ‘jarvis_tokenizer’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:16:59.927346 71 model_repository_manager.cc:1239] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:05.648413 71 model_repository_manager.cc:1239] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline’ version 1
I0528 13:17:05.851636 71 plan_backend.cc:365] Creating instance jarvis-trt-tacotron2_encoder_0_0_gpu0 on GPU 0 (8.6) using model.plan
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:05.931567 71 plan_backend.cc:748] Created instance jarvis-trt-tacotron2_encoder_0_0_gpu0 on GPU 0 with stream priority 0
I0528 13:17:05.936842 71 model_repository_manager.cc:1239] successfully loaded ‘jarvis-trt-tacotron2_encoder’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:07.404264 71 model_repository_manager.cc:1239] successfully loaded ‘tacotron2_decoder_postnet’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:08.661343 71 plan_backend.cc:365] Creating instance jarvis-trt-jasper_0_0_gpu0 on GPU 0 (8.6) using model.plan
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:11.344927 71 plan_backend.cc:744] Created instance jarvis-trt-jasper_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0528 13:17:11.369643 71 model_repository_manager.cc:1239] successfully loaded ‘jarvis-trt-jasper’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:12.281704 71 plan_backend.cc:365] Creating instance jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (8.6) using model.plan
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:13.219452 71 plan_backend.cc:744] Created instance jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0528 13:17:13.244671 71 model_repository_manager.cc:1239] successfully loaded ‘jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:14.153044 71 plan_backend.cc:365] Creating instance jarvis-trt-jarvis_qa-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (8.6) using model.plan
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:15.098166 71 plan_backend.cc:744] Created instance jarvis-trt-jarvis_qa-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0528 13:17:15.122888 71 model_repository_manager.cc:1239] successfully loaded ‘jarvis-trt-jarvis_qa-nn-bert-base-uncased’ version 1
I0528 13:17:15.131145 71 model_repository_manager.cc:1065] loading: jarvis_qa:1
I0528 13:17:15.234379 71 model_repository_manager.cc:1065] loading: jarvis_text_classification_domain:1
I0528 13:17:15.337609 71 model_repository_manager.cc:1239] successfully loaded ‘jarvis_text_classification_domain’ version 1
I0528 13:17:15.337791 71 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-offline:1
I0528 13:17:15.438087 71 model_repository_manager.cc:1239] successfully loaded ‘jarvis_qa’ version 1
I0528 13:17:15.438330 71 model_repository_manager.cc:1239] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-offline’ version 1
I0528 13:17:15.438406 71 server.cc:500]
±-----------------±-----+
| Repository Agent | Path |
±-----------------±-----+
±-----------------±-----+

I0528 13:17:15.438455 71 server.cc:527]
±------------------------±----------------------------------------------------------------------------------------±-------+
| Backend | Path | Config |
±------------------------±----------------------------------------------------------------------------------------±-------+
| onnxruntime | /opt/tritonserver/backends/onnxruntime/libtriton_onnxruntime.so | {} |
| jarvis_tts_taco_postnet | /opt/tritonserver/backends/jarvis_tts_taco_postnet/libtriton_jarvis_tts_taco_postnet.so | {} |
±------------------------±----------------------------------------------------------------------------------------±-------+

I0528 13:17:15.438573 71 server.cc:570]
±--------------------------------------------------------------------------------------------±--------±-------+
| Model | Version | Status |
±--------------------------------------------------------------------------------------------±--------±-------+
| jarvis-trt-jarvis_qa-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jasper | 1 | READY |
| jarvis-trt-tacotron2_encoder | 1 | READY |
| jarvis_qa | 1 | READY |
| jarvis_qa_postprocessor | 1 | READY |
| jarvis_qa_preprocessor | 1 | READY |
| jarvis_text_classification_domain | 1 | READY |
| jarvis_tokenizer | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline | 1 | READY |
| tacotron2_decoder_postnet | 1 | READY |
| tts_preprocessor | 1 | READY |
±--------------------------------------------------------------------------------------------±--------±-------+

I0528 13:17:15.441877 71 tritonserver.cc:1658]
±---------------------------------±---------------------------------------------------------------------------------------------------------------------------------------------------+
| Option | Value |
±---------------------------------±---------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.8.0 |
| server_extensions | classification sequence model_repository schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics |
| model_repository_path[0] | /data/models |
| model_control_mode | MODE_NONE |
| strict_model_config | 1 |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 1000000000 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
±---------------------------------±---------------------------------------------------------------------------------------------------------------------------------------------------+

I0528 13:17:15.441892 71 server.cc:233] Waiting for in-flight requests to complete.
I0528 13:17:15.441897 71 model_repository_manager.cc:1098] unloading: tts_preprocessor:1
I0528 13:17:15.441933 71 model_repository_manager.cc:1098] unloading: jarvis-trt-jasper:1
I0528 13:17:15.441985 71 model_repository_manager.cc:1098] unloading: jarvis-trt-jarvis_qa-nn-bert-base-uncased:1
I0528 13:17:15.442034 71 model_repository_manager.cc:1098] unloading: jarvis-trt-tacotron2_encoder:1
I0528 13:17:15.442130 71 model_repository_manager.cc:1098] unloading: jarvis_qa_postprocessor:1
I0528 13:17:15.442180 71 model_repository_manager.cc:1098] unloading: jarvis_qa_preprocessor:1
I0528 13:17:15.442228 71 model_repository_manager.cc:1098] unloading: jarvis_text_classification_domain:1
I0528 13:17:15.442298 71 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-offline:1
I0528 13:17:15.442425 71 model_repository_manager.cc:1098] unloading: jarvis_tokenizer:1
I0528 13:17:15.442510 71 model_repository_manager.cc:1098] unloading: tacotron2_decoder_postnet:1
I0528 13:17:15.442557 71 model_repository_manager.cc:1222] successfully unloaded ‘tts_preprocessor’ version 1
I0528 13:17:15.442599 71 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline:1
I0528 13:17:15.442674 71 model_repository_manager.cc:1098] unloading: jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased:1
I0528 13:17:15.442768 71 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_text_classification_domain’ version 1
I0528 13:17:15.442813 71 model_repository_manager.cc:1222] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-offline’ version 1
I0528 13:17:15.442823 71 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline:1
I0528 13:17:15.442916 71 model_repository_manager.cc:1098] unloading: jarvis_qa:1
I0528 13:17:15.442936 71 tacotron-decoder-postnet.cc:1000] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0528 13:17:15.442977 71 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline:1
I0528 13:17:15.443056 71 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_qa’ version 1
I0528 13:17:15.443163 71 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_qa_postprocessor’ version 1
I0528 13:17:15.443312 71 server.cc:248] Timeout 30: Found 10 live models and 0 in-flight non-inference requests
I0528 13:17:15.456938 71 model_repository_manager.cc:1222] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline’ version 1
I0528 13:17:15.456949 71 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_qa_preprocessor’ version 1
I0528 13:17:15.457269 71 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_tokenizer’ version 1
I0528 13:17:15.464726 71 model_repository_manager.cc:1222] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline’ version 1
I0528 13:17:15.468192 71 model_repository_manager.cc:1222] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline’ version 1
I0528 13:17:15.477466 71 model_repository_manager.cc:1222] successfully unloaded ‘jarvis-trt-tacotron2_encoder’ version 1
I0528 13:17:15.487915 71 model_repository_manager.cc:1222] successfully unloaded ‘jarvis-trt-jasper’ version 1
I0528 13:17:15.510058 71 model_repository_manager.cc:1222] successfully unloaded ‘jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased’ version 1
I0528 13:17:15.518931 71 model_repository_manager.cc:1222] successfully unloaded ‘jarvis-trt-jarvis_qa-nn-bert-base-uncased’ version 1

Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:16.444122 71 server.cc:248] Timeout 29: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:17.444511 71 server.cc:248] Timeout 28: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:18.444644 71 server.cc:248] Timeout 27: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:19.444773 71 server.cc:248] Timeout 26: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:20.445419 71 server.cc:248] Timeout 25: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:21.446069 71 server.cc:248] Timeout 24: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:22.446723 71 server.cc:248] Timeout 23: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:23.447358 71 server.cc:248] Timeout 22: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:24.448019 71 server.cc:248] Timeout 21: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:25.448688 71 server.cc:248] Timeout 20: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:26.449568 71 server.cc:248] Timeout 19: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:27.449827 71 server.cc:248] Timeout 18: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:28.449970 71 server.cc:248] Timeout 17: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:29.450224 71 server.cc:248] Timeout 16: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:30.450871 71 server.cc:248] Timeout 15: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:31.451031 71 server.cc:248] Timeout 14: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:32.451893 71 server.cc:248] Timeout 13: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:33.452745 71 server.cc:248] Timeout 12: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:34.453389 71 server.cc:248] Timeout 11: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:35.454043 71 server.cc:248] Timeout 10: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:36.454674 71 server.cc:248] Timeout 9: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:37.455326 71 server.cc:248] Timeout 8: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:38.456211 71 server.cc:248] Timeout 7: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:39.456345 71 server.cc:248] Timeout 6: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:40.456744 71 server.cc:248] Timeout 5: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:41.456960 71 server.cc:248] Timeout 4: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:42.457140 71 server.cc:248] Timeout 3: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:43.457275 71 server.cc:248] Timeout 2: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:44.457415 71 server.cc:248] Timeout 1: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 13:17:45.457563 71 server.cc:248] Timeout 0: Found 1 live models and 0 in-flight non-inference requests
error: creating server: Internal - failed to load all models
Jarvis waiting for Triton server to load all models…retrying in 1 second
Triton server died before reaching ready state. Terminating Jarvis startup.
Check Triton logs with: docker logs
/opt/jarvis/bin/start-jarvis: line 1: kill: (71) - No such process

########################################################

Any updates on this issue ?
thx a lot fot your help

sincerely,

7

Hi @iagogb and @xavcec2000,
Could you please try the latest Riva release and let us know in case issue persist?

Thanks