Hello, im trying to execute bash jarvis_start.sh but im getting Health ready check failed error after many lines of Waiting for Jarvis server to load all models…retrying in 1- seconds. Would really appreciate some help. Below, i have included the output of running the folowing cli commands:
- bash jarvis_init.sh
- nvidia-smi
- bash jarvis_start.sh
bash jarvis_init.sh output
Logging into NGC docker registry if necessary…
Pulling required docker images if necessary…
Note: This may take some time, depending on the speed of your Internet connection.
Pulling Jarvis Speech Server images.
Pulling nvcr.io/nvidia/jarvis/jarvis-speech:1.1.0-beta-server. This may take some time…
Pulling nvcr.io/nvidia/jarvis/jarvis-speech-client:1.1.0-beta. This may take some time…
Pulling nvcr.io/nvidia/jarvis/jarvis-speech:1.1.0-beta-servicemaker. This may take some time…
Downloading models (JMIRs) from NGC…
Note: this may take some time, depending on the speed of your Internet connection.
To skip this process and use existing JMIRs set the location and corresponding flag in config.sh.
==========================
== Jarvis Speech Skills ==
NVIDIA Release (build 21060478)
Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.
Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying
project or file.
NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be
insufficient for the inference server. NVIDIA recommends the use of the following flags:
nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 …
/data/artifacts /opt/jarvis
Downloading nvidia/jarvis/jmir_punctuation:1.0.0-b.1…
Downloaded 418.11 MB in 38s, Download speed: 10.99 MB/s
Transfer id: jmir_punctuation_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_punctuation_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 418.11 MB
Started at: 2021-05-24 02:45:03.773658
Completed at: 2021-05-24 02:45:41.822732
Duration taken: 38s
Downloading nvidia/jarvis/jmir_jarvis_asr_jasper_english_streaming:1.0.0-b.1…
Downloaded 1.98 GB in 3m 27s, Download speed: 9.8 MB/s
Transfer id: jmir_jarvis_asr_jasper_english_streaming_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_jarvis_asr_jasper_english_streaming_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 1.98 GB
Started at: 2021-05-24 02:45:50.887673
Completed at: 2021-05-24 02:49:18.125809
Duration taken: 3m 27s
Downloading nvidia/jarvis/jmir_jarvis_asr_jasper_english_offline:1.0.0-b.1…
Downloaded 1.98 GB in 2m 35s, Download speed: 13.09 MB/s
Transfer id: jmir_jarvis_asr_jasper_english_offline_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_jarvis_asr_jasper_english_offline_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 1.98 GB
Started at: 2021-05-24 02:49:26.484392
Completed at: 2021-05-24 02:52:01.651045
Duration taken: 2m 35s
Directory jmir_punctuation_v1.0.0-b.1 already exists, skipping. Use ‘–force’ option to override.
Downloading nvidia/jarvis/jmir_named_entity_recognition:1.0.0-b.1…
Downloaded 420.38 MB in 38s, Download speed: 11.05 MB/s
Transfer id: jmir_named_entity_recognition_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_named_entity_recognition_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 420.38 MB
Started at: 2021-05-24 02:52:10.191968
Completed at: 2021-05-24 02:52:48.233847
Duration taken: 38s
Downloading nvidia/jarvis/jmir_intent_slot:1.0.0-b.1…
Downloaded 422.71 MB in 1m 5s, Download speed: 6.5 MB/s
Transfer id: jmir_intent_slot_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_intent_slot_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 422.71 MB
Started at: 2021-05-24 02:52:55.847139
Completed at: 2021-05-24 02:54:00.925772
Duration taken: 1m 5s
Downloading nvidia/jarvis/jmir_question_answering:1.0.0-b.1…
Downloaded 418.06 MB in 41s, Download speed: 10.18 MB/s
Transfer id: jmir_question_answering_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_question_answering_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 418.06 MB
Started at: 2021-05-24 02:54:08.527265
Completed at: 2021-05-24 02:54:49.574643
Duration taken: 41s
Downloading nvidia/jarvis/jmir_text_classification:1.0.0-b.1…
Downloaded 420.27 MB in 40s, Download speed: 10.49 MB/s
Transfer id: jmir_text_classification_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_text_classification_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 420.27 MB
Started at: 2021-05-24 02:54:58.275814
Completed at: 2021-05-24 02:55:38.322575
Duration taken: 40s
Downloading nvidia/jarvis/jmir_jarvis_tts_ljspeech:1.0.0-b.1…
Downloaded 527.36 MB in 49s, Download speed: 10.75 MB/s
Transfer id: jmir_jarvis_tts_ljspeech_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_jarvis_tts_ljspeech_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 527.36 MB
Started at: 2021-05-24 02:55:46.052547
Completed at: 2021-05-24 02:56:35.111061
Duration taken: 49s
/opt/jarvis
Converting JMIRs at jarvis-model-repo/jmir to Jarvis Model repository.
- docker run --init -it --rm --gpus ‘“device=0”’ -v jarvis-model-repo:/data -e MODEL_DEPLOY_KEY=tlt_encode --name jarvis-service-maker nvcr.io/nvidia/jarvis/jarvis-speech:1.1.0-beta-servicemaker deploy_all_models /data/jmir /data/models
==========================
== Jarvis Speech Skills ==
NVIDIA Release (build 21060478)
Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.
Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying
project or file.
NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be
insufficient for the inference server. NVIDIA recommends the use of the following flags:
nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 …
2021-05-24 02:56:39,137 [INFO] Writing Jarvis model repository to ‘/data/models’…
2021-05-24 02:56:39,137 [INFO] The jarvis model repo target directory is /data/models
2021-05-24 02:56:39,953 [INFO] Extract_binaries for tokenizer → /data/models/jarvis_tokenizer/1
2021-05-24 02:56:40,778 [INFO] Extract_binaries for language_model → /data/models/jarvis-trt-jarvis_ner-nn-bert-base-uncased/1
2021-05-24 02:56:44,460 [INFO] Building TRT engine from PyTorch Checkpoint
2021-05-24 02:57:28,821 [INFO] NER classes: 13
2021-05-24 02:57:28,821 [INFO] Extract_binaries for label_tokens → /data/models/jarvis_ner_label_tokens/1
2021-05-24 02:57:29,640 [INFO] Extract_binaries for detokenizer → /data/models/jarvis_detokenize/1
2021-05-24 02:57:29,640 [INFO] Extract_binaries for self → /data/models/jarvis_ner/1
2021-05-24 02:57:30,532 [WARNING] /data/models/jarvis_tokenizer already exists, skipping deployment. To force deployment rerun with -f or remove the /data/models/jarvis_tokenizer
2021-05-24 02:57:30,532 [INFO] Extract_binaries for language_model → /data/models/jarvis-trt-jarvis_punctuation-nn-bert-base-uncased/1
2021-05-24 02:57:33,555 [INFO] Building TRT engine from PyTorch Checkpoint
2021-05-24 02:58:15,971 [INFO] Capit dimensions:(-1, 128, 2)
2021-05-24 02:58:15,971 [INFO] Punct dimensions:4
2021-05-24 02:58:15,972 [INFO] Extract_binaries for label_tokens_punct → /data/models/jarvis_punctuation_label_tokens_punct/1
2021-05-24 02:58:16,791 [INFO] Extract_binaries for label_tokens_cap → /data/models/jarvis_punctuation_label_tokens_cap/1
2021-05-24 02:58:17,614 [WARNING] /data/models/jarvis_detokenize already exists, skipping deployment. To force deployment rerun with -f or remove the /data/models/jarvis_detokenize
2021-05-24 02:58:17,615 [INFO] Extract_binaries for self → /data/models/jarvis_punctuation/1
2021-05-24 02:58:25,370 [WARNING] Value for attribute use_subword not found in config, setting to default
2021-05-24 02:58:25,370 [WARNING] Value for attribute vocab_file not found in config, setting to default
2021-05-24 02:58:25,370 [WARNING] Value for attribute model_class not found in config, setting to default
2021-05-24 02:58:25,370 [WARNING] Value for attribute use_subword not found in config, setting to default
2021-05-24 02:58:25,370 [WARNING] Value for attribute use_subword not found in config, setting to default
2021-05-24 02:58:25,370 [WARNING] Value for attribute use_subword not found in config, setting to default
2021-05-24 02:58:25,533 [INFO] Extract_binaries for featurizer → /data/models/jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming/1
2021-05-24 02:58:25,534 [INFO] Extract_binaries for nn → /data/models/jarvis-trt-jasper/1
2021-05-24 02:58:41,361 [INFO] Building TRT engine from ONNX file
2021-05-24 02:58:49,962 [INFO] Optimizing for FP16
[libprotobuf WARNING /home/jenkins/workspace/OSS/L0_MergeRequest/oss/build/third_party.protobuf/src/third_party.protobuf/src/google/protobuf/io/coded_stream.cc:604] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING /home/jenkins/workspace/OSS/L0_MergeRequest/oss/build/third_party.protobuf/src/third_party.protobuf/src/google/protobuf/io/coded_stream.cc:81] The total number of bytes read was 1331055768
[TensorRT] WARNING: /home/jenkins/workspace/OSS/L0_MergeRequest/oss/parsers/onnx/onnx2trt_utils.cpp:226: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
2021-05-24 03:00:46,295 [INFO] Extract_binaries for vad → /data/models/jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming/1
2021-05-24 03:00:46,295 [INFO] Extract_binaries for lm_decoder → /data/models/jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming/1
2021-05-24 03:00:54,035 [INFO] Extract_binaries for self → /data/models/jasper-asr-trt-ensemble-vad-streaming/1
2021-05-24 03:00:57,074 [INFO] Extract_binaries for preprocessor → /data/models/tts_preprocessor/1
2021-05-24 03:01:00,109 [INFO] Extract_binaries for encoder → /data/models/jarvis-trt-tacotron2_encoder/1
Available devices:
Device: 0 : ‘GeForce RTX 2080 Ti’, 68 SMs, support Co-op Launch ← [ ACTIVE ]
Conv selected stable alg 104
Conv selected stable alg 104
Conv selected stable alg 104
Selected stable alg 0
FC selected stable alg 35
2021-05-24 03:01:54,484 [INFO] Extract_binaries for decoder → /data/models/tacotron2_decoder_postnet/1
Available devices:
Device: 0 : ‘GeForce RTX 2080 Ti’, 68 SMs, support Co-op Launch ← [ ACTIVE ]
Conv selected stable alg 104
Conv selected stable alg 104
Conv selected stable alg 104
Selected stable alg 0
FC selected stable alg 35
2021-05-24 03:03:10,515 [INFO] Extract_binaries for waveglow → /data/models/jarvis-trt-waveglow/1
Available devices:
Device: 0 : ‘GeForce RTX 2080 Ti’, 68 SMs, support Co-op Launch ← [ ACTIVE ]
Tensor ‘spect’ ={1 1 80 80 }
Tensor ‘spect’ ={8 1 80 80 }
Tensor ‘z’ ={1 8 2656 1 }
Tensor ‘z’ ={8 8 2656 1 }
2021-05-24 03:08:27,788 [INFO] Extract_binaries for denoiser → /data/models/waveglow_denoiser/1
Available devices:
Device: 0 : ‘GeForce RTX 2080 Ti’, 68 SMs, support Co-op Launch ← [ ACTIVE ]
2021-05-24 03:08:46,626 [INFO] Extract_binaries for self → /data/models/tacotron2_ensemble/1
2021-05-24 03:08:47,412 [WARNING] /data/models/jarvis_tokenizer already exists, skipping deployment. To force deployment rerun with -f or remove the /data/models/jarvis_tokenizer
2021-05-24 03:08:47,412 [INFO] Extract_binaries for language_model → /data/models/jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased/1
2021-05-24 03:08:50,895 [INFO] Building TRT engine from PyTorch Checkpoint
2021-05-24 03:09:34,797 [INFO] Intent classes: 18
2021-05-24 03:09:34,797 [INFO] Entity classes: 31
2021-05-24 03:09:34,798 [INFO] Extract_binaries for label_tokens → /data/models/jarvis_label_tokens_weather/1
2021-05-24 03:09:35,581 [WARNING] /data/models/jarvis_detokenize already exists, skipping deployment. To force deployment rerun with -f or remove the /data/models/jarvis_detokenize
2021-05-24 03:09:35,581 [INFO] Extract_binaries for self → /data/models/jarvis_intent_weather/1
2021-05-24 03:09:43,629 [WARNING] Value for attribute use_subword not found in config, setting to default
2021-05-24 03:09:43,629 [WARNING] Value for attribute vocab_file not found in config, setting to default
2021-05-24 03:09:43,629 [WARNING] Value for attribute model_class not found in config, setting to default
2021-05-24 03:09:43,629 [WARNING] Value for attribute use_subword not found in config, setting to default
2021-05-24 03:09:43,629 [WARNING] Value for attribute use_subword not found in config, setting to default
2021-05-24 03:09:43,629 [WARNING] Value for attribute use_subword not found in config, setting to default
2021-05-24 03:09:43,791 [INFO] Extract_binaries for featurizer → /data/models/jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline/1
2021-05-24 03:09:43,793 [WARNING] /data/models/jarvis-trt-jasper already exists, skipping deployment. To force deployment rerun with -f or remove the /data/models/jarvis-trt-jasper
2021-05-24 03:09:43,793 [INFO] Extract_binaries for vad → /data/models/jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline/1
2021-05-24 03:09:43,793 [INFO] Extract_binaries for lm_decoder → /data/models/jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline/1
2021-05-24 03:09:51,354 [INFO] Extract_binaries for self → /data/models/jasper-asr-trt-ensemble-vad-streaming-offline/1
2021-05-24 03:09:52,137 [INFO] Extract_binaries for tokenizer → /data/models/jarvis_qa_preprocessor/1
2021-05-24 03:09:52,923 [INFO] Extract_binaries for language_model → /data/models/jarvis-trt-jarvis_qa-nn-bert-base-uncased/1
2021-05-24 03:09:55,759 [INFO] Building TRT engine from PyTorch Checkpoint
2021-05-24 03:10:58,983 [INFO] QA dimensions:(-1, 384, 2, 1, 1)
2021-05-24 03:10:58,984 [INFO] Extract_binaries for token_classifier → /data/models/jarvis_qa_postprocessor/1
2021-05-24 03:10:58,984 [INFO] Extract_binaries for self → /data/models/jarvis_qa/1
2021-05-24 03:10:59,762 [WARNING] /data/models/jarvis_tokenizer already exists, skipping deployment. To force deployment rerun with -f or remove the /data/models/jarvis_tokenizer
2021-05-24 03:10:59,762 [INFO] Extract_binaries for language_model → /data/models/jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased/1
2021-05-24 03:11:03,238 [INFO] Building TRT engine from PyTorch Checkpoint
2021-05-24 03:11:45,251 [INFO] Text Classification classes:4
2021-05-24 03:11:45,252 [INFO] Extract_binaries for self → /data/models/jarvis_text_classification_domain/1
-
echo
-
echo ‘Jarvis initialization complete. Run ./jarvis_start.sh to launch services.’
Jarvis initialization complete. Run ./jarvis_start.sh to launch services.
nvidia-smi output
Mon May 24 11:53:29 2021
±----------------------------------------------------------------------------+
| NVIDIA-SMI 460.80 Driver Version: 460.80 CUDA Version: 11.2 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 208… Off | 00000000:01:00.0 On | N/A |
| 26% 34C P8 21W / 260W | 409MiB / 11016MiB | 15% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1127 G /usr/lib/xorg/Xorg 329MiB |
| 0 N/A N/A 1372 G /usr/bin/gnome-shell 59MiB |
| 0 N/A N/A 16842 G /usr/lib/firefox/firefox 2MiB |
| 0 N/A N/A 18598 G …AAAAAAAAA= --shared-files 10MiB |
| 0 N/A N/A 20164 G /usr/lib/firefox/firefox 2MiB |
±----------------------------------------------------------------------------+
Below is the docker log
==========================
== Jarvis Speech Skills ==
NVIDIA Release 21.04 (build 22933806)
Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.
Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying
project or file.
NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be
insufficient for the inference server. NVIDIA recommends the use of the following flags:
nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 …
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:32:54.466813 70 metrics.cc:221] Collecting metrics for GPU 0: GeForce RTX 2080 Ti
I0524 03:32:54.469289 70 onnxruntime.cc:1728] TRITONBACKEND_Initialize: onnxruntime
I0524 03:32:54.469299 70 onnxruntime.cc:1738] Triton TRITONBACKEND API version: 1.0
I0524 03:32:54.469301 70 onnxruntime.cc:1744] 'onnxruntime' TRITONBACKEND API version: 1.0
I0524 03:32:54.585770 70 pinned_memory_manager.cc:205] Pinned memory pool is created at '0x7fb3da000000' with size 268435456
I0524 03:32:54.585978 70 cuda_memory_manager.cc:103] CUDA memory pool is created on device 0 with size 1000000000
I0524 03:32:54.594255 70 model_repository_manager.cc:1065] loading: jarvis-trt-jarvis_punctuation-nn-bert-base-uncased:1
I0524 03:32:54.694451 70 model_repository_manager.cc:1065] loading: jarvis-trt-jarvis_ner-nn-bert-base-uncased:1
I0524 03:32:54.794679 70 model_repository_manager.cc:1065] loading: jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased:1
I0524 03:32:54.894908 70 model_repository_manager.cc:1065] loading: jarvis-trt-jarvis_qa-nn-bert-base-uncased:1
I0524 03:32:54.995113 70 model_repository_manager.cc:1065] loading: jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased:1
I0524 03:32:55.095390 70 model_repository_manager.cc:1065] loading: jarvis-trt-jasper:1
I0524 03:32:55.195794 70 model_repository_manager.cc:1065] loading: jarvis-trt-tacotron2_encoder:1
I0524 03:32:55.296231 70 model_repository_manager.cc:1065] loading: jarvis-trt-waveglow:1
I0524 03:32:55.396647 70 model_repository_manager.cc:1065] loading: jarvis_detokenize:1
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:32:55.500753 70 model_repository_manager.cc:1065] loading: jarvis_label_tokens_weather:1
I0524 03:32:55.501103 70 custom_backend.cc:198] Creating instance jarvis_detokenize_0_0_cpu on CPU using libtriton_jarvis_nlp_detokenizer.so
I0524 03:32:55.502062 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_detokenize' version 1
I0524 03:32:55.608290 70 model_repository_manager.cc:1065] loading: jarvis_ner_label_tokens:1
I0524 03:32:55.608574 70 custom_backend.cc:198] Creating instance jarvis_label_tokens_weather_0_0_cpu on CPU using libtriton_jarvis_nlp_seqlabel.so
I0524 03:32:55.609726 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_label_tokens_weather' version 1
I0524 03:32:55.708591 70 model_repository_manager.cc:1065] loading: jarvis_punctuation_gen_output:1
I0524 03:32:55.708764 70 custom_backend.cc:198] Creating instance jarvis_ner_label_tokens_0_0_cpu on CPU using libtriton_jarvis_nlp_seqlabel.so
I0524 03:32:55.709033 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_ner_label_tokens' version 1
I0524 03:32:55.808836 70 model_repository_manager.cc:1065] loading: jarvis_punctuation_label_tokens_cap:1
I0524 03:32:55.808992 70 custom_backend.cc:198] Creating instance jarvis_punctuation_gen_output_0_0_cpu on CPU using libtriton_jarvis_nlp_punctuation.so
I0524 03:32:55.809563 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_punctuation_gen_output' version 1
I0524 03:32:55.922997 70 model_repository_manager.cc:1065] loading: jarvis_punctuation_label_tokens_punct:1
I0524 03:32:55.923108 70 custom_backend.cc:198] Creating instance jarvis_punctuation_label_tokens_cap_0_0_cpu on CPU using libtriton_jarvis_nlp_seqlabel.so
I0524 03:32:55.923359 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_punctuation_label_tokens_cap' version 1
I0524 03:32:56.023186 70 model_repository_manager.cc:1065] loading: jarvis_punctuation_merge_labels:1
I0524 03:32:56.023299 70 custom_backend.cc:198] Creating instance jarvis_punctuation_label_tokens_punct_0_0_cpu on CPU using libtriton_jarvis_nlp_seqlabel.so
I0524 03:32:56.023497 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_punctuation_label_tokens_punct' version 1
I0524 03:32:56.123390 70 model_repository_manager.cc:1065] loading: jarvis_qa_postprocessor:1
I0524 03:32:56.123534 70 custom_backend.cc:198] Creating instance jarvis_punctuation_merge_labels_0_0_cpu on CPU using libtriton_jarvis_nlp_labels.so
I0524 03:32:56.124048 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_punctuation_merge_labels' version 1
I0524 03:32:56.223588 70 model_repository_manager.cc:1065] loading: jarvis_qa_preprocessor:1
I0524 03:32:56.223783 70 custom_backend.cc:198] Creating instance jarvis_qa_postprocessor_0_0_cpu on CPU using libtriton_jarvis_nlp_qa.so
I0524 03:32:56.226974 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_qa_postprocessor' version 1
I0524 03:32:56.323788 70 model_repository_manager.cc:1065] loading: jarvis_tokenizer:1
I0524 03:32:56.323910 70 custom_backend.cc:198] Creating instance jarvis_qa_preprocessor_0_0_cpu on CPU using libtriton_jarvis_nlp_tokenizer.so
I0524 03:32:56.334857 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_qa_preprocessor' version 1
I0524 03:32:56.423986 70 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming:1
I0524 03:32:56.424206 70 custom_backend.cc:198] Creating instance jarvis_tokenizer_0_0_cpu on CPU using libtriton_jarvis_nlp_tokenizer.so
I0524 03:32:56.432404 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_tokenizer' version 1
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:32:56.524216 70 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming:1
I0524 03:32:56.524436 70 custom_backend.cc:198] Creating instance jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming_0_0_cpu on CPU using libtriton_jarvis_asr_decoder_cpu.so
E:decoder_context.cc:696: vocabulary parameter will be deprecated. Please use vocab_file instead.
I0524 03:32:56.624430 70 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline:1
I0524 03:32:56.624626 70 custom_backend.cc:201] Creating instance jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming_0_0_gpu0 on GPU 0 (7.5) using libtriton_jarvis_asr_features.so
I0524 03:32:56.724663 70 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline:1
I0524 03:32:56.724836 70 custom_backend.cc:198] Creating instance jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline_0_0_cpu on CPU using libtriton_jarvis_asr_decoder_cpu.so
E:decoder_context.cc:696: vocabulary parameter will be deprecated. Please use vocab_file instead.
I0524 03:32:56.824923 70 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline:1
I0524 03:32:56.825164 70 custom_backend.cc:201] Creating instance jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline_0_0_gpu0 on GPU 0 (7.5) using libtriton_jarvis_asr_features.so
I0524 03:32:56.925150 70 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming:1
I0524 03:32:56.925277 70 custom_backend.cc:198] Creating instance jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline_0_0_cpu on CPU using libtriton_jarvis_asr_vad.so
E:voice-activity-detector.cc:414: vocabulary parameter will be deprecated. Please use vocab_file instead.
I0524 03:32:56.927910 70 model_repository_manager.cc:1239] successfully loaded 'jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline' version 1
I0524 03:32:57.025406 70 model_repository_manager.cc:1065] loading: tacotron2_decoder_postnet:1
I0524 03:32:57.025569 70 custom_backend.cc:198] Creating instance jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming_0_0_cpu on CPU using libtriton_jarvis_asr_vad.so
E:voice-activity-detector.cc:414: vocabulary parameter will be deprecated. Please use vocab_file instead.
I0524 03:32:57.027931 70 model_repository_manager.cc:1239] successfully loaded 'jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming' version 1
I0524 03:32:57.125647 70 model_repository_manager.cc:1065] loading: tts_preprocessor:1
I0524 03:32:57.225862 70 model_repository_manager.cc:1065] loading: waveglow_denoiser:1
I0524 03:32:57.226098 70 custom_backend.cc:201] Creating instance tts_preprocessor_0_0_gpu0 on GPU 0 (7.5) using libtriton_jarvis_tts_preprocessor.so
I0524 03:32:57.326387 70 custom_backend.cc:201] Creating instance waveglow_denoiser_0_0_gpu0 on GPU 0 (7.5) using libtriton_jarvis_tts_denoiser.so
> Jarvis waiting for Triton server to load all models…retrying in 1 second
> Jarvis waiting for Triton server to load all models…retrying in 1 second
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:32:59.739185 70 tacotron-decoder-postnet.cc:873] TRITONBACKEND_ModelInitialize: tacotron2_decoder_postnet (version 1)
I0524 03:32:59.740133 70 tacotron-decoder-postnet.cc:767] model configuration:
{
"name": "tacotron2_decoder_postnet",
"platform": "",
"backend": "jarvis_tts_taco_postnet",
"version_policy": {
"latest": {
"num_versions": 1
}
},
"max_batch_size": 8,
"input": [
{
"name": "input_decoder",
"data_type": "TYPE_FP32",
"format": "FORMAT_NONE",
"dims": [
1,
400,
512
],
"is_shape_tensor": false,
"allow_ragged_batch": false
},
{
"name": "input_processed_decoder",
"data_type": "TYPE_FP32",
"format": "FORMAT_NONE",
"dims": [
400,
128,
1,
1
],
"is_shape_tensor": false,
"allow_ragged_batch": false
},
{
"name": "input_num_characters",
"data_type": "TYPE_INT32",
"format": "FORMAT_NONE",
"dims": [
1
],
"is_shape_tensor": false,
"allow_ragged_batch": false
}
],
"output": [
{
"name": "spectrogram_chunk",
"data_type": "TYPE_FP32",
"dims": [
1,
80,
80
],
"label_filename": "",
"is_shape_tensor": false
},
{
"name": "z",
"data_type": "TYPE_FP32",
"dims": [
8,
2656,
1
],
"label_filename": "",
"is_shape_tensor": false
},
{
"name": "num_valid_samples",
"data_type": "TYPE_INT32",
"dims": [
1
],
"label_filename": "",
"is_shape_tensor": false
},
{
"name": "end_flag",
"data_type": "TYPE_INT32",
"dims": [
1
],
"label_filename": "",
"is_shape_tensor": false
}
],
"batch_input": ,
"batch_output": ,
"optimization": {
"priority": "PRIORITY_DEFAULT",
"input_pinned_memory": {
"enable": true
},
"output_pinned_memory": {
"enable": true
},
"gather_kernel_buffer_threshold": 0,
"eager_batching": false
},
"sequence_batching": {
"oldest": {
"max_candidate_sequences": 8,
"preferred_batch_size": [
8
],
"max_queue_delay_microseconds": 100
},
"max_sequence_idle_microseconds": 60000000,
"control_input": [
{
"name": "START",
"control": [
{
"kind": "CONTROL_SEQUENCE_START",
"int32_false_true": [
0,
1
],
"fp32_false_true": ,
"data_type": "TYPE_INVALID"
}
]
},
{
"name": "READY",
"control": [
{
"kind": "CONTROL_SEQUENCE_READY",
"int32_false_true": [
0,
1
],
"fp32_false_true": ,
"data_type": "TYPE_INVALID"
}
]
},
{
"name": "END",
"control": [
{
"kind": "CONTROL_SEQUENCE_END",
"int32_false_true": [
0,
1
],
"fp32_false_true": ,
"data_type": "TYPE_INVALID"
}
]
},
{
"name": "CORRID",
"control": [
{
"kind": "CONTROL_SEQUENCE_CORRID",
"int32_false_true": ,
"fp32_false_true": ,
"data_type": "TYPE_UINT64"
}
]
}
]
},
"instance_group": [
{
"name": "tacotron2_decoder_postnet_0",
"kind": "KIND_GPU",
"count": 1,
"gpus": [
0
],
"profile":
}
],
"default_model_filename": "",
"cc_model_filenames": {},
"metric_tags": {},
"parameters": {
"z_dim0": {
"string_value": "8"
},
"z_dim1": {
"string_value": "2656"
},
"tacotron_decoder_engine": {
"string_value": "/data/models/tacotron2_decoder_postnet/1/model.plan"
},
"num_mels": {
"string_value": "80"
},
"encoding_dimension": {
"string_value": "512"
},
"max_execution_batch_size": {
"string_value": "8"
},
"chunk_length": {
"string_value": "80"
},
"max_input_length": {
"string_value": "400"
},
"attention_dimension": {
"string_value": "128"
},
"num_samples_per_frame": {
"string_value": "256"
}
},
"model_warmup": ,
"model_transaction_policy": {
"decoupled": true
}
}
I0524 03:32:59.740171 70 model_repository_manager.cc:1239] successfully loaded 'tts_preprocessor' version 1
I0524 03:32:59.740212 70 tacotron-decoder-postnet.cc:927] TRITONBACKEND_ModelInstanceInitialize: tacotron2_decoder_postnet_0 (device 0)
I0524 03:32:59.838815 70 model_repository_manager.cc:1239] successfully loaded 'jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming' version 1
I0524 03:33:00.040129 70 model_repository_manager.cc:1239] successfully loaded 'jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline' version 1
> Jarvis waiting for Triton server to load all models…retrying in 1 second
> Jarvis waiting for Triton server to load all models…retrying in 1 second
> Jarvis waiting for Triton server to load all models…retrying in 1 second
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:04.015767 70 model_repository_manager.cc:1239] successfully loaded 'jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming' version 1
I0524 03:33:04.015806 70 model_repository_manager.cc:1239] successfully loaded 'jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline' version 1
I0524 03:33:04.022598 70 plan_backend.cc:365] Creating instance jarvis-trt-tacotron2_encoder_0_0_gpu0 on GPU 0 (7.5) using model.plan
I0524 03:33:04.036793 70 model_repository_manager.cc:1239] successfully loaded 'waveglow_denoiser' version 1
I0524 03:33:04.037190 70 plan_backend.cc:748] Created instance jarvis-trt-tacotron2_encoder_0_0_gpu0 on GPU 0 with stream priority 0
I0524 03:33:04.040128 70 model_repository_manager.cc:1239] successfully loaded 'jarvis-trt-tacotron2_encoder' version 1
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:04.800884 70 plan_backend.cc:365] Creating instance jarvis-trt-jarvis_punctuation-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (7.5) using model.plan
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:05.552078 70 model_repository_manager.cc:1239] successfully loaded 'tacotron2_decoder_postnet' version 1
I0524 03:33:05.585996 70 plan_backend.cc:744] Created instance jarvis-trt-jarvis_punctuation-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0524 03:33:05.602902 70 model_repository_manager.cc:1239] successfully loaded 'jarvis-trt-jarvis_punctuation-nn-bert-base-uncased' version 1
I0524 03:33:06.352490 70 plan_backend.cc:365] Creating instance jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (7.5) using model.plan
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:07.129626 70 plan_backend.cc:744] Created instance jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0524 03:33:07.147901 70 model_repository_manager.cc:1239] successfully loaded 'jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased' version 1
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:07.883088 70 plan_backend.cc:365] Creating instance jarvis-trt-jarvis_ner-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (7.5) using model.plan
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:08.653217 70 plan_backend.cc:744] Created instance jarvis-trt-jarvis_ner-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0524 03:33:08.670945 70 model_repository_manager.cc:1239] successfully loaded 'jarvis-trt-jarvis_ner-nn-bert-base-uncased' version 1
I0524 03:33:09.401029 70 plan_backend.cc:365] Creating instance jarvis-trt-jarvis_qa-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (7.5) using model.plan
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:10.171243 70 plan_backend.cc:744] Created instance jarvis-trt-jarvis_qa-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0524 03:33:10.190198 70 model_repository_manager.cc:1239] successfully loaded 'jarvis-trt-jarvis_qa-nn-bert-base-uncased' version 1
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:10.925424 70 plan_backend.cc:365] Creating instance jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (7.5) using model.plan
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:11.698037 70 plan_backend.cc:744] Created instance jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0524 03:33:11.716006 70 model_repository_manager.cc:1239] successfully loaded 'jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased' version 1
> Jarvis waiting for Triton server to load all models…retrying in 1 second
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:13.868273 70 plan_backend.cc:365] Creating instance jarvis-trt-jasper_0_0_gpu0 on GPU 0 (7.5) using model.plan
> Jarvis waiting for Triton server to load all models…retrying in 1 second
> Jarvis waiting for Triton server to load all models…retrying in 1 second
E0524 03:33:16.038101 70 logging.cc:43] …/rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
E0524 03:33:16.038607 70 logging.cc:43] INTERNAL_ERROR: std::exception
I0524 03:33:16.039370 70 plan_backend.cc:744] Created instance jarvis-trt-jasper_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0524 03:33:16.057359 70 model_repository_manager.cc:1239] successfully loaded 'jarvis-trt-jasper' version 1
E0524 03:33:16.332191 70 logging.cc:43] …/rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
E0524 03:33:16.332523 70 logging.cc:43] INVALID_STATE: std::exception
E0524 03:33:16.332679 70 logging.cc:43] INVALID_CONFIG: Deserialize the cuda engine failed.
E0524 03:33:16.347702 70 model_repository_manager.cc:1242] failed to load 'jarvis-trt-waveglow' version 1: Internal: unable to create TensorRT engine
E0524 03:33:16.348634 70 model_repository_manager.cc:1430] Invalid argument: ensemble 'tacotron2_ensemble' depends on 'jarvis-trt-waveglow' which has no loaded version
I0524 03:33:16.348698 70 model_repository_manager.cc:1065] loading: jarvis_intent_weather:1
I0524 03:33:16.448843 70 model_repository_manager.cc:1065] loading: jarvis_ner:1
I0524 03:33:16.549389 70 model_repository_manager.cc:1065] loading: jarvis_punctuation:1
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:16.649955 70 model_repository_manager.cc:1065] loading: jarvis_qa:1
I0524 03:33:16.750459 70 model_repository_manager.cc:1065] loading: jarvis_text_classification_domain:1
I0524 03:33:16.851070 70 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming:1
I0524 03:33:16.851234 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_text_classification_domain' version 1
I0524 03:33:16.951649 70 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-offline:1
I0524 03:33:17.052078 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_ner' version 1
I0524 03:33:17.052167 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_qa' version 1
I0524 03:33:17.052229 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_intent_weather' version 1
I0524 03:33:17.052325 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_punctuation' version 1
I0524 03:33:17.052421 70 model_repository_manager.cc:1239] successfully loaded 'jasper-asr-trt-ensemble-vad-streaming' version 1
I0524 03:33:17.052630 70 model_repository_manager.cc:1239] successfully loaded 'jasper-asr-trt-ensemble-vad-streaming-offline' version 1
I0524 03:33:17.052882 70 server.cc:500]
±-----------------±-----+
| Repository Agent | Path |
±-----------------±-----+
±-----------------±-----+
I0524 03:33:17.053017 70 server.cc:527]
±------------------------±----------------------------------------------------------------------------------------±-------+
| Backend | Path | Config |
±------------------------±----------------------------------------------------------------------------------------±-------+
| onnxruntime | /opt/tritonserver/backends/onnxruntime/libtriton_onnxruntime.so | {} |
| jarvis_tts_taco_postnet | /opt/tritonserver/backends/jarvis_tts_taco_postnet/libtriton_jarvis_tts_taco_postnet.so | {} |
±------------------------±----------------------------------------------------------------------------------------±-------+
I0524 03:33:17.053696 70 server.cc:570]
±--------------------------------------------------------------------------------------------±--------±--------------------------------------------------------+
| Model | Version | Status |
±--------------------------------------------------------------------------------------------±--------±--------------------------------------------------------+
| jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jarvis_ner-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jarvis_punctuation-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jarvis_qa-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jasper | 1 | READY |
| jarvis-trt-tacotron2_encoder | 1 | READY |
| jarvis-trt-waveglow | 1 | UNAVAILABLE: Internal: unable to create TensorRT engine |
| jarvis_detokenize | 1 | READY |
| jarvis_intent_weather | 1 | READY |
| jarvis_label_tokens_weather | 1 | READY |
| jarvis_ner | 1 | READY |
| jarvis_ner_label_tokens | 1 | READY |
| jarvis_punctuation | 1 | READY |
| jarvis_punctuation_gen_output | 1 | READY |
| jarvis_punctuation_label_tokens_cap | 1 | READY |
| jarvis_punctuation_label_tokens_punct | 1 | READY |
| jarvis_punctuation_merge_labels | 1 | READY |
| jarvis_qa | 1 | READY |
| jarvis_qa_postprocessor | 1 | READY |
| jarvis_qa_preprocessor | 1 | READY |
| jarvis_text_classification_domain | 1 | READY |
| jarvis_tokenizer | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming | 1 | READY |
| tacotron2_decoder_postnet | 1 | READY |
| tts_preprocessor | 1 | READY |
| waveglow_denoiser | 1 | READY |
±--------------------------------------------------------------------------------------------±--------±--------------------------------------------------------+
I0524 03:33:17.054029 70 tritonserver.cc:1658]
±---------------------------------±---------------------------------------------------------------------------------------------------------------------------------------------------+
| Option | Value |
±---------------------------------±---------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.8.0 |
| server_extensions | classification sequence model_repository schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics |
| model_repository_path[0] | /data/models |
| model_control_mode | MODE_NONE |
| strict_model_config | 1 |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 1000000000 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
±---------------------------------±---------------------------------------------------------------------------------------------------------------------------------------------------+
I0524 03:33:17.054058 70 server.cc:233] Waiting for in-flight requests to complete.
I0524 03:33:17.054072 70 model_repository_manager.cc:1098] unloading: waveglow_denoiser:1
I0524 03:33:17.054157 70 model_repository_manager.cc:1098] unloading: tts_preprocessor:1
I0524 03:33:17.054332 70 model_repository_manager.cc:1098] unloading: tacotron2_decoder_postnet:1
I0524 03:33:17.054677 70 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline:1
I0524 03:33:17.055281 70 model_repository_manager.cc:1098] unloading: jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased:1
I0524 03:33:17.055399 70 tacotron-decoder-postnet.cc:1000] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0524 03:33:17.055476 70 model_repository_manager.cc:1098] unloading: jarvis-trt-jarvis_qa-nn-bert-base-uncased:1
I0524 03:33:17.055991 70 model_repository_manager.cc:1098] unloading: jarvis-trt-jarvis_ner-nn-bert-base-uncased:1
I0524 03:33:17.056307 70 model_repository_manager.cc:1222] successfully unloaded 'tts_preprocessor' version 1
I0524 03:33:17.056486 70 model_repository_manager.cc:1098] unloading: jarvis-trt-jasper:1
I0524 03:33:17.056583 70 model_repository_manager.cc:1098] unloading: jarvis-trt-jarvis_punctuation-nn-bert-base-uncased:1
I0524 03:33:17.056765 70 model_repository_manager.cc:1098] unloading: jarvis_punctuation_label_tokens_punct:1
I0524 03:33:17.056922 70 model_repository_manager.cc:1098] unloading: jarvis_ner_label_tokens:1
I0524 03:33:17.057074 70 model_repository_manager.cc:1098] unloading: jarvis_qa:1
I0524 03:33:17.057103 70 model_repository_manager.cc:1098] unloading: jarvis_label_tokens_weather:1
I0524 03:33:17.057133 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_qa' version 1
I0524 03:33:17.057184 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_punctuation_label_tokens_punct' version 1
I0524 03:33:17.057186 70 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline:1
I0524 03:33:17.057229 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_ner_label_tokens' version 1
I0524 03:33:17.057248 70 model_repository_manager.cc:1098] unloading: jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased:1
I0524 03:33:17.057274 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_label_tokens_weather' version 1
I0524 03:33:17.057294 70 model_repository_manager.cc:1098] unloading: jarvis_punctuation:1
I0524 03:33:17.057312 70 model_repository_manager.cc:1098] unloading: jarvis_intent_weather:1
I0524 03:33:17.057344 70 model_repository_manager.cc:1098] unloading: jarvis_punctuation_gen_output:1
I0524 03:33:17.057376 70 model_repository_manager.cc:1098] unloading: jarvis_punctuation_label_tokens_cap:1
I0524 03:33:17.057409 70 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline:1
I0524 03:33:17.057451 70 model_repository_manager.cc:1098] unloading: jarvis_punctuation_merge_labels:1
I0524 03:33:17.057502 70 model_repository_manager.cc:1098] unloading: jarvis_qa_postprocessor:1
I0524 03:33:17.057518 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_intent_weather' version 1
I0524 03:33:17.057538 70 model_repository_manager.cc:1098] unloading: jarvis_ner:1
I0524 03:33:17.057586 70 model_repository_manager.cc:1098] unloading: jarvis_qa_preprocessor:1
I0524 03:33:17.057595 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_punctuation' version 1
I0524 03:33:17.057625 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_ner' version 1
I0524 03:33:17.057635 70 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming:1
I0524 03:33:17.057653 70 model_repository_manager.cc:1098] unloading: jarvis_detokenize:1
I0524 03:33:17.059127 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_punctuation_label_tokens_cap' version 1
I0524 03:33:17.070820 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_punctuation_gen_output' version 1
I0524 03:33:17.071402 70 model_repository_manager.cc:1098] unloading: jarvis_text_classification_domain:1
I0524 03:33:17.071423 70 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming:1
I0524 03:33:17.071440 70 model_repository_manager.cc:1098] unloading: jarvis-trt-tacotron2_encoder:1
I0524 03:33:17.071464 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_punctuation_merge_labels' version 1
I0524 03:33:17.071476 70 model_repository_manager.cc:1098] unloading: jarvis_tokenizer:1
I0524 03:33:17.071468 70 model_repository_manager.cc:1222] successfully unloaded 'jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline' version 1
I0524 03:33:17.071662 70 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming:1
I0524 03:33:17.071688 70 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming:1
I0524 03:33:17.071794 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_qa_postprocessor' version 1
I0524 03:33:17.071830 70 model_repository_manager.cc:1222] successfully unloaded 'jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline' version 1
I0524 03:33:17.071848 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_qa_preprocessor' version 1
I0524 03:33:17.072025 70 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-offline:1
I0524 03:33:17.072062 70 server.cc:248] Timeout 30: Found 18 live models and 0 in-flight non-inference requests
I0524 03:33:17.072102 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_detokenize' version 1
I0524 03:33:17.083186 70 model_repository_manager.cc:1222] successfully unloaded 'jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming' version 1
I0524 03:33:17.083285 70 model_repository_manager.cc:1222] successfully unloaded 'jasper-asr-trt-ensemble-vad-streaming' version 1
I0524 03:33:17.083294 70 model_repository_manager.cc:1222] successfully unloaded 'jasper-asr-trt-ensemble-vad-streaming-offline' version 1
I0524 03:33:17.083636 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_tokenizer' version 1
I0524 03:33:17.084985 70 model_repository_manager.cc:1222] successfully unloaded 'jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming' version 1
I0524 03:33:17.085914 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_text_classification_domain' version 1
I0524 03:33:17.100747 70 model_repository_manager.cc:1222] successfully unloaded 'waveglow_denoiser' version 1
I0524 03:33:17.107792 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis-trt-jasper' version 1
I0524 03:33:17.159940 70 model_repository_manager.cc:1222] successfully unloaded 'jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming' version 1
I0524 03:33:17.167304 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis-trt-jarvis_ner-nn-bert-base-uncased' version 1
I0524 03:33:17.167384 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis-trt-jarvis_punctuation-nn-bert-base-uncased' version 1
I0524 03:33:17.167415 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis-trt-jarvis_qa-nn-bert-base-uncased' version 1
I0524 03:33:17.170428 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased' version 1
I0524 03:33:17.171452 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis-trt-tacotron2_encoder' version 1
I0524 03:33:17.185409 70 model_repository_manager.cc:1222] successfully unloaded 'jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline' version 1
I0524 03:33:17.188750 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased' version 1
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:18.072234 70 server.cc:248] Timeout 29: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:19.072440 70 server.cc:248] Timeout 28: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:20.072649 70 server.cc:248] Timeout 27: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:21.072849 70 server.cc:248] Timeout 26: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:22.073053 70 server.cc:248] Timeout 25: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:23.073256 70 server.cc:248] Timeout 24: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:24.073467 70 server.cc:248] Timeout 23: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:25.073684 70 server.cc:248] Timeout 22: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:26.073897 70 server.cc:248] Timeout 21: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:27.074118 70 server.cc:248] Timeout 20: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:28.074337 70 server.cc:248] Timeout 19: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:29.074547 70 server.cc:248] Timeout 18: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:30.074765 70 server.cc:248] Timeout 17: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:31.074976 70 server.cc:248] Timeout 16: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:32.075183 70 server.cc:248] Timeout 15: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:33.075389 70 server.cc:248] Timeout 14: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:34.075598 70 server.cc:248] Timeout 13: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:35.075810 70 server.cc:248] Timeout 12: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:36.076021 70 server.cc:248] Timeout 11: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:37.076229 70 server.cc:248] Timeout 10: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:38.076435 70 server.cc:248] Timeout 9: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:39.076648 70 server.cc:248] Timeout 8: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:40.076858 70 server.cc:248] Timeout 7: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:41.077068 70 server.cc:248] Timeout 6: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:42.077272 70 server.cc:248] Timeout 5: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:43.077475 70 server.cc:248] Timeout 4: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:44.077639 70 server.cc:248] Timeout 3: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:45.077836 70 server.cc:248] Timeout 2: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:46.078067 70 server.cc:248] Timeout 1: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:47.078278 70 server.cc:248] Timeout 0: Found 1 live models and 0 in-flight non-inference requests
error: creating server: Internal - failed to load all models
> Jarvis waiting for Triton server to load all models…retrying in 1 second
> Triton server died before reaching ready state. Terminating Jarvis startup.
Check Triton logs with: docker logs
/opt/jarvis/bin/start-jarvis: line 1: kill: (70) - No such process