Health ready check failed error whe running bash jarvis_start.sh

Hello, im trying to execute bash jarvis_start.sh but im getting Health ready check failed error after many lines of Waiting for Jarvis server to load all models…retrying in 1- seconds. Would really appreciate some help. Below, i have included the output of running the folowing cli commands:

  1. bash jarvis_init.sh
  2. nvidia-smi
  3. bash jarvis_start.sh

bash jarvis_init.sh output

Logging into NGC docker registry if necessary…
Pulling required docker images if necessary…
Note: This may take some time, depending on the speed of your Internet connection.

Pulling Jarvis Speech Server images.
Pulling nvcr.io/nvidia/jarvis/jarvis-speech:1.1.0-beta-server. This may take some time…
Pulling nvcr.io/nvidia/jarvis/jarvis-speech-client:1.1.0-beta. This may take some time…
Pulling nvcr.io/nvidia/jarvis/jarvis-speech:1.1.0-beta-servicemaker. This may take some time…

Downloading models (JMIRs) from NGC…
Note: this may take some time, depending on the speed of your Internet connection.
To skip this process and use existing JMIRs set the location and corresponding flag in config.sh.

==========================
== Jarvis Speech Skills ==

NVIDIA Release (build 21060478)

Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying
project or file.

NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be
insufficient for the inference server. NVIDIA recommends the use of the following flags:
nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 …

/data/artifacts /opt/jarvis

Downloading nvidia/jarvis/jmir_punctuation:1.0.0-b.1…
Downloaded 418.11 MB in 38s, Download speed: 10.99 MB/s


Transfer id: jmir_punctuation_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_punctuation_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 418.11 MB
Started at: 2021-05-24 02:45:03.773658
Completed at: 2021-05-24 02:45:41.822732
Duration taken: 38s

Downloading nvidia/jarvis/jmir_jarvis_asr_jasper_english_streaming:1.0.0-b.1…
Downloaded 1.98 GB in 3m 27s, Download speed: 9.8 MB/s


Transfer id: jmir_jarvis_asr_jasper_english_streaming_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_jarvis_asr_jasper_english_streaming_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 1.98 GB
Started at: 2021-05-24 02:45:50.887673
Completed at: 2021-05-24 02:49:18.125809
Duration taken: 3m 27s

Downloading nvidia/jarvis/jmir_jarvis_asr_jasper_english_offline:1.0.0-b.1…
Downloaded 1.98 GB in 2m 35s, Download speed: 13.09 MB/s


Transfer id: jmir_jarvis_asr_jasper_english_offline_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_jarvis_asr_jasper_english_offline_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 1.98 GB
Started at: 2021-05-24 02:49:26.484392
Completed at: 2021-05-24 02:52:01.651045
Duration taken: 2m 35s

Directory jmir_punctuation_v1.0.0-b.1 already exists, skipping. Use ‘–force’ option to override.

Downloading nvidia/jarvis/jmir_named_entity_recognition:1.0.0-b.1…
Downloaded 420.38 MB in 38s, Download speed: 11.05 MB/s


Transfer id: jmir_named_entity_recognition_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_named_entity_recognition_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 420.38 MB
Started at: 2021-05-24 02:52:10.191968
Completed at: 2021-05-24 02:52:48.233847
Duration taken: 38s

Downloading nvidia/jarvis/jmir_intent_slot:1.0.0-b.1…
Downloaded 422.71 MB in 1m 5s, Download speed: 6.5 MB/s


Transfer id: jmir_intent_slot_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_intent_slot_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 422.71 MB
Started at: 2021-05-24 02:52:55.847139
Completed at: 2021-05-24 02:54:00.925772
Duration taken: 1m 5s

Downloading nvidia/jarvis/jmir_question_answering:1.0.0-b.1…
Downloaded 418.06 MB in 41s, Download speed: 10.18 MB/s


Transfer id: jmir_question_answering_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_question_answering_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 418.06 MB
Started at: 2021-05-24 02:54:08.527265
Completed at: 2021-05-24 02:54:49.574643
Duration taken: 41s

Downloading nvidia/jarvis/jmir_text_classification:1.0.0-b.1…
Downloaded 420.27 MB in 40s, Download speed: 10.49 MB/s


Transfer id: jmir_text_classification_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_text_classification_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 420.27 MB
Started at: 2021-05-24 02:54:58.275814
Completed at: 2021-05-24 02:55:38.322575
Duration taken: 40s

Downloading nvidia/jarvis/jmir_jarvis_tts_ljspeech:1.0.0-b.1…
Downloaded 527.36 MB in 49s, Download speed: 10.75 MB/s


Transfer id: jmir_jarvis_tts_ljspeech_v1.0.0-b.1 Download status: Completed.
Downloaded local path: /data/artifacts/jmir_jarvis_tts_ljspeech_v1.0.0-b.1
Total files downloaded: 1
Total downloaded size: 527.36 MB
Started at: 2021-05-24 02:55:46.052547
Completed at: 2021-05-24 02:56:35.111061
Duration taken: 49s

/opt/jarvis

Converting JMIRs at jarvis-model-repo/jmir to Jarvis Model repository.

==========================
== Jarvis Speech Skills ==

NVIDIA Release (build 21060478)

Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying
project or file.

NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be
insufficient for the inference server. NVIDIA recommends the use of the following flags:
nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 …

2021-05-24 02:56:39,137 [INFO] Writing Jarvis model repository to ‘/data/models’…
2021-05-24 02:56:39,137 [INFO] The jarvis model repo target directory is /data/models
2021-05-24 02:56:39,953 [INFO] Extract_binaries for tokenizer → /data/models/jarvis_tokenizer/1
2021-05-24 02:56:40,778 [INFO] Extract_binaries for language_model → /data/models/jarvis-trt-jarvis_ner-nn-bert-base-uncased/1
2021-05-24 02:56:44,460 [INFO] Building TRT engine from PyTorch Checkpoint
2021-05-24 02:57:28,821 [INFO] NER classes: 13
2021-05-24 02:57:28,821 [INFO] Extract_binaries for label_tokens → /data/models/jarvis_ner_label_tokens/1
2021-05-24 02:57:29,640 [INFO] Extract_binaries for detokenizer → /data/models/jarvis_detokenize/1
2021-05-24 02:57:29,640 [INFO] Extract_binaries for self → /data/models/jarvis_ner/1
2021-05-24 02:57:30,532 [WARNING] /data/models/jarvis_tokenizer already exists, skipping deployment. To force deployment rerun with -f or remove the /data/models/jarvis_tokenizer
2021-05-24 02:57:30,532 [INFO] Extract_binaries for language_model → /data/models/jarvis-trt-jarvis_punctuation-nn-bert-base-uncased/1
2021-05-24 02:57:33,555 [INFO] Building TRT engine from PyTorch Checkpoint
2021-05-24 02:58:15,971 [INFO] Capit dimensions:(-1, 128, 2)
2021-05-24 02:58:15,971 [INFO] Punct dimensions:4
2021-05-24 02:58:15,972 [INFO] Extract_binaries for label_tokens_punct → /data/models/jarvis_punctuation_label_tokens_punct/1
2021-05-24 02:58:16,791 [INFO] Extract_binaries for label_tokens_cap → /data/models/jarvis_punctuation_label_tokens_cap/1
2021-05-24 02:58:17,614 [WARNING] /data/models/jarvis_detokenize already exists, skipping deployment. To force deployment rerun with -f or remove the /data/models/jarvis_detokenize
2021-05-24 02:58:17,615 [INFO] Extract_binaries for self → /data/models/jarvis_punctuation/1
2021-05-24 02:58:25,370 [WARNING] Value for attribute use_subword not found in config, setting to default
2021-05-24 02:58:25,370 [WARNING] Value for attribute vocab_file not found in config, setting to default
2021-05-24 02:58:25,370 [WARNING] Value for attribute model_class not found in config, setting to default
2021-05-24 02:58:25,370 [WARNING] Value for attribute use_subword not found in config, setting to default
2021-05-24 02:58:25,370 [WARNING] Value for attribute use_subword not found in config, setting to default
2021-05-24 02:58:25,370 [WARNING] Value for attribute use_subword not found in config, setting to default
2021-05-24 02:58:25,533 [INFO] Extract_binaries for featurizer → /data/models/jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming/1
2021-05-24 02:58:25,534 [INFO] Extract_binaries for nn → /data/models/jarvis-trt-jasper/1
2021-05-24 02:58:41,361 [INFO] Building TRT engine from ONNX file
2021-05-24 02:58:49,962 [INFO] Optimizing for FP16
[libprotobuf WARNING /home/jenkins/workspace/OSS/L0_MergeRequest/oss/build/third_party.protobuf/src/third_party.protobuf/src/google/protobuf/io/coded_stream.cc:604] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING /home/jenkins/workspace/OSS/L0_MergeRequest/oss/build/third_party.protobuf/src/third_party.protobuf/src/google/protobuf/io/coded_stream.cc:81] The total number of bytes read was 1331055768
[TensorRT] WARNING: /home/jenkins/workspace/OSS/L0_MergeRequest/oss/parsers/onnx/onnx2trt_utils.cpp:226: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
2021-05-24 03:00:46,295 [INFO] Extract_binaries for vad → /data/models/jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming/1
2021-05-24 03:00:46,295 [INFO] Extract_binaries for lm_decoder → /data/models/jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming/1
2021-05-24 03:00:54,035 [INFO] Extract_binaries for self → /data/models/jasper-asr-trt-ensemble-vad-streaming/1
2021-05-24 03:00:57,074 [INFO] Extract_binaries for preprocessor → /data/models/tts_preprocessor/1
2021-05-24 03:01:00,109 [INFO] Extract_binaries for encoder → /data/models/jarvis-trt-tacotron2_encoder/1
Available devices:
Device: 0 : ‘GeForce RTX 2080 Ti’, 68 SMs, support Co-op Launch ← [ ACTIVE ]
Conv selected stable alg 104
Conv selected stable alg 104
Conv selected stable alg 104
Selected stable alg 0
FC selected stable alg 35
2021-05-24 03:01:54,484 [INFO] Extract_binaries for decoder → /data/models/tacotron2_decoder_postnet/1
Available devices:
Device: 0 : ‘GeForce RTX 2080 Ti’, 68 SMs, support Co-op Launch ← [ ACTIVE ]
Conv selected stable alg 104
Conv selected stable alg 104
Conv selected stable alg 104
Selected stable alg 0
FC selected stable alg 35
2021-05-24 03:03:10,515 [INFO] Extract_binaries for waveglow → /data/models/jarvis-trt-waveglow/1
Available devices:
Device: 0 : ‘GeForce RTX 2080 Ti’, 68 SMs, support Co-op Launch ← [ ACTIVE ]
Tensor ‘spect’ ={1 1 80 80 }
Tensor ‘spect’ ={8 1 80 80 }
Tensor ‘z’ ={1 8 2656 1 }
Tensor ‘z’ ={8 8 2656 1 }
2021-05-24 03:08:27,788 [INFO] Extract_binaries for denoiser → /data/models/waveglow_denoiser/1
Available devices:
Device: 0 : ‘GeForce RTX 2080 Ti’, 68 SMs, support Co-op Launch ← [ ACTIVE ]
2021-05-24 03:08:46,626 [INFO] Extract_binaries for self → /data/models/tacotron2_ensemble/1
2021-05-24 03:08:47,412 [WARNING] /data/models/jarvis_tokenizer already exists, skipping deployment. To force deployment rerun with -f or remove the /data/models/jarvis_tokenizer
2021-05-24 03:08:47,412 [INFO] Extract_binaries for language_model → /data/models/jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased/1
2021-05-24 03:08:50,895 [INFO] Building TRT engine from PyTorch Checkpoint
2021-05-24 03:09:34,797 [INFO] Intent classes: 18
2021-05-24 03:09:34,797 [INFO] Entity classes: 31
2021-05-24 03:09:34,798 [INFO] Extract_binaries for label_tokens → /data/models/jarvis_label_tokens_weather/1
2021-05-24 03:09:35,581 [WARNING] /data/models/jarvis_detokenize already exists, skipping deployment. To force deployment rerun with -f or remove the /data/models/jarvis_detokenize
2021-05-24 03:09:35,581 [INFO] Extract_binaries for self → /data/models/jarvis_intent_weather/1
2021-05-24 03:09:43,629 [WARNING] Value for attribute use_subword not found in config, setting to default
2021-05-24 03:09:43,629 [WARNING] Value for attribute vocab_file not found in config, setting to default
2021-05-24 03:09:43,629 [WARNING] Value for attribute model_class not found in config, setting to default
2021-05-24 03:09:43,629 [WARNING] Value for attribute use_subword not found in config, setting to default
2021-05-24 03:09:43,629 [WARNING] Value for attribute use_subword not found in config, setting to default
2021-05-24 03:09:43,629 [WARNING] Value for attribute use_subword not found in config, setting to default
2021-05-24 03:09:43,791 [INFO] Extract_binaries for featurizer → /data/models/jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline/1
2021-05-24 03:09:43,793 [WARNING] /data/models/jarvis-trt-jasper already exists, skipping deployment. To force deployment rerun with -f or remove the /data/models/jarvis-trt-jasper
2021-05-24 03:09:43,793 [INFO] Extract_binaries for vad → /data/models/jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline/1
2021-05-24 03:09:43,793 [INFO] Extract_binaries for lm_decoder → /data/models/jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline/1
2021-05-24 03:09:51,354 [INFO] Extract_binaries for self → /data/models/jasper-asr-trt-ensemble-vad-streaming-offline/1
2021-05-24 03:09:52,137 [INFO] Extract_binaries for tokenizer → /data/models/jarvis_qa_preprocessor/1
2021-05-24 03:09:52,923 [INFO] Extract_binaries for language_model → /data/models/jarvis-trt-jarvis_qa-nn-bert-base-uncased/1
2021-05-24 03:09:55,759 [INFO] Building TRT engine from PyTorch Checkpoint
2021-05-24 03:10:58,983 [INFO] QA dimensions:(-1, 384, 2, 1, 1)
2021-05-24 03:10:58,984 [INFO] Extract_binaries for token_classifier → /data/models/jarvis_qa_postprocessor/1
2021-05-24 03:10:58,984 [INFO] Extract_binaries for self → /data/models/jarvis_qa/1
2021-05-24 03:10:59,762 [WARNING] /data/models/jarvis_tokenizer already exists, skipping deployment. To force deployment rerun with -f or remove the /data/models/jarvis_tokenizer
2021-05-24 03:10:59,762 [INFO] Extract_binaries for language_model → /data/models/jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased/1
2021-05-24 03:11:03,238 [INFO] Building TRT engine from PyTorch Checkpoint
2021-05-24 03:11:45,251 [INFO] Text Classification classes:4
2021-05-24 03:11:45,252 [INFO] Extract_binaries for self → /data/models/jarvis_text_classification_domain/1

  • echo

  • echo ‘Jarvis initialization complete. Run ./jarvis_start.sh to launch services.’
    Jarvis initialization complete. Run ./jarvis_start.sh to launch services.

nvidia-smi output

Mon May 24 11:53:29 2021
±----------------------------------------------------------------------------+
| NVIDIA-SMI 460.80 Driver Version: 460.80 CUDA Version: 11.2 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 208… Off | 00000000:01:00.0 On | N/A |
| 26% 34C P8 21W / 260W | 409MiB / 11016MiB | 15% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1127 G /usr/lib/xorg/Xorg 329MiB |
| 0 N/A N/A 1372 G /usr/bin/gnome-shell 59MiB |
| 0 N/A N/A 16842 G /usr/lib/firefox/firefox 2MiB |
| 0 N/A N/A 18598 G …AAAAAAAAA= --shared-files 10MiB |
| 0 N/A N/A 20164 G /usr/lib/firefox/firefox 2MiB |
±----------------------------------------------------------------------------+

Below is the docker log

==========================
== Jarvis Speech Skills ==

NVIDIA Release 21.04 (build 22933806)

Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying
project or file.

NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be
insufficient for the inference server. NVIDIA recommends the use of the following flags:
nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 …

> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:32:54.466813 70 metrics.cc:221] Collecting metrics for GPU 0: GeForce RTX 2080 Ti
I0524 03:32:54.469289 70 onnxruntime.cc:1728] TRITONBACKEND_Initialize: onnxruntime
I0524 03:32:54.469299 70 onnxruntime.cc:1738] Triton TRITONBACKEND API version: 1.0
I0524 03:32:54.469301 70 onnxruntime.cc:1744] 'onnxruntime' TRITONBACKEND API version: 1.0
I0524 03:32:54.585770 70 pinned_memory_manager.cc:205] Pinned memory pool is created at '0x7fb3da000000' with size 268435456
I0524 03:32:54.585978 70 cuda_memory_manager.cc:103] CUDA memory pool is created on device 0 with size 1000000000
I0524 03:32:54.594255 70 model_repository_manager.cc:1065] loading: jarvis-trt-jarvis_punctuation-nn-bert-base-uncased:1
I0524 03:32:54.694451 70 model_repository_manager.cc:1065] loading: jarvis-trt-jarvis_ner-nn-bert-base-uncased:1
I0524 03:32:54.794679 70 model_repository_manager.cc:1065] loading: jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased:1
I0524 03:32:54.894908 70 model_repository_manager.cc:1065] loading: jarvis-trt-jarvis_qa-nn-bert-base-uncased:1
I0524 03:32:54.995113 70 model_repository_manager.cc:1065] loading: jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased:1
I0524 03:32:55.095390 70 model_repository_manager.cc:1065] loading: jarvis-trt-jasper:1
I0524 03:32:55.195794 70 model_repository_manager.cc:1065] loading: jarvis-trt-tacotron2_encoder:1
I0524 03:32:55.296231 70 model_repository_manager.cc:1065] loading: jarvis-trt-waveglow:1
I0524 03:32:55.396647 70 model_repository_manager.cc:1065] loading: jarvis_detokenize:1
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:32:55.500753 70 model_repository_manager.cc:1065] loading: jarvis_label_tokens_weather:1
I0524 03:32:55.501103 70 custom_backend.cc:198] Creating instance jarvis_detokenize_0_0_cpu on CPU using libtriton_jarvis_nlp_detokenizer.so
I0524 03:32:55.502062 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_detokenize' version 1
I0524 03:32:55.608290 70 model_repository_manager.cc:1065] loading: jarvis_ner_label_tokens:1
I0524 03:32:55.608574 70 custom_backend.cc:198] Creating instance jarvis_label_tokens_weather_0_0_cpu on CPU using libtriton_jarvis_nlp_seqlabel.so
I0524 03:32:55.609726 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_label_tokens_weather' version 1
I0524 03:32:55.708591 70 model_repository_manager.cc:1065] loading: jarvis_punctuation_gen_output:1
I0524 03:32:55.708764 70 custom_backend.cc:198] Creating instance jarvis_ner_label_tokens_0_0_cpu on CPU using libtriton_jarvis_nlp_seqlabel.so
I0524 03:32:55.709033 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_ner_label_tokens' version 1
I0524 03:32:55.808836 70 model_repository_manager.cc:1065] loading: jarvis_punctuation_label_tokens_cap:1
I0524 03:32:55.808992 70 custom_backend.cc:198] Creating instance jarvis_punctuation_gen_output_0_0_cpu on CPU using libtriton_jarvis_nlp_punctuation.so
I0524 03:32:55.809563 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_punctuation_gen_output' version 1
I0524 03:32:55.922997 70 model_repository_manager.cc:1065] loading: jarvis_punctuation_label_tokens_punct:1
I0524 03:32:55.923108 70 custom_backend.cc:198] Creating instance jarvis_punctuation_label_tokens_cap_0_0_cpu on CPU using libtriton_jarvis_nlp_seqlabel.so
I0524 03:32:55.923359 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_punctuation_label_tokens_cap' version 1
I0524 03:32:56.023186 70 model_repository_manager.cc:1065] loading: jarvis_punctuation_merge_labels:1
I0524 03:32:56.023299 70 custom_backend.cc:198] Creating instance jarvis_punctuation_label_tokens_punct_0_0_cpu on CPU using libtriton_jarvis_nlp_seqlabel.so
I0524 03:32:56.023497 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_punctuation_label_tokens_punct' version 1
I0524 03:32:56.123390 70 model_repository_manager.cc:1065] loading: jarvis_qa_postprocessor:1
I0524 03:32:56.123534 70 custom_backend.cc:198] Creating instance jarvis_punctuation_merge_labels_0_0_cpu on CPU using libtriton_jarvis_nlp_labels.so
I0524 03:32:56.124048 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_punctuation_merge_labels' version 1
I0524 03:32:56.223588 70 model_repository_manager.cc:1065] loading: jarvis_qa_preprocessor:1
I0524 03:32:56.223783 70 custom_backend.cc:198] Creating instance jarvis_qa_postprocessor_0_0_cpu on CPU using libtriton_jarvis_nlp_qa.so
I0524 03:32:56.226974 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_qa_postprocessor' version 1
I0524 03:32:56.323788 70 model_repository_manager.cc:1065] loading: jarvis_tokenizer:1
I0524 03:32:56.323910 70 custom_backend.cc:198] Creating instance jarvis_qa_preprocessor_0_0_cpu on CPU using libtriton_jarvis_nlp_tokenizer.so
I0524 03:32:56.334857 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_qa_preprocessor' version 1
I0524 03:32:56.423986 70 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming:1
I0524 03:32:56.424206 70 custom_backend.cc:198] Creating instance jarvis_tokenizer_0_0_cpu on CPU using libtriton_jarvis_nlp_tokenizer.so
I0524 03:32:56.432404 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_tokenizer' version 1
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:32:56.524216 70 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming:1
I0524 03:32:56.524436 70 custom_backend.cc:198] Creating instance jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming_0_0_cpu on CPU using libtriton_jarvis_asr_decoder_cpu.so
E:decoder_context.cc:696: vocabulary parameter will be deprecated. Please use vocab_file instead.
I0524 03:32:56.624430 70 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline:1
I0524 03:32:56.624626 70 custom_backend.cc:201] Creating instance jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming_0_0_gpu0 on GPU 0 (7.5) using libtriton_jarvis_asr_features.so
I0524 03:32:56.724663 70 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline:1
I0524 03:32:56.724836 70 custom_backend.cc:198] Creating instance jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline_0_0_cpu on CPU using libtriton_jarvis_asr_decoder_cpu.so
E:decoder_context.cc:696: vocabulary parameter will be deprecated. Please use vocab_file instead.
I0524 03:32:56.824923 70 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline:1
I0524 03:32:56.825164 70 custom_backend.cc:201] Creating instance jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline_0_0_gpu0 on GPU 0 (7.5) using libtriton_jarvis_asr_features.so
I0524 03:32:56.925150 70 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming:1
I0524 03:32:56.925277 70 custom_backend.cc:198] Creating instance jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline_0_0_cpu on CPU using libtriton_jarvis_asr_vad.so
E:voice-activity-detector.cc:414: vocabulary parameter will be deprecated. Please use vocab_file instead.
I0524 03:32:56.927910 70 model_repository_manager.cc:1239] successfully loaded 'jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline' version 1
I0524 03:32:57.025406 70 model_repository_manager.cc:1065] loading: tacotron2_decoder_postnet:1
I0524 03:32:57.025569 70 custom_backend.cc:198] Creating instance jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming_0_0_cpu on CPU using libtriton_jarvis_asr_vad.so
E:voice-activity-detector.cc:414: vocabulary parameter will be deprecated. Please use vocab_file instead.
I0524 03:32:57.027931 70 model_repository_manager.cc:1239] successfully loaded 'jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming' version 1
I0524 03:32:57.125647 70 model_repository_manager.cc:1065] loading: tts_preprocessor:1
I0524 03:32:57.225862 70 model_repository_manager.cc:1065] loading: waveglow_denoiser:1
I0524 03:32:57.226098 70 custom_backend.cc:201] Creating instance tts_preprocessor_0_0_gpu0 on GPU 0 (7.5) using libtriton_jarvis_tts_preprocessor.so
I0524 03:32:57.326387 70 custom_backend.cc:201] Creating instance waveglow_denoiser_0_0_gpu0 on GPU 0 (7.5) using libtriton_jarvis_tts_denoiser.so
> Jarvis waiting for Triton server to load all models…retrying in 1 second
> Jarvis waiting for Triton server to load all models…retrying in 1 second
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:32:59.739185 70 tacotron-decoder-postnet.cc:873] TRITONBACKEND_ModelInitialize: tacotron2_decoder_postnet (version 1)
I0524 03:32:59.740133 70 tacotron-decoder-postnet.cc:767] model configuration:
{
"name": "tacotron2_decoder_postnet",
"platform": "",
"backend": "jarvis_tts_taco_postnet",
"version_policy": {
"latest": {
"num_versions": 1
}
},
"max_batch_size": 8,
"input": [
{
"name": "input_decoder",
"data_type": "TYPE_FP32",
"format": "FORMAT_NONE",
"dims": [
1,
400,
512
],
"is_shape_tensor": false,
"allow_ragged_batch": false
},
{
"name": "input_processed_decoder",
"data_type": "TYPE_FP32",
"format": "FORMAT_NONE",
"dims": [
400,
128,
1,
1
],
"is_shape_tensor": false,
"allow_ragged_batch": false
},
{
"name": "input_num_characters",
"data_type": "TYPE_INT32",
"format": "FORMAT_NONE",
"dims": [
1
],
"is_shape_tensor": false,
"allow_ragged_batch": false
}
],
"output": [
{
"name": "spectrogram_chunk",
"data_type": "TYPE_FP32",
"dims": [
1,
80,
80
],
"label_filename": "",
"is_shape_tensor": false
},
{
"name": "z",
"data_type": "TYPE_FP32",
"dims": [
8,
2656,
1
],
"label_filename": "",
"is_shape_tensor": false
},
{
"name": "num_valid_samples",
"data_type": "TYPE_INT32",
"dims": [
1
],
"label_filename": "",
"is_shape_tensor": false
},
{
"name": "end_flag",
"data_type": "TYPE_INT32",
"dims": [
1
],
"label_filename": "",
"is_shape_tensor": false
}
],
"batch_input": ,
"batch_output": ,
"optimization": {
"priority": "PRIORITY_DEFAULT",
"input_pinned_memory": {
"enable": true
},
"output_pinned_memory": {
"enable": true
},
"gather_kernel_buffer_threshold": 0,
"eager_batching": false
},
"sequence_batching": {
"oldest": {
"max_candidate_sequences": 8,
"preferred_batch_size": [
8
],
"max_queue_delay_microseconds": 100
},
"max_sequence_idle_microseconds": 60000000,
"control_input": [
{
"name": "START",
"control": [
{
"kind": "CONTROL_SEQUENCE_START",
"int32_false_true": [
0,
1
],
"fp32_false_true": ,
"data_type": "TYPE_INVALID"
}
]
},
{
"name": "READY",
"control": [
{
"kind": "CONTROL_SEQUENCE_READY",
"int32_false_true": [
0,
1
],
"fp32_false_true": ,
"data_type": "TYPE_INVALID"
}
]
},
{
"name": "END",
"control": [
{
"kind": "CONTROL_SEQUENCE_END",
"int32_false_true": [
0,
1
],
"fp32_false_true": ,
"data_type": "TYPE_INVALID"
}
]
},
{
"name": "CORRID",
"control": [
{
"kind": "CONTROL_SEQUENCE_CORRID",
"int32_false_true": ,
"fp32_false_true": ,
"data_type": "TYPE_UINT64"
}
]
}
]
},
"instance_group": [
{
"name": "tacotron2_decoder_postnet_0",
"kind": "KIND_GPU",
"count": 1,
"gpus": [
0
],
"profile":
}
],
"default_model_filename": "",
"cc_model_filenames": {},
"metric_tags": {},
"parameters": {
"z_dim0": {
"string_value": "8"
},
"z_dim1": {
"string_value": "2656"
},
"tacotron_decoder_engine": {
"string_value": "/data/models/tacotron2_decoder_postnet/1/model.plan"
},
"num_mels": {
"string_value": "80"
},
"encoding_dimension": {
"string_value": "512"
},
"max_execution_batch_size": {
"string_value": "8"
},
"chunk_length": {
"string_value": "80"
},
"max_input_length": {
"string_value": "400"
},
"attention_dimension": {
"string_value": "128"
},
"num_samples_per_frame": {
"string_value": "256"
}
},
"model_warmup": ,
"model_transaction_policy": {
"decoupled": true
}
}
I0524 03:32:59.740171 70 model_repository_manager.cc:1239] successfully loaded 'tts_preprocessor' version 1
I0524 03:32:59.740212 70 tacotron-decoder-postnet.cc:927] TRITONBACKEND_ModelInstanceInitialize: tacotron2_decoder_postnet_0 (device 0)
I0524 03:32:59.838815 70 model_repository_manager.cc:1239] successfully loaded 'jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming' version 1
I0524 03:33:00.040129 70 model_repository_manager.cc:1239] successfully loaded 'jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline' version 1
> Jarvis waiting for Triton server to load all models…retrying in 1 second
> Jarvis waiting for Triton server to load all models…retrying in 1 second
> Jarvis waiting for Triton server to load all models…retrying in 1 second
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:04.015767 70 model_repository_manager.cc:1239] successfully loaded 'jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming' version 1
I0524 03:33:04.015806 70 model_repository_manager.cc:1239] successfully loaded 'jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline' version 1
I0524 03:33:04.022598 70 plan_backend.cc:365] Creating instance jarvis-trt-tacotron2_encoder_0_0_gpu0 on GPU 0 (7.5) using model.plan
I0524 03:33:04.036793 70 model_repository_manager.cc:1239] successfully loaded 'waveglow_denoiser' version 1
I0524 03:33:04.037190 70 plan_backend.cc:748] Created instance jarvis-trt-tacotron2_encoder_0_0_gpu0 on GPU 0 with stream priority 0
I0524 03:33:04.040128 70 model_repository_manager.cc:1239] successfully loaded 'jarvis-trt-tacotron2_encoder' version 1
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:04.800884 70 plan_backend.cc:365] Creating instance jarvis-trt-jarvis_punctuation-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (7.5) using model.plan
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:05.552078 70 model_repository_manager.cc:1239] successfully loaded 'tacotron2_decoder_postnet' version 1
I0524 03:33:05.585996 70 plan_backend.cc:744] Created instance jarvis-trt-jarvis_punctuation-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0524 03:33:05.602902 70 model_repository_manager.cc:1239] successfully loaded 'jarvis-trt-jarvis_punctuation-nn-bert-base-uncased' version 1
I0524 03:33:06.352490 70 plan_backend.cc:365] Creating instance jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (7.5) using model.plan
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:07.129626 70 plan_backend.cc:744] Created instance jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0524 03:33:07.147901 70 model_repository_manager.cc:1239] successfully loaded 'jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased' version 1
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:07.883088 70 plan_backend.cc:365] Creating instance jarvis-trt-jarvis_ner-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (7.5) using model.plan
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:08.653217 70 plan_backend.cc:744] Created instance jarvis-trt-jarvis_ner-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0524 03:33:08.670945 70 model_repository_manager.cc:1239] successfully loaded 'jarvis-trt-jarvis_ner-nn-bert-base-uncased' version 1
I0524 03:33:09.401029 70 plan_backend.cc:365] Creating instance jarvis-trt-jarvis_qa-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (7.5) using model.plan
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:10.171243 70 plan_backend.cc:744] Created instance jarvis-trt-jarvis_qa-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0524 03:33:10.190198 70 model_repository_manager.cc:1239] successfully loaded 'jarvis-trt-jarvis_qa-nn-bert-base-uncased' version 1
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:10.925424 70 plan_backend.cc:365] Creating instance jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (7.5) using model.plan
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:11.698037 70 plan_backend.cc:744] Created instance jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0524 03:33:11.716006 70 model_repository_manager.cc:1239] successfully loaded 'jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased' version 1
> Jarvis waiting for Triton server to load all models…retrying in 1 second
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:13.868273 70 plan_backend.cc:365] Creating instance jarvis-trt-jasper_0_0_gpu0 on GPU 0 (7.5) using model.plan
> Jarvis waiting for Triton server to load all models…retrying in 1 second
> Jarvis waiting for Triton server to load all models…retrying in 1 second
E0524 03:33:16.038101 70 logging.cc:43] …/rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
E0524 03:33:16.038607 70 logging.cc:43] INTERNAL_ERROR: std::exception
I0524 03:33:16.039370 70 plan_backend.cc:744] Created instance jarvis-trt-jasper_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0524 03:33:16.057359 70 model_repository_manager.cc:1239] successfully loaded 'jarvis-trt-jasper' version 1
E0524 03:33:16.332191 70 logging.cc:43] …/rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
E0524 03:33:16.332523 70 logging.cc:43] INVALID_STATE: std::exception
E0524 03:33:16.332679 70 logging.cc:43] INVALID_CONFIG: Deserialize the cuda engine failed.
E0524 03:33:16.347702 70 model_repository_manager.cc:1242] failed to load 'jarvis-trt-waveglow' version 1: Internal: unable to create TensorRT engine
E0524 03:33:16.348634 70 model_repository_manager.cc:1430] Invalid argument: ensemble 'tacotron2_ensemble' depends on 'jarvis-trt-waveglow' which has no loaded version
I0524 03:33:16.348698 70 model_repository_manager.cc:1065] loading: jarvis_intent_weather:1
I0524 03:33:16.448843 70 model_repository_manager.cc:1065] loading: jarvis_ner:1
I0524 03:33:16.549389 70 model_repository_manager.cc:1065] loading: jarvis_punctuation:1
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:16.649955 70 model_repository_manager.cc:1065] loading: jarvis_qa:1
I0524 03:33:16.750459 70 model_repository_manager.cc:1065] loading: jarvis_text_classification_domain:1
I0524 03:33:16.851070 70 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming:1
I0524 03:33:16.851234 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_text_classification_domain' version 1
I0524 03:33:16.951649 70 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-offline:1
I0524 03:33:17.052078 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_ner' version 1
I0524 03:33:17.052167 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_qa' version 1
I0524 03:33:17.052229 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_intent_weather' version 1
I0524 03:33:17.052325 70 model_repository_manager.cc:1239] successfully loaded 'jarvis_punctuation' version 1
I0524 03:33:17.052421 70 model_repository_manager.cc:1239] successfully loaded 'jasper-asr-trt-ensemble-vad-streaming' version 1
I0524 03:33:17.052630 70 model_repository_manager.cc:1239] successfully loaded 'jasper-asr-trt-ensemble-vad-streaming-offline' version 1
I0524 03:33:17.052882 70 server.cc:500]
±-----------------±-----+
| Repository Agent | Path |
±-----------------±-----+
±-----------------±-----+

I0524 03:33:17.053017 70 server.cc:527]
±------------------------±----------------------------------------------------------------------------------------±-------+
| Backend | Path | Config |
±------------------------±----------------------------------------------------------------------------------------±-------+
| onnxruntime | /opt/tritonserver/backends/onnxruntime/libtriton_onnxruntime.so | {} |
| jarvis_tts_taco_postnet | /opt/tritonserver/backends/jarvis_tts_taco_postnet/libtriton_jarvis_tts_taco_postnet.so | {} |
±------------------------±----------------------------------------------------------------------------------------±-------+

I0524 03:33:17.053696 70 server.cc:570]
±--------------------------------------------------------------------------------------------±--------±--------------------------------------------------------+
| Model | Version | Status |
±--------------------------------------------------------------------------------------------±--------±--------------------------------------------------------+
| jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jarvis_ner-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jarvis_punctuation-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jarvis_qa-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jasper | 1 | READY |
| jarvis-trt-tacotron2_encoder | 1 | READY |
| jarvis-trt-waveglow | 1 | UNAVAILABLE: Internal: unable to create TensorRT engine |
| jarvis_detokenize | 1 | READY |
| jarvis_intent_weather | 1 | READY |
| jarvis_label_tokens_weather | 1 | READY |
| jarvis_ner | 1 | READY |
| jarvis_ner_label_tokens | 1 | READY |
| jarvis_punctuation | 1 | READY |
| jarvis_punctuation_gen_output | 1 | READY |
| jarvis_punctuation_label_tokens_cap | 1 | READY |
| jarvis_punctuation_label_tokens_punct | 1 | READY |
| jarvis_punctuation_merge_labels | 1 | READY |
| jarvis_qa | 1 | READY |
| jarvis_qa_postprocessor | 1 | READY |
| jarvis_qa_preprocessor | 1 | READY |
| jarvis_text_classification_domain | 1 | READY |
| jarvis_tokenizer | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming | 1 | READY |
| tacotron2_decoder_postnet | 1 | READY |
| tts_preprocessor | 1 | READY |
| waveglow_denoiser | 1 | READY |
±--------------------------------------------------------------------------------------------±--------±--------------------------------------------------------+

I0524 03:33:17.054029 70 tritonserver.cc:1658]
±---------------------------------±---------------------------------------------------------------------------------------------------------------------------------------------------+
| Option | Value |
±---------------------------------±---------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.8.0 |
| server_extensions | classification sequence model_repository schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics |
| model_repository_path[0] | /data/models |
| model_control_mode | MODE_NONE |
| strict_model_config | 1 |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 1000000000 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
±---------------------------------±---------------------------------------------------------------------------------------------------------------------------------------------------+

I0524 03:33:17.054058 70 server.cc:233] Waiting for in-flight requests to complete.
I0524 03:33:17.054072 70 model_repository_manager.cc:1098] unloading: waveglow_denoiser:1
I0524 03:33:17.054157 70 model_repository_manager.cc:1098] unloading: tts_preprocessor:1
I0524 03:33:17.054332 70 model_repository_manager.cc:1098] unloading: tacotron2_decoder_postnet:1
I0524 03:33:17.054677 70 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline:1
I0524 03:33:17.055281 70 model_repository_manager.cc:1098] unloading: jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased:1
I0524 03:33:17.055399 70 tacotron-decoder-postnet.cc:1000] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0524 03:33:17.055476 70 model_repository_manager.cc:1098] unloading: jarvis-trt-jarvis_qa-nn-bert-base-uncased:1
I0524 03:33:17.055991 70 model_repository_manager.cc:1098] unloading: jarvis-trt-jarvis_ner-nn-bert-base-uncased:1
I0524 03:33:17.056307 70 model_repository_manager.cc:1222] successfully unloaded 'tts_preprocessor' version 1
I0524 03:33:17.056486 70 model_repository_manager.cc:1098] unloading: jarvis-trt-jasper:1
I0524 03:33:17.056583 70 model_repository_manager.cc:1098] unloading: jarvis-trt-jarvis_punctuation-nn-bert-base-uncased:1
I0524 03:33:17.056765 70 model_repository_manager.cc:1098] unloading: jarvis_punctuation_label_tokens_punct:1
I0524 03:33:17.056922 70 model_repository_manager.cc:1098] unloading: jarvis_ner_label_tokens:1
I0524 03:33:17.057074 70 model_repository_manager.cc:1098] unloading: jarvis_qa:1
I0524 03:33:17.057103 70 model_repository_manager.cc:1098] unloading: jarvis_label_tokens_weather:1
I0524 03:33:17.057133 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_qa' version 1
I0524 03:33:17.057184 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_punctuation_label_tokens_punct' version 1
I0524 03:33:17.057186 70 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline:1
I0524 03:33:17.057229 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_ner_label_tokens' version 1
I0524 03:33:17.057248 70 model_repository_manager.cc:1098] unloading: jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased:1
I0524 03:33:17.057274 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_label_tokens_weather' version 1
I0524 03:33:17.057294 70 model_repository_manager.cc:1098] unloading: jarvis_punctuation:1
I0524 03:33:17.057312 70 model_repository_manager.cc:1098] unloading: jarvis_intent_weather:1
I0524 03:33:17.057344 70 model_repository_manager.cc:1098] unloading: jarvis_punctuation_gen_output:1
I0524 03:33:17.057376 70 model_repository_manager.cc:1098] unloading: jarvis_punctuation_label_tokens_cap:1
I0524 03:33:17.057409 70 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline:1
I0524 03:33:17.057451 70 model_repository_manager.cc:1098] unloading: jarvis_punctuation_merge_labels:1
I0524 03:33:17.057502 70 model_repository_manager.cc:1098] unloading: jarvis_qa_postprocessor:1
I0524 03:33:17.057518 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_intent_weather' version 1
I0524 03:33:17.057538 70 model_repository_manager.cc:1098] unloading: jarvis_ner:1
I0524 03:33:17.057586 70 model_repository_manager.cc:1098] unloading: jarvis_qa_preprocessor:1
I0524 03:33:17.057595 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_punctuation' version 1
I0524 03:33:17.057625 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_ner' version 1
I0524 03:33:17.057635 70 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming:1
I0524 03:33:17.057653 70 model_repository_manager.cc:1098] unloading: jarvis_detokenize:1
I0524 03:33:17.059127 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_punctuation_label_tokens_cap' version 1
I0524 03:33:17.070820 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_punctuation_gen_output' version 1
I0524 03:33:17.071402 70 model_repository_manager.cc:1098] unloading: jarvis_text_classification_domain:1
I0524 03:33:17.071423 70 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming:1
I0524 03:33:17.071440 70 model_repository_manager.cc:1098] unloading: jarvis-trt-tacotron2_encoder:1
I0524 03:33:17.071464 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_punctuation_merge_labels' version 1
I0524 03:33:17.071476 70 model_repository_manager.cc:1098] unloading: jarvis_tokenizer:1
I0524 03:33:17.071468 70 model_repository_manager.cc:1222] successfully unloaded 'jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline' version 1
I0524 03:33:17.071662 70 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming:1
I0524 03:33:17.071688 70 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming:1
I0524 03:33:17.071794 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_qa_postprocessor' version 1
I0524 03:33:17.071830 70 model_repository_manager.cc:1222] successfully unloaded 'jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline' version 1
I0524 03:33:17.071848 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_qa_preprocessor' version 1
I0524 03:33:17.072025 70 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-offline:1
I0524 03:33:17.072062 70 server.cc:248] Timeout 30: Found 18 live models and 0 in-flight non-inference requests
I0524 03:33:17.072102 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_detokenize' version 1
I0524 03:33:17.083186 70 model_repository_manager.cc:1222] successfully unloaded 'jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming' version 1
I0524 03:33:17.083285 70 model_repository_manager.cc:1222] successfully unloaded 'jasper-asr-trt-ensemble-vad-streaming' version 1
I0524 03:33:17.083294 70 model_repository_manager.cc:1222] successfully unloaded 'jasper-asr-trt-ensemble-vad-streaming-offline' version 1
I0524 03:33:17.083636 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_tokenizer' version 1
I0524 03:33:17.084985 70 model_repository_manager.cc:1222] successfully unloaded 'jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming' version 1
I0524 03:33:17.085914 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis_text_classification_domain' version 1
I0524 03:33:17.100747 70 model_repository_manager.cc:1222] successfully unloaded 'waveglow_denoiser' version 1
I0524 03:33:17.107792 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis-trt-jasper' version 1
I0524 03:33:17.159940 70 model_repository_manager.cc:1222] successfully unloaded 'jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming' version 1
I0524 03:33:17.167304 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis-trt-jarvis_ner-nn-bert-base-uncased' version 1
I0524 03:33:17.167384 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis-trt-jarvis_punctuation-nn-bert-base-uncased' version 1
I0524 03:33:17.167415 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis-trt-jarvis_qa-nn-bert-base-uncased' version 1
I0524 03:33:17.170428 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased' version 1
I0524 03:33:17.171452 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis-trt-tacotron2_encoder' version 1
I0524 03:33:17.185409 70 model_repository_manager.cc:1222] successfully unloaded 'jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline' version 1
I0524 03:33:17.188750 70 model_repository_manager.cc:1222] successfully unloaded 'jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased' version 1
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:18.072234 70 server.cc:248] Timeout 29: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:19.072440 70 server.cc:248] Timeout 28: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:20.072649 70 server.cc:248] Timeout 27: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:21.072849 70 server.cc:248] Timeout 26: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:22.073053 70 server.cc:248] Timeout 25: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:23.073256 70 server.cc:248] Timeout 24: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:24.073467 70 server.cc:248] Timeout 23: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:25.073684 70 server.cc:248] Timeout 22: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:26.073897 70 server.cc:248] Timeout 21: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:27.074118 70 server.cc:248] Timeout 20: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:28.074337 70 server.cc:248] Timeout 19: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:29.074547 70 server.cc:248] Timeout 18: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:30.074765 70 server.cc:248] Timeout 17: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:31.074976 70 server.cc:248] Timeout 16: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:32.075183 70 server.cc:248] Timeout 15: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:33.075389 70 server.cc:248] Timeout 14: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:34.075598 70 server.cc:248] Timeout 13: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:35.075810 70 server.cc:248] Timeout 12: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:36.076021 70 server.cc:248] Timeout 11: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:37.076229 70 server.cc:248] Timeout 10: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:38.076435 70 server.cc:248] Timeout 9: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:39.076648 70 server.cc:248] Timeout 8: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:40.076858 70 server.cc:248] Timeout 7: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:41.077068 70 server.cc:248] Timeout 6: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:42.077272 70 server.cc:248] Timeout 5: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:43.077475 70 server.cc:248] Timeout 4: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:44.077639 70 server.cc:248] Timeout 3: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:45.077836 70 server.cc:248] Timeout 2: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:46.078067 70 server.cc:248] Timeout 1: Found 1 live models and 0 in-flight non-inference requests
> Jarvis waiting for Triton server to load all models…retrying in 1 second
I0524 03:33:47.078278 70 server.cc:248] Timeout 0: Found 1 live models and 0 in-flight non-inference requests
error: creating server: Internal - failed to load all models
> Jarvis waiting for Triton server to load all models…retrying in 1 second
> Triton server died before reaching ready state. Terminating Jarvis startup.
Check Triton logs with: docker logs
/opt/jarvis/bin/start-jarvis: line 1: kill: (70) - No such process

Could you please try commenting out all the NLP models except 1 and see if that deploys successfully on your setup.

Thanks

Hello SunilJB, thank you for your reply!
Sorry im quite a newbie to this so this is what i interpreted your instructions to be, please tell me if i did anything wrong.

I opened config.sh using vim and changed the models_nlp section, this is how my config.sh looks like now:

Ports to expose for Jarvis services

jarvis_speech_api_port=“50051”
jarvis_vision_api_port=“60051”

NGC orgs

jarvis_ngc_org=“nvidia”
jarvis_ngc_team=“jarvis”
jarvis_ngc_image_version=“1.1.0-beta”
jarvis_ngc_model_version=“1.0.0-b.1”

Pre-built models listed below will be downloaded from NGC. If models already exist in $jarvis-jmir

then models can be commented out to skip download from NGC

models_asr=(

Punctuation model

"${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_punctuation:${jarvis_ngc_model_version}"

Jasper Streaming w/ CPU decoder, best latency configuration

"${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_jasper_english_streaming:${jarvis_ngc_model_version}"

Jasper Streaming w/ CPU decoder, best throughput configuration

“${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_jasper_english_streaming_throughput:${jarvis_ngc_model_version}”

Jasper Offline w/ CPU decoder

"${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_jasper_english_offline:${jarvis_ngc_model_version}"

Quarztnet Streaming w/ CPU decoder, best latency configuration

“${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_quartznet_english_streaming:${jarvis_ngc_model_version}”

Quarztnet Streaming w/ CPU decoder, best throughput configuration

“${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_quartznet_english_streaming_throughput:${jarvis_ngc_model_version}”

Quarztnet Offline w/ CPU decoder

“${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_quartznet_english_offline:${jarvis_ngc_model_version}”

Jasper Streaming w/ GPU decoder, best latency configuration

“${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_jasper_english_streaming_gpu_decoder:${jarvis_ngc_model_version}”

Jasper Streaming w/ GPU decoder, best throughput configuration

“${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_jasper_english_streaming_throughput_gpu_decoder:${jarvis_ngc_model_version}”

Jasper Offline w/ GPU decoder

“${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_jasper_english_offline_gpu_decoder:${jarvis_ngc_model_version}”

)

models_nlp=(
#"${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_punctuation:${jarvis_ngc_model_version}"
#"${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_named_entity_recognition:${jarvis_ngc_model_version}"
#"${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_intent_slot:${jarvis_ngc_model_version}"
#"${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_question_answering:${jarvis_ngc_model_version}"
“${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_text_classification:${jarvis_ngc_model_version}”
)
models_tts=(
“${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_tts_ljspeech:${jarvis_ngc_model_version}”
)

NGC_TARGET=${jarvis_ngc_org}
if [[ ! -z ${jarvis_ngc_team} ]]; then
NGC_TARGET="${NGC_TARGET}/${jarvis_ngc_team}"
else
team=""""
fi

define docker images required to run Jarvis

image_client=“nvcr.io/${NGC_TARGET}/jarvis-speech-client:${jarvis_ngc_image_version}
image_speech_api=“nvcr.io/${NGC_TARGET}/jarvis-speech:${jarvis_ngc_image_version}-server

define docker images required to setup Jarvis

image_init_speech=“nvcr.io/${NGC_TARGET}/jarvis-speech:${jarvis_ngc_image_version}-servicemaker

daemon names

jarvis_daemon_speech=“jarvis-speech”
jarvis_daemon_client=“jarvis-client”

I got the same error and this is what the logs look like

==========================
== Jarvis Speech Skills ==

NVIDIA Release 21.04 (build 22933806)

Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying
project or file.

NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be
insufficient for the inference server. NVIDIA recommends the use of the following flags:
nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 …

Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:30:37.906968 69 metrics.cc:221] Collecting metrics for GPU 0: NVIDIA GeForce RTX 2080 Ti
I0527 03:30:37.922599 69 onnxruntime.cc:1728] TRITONBACKEND_Initialize: onnxruntime
I0527 03:30:37.922767 69 onnxruntime.cc:1738] Triton TRITONBACKEND API version: 1.0
I0527 03:30:37.922771 69 onnxruntime.cc:1744] ‘onnxruntime’ TRITONBACKEND API version: 1.0
I0527 03:30:38.048329 69 pinned_memory_manager.cc:205] Pinned memory pool is created at ‘0x7f2e6a000000’ with size 268435456
I0527 03:30:38.048761 69 cuda_memory_manager.cc:103] CUDA memory pool is created on device 0 with size 1000000000
I0527 03:30:38.066276 69 model_repository_manager.cc:1065] loading: jarvis-trt-jarvis_punctuation-nn-bert-base-uncased:1
I0527 03:30:38.166711 69 model_repository_manager.cc:1065] loading: jarvis-trt-jarvis_ner-nn-bert-base-uncased:1
I0527 03:30:38.267020 69 model_repository_manager.cc:1065] loading: jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased:1
I0527 03:30:38.367230 69 model_repository_manager.cc:1065] loading: jarvis-trt-jarvis_qa-nn-bert-base-uncased:1
I0527 03:30:38.467457 69 model_repository_manager.cc:1065] loading: jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased:1
I0527 03:30:38.567673 69 model_repository_manager.cc:1065] loading: jarvis-trt-jasper:1
I0527 03:30:38.667913 69 model_repository_manager.cc:1065] loading: jarvis-trt-tacotron2_encoder:1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:30:38.768143 69 model_repository_manager.cc:1065] loading: jarvis-trt-waveglow:1
I0527 03:30:38.868427 69 model_repository_manager.cc:1065] loading: jarvis_detokenize:1
I0527 03:30:38.968679 69 model_repository_manager.cc:1065] loading: jarvis_label_tokens_weather:1
I0527 03:30:38.968904 69 custom_backend.cc:198] Creating instance jarvis_detokenize_0_0_cpu on CPU using libtriton_jarvis_nlp_detokenizer.so
I0527 03:30:38.998003 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis_detokenize’ version 1
I0527 03:30:39.068956 69 model_repository_manager.cc:1065] loading: jarvis_ner_label_tokens:1
I0527 03:30:39.069112 69 custom_backend.cc:198] Creating instance jarvis_label_tokens_weather_0_0_cpu on CPU using libtriton_jarvis_nlp_seqlabel.so
I0527 03:30:39.091327 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis_label_tokens_weather’ version 1
I0527 03:30:39.169144 69 model_repository_manager.cc:1065] loading: jarvis_punctuation_gen_output:1
I0527 03:30:39.169314 69 custom_backend.cc:198] Creating instance jarvis_ner_label_tokens_0_0_cpu on CPU using libtriton_jarvis_nlp_seqlabel.so
I0527 03:30:39.170446 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis_ner_label_tokens’ version 1
I0527 03:30:39.269355 69 model_repository_manager.cc:1065] loading: jarvis_punctuation_label_tokens_cap:1
I0527 03:30:39.269530 69 custom_backend.cc:198] Creating instance jarvis_punctuation_gen_output_0_0_cpu on CPU using libtriton_jarvis_nlp_punctuation.so
I0527 03:30:39.285083 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis_punctuation_gen_output’ version 1
I0527 03:30:39.369704 69 model_repository_manager.cc:1065] loading: jarvis_punctuation_label_tokens_punct:1
I0527 03:30:39.369836 69 custom_backend.cc:198] Creating instance jarvis_punctuation_label_tokens_cap_0_0_cpu on CPU using libtriton_jarvis_nlp_seqlabel.so
I0527 03:30:39.370327 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis_punctuation_label_tokens_cap’ version 1
I0527 03:30:39.469954 69 model_repository_manager.cc:1065] loading: jarvis_punctuation_merge_labels:1
I0527 03:30:39.470101 69 custom_backend.cc:198] Creating instance jarvis_punctuation_label_tokens_punct_0_0_cpu on CPU using libtriton_jarvis_nlp_seqlabel.so
I0527 03:30:39.471096 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis_punctuation_label_tokens_punct’ version 1
I0527 03:30:39.570231 69 model_repository_manager.cc:1065] loading: jarvis_qa_postprocessor:1
I0527 03:30:39.570375 69 custom_backend.cc:198] Creating instance jarvis_punctuation_merge_labels_0_0_cpu on CPU using libtriton_jarvis_nlp_labels.so
I0527 03:30:39.583090 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis_punctuation_merge_labels’ version 1
I0527 03:30:39.670480 69 model_repository_manager.cc:1065] loading: jarvis_qa_preprocessor:1
I0527 03:30:39.670700 69 custom_backend.cc:198] Creating instance jarvis_qa_postprocessor_0_0_cpu on CPU using libtriton_jarvis_nlp_qa.so
I0527 03:30:39.689564 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis_qa_postprocessor’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:30:39.770718 69 model_repository_manager.cc:1065] loading: jarvis_tokenizer:1
I0527 03:30:39.770852 69 custom_backend.cc:198] Creating instance jarvis_qa_preprocessor_0_0_cpu on CPU using libtriton_jarvis_nlp_tokenizer.so
I0527 03:30:39.789336 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis_qa_preprocessor’ version 1
I0527 03:30:39.870940 69 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming:1
I0527 03:30:39.871046 69 custom_backend.cc:198] Creating instance jarvis_tokenizer_0_0_cpu on CPU using libtriton_jarvis_nlp_tokenizer.so
I0527 03:30:39.887209 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis_tokenizer’ version 1
I0527 03:30:39.971160 69 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming:1
I0527 03:30:39.971282 69 custom_backend.cc:198] Creating instance jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming_0_0_cpu on CPU using libtriton_jarvis_asr_decoder_cpu.so
E:decoder_context.cc:696: vocabulary parameter will be deprecated. Please use vocab_file instead.
I0527 03:30:40.071370 69 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline:1
I0527 03:30:40.071571 69 custom_backend.cc:201] Creating instance jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming_0_0_gpu0 on GPU 0 (7.5) using libtriton_jarvis_asr_features.so
I0527 03:30:40.171579 69 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline:1
I0527 03:30:40.171720 69 custom_backend.cc:198] Creating instance jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline_0_0_cpu on CPU using libtriton_jarvis_asr_decoder_cpu.so
E:decoder_context.cc:696: vocabulary parameter will be deprecated. Please use vocab_file instead.
I0527 03:30:40.271783 69 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline:1
I0527 03:30:40.271993 69 custom_backend.cc:201] Creating instance jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline_0_0_gpu0 on GPU 0 (7.5) using libtriton_jarvis_asr_features.so
I0527 03:30:40.371987 69 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming:1
I0527 03:30:40.372101 69 custom_backend.cc:198] Creating instance jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline_0_0_cpu on CPU using libtriton_jarvis_asr_vad.so
E:voice-activity-detector.cc:414: vocabulary parameter will be deprecated. Please use vocab_file instead.
I0527 03:30:40.379255 69 model_repository_manager.cc:1239] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline’ version 1
I0527 03:30:40.472194 69 model_repository_manager.cc:1065] loading: tacotron2_decoder_postnet:1
I0527 03:30:40.472316 69 custom_backend.cc:198] Creating instance jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming_0_0_cpu on CPU using libtriton_jarvis_asr_vad.so
E:voice-activity-detector.cc:414: vocabulary parameter will be deprecated. Please use vocab_file instead.
I0527 03:30:40.474539 69 model_repository_manager.cc:1239] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming’ version 1
I0527 03:30:40.572401 69 model_repository_manager.cc:1065] loading: tts_preprocessor:1
I0527 03:30:40.672591 69 model_repository_manager.cc:1065] loading: waveglow_denoiser:1
I0527 03:30:40.672792 69 custom_backend.cc:201] Creating instance tts_preprocessor_0_0_gpu0 on GPU 0 (7.5) using libtriton_jarvis_tts_preprocessor.so
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:30:40.772971 69 custom_backend.cc:201] Creating instance waveglow_denoiser_0_0_gpu0 on GPU 0 (7.5) using libtriton_jarvis_tts_denoiser.so
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:30:42.659712 69 tacotron-decoder-postnet.cc:873] TRITONBACKEND_ModelInitialize: tacotron2_decoder_postnet (version 1)
I0527 03:30:42.661048 69 tacotron-decoder-postnet.cc:767] model configuration:
{
“name”: “tacotron2_decoder_postnet”,
“platform”: “”,
“backend”: “jarvis_tts_taco_postnet”,
“version_policy”: {
“latest”: {
“num_versions”: 1
}
},
“max_batch_size”: 8,
“input”: [
{
“name”: “input_decoder”,
“data_type”: “TYPE_FP32”,
“format”: “FORMAT_NONE”,
“dims”: [
1,
400,
512
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false
},
{
“name”: “input_processed_decoder”,
“data_type”: “TYPE_FP32”,
“format”: “FORMAT_NONE”,
“dims”: [
400,
128,
1,
1
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false
},
{
“name”: “input_num_characters”,
“data_type”: “TYPE_INT32”,
“format”: “FORMAT_NONE”,
“dims”: [
1
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false
}
],
“output”: [
{
“name”: “spectrogram_chunk”,
“data_type”: “TYPE_FP32”,
“dims”: [
1,
80,
80
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “z”,
“data_type”: “TYPE_FP32”,
“dims”: [
8,
2656,
1
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “num_valid_samples”,
“data_type”: “TYPE_INT32”,
“dims”: [
1
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “end_flag”,
“data_type”: “TYPE_INT32”,
“dims”: [
1
],
“label_filename”: “”,
“is_shape_tensor”: false
}
],
“batch_input”: ,
“batch_output”: ,
“optimization”: {
“priority”: “PRIORITY_DEFAULT”,
“input_pinned_memory”: {
“enable”: true
},
“output_pinned_memory”: {
“enable”: true
},
“gather_kernel_buffer_threshold”: 0,
“eager_batching”: false
},
“sequence_batching”: {
“oldest”: {
“max_candidate_sequences”: 8,
“preferred_batch_size”: [
8
],
“max_queue_delay_microseconds”: 100
},
“max_sequence_idle_microseconds”: 60000000,
“control_input”: [
{
“name”: “START”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_START”,
“int32_false_true”: [
0,
1
],
“fp32_false_true”: ,
“data_type”: “TYPE_INVALID”
}
]
},
{
“name”: “READY”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_READY”,
“int32_false_true”: [
0,
1
],
“fp32_false_true”: ,
“data_type”: “TYPE_INVALID”
}
]
},
{
“name”: “END”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_END”,
“int32_false_true”: [
0,
1
],
“fp32_false_true”: ,
“data_type”: “TYPE_INVALID”
}
]
},
{
“name”: “CORRID”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_CORRID”,
“int32_false_true”: ,
“fp32_false_true”: ,
“data_type”: “TYPE_UINT64”
}
]
}
]
},
“instance_group”: [
{
“name”: “tacotron2_decoder_postnet_0”,
“kind”: “KIND_GPU”,
“count”: 1,
“gpus”: [
0
],
“profile”:
}
],
“default_model_filename”: “”,
“cc_model_filenames”: {},
“metric_tags”: {},
“parameters”: {
“max_execution_batch_size”: {
“string_value”: “8”
},
“max_input_length”: {
“string_value”: “400”
},
“chunk_length”: {
“string_value”: “80”
},
“attention_dimension”: {
“string_value”: “128”
},
“num_samples_per_frame”: {
“string_value”: “256”
},
“z_dim0”: {
“string_value”: “8”
},
“z_dim1”: {
“string_value”: “2656”
},
“encoding_dimension”: {
“string_value”: “512”
},
“num_mels”: {
“string_value”: “80”
},
“tacotron_decoder_engine”: {
“string_value”: “/data/models/tacotron2_decoder_postnet/1/model.plan”
}
},
“model_warmup”: ,
“model_transaction_policy”: {
“decoupled”: true
}
}
I0527 03:30:42.661131 69 tacotron-decoder-postnet.cc:927] TRITONBACKEND_ModelInstanceInitialize: tacotron2_decoder_postnet_0 (device 0)
I0527 03:30:42.664203 69 model_repository_manager.cc:1239] successfully loaded ‘tts_preprocessor’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:30:44.209146 69 model_repository_manager.cc:1239] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming’ version 1
I0527 03:30:44.415246 69 model_repository_manager.cc:1239] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:30:49.024453 69 model_repository_manager.cc:1239] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming’ version 1
I0527 03:30:49.024608 69 model_repository_manager.cc:1239] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline’ version 1
I0527 03:30:49.026945 69 plan_backend.cc:365] Creating instance jarvis-trt-tacotron2_encoder_0_0_gpu0 on GPU 0 (7.5) using model.plan
I0527 03:30:49.119435 69 model_repository_manager.cc:1239] successfully loaded ‘waveglow_denoiser’ version 1
I0527 03:30:49.122158 69 plan_backend.cc:748] Created instance jarvis-trt-tacotron2_encoder_0_0_gpu0 on GPU 0 with stream priority 0
I0527 03:30:49.124819 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis-trt-tacotron2_encoder’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:30:49.881052 69 plan_backend.cc:365] Creating instance jarvis-trt-jarvis_punctuation-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (7.5) using model.plan
I0527 03:30:50.550431 69 model_repository_manager.cc:1239] successfully loaded ‘tacotron2_decoder_postnet’ version 1
I0527 03:30:50.651547 69 plan_backend.cc:744] Created instance jarvis-trt-jarvis_punctuation-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0527 03:30:50.666019 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis-trt-jarvis_punctuation-nn-bert-base-uncased’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:30:51.386640 69 plan_backend.cc:365] Creating instance jarvis-trt-jarvis_ner-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (7.5) using model.plan
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:30:52.139474 69 plan_backend.cc:744] Created instance jarvis-trt-jarvis_ner-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0527 03:30:52.153591 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis-trt-jarvis_ner-nn-bert-base-uncased’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:30:52.895494 69 plan_backend.cc:365] Creating instance jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (7.5) using model.plan
I0527 03:30:53.666302 69 plan_backend.cc:744] Created instance jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0527 03:30:53.685302 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:30:54.420089 69 plan_backend.cc:365] Creating instance jarvis-trt-jarvis_qa-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (7.5) using model.plan
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:30:55.194738 69 plan_backend.cc:744] Created instance jarvis-trt-jarvis_qa-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0527 03:30:55.209884 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis-trt-jarvis_qa-nn-bert-base-uncased’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:30:55.958087 69 plan_backend.cc:365] Creating instance jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (7.5) using model.plan
I0527 03:30:56.725987 69 plan_backend.cc:744] Created instance jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0527 03:30:56.740745 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:30:57.381317 69 plan_backend.cc:365] Creating instance jarvis-trt-waveglow_0_0_gpu0 on GPU 0 (7.5) using model.plan
Jarvis waiting for Triton server to load all models…retrying in 1 second
E0527 03:30:58.020863 69 logging.cc:43] …/rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
E0527 03:30:58.025513 69 logging.cc:43] FAILED_ALLOCATION: std::exception
E0527 03:30:58.056410 69 model_repository_manager.cc:1242] failed to load ‘jarvis-trt-waveglow’ version 1: Internal: unable to create TensorRT context
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:00.249184 69 plan_backend.cc:365] Creating instance jarvis-trt-jasper_0_0_gpu0 on GPU 0 (7.5) using model.plan
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
E0527 03:31:02.452137 69 logging.cc:43] …/rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
E0527 03:31:02.452164 69 logging.cc:43] INTERNAL_ERROR: std::exception
I0527 03:31:02.452883 69 plan_backend.cc:744] Created instance jarvis-trt-jasper_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0527 03:31:02.468530 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis-trt-jasper’ version 1
E0527 03:31:02.469444 69 model_repository_manager.cc:1430] Invalid argument: ensemble ‘tacotron2_ensemble’ depends on ‘jarvis-trt-waveglow’ which has no loaded version
I0527 03:31:02.469521 69 model_repository_manager.cc:1065] loading: jarvis_intent_weather:1
I0527 03:31:02.569851 69 model_repository_manager.cc:1065] loading: jarvis_ner:1
I0527 03:31:02.670422 69 model_repository_manager.cc:1065] loading: jarvis_punctuation:1
I0527 03:31:02.770979 69 model_repository_manager.cc:1065] loading: jarvis_qa:1
I0527 03:31:02.871506 69 model_repository_manager.cc:1065] loading: jarvis_text_classification_domain:1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:02.972101 69 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming:1
I0527 03:31:02.972568 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis_text_classification_domain’ version 1
I0527 03:31:03.072688 69 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-offline:1
I0527 03:31:03.173099 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis_ner’ version 1
I0527 03:31:03.173149 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis_qa’ version 1
I0527 03:31:03.173298 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis_punctuation’ version 1
I0527 03:31:03.173349 69 model_repository_manager.cc:1239] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming’ version 1
I0527 03:31:03.173420 69 model_repository_manager.cc:1239] successfully loaded ‘jarvis_intent_weather’ version 1
I0527 03:31:03.173640 69 model_repository_manager.cc:1239] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-offline’ version 1
I0527 03:31:03.173856 69 server.cc:500]
±-----------------±-----+
| Repository Agent | Path |
±-----------------±-----+
±-----------------±-----+

I0527 03:31:03.173986 69 server.cc:527]
±------------------------±----------------------------------------------------------------------------------------±-------+
| Backend | Path | Config |
±------------------------±----------------------------------------------------------------------------------------±-------+
| onnxruntime | /opt/tritonserver/backends/onnxruntime/libtriton_onnxruntime.so | {} |
| jarvis_tts_taco_postnet | /opt/tritonserver/backends/jarvis_tts_taco_postnet/libtriton_jarvis_tts_taco_postnet.so | {} |
±------------------------±----------------------------------------------------------------------------------------±-------+

I0527 03:31:03.174632 69 server.cc:570]
±--------------------------------------------------------------------------------------------±--------±---------------------------------------------------------+
| Model | Version | Status |
±--------------------------------------------------------------------------------------------±--------±---------------------------------------------------------+
| jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jarvis_ner-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jarvis_punctuation-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jarvis_qa-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jasper | 1 | READY |
| jarvis-trt-tacotron2_encoder | 1 | READY |
| jarvis-trt-waveglow | 1 | UNAVAILABLE: Internal: unable to create TensorRT context |
| jarvis_detokenize | 1 | READY |
| jarvis_intent_weather | 1 | READY |
| jarvis_label_tokens_weather | 1 | READY |
| jarvis_ner | 1 | READY |
| jarvis_ner_label_tokens | 1 | READY |
| jarvis_punctuation | 1 | READY |
| jarvis_punctuation_gen_output | 1 | READY |
| jarvis_punctuation_label_tokens_cap | 1 | READY |
| jarvis_punctuation_label_tokens_punct | 1 | READY |
| jarvis_punctuation_merge_labels | 1 | READY |
| jarvis_qa | 1 | READY |
| jarvis_qa_postprocessor | 1 | READY |
| jarvis_qa_preprocessor | 1 | READY |
| jarvis_text_classification_domain | 1 | READY |
| jarvis_tokenizer | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming | 1 | READY |
| tacotron2_decoder_postnet | 1 | READY |
| tts_preprocessor | 1 | READY |
| waveglow_denoiser | 1 | READY |
±--------------------------------------------------------------------------------------------±--------±---------------------------------------------------------+

I0527 03:31:03.174969 69 tritonserver.cc:1658]
±---------------------------------±---------------------------------------------------------------------------------------------------------------------------------------------------+
| Option | Value |
±---------------------------------±---------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.8.0 |
| server_extensions | classification sequence model_repository schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics |
| model_repository_path[0] | /data/models |
| model_control_mode | MODE_NONE |
| strict_model_config | 1 |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 1000000000 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
±---------------------------------±---------------------------------------------------------------------------------------------------------------------------------------------------+

I0527 03:31:03.174995 69 server.cc:233] Waiting for in-flight requests to complete.
I0527 03:31:03.175013 69 model_repository_manager.cc:1098] unloading: waveglow_denoiser:1
I0527 03:31:03.175104 69 model_repository_manager.cc:1098] unloading: tts_preprocessor:1
I0527 03:31:03.175278 69 model_repository_manager.cc:1098] unloading: tacotron2_decoder_postnet:1
I0527 03:31:03.176002 69 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline:1
I0527 03:31:03.176190 69 model_repository_manager.cc:1098] unloading: jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased:1
I0527 03:31:03.176471 69 model_repository_manager.cc:1098] unloading: jarvis-trt-jarvis_qa-nn-bert-base-uncased:1
I0527 03:31:03.176677 69 model_repository_manager.cc:1222] successfully unloaded ‘tts_preprocessor’ version 1
I0527 03:31:03.176869 69 tacotron-decoder-postnet.cc:1000] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0527 03:31:03.176983 69 model_repository_manager.cc:1098] unloading: jarvis-trt-jarvis_ner-nn-bert-base-uncased:1
I0527 03:31:03.177058 69 model_repository_manager.cc:1098] unloading: jarvis-trt-jasper:1
I0527 03:31:03.177214 69 model_repository_manager.cc:1098] unloading: jarvis-trt-jarvis_punctuation-nn-bert-base-uncased:1
I0527 03:31:03.177358 69 model_repository_manager.cc:1098] unloading: jarvis_punctuation_label_tokens_punct:1
I0527 03:31:03.177515 69 model_repository_manager.cc:1098] unloading: jarvis_ner_label_tokens:1
I0527 03:31:03.177660 69 model_repository_manager.cc:1098] unloading: jarvis_qa:1
I0527 03:31:03.177828 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_punctuation_label_tokens_punct’ version 1
I0527 03:31:03.177931 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_ner_label_tokens’ version 1
I0527 03:31:03.177950 69 model_repository_manager.cc:1098] unloading: jarvis_label_tokens_weather:1
I0527 03:31:03.178066 69 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline:1
I0527 03:31:03.178131 69 model_repository_manager.cc:1098] unloading: jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased:1
I0527 03:31:03.178321 69 model_repository_manager.cc:1098] unloading: jarvis_punctuation:1
I0527 03:31:03.178405 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_label_tokens_weather’ version 1
I0527 03:31:03.178590 69 model_repository_manager.cc:1098] unloading: jarvis_intent_weather:1
I0527 03:31:03.178677 69 model_repository_manager.cc:1098] unloading: jarvis_punctuation_gen_output:1
I0527 03:31:03.178753 69 model_repository_manager.cc:1098] unloading: jarvis_punctuation_label_tokens_cap:1
I0527 03:31:03.178786 69 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline:1
I0527 03:31:03.178809 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_punctuation’ version 1
I0527 03:31:03.178829 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_intent_weather’ version 1
I0527 03:31:03.178847 69 model_repository_manager.cc:1098] unloading: jarvis_punctuation_merge_labels:1
I0527 03:31:03.178869 69 model_repository_manager.cc:1098] unloading: jarvis_qa_postprocessor:1
I0527 03:31:03.178921 69 model_repository_manager.cc:1098] unloading: jarvis_ner:1
I0527 03:31:03.178950 69 model_repository_manager.cc:1098] unloading: jarvis_qa_preprocessor:1
I0527 03:31:03.178989 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_ner’ version 1
I0527 03:31:03.178995 69 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming:1
I0527 03:31:03.179002 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_qa’ version 1
I0527 03:31:03.206830 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_punctuation_gen_output’ version 1
I0527 03:31:03.209378 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_punctuation_label_tokens_cap’ version 1
I0527 03:31:03.209429 69 model_repository_manager.cc:1222] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline’ version 1
I0527 03:31:03.210164 69 model_repository_manager.cc:1098] unloading: jarvis_detokenize:1
I0527 03:31:03.210389 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_qa_postprocessor’ version 1
I0527 03:31:03.211186 69 model_repository_manager.cc:1098] unloading: jarvis_text_classification_domain:1
I0527 03:31:03.211219 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_punctuation_merge_labels’ version 1
I0527 03:31:03.211264 69 model_repository_manager.cc:1222] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline’ version 1
I0527 03:31:03.211424 69 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming:1
I0527 03:31:03.211460 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_detokenize’ version 1
I0527 03:31:03.211484 69 model_repository_manager.cc:1098] unloading: jarvis-trt-tacotron2_encoder:1
I0527 03:31:03.211502 69 model_repository_manager.cc:1098] unloading: jarvis_tokenizer:1
I0527 03:31:03.211632 69 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming:1
I0527 03:31:03.211700 69 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming:1
I0527 03:31:03.211732 69 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-offline:1
I0527 03:31:03.211779 69 server.cc:248] Timeout 30: Found 18 live models and 0 in-flight non-inference requests
I0527 03:31:03.211944 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_qa_preprocessor’ version 1
I0527 03:31:03.213593 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_text_classification_domain’ version 1
I0527 03:31:03.240192 69 model_repository_manager.cc:1222] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming’ version 1
I0527 03:31:03.240666 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_tokenizer’ version 1
I0527 03:31:03.241717 69 model_repository_manager.cc:1222] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming’ version 1
I0527 03:31:03.241728 69 model_repository_manager.cc:1222] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-offline’ version 1
I0527 03:31:03.241968 69 model_repository_manager.cc:1222] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming’ version 1
I0527 03:31:03.243613 69 model_repository_manager.cc:1222] successfully unloaded ‘waveglow_denoiser’ version 1
I0527 03:31:03.254561 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis-trt-tacotron2_encoder’ version 1
I0527 03:31:03.282027 69 model_repository_manager.cc:1222] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline’ version 1
I0527 03:31:03.303263 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis-trt-jasper’ version 1
I0527 03:31:03.303274 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased’ version 1
I0527 03:31:03.320338 69 model_repository_manager.cc:1222] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming’ version 1
I0527 03:31:03.352773 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis-trt-jarvis_punctuation-nn-bert-base-uncased’ version 1
I0527 03:31:03.352836 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis-trt-jarvis_intent_weather-nn-bert-base-uncased’ version 1
I0527 03:31:03.352926 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis-trt-jarvis_qa-nn-bert-base-uncased’ version 1
I0527 03:31:03.354984 69 model_repository_manager.cc:1222] successfully unloaded ‘jarvis-trt-jarvis_ner-nn-bert-base-uncased’ version 1

Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:04.211947 69 server.cc:248] Timeout 29: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:05.212153 69 server.cc:248] Timeout 28: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:06.212355 69 server.cc:248] Timeout 27: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:07.212561 69 server.cc:248] Timeout 26: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:08.212791 69 server.cc:248] Timeout 25: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:09.212955 69 server.cc:248] Timeout 24: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:10.213119 69 server.cc:248] Timeout 23: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:11.213325 69 server.cc:248] Timeout 22: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:12.213530 69 server.cc:248] Timeout 21: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:13.213732 69 server.cc:248] Timeout 20: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:14.213931 69 server.cc:248] Timeout 19: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:15.214132 69 server.cc:248] Timeout 18: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:16.214337 69 server.cc:248] Timeout 17: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:17.214539 69 server.cc:248] Timeout 16: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:18.214740 69 server.cc:248] Timeout 15: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:19.214941 69 server.cc:248] Timeout 14: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:20.215144 69 server.cc:248] Timeout 13: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:21.215345 69 server.cc:248] Timeout 12: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:22.215551 69 server.cc:248] Timeout 11: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:23.215758 69 server.cc:248] Timeout 10: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:24.215993 69 server.cc:248] Timeout 9: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:25.216194 69 server.cc:248] Timeout 8: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:26.216392 69 server.cc:248] Timeout 7: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:27.216590 69 server.cc:248] Timeout 6: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:28.216812 69 server.cc:248] Timeout 5: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:29.217010 69 server.cc:248] Timeout 4: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:30.217197 69 server.cc:248] Timeout 3: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:31.217381 69 server.cc:248] Timeout 2: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:32.217548 69 server.cc:248] Timeout 1: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0527 03:31:33.217716 69 server.cc:248] Timeout 0: Found 1 live models and 0 in-flight non-inference requests
error: creating server: Internal - failed to load all models
Jarvis waiting for Triton server to load all models…retrying in 1 second
Triton server died before reaching ready state. Terminating Jarvis startup.
Check Triton logs with: docker logs
/opt/jarvis/bin/start-jarvis: line 1: kill: (69) - No such process

I tried looking for some answers and they mentioned about reducing batchsize or worksize but im not too sure on what it is and how to do it. Do you think thats worth a try? If yes would you kindly give me some instructions on how to do that? Thank you in advance

Could you please try to run jarvis_clean.sh and then restart the jarvis_init.sh steps with new config file?

Thanks

Hi, i did jarvis_clean, jarvis_init and jarvis_start. I didnt comment out anything in the config file, and I got the same error (was i supposed to also comment out all except 1 ngl model?). Also, i watched this video Getting Started with Conversational AI with NVIDIA's Jarvis Platform - YouTube and the guy said that jarvis wont run on a rtx 2080 ti which is what i have because it needs minimally 16gb of ram on the gpu?

Also, any thoughts on this:

Anyways, here is my docker logs:

== Jarvis Speech Skills ==

NVIDIA Release 21.04 (build 22933806)

Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying
project or file.

NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be
insufficient for the inference server. NVIDIA recommends the use of the following flags:
nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 …

Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:42:48.173754 68 metrics.cc:221] Collecting metrics for GPU 0: NVIDIA GeForce RTX 2080 Ti
I0528 06:42:48.199061 68 onnxruntime.cc:1728] TRITONBACKEND_Initialize: onnxruntime
I0528 06:42:48.199228 68 onnxruntime.cc:1738] Triton TRITONBACKEND API version: 1.0
I0528 06:42:48.199232 68 onnxruntime.cc:1744] ‘onnxruntime’ TRITONBACKEND API version: 1.0
I0528 06:42:48.321821 68 pinned_memory_manager.cc:205] Pinned memory pool is created at ‘0x7fb85e000000’ with size 268435456
I0528 06:42:48.322249 68 cuda_memory_manager.cc:103] CUDA memory pool is created on device 0 with size 1000000000
I0528 06:42:48.337948 68 model_repository_manager.cc:1065] loading: jarvis-trt-jasper:1
I0528 06:42:48.438373 68 model_repository_manager.cc:1065] loading: jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased:1
I0528 06:42:48.538610 68 model_repository_manager.cc:1065] loading: jarvis-trt-jarvis_punctuation-nn-bert-base-uncased:1
I0528 06:42:48.638827 68 model_repository_manager.cc:1065] loading: jarvis-trt-tacotron2_encoder:1
I0528 06:42:48.739015 68 model_repository_manager.cc:1065] loading: jarvis-trt-waveglow:1
I0528 06:42:48.839228 68 model_repository_manager.cc:1065] loading: jarvis_detokenize:1
I0528 06:42:48.939422 68 model_repository_manager.cc:1065] loading: jarvis_punctuation_gen_output:1
I0528 06:42:48.939542 68 custom_backend.cc:198] Creating instance jarvis_detokenize_0_0_cpu on CPU using libtriton_jarvis_nlp_detokenizer.so
I0528 06:42:48.950654 68 model_repository_manager.cc:1239] successfully loaded ‘jarvis_detokenize’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:42:49.039639 68 model_repository_manager.cc:1065] loading: jarvis_punctuation_label_tokens_cap:1
I0528 06:42:49.039838 68 custom_backend.cc:198] Creating instance jarvis_punctuation_gen_output_0_0_cpu on CPU using libtriton_jarvis_nlp_punctuation.so
I0528 06:42:49.048086 68 model_repository_manager.cc:1239] successfully loaded ‘jarvis_punctuation_gen_output’ version 1
I0528 06:42:49.139902 68 model_repository_manager.cc:1065] loading: jarvis_punctuation_label_tokens_punct:1
I0528 06:42:49.140027 68 custom_backend.cc:198] Creating instance jarvis_punctuation_label_tokens_cap_0_0_cpu on CPU using libtriton_jarvis_nlp_seqlabel.so
I0528 06:42:49.150652 68 model_repository_manager.cc:1239] successfully loaded ‘jarvis_punctuation_label_tokens_cap’ version 1
I0528 06:42:49.240170 68 model_repository_manager.cc:1065] loading: jarvis_punctuation_merge_labels:1
I0528 06:42:49.240326 68 custom_backend.cc:198] Creating instance jarvis_punctuation_label_tokens_punct_0_0_cpu on CPU using libtriton_jarvis_nlp_seqlabel.so
I0528 06:42:49.240782 68 model_repository_manager.cc:1239] successfully loaded ‘jarvis_punctuation_label_tokens_punct’ version 1
I0528 06:42:49.340519 68 model_repository_manager.cc:1065] loading: jarvis_tokenizer:1
I0528 06:42:49.340637 68 custom_backend.cc:198] Creating instance jarvis_punctuation_merge_labels_0_0_cpu on CPU using libtriton_jarvis_nlp_labels.so
I0528 06:42:49.348793 68 model_repository_manager.cc:1239] successfully loaded ‘jarvis_punctuation_merge_labels’ version 1
I0528 06:42:49.440734 68 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming:1
I0528 06:42:49.440868 68 custom_backend.cc:198] Creating instance jarvis_tokenizer_0_0_cpu on CPU using libtriton_jarvis_nlp_tokenizer.so
I0528 06:42:49.473088 68 model_repository_manager.cc:1239] successfully loaded ‘jarvis_tokenizer’ version 1
I0528 06:42:49.540960 68 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming:1
I0528 06:42:49.541075 68 custom_backend.cc:198] Creating instance jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming_0_0_cpu on CPU using libtriton_jarvis_asr_decoder_cpu.so
E:decoder_context.cc:696: vocabulary parameter will be deprecated. Please use vocab_file instead.
I0528 06:42:49.641172 68 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline:1
I0528 06:42:49.641350 68 custom_backend.cc:201] Creating instance jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming_0_0_gpu0 on GPU 0 (7.5) using libtriton_jarvis_asr_features.so
I0528 06:42:49.741437 68 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline:1
I0528 06:42:49.741529 68 custom_backend.cc:198] Creating instance jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline_0_0_cpu on CPU using libtriton_jarvis_asr_decoder_cpu.so
E:decoder_context.cc:696: vocabulary parameter will be deprecated. Please use vocab_file instead.
I0528 06:42:49.841632 68 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline:1
I0528 06:42:49.841833 68 custom_backend.cc:201] Creating instance jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline_0_0_gpu0 on GPU 0 (7.5) using libtriton_jarvis_asr_features.so
I0528 06:42:49.941832 68 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming:1
I0528 06:42:49.941986 68 custom_backend.cc:198] Creating instance jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline_0_0_cpu on CPU using libtriton_jarvis_asr_vad.so
E:voice-activity-detector.cc:414: vocabulary parameter will be deprecated. Please use vocab_file instead.
I0528 06:42:49.948650 68 model_repository_manager.cc:1239] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:42:50.042010 68 model_repository_manager.cc:1065] loading: tacotron2_decoder_postnet:1
I0528 06:42:50.042105 68 custom_backend.cc:198] Creating instance jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming_0_0_cpu on CPU using libtriton_jarvis_asr_vad.so
E:voice-activity-detector.cc:414: vocabulary parameter will be deprecated. Please use vocab_file instead.
I0528 06:42:50.044335 68 model_repository_manager.cc:1239] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming’ version 1
I0528 06:42:50.142199 68 model_repository_manager.cc:1065] loading: tts_preprocessor:1
I0528 06:42:50.242378 68 model_repository_manager.cc:1065] loading: waveglow_denoiser:1
I0528 06:42:50.242615 68 custom_backend.cc:201] Creating instance tts_preprocessor_0_0_gpu0 on GPU 0 (7.5) using libtriton_jarvis_tts_preprocessor.so
I0528 06:42:50.342798 68 custom_backend.cc:201] Creating instance waveglow_denoiser_0_0_gpu0 on GPU 0 (7.5) using libtriton_jarvis_tts_denoiser.so
I0528 06:42:51.011407 68 tacotron-decoder-postnet.cc:873] TRITONBACKEND_ModelInitialize: tacotron2_decoder_postnet (version 1)
I0528 06:42:51.012564 68 tacotron-decoder-postnet.cc:767] model configuration:
{
“name”: “tacotron2_decoder_postnet”,
“platform”: “”,
“backend”: “jarvis_tts_taco_postnet”,
“version_policy”: {
“latest”: {
“num_versions”: 1
}
},
“max_batch_size”: 8,
“input”: [
{
“name”: “input_decoder”,
“data_type”: “TYPE_FP32”,
“format”: “FORMAT_NONE”,
“dims”: [
1,
400,
512
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false
},
{
“name”: “input_processed_decoder”,
“data_type”: “TYPE_FP32”,
“format”: “FORMAT_NONE”,
“dims”: [
400,
128,
1,
1
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false
},
{
“name”: “input_num_characters”,
“data_type”: “TYPE_INT32”,
“format”: “FORMAT_NONE”,
“dims”: [
1
],
“is_shape_tensor”: false,
“allow_ragged_batch”: false
}
],
“output”: [
{
“name”: “spectrogram_chunk”,
“data_type”: “TYPE_FP32”,
“dims”: [
1,
80,
80
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “z”,
“data_type”: “TYPE_FP32”,
“dims”: [
8,
2656,
1
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “num_valid_samples”,
“data_type”: “TYPE_INT32”,
“dims”: [
1
],
“label_filename”: “”,
“is_shape_tensor”: false
},
{
“name”: “end_flag”,
“data_type”: “TYPE_INT32”,
“dims”: [
1
],
“label_filename”: “”,
“is_shape_tensor”: false
}
],
“batch_input”: ,
“batch_output”: ,
“optimization”: {
“priority”: “PRIORITY_DEFAULT”,
“input_pinned_memory”: {
“enable”: true
},
“output_pinned_memory”: {
“enable”: true
},
“gather_kernel_buffer_threshold”: 0,
“eager_batching”: false
},
“sequence_batching”: {
“oldest”: {
“max_candidate_sequences”: 8,
“preferred_batch_size”: [
8
],
“max_queue_delay_microseconds”: 100
},
“max_sequence_idle_microseconds”: 60000000,
“control_input”: [
{
“name”: “START”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_START”,
“int32_false_true”: [
0,
1
],
“fp32_false_true”: ,
“data_type”: “TYPE_INVALID”
}
]
},
{
“name”: “READY”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_READY”,
“int32_false_true”: [
0,
1
],
“fp32_false_true”: ,
“data_type”: “TYPE_INVALID”
}
]
},
{
“name”: “END”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_END”,
“int32_false_true”: [
0,
1
],
“fp32_false_true”: ,
“data_type”: “TYPE_INVALID”
}
]
},
{
“name”: “CORRID”,
“control”: [
{
“kind”: “CONTROL_SEQUENCE_CORRID”,
“int32_false_true”: ,
“fp32_false_true”: ,
“data_type”: “TYPE_UINT64”
}
]
}
]
},
“instance_group”: [
{
“name”: “tacotron2_decoder_postnet_0”,
“kind”: “KIND_GPU”,
“count”: 1,
“gpus”: [
0
],
“profile”:
}
],
“default_model_filename”: “”,
“cc_model_filenames”: {},
“metric_tags”: {},
“parameters”: {
“num_samples_per_frame”: {
“string_value”: “256”
},
“z_dim0”: {
“string_value”: “8”
},
“z_dim1”: {
“string_value”: “2656”
},
“encoding_dimension”: {
“string_value”: “512”
},
“num_mels”: {
“string_value”: “80”
},
“tacotron_decoder_engine”: {
“string_value”: “/data/models/tacotron2_decoder_postnet/1/model.plan”
},
“max_execution_batch_size”: {
“string_value”: “8”
},
“max_input_length”: {
“string_value”: “400”
},
“chunk_length”: {
“string_value”: “80”
},
“attention_dimension”: {
“string_value”: “128”
}
},
“model_warmup”: ,
“model_transaction_policy”: {
“decoupled”: true
}
}
I0528 06:42:51.012668 68 tacotron-decoder-postnet.cc:927] TRITONBACKEND_ModelInstanceInitialize: tacotron2_decoder_postnet_0 (device 0)
I0528 06:42:51.016127 68 model_repository_manager.cc:1239] successfully loaded ‘tts_preprocessor’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:42:53.910252 68 model_repository_manager.cc:1239] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming’ version 1
I0528 06:42:54.002517 68 model_repository_manager.cc:1239] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:42:58.821855 68 model_repository_manager.cc:1239] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline’ version 1
I0528 06:42:58.823266 68 model_repository_manager.cc:1239] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming’ version 1
I0528 06:42:59.030225 68 plan_backend.cc:365] Creating instance jarvis-trt-tacotron2_encoder_0_0_gpu0 on GPU 0 (7.5) using model.plan
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:42:59.127575 68 model_repository_manager.cc:1239] successfully loaded ‘waveglow_denoiser’ version 1
I0528 06:42:59.128536 68 plan_backend.cc:748] Created instance jarvis-trt-tacotron2_encoder_0_0_gpu0 on GPU 0 with stream priority 0
I0528 06:42:59.131215 68 model_repository_manager.cc:1239] successfully loaded ‘jarvis-trt-tacotron2_encoder’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:00.603557 68 model_repository_manager.cc:1239] successfully loaded ‘tacotron2_decoder_postnet’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:01.421051 68 plan_backend.cc:365] Creating instance jarvis-trt-jasper_0_0_gpu0 on GPU 0 (7.5) using model.plan
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:03.677059 68 plan_backend.cc:744] Created instance jarvis-trt-jasper_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0528 06:43:03.691754 68 model_repository_manager.cc:1239] successfully loaded ‘jarvis-trt-jasper’ version 1
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:04.468555 68 plan_backend.cc:365] Creating instance jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (7.5) using model.plan
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:05.275445 68 plan_backend.cc:744] Created instance jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0528 06:43:05.290725 68 model_repository_manager.cc:1239] successfully loaded ‘jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased’ version 1
I0528 06:43:05.973890 68 plan_backend.cc:365] Creating instance jarvis-trt-waveglow_0_0_gpu0 on GPU 0 (7.5) using model.plan
Jarvis waiting for Triton server to load all models…retrying in 1 second
E0528 06:43:06.612047 68 logging.cc:43] …/rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
E0528 06:43:06.615641 68 logging.cc:43] FAILED_ALLOCATION: std::exception
E0528 06:43:06.650372 68 model_repository_manager.cc:1242] failed to load ‘jarvis-trt-waveglow’ version 1: Internal: unable to create TensorRT context
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:07.390547 68 plan_backend.cc:365] Creating instance jarvis-trt-jarvis_punctuation-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (7.5) using model.plan
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:08.191145 68 plan_backend.cc:744] Created instance jarvis-trt-jarvis_punctuation-nn-bert-base-uncased_0_0_gpu0 on GPU 0 with stream priority 0 and optimization profile default[0];
I0528 06:43:08.207277 68 model_repository_manager.cc:1239] successfully loaded ‘jarvis-trt-jarvis_punctuation-nn-bert-base-uncased’ version 1
E0528 06:43:08.207902 68 model_repository_manager.cc:1430] Invalid argument: ensemble ‘tacotron2_ensemble’ depends on ‘jarvis-trt-waveglow’ which has no loaded version
I0528 06:43:08.207970 68 model_repository_manager.cc:1065] loading: jarvis_punctuation:1
I0528 06:43:08.308117 68 model_repository_manager.cc:1065] loading: jarvis_text_classification_domain:1
I0528 06:43:08.408416 68 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming:1
I0528 06:43:08.408955 68 model_repository_manager.cc:1239] successfully loaded ‘jarvis_text_classification_domain’ version 1
I0528 06:43:08.508757 68 model_repository_manager.cc:1065] loading: jasper-asr-trt-ensemble-vad-streaming-offline:1
I0528 06:43:08.608998 68 model_repository_manager.cc:1239] successfully loaded ‘jarvis_punctuation’ version 1
I0528 06:43:08.609008 68 model_repository_manager.cc:1239] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming’ version 1
I0528 06:43:08.609058 68 model_repository_manager.cc:1239] successfully loaded ‘jasper-asr-trt-ensemble-vad-streaming-offline’ version 1
I0528 06:43:08.609147 68 server.cc:500]
±-----------------±-----+
| Repository Agent | Path |
±-----------------±-----+
±-----------------±-----+

I0528 06:43:08.609186 68 server.cc:527]
±------------------------±----------------------------------------------------------------------------------------±-------+
| Backend | Path | Config |
±------------------------±----------------------------------------------------------------------------------------±-------+
| onnxruntime | /opt/tritonserver/backends/onnxruntime/libtriton_onnxruntime.so | {} |
| jarvis_tts_taco_postnet | /opt/tritonserver/backends/jarvis_tts_taco_postnet/libtriton_jarvis_tts_taco_postnet.so | {} |
±------------------------±----------------------------------------------------------------------------------------±-------+

I0528 06:43:08.609284 68 server.cc:570]
±--------------------------------------------------------------------------------------------±--------±---------------------------------------------------------+
| Model | Version | Status |
±--------------------------------------------------------------------------------------------±--------±---------------------------------------------------------+
| jarvis-trt-jarvis_punctuation-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased | 1 | READY |
| jarvis-trt-jasper | 1 | READY |
| jarvis-trt-tacotron2_encoder | 1 | READY |
| jarvis-trt-waveglow | 1 | UNAVAILABLE: Internal: unable to create TensorRT context |
| jarvis_detokenize | 1 | READY |
| jarvis_punctuation | 1 | READY |
| jarvis_punctuation_gen_output | 1 | READY |
| jarvis_punctuation_label_tokens_cap | 1 | READY |
| jarvis_punctuation_label_tokens_punct | 1 | READY |
| jarvis_punctuation_merge_labels | 1 | READY |
| jarvis_text_classification_domain | 1 | READY |
| jarvis_tokenizer | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline | 1 | READY |
| jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming | 1 | READY |
| tacotron2_decoder_postnet | 1 | READY |
| tts_preprocessor | 1 | READY |
| waveglow_denoiser | 1 | READY |
±--------------------------------------------------------------------------------------------±--------±---------------------------------------------------------+

I0528 06:43:08.609341 68 tritonserver.cc:1658]
±---------------------------------±---------------------------------------------------------------------------------------------------------------------------------------------------+
| Option | Value |
±---------------------------------±---------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.8.0 |
| server_extensions | classification sequence model_repository schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics |
| model_repository_path[0] | /data/models |
| model_control_mode | MODE_NONE |
| strict_model_config | 1 |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 1000000000 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
±---------------------------------±---------------------------------------------------------------------------------------------------------------------------------------------------+

I0528 06:43:08.609346 68 server.cc:233] Waiting for in-flight requests to complete.
I0528 06:43:08.609349 68 model_repository_manager.cc:1098] unloading: tts_preprocessor:1
I0528 06:43:08.609370 68 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline:1
I0528 06:43:08.609400 68 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-offline:1
I0528 06:43:08.609456 68 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming:1
I0528 06:43:08.609487 68 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming:1
I0528 06:43:08.609527 68 model_repository_manager.cc:1222] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-offline’ version 1
I0528 06:43:08.609569 68 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming:1
I0528 06:43:08.609715 68 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming:1
I0528 06:43:08.609726 68 model_repository_manager.cc:1222] successfully unloaded ‘tts_preprocessor’ version 1
I0528 06:43:08.609749 68 model_repository_manager.cc:1098] unloading: jarvis_tokenizer:1
I0528 06:43:08.609768 68 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline:1
I0528 06:43:08.609779 68 model_repository_manager.cc:1222] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming’ version 1
I0528 06:43:08.609824 68 model_repository_manager.cc:1098] unloading: jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased:1
I0528 06:43:08.609862 68 model_repository_manager.cc:1098] unloading: tacotron2_decoder_postnet:1
I0528 06:43:08.638348 68 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_tokenizer’ version 1
I0528 06:43:08.638650 68 model_repository_manager.cc:1098] unloading: jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline:1
I0528 06:43:08.638683 68 model_repository_manager.cc:1098] unloading: jarvis-trt-jarvis_punctuation-nn-bert-base-uncased:1
I0528 06:43:08.638717 68 model_repository_manager.cc:1098] unloading: jarvis_punctuation_label_tokens_punct:1
I0528 06:43:08.638778 68 model_repository_manager.cc:1098] unloading: jarvis-trt-jasper:1
I0528 06:43:08.639079 68 model_repository_manager.cc:1222] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming’ version 1
I0528 06:43:08.640008 68 model_repository_manager.cc:1222] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline’ version 1
I0528 06:43:08.668981 68 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_punctuation_label_tokens_punct’ version 1
I0528 06:43:08.668994 68 model_repository_manager.cc:1098] unloading: jarvis_punctuation_gen_output:1
I0528 06:43:08.669023 68 tacotron-decoder-postnet.cc:1000] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0528 06:43:08.671092 68 model_repository_manager.cc:1098] unloading: jarvis-trt-tacotron2_encoder:1
I0528 06:43:08.671108 68 model_repository_manager.cc:1222] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline’ version 1
I0528 06:43:08.671120 68 model_repository_manager.cc:1098] unloading: waveglow_denoiser:1
I0528 06:43:08.671178 68 model_repository_manager.cc:1098] unloading: jarvis_detokenize:1
I0528 06:43:08.671242 68 model_repository_manager.cc:1098] unloading: jarvis_punctuation:1
I0528 06:43:08.671432 68 model_repository_manager.cc:1222] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming’ version 1
I0528 06:43:08.671489 68 model_repository_manager.cc:1098] unloading: jarvis_punctuation_merge_labels:1
I0528 06:43:08.671511 68 model_repository_manager.cc:1098] unloading: jarvis_text_classification_domain:1
I0528 06:43:08.671544 68 model_repository_manager.cc:1098] unloading: jarvis_punctuation_label_tokens_cap:1
I0528 06:43:08.671569 68 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_punctuation_gen_output’ version 1
I0528 06:43:08.671592 68 server.cc:248] Timeout 30: Found 13 live models and 0 in-flight non-inference requests
I0528 06:43:08.671673 68 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_detokenize’ version 1
I0528 06:43:08.672296 68 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_punctuation’ version 1
I0528 06:43:08.672347 68 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_text_classification_domain’ version 1
I0528 06:43:08.672478 68 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_punctuation_merge_labels’ version 1
I0528 06:43:08.672621 68 model_repository_manager.cc:1222] successfully unloaded ‘jarvis_punctuation_label_tokens_cap’ version 1
I0528 06:43:08.693018 68 model_repository_manager.cc:1222] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-feature-extractor-streaming’ version 1
I0528 06:43:08.736403 68 model_repository_manager.cc:1222] successfully unloaded ‘waveglow_denoiser’ version 1
I0528 06:43:08.736555 68 model_repository_manager.cc:1222] successfully unloaded ‘jarvis-trt-jarvis_text_classification_domain-nn-bert-base-uncased’ version 1
I0528 06:43:08.737655 68 model_repository_manager.cc:1222] successfully unloaded ‘jasper-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline’ version 1
I0528 06:43:08.741897 68 model_repository_manager.cc:1222] successfully unloaded ‘jarvis-trt-jasper’ version 1
I0528 06:43:08.741934 68 model_repository_manager.cc:1222] successfully unloaded ‘jarvis-trt-jarvis_punctuation-nn-bert-base-uncased’ version 1
I0528 06:43:08.747034 68 model_repository_manager.cc:1222] successfully unloaded ‘jarvis-trt-tacotron2_encoder’ version 1

Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:09.671753 68 server.cc:248] Timeout 29: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:10.671883 68 server.cc:248] Timeout 28: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:11.672031 68 server.cc:248] Timeout 27: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:12.672152 68 server.cc:248] Timeout 26: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:13.672329 68 server.cc:248] Timeout 25: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:14.672490 68 server.cc:248] Timeout 24: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:15.672663 68 server.cc:248] Timeout 23: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:16.672825 68 server.cc:248] Timeout 22: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:17.672942 68 server.cc:248] Timeout 21: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:18.673105 68 server.cc:248] Timeout 20: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:19.673282 68 server.cc:248] Timeout 19: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:20.673465 68 server.cc:248] Timeout 18: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:21.673567 68 server.cc:248] Timeout 17: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:22.673681 68 server.cc:248] Timeout 16: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:23.673819 68 server.cc:248] Timeout 15: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:24.673937 68 server.cc:248] Timeout 14: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:25.674056 68 server.cc:248] Timeout 13: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:26.674190 68 server.cc:248] Timeout 12: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:27.674427 68 server.cc:248] Timeout 11: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:28.674623 68 server.cc:248] Timeout 10: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:29.674768 68 server.cc:248] Timeout 9: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:30.674953 68 server.cc:248] Timeout 8: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:31.675165 68 server.cc:248] Timeout 7: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:32.675283 68 server.cc:248] Timeout 6: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:33.675403 68 server.cc:248] Timeout 5: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:34.675517 68 server.cc:248] Timeout 4: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:35.675728 68 server.cc:248] Timeout 3: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:36.675888 68 server.cc:248] Timeout 2: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:37.676026 68 server.cc:248] Timeout 1: Found 1 live models and 0 in-flight non-inference requests
Jarvis waiting for Triton server to load all models…retrying in 1 second
I0528 06:43:38.676227 68 server.cc:248] Timeout 0: Found 1 live models and 0 in-flight non-inference requests
error: creating server: Internal - failed to load all models
Jarvis waiting for Triton server to load all models…retrying in 1 second
Jarvis waiting for Triton server to load all models…retrying in 1 second
Triton server died before reaching ready state. Terminating Jarvis startup.
Check Triton logs with: docker logs
/opt/jarvis/bin/start-jarvis: line 1: kill: (68) - No such process