Hello,
I just wanna start the jarvis server with jarvis_init.sh
and then jarvis_start.sh. When running jarvis_start.sh it fails with the message:
Health ready check failed.
Check Jarvis logs with: docker logs jarvis-speech
i already tried to run jarvis_clean.sh to remove old installations like its mentioned in this topic:https://forums.developer.nvidia.com/t/waiting-for-jarvis-server-to-load-all-models-retrying-in-10-seconds/175969/2
and i tried it by removing the docker volume, like its mentioned in this topic: https://forums.developer.nvidia.com/t/jarvis-triton-server-died-before-reaching-ready-state-terminating-jarvis-startup/176980/4
The output of the docker logs:
==========================
== Jarvis Speech Skills ==
==========================
NVIDIA Release 21.05 (build 23858942)
Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.
Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying
project or file.
ERROR: This container was built for NVIDIA Driver Release 465.19 or later, but
version 460.84 was detected and compatibility mode is UNAVAILABLE.
[[Forward compatibility was attempted on non supported HW (CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE) cuInit()=804]]
NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be
insufficient for the inference server. NVIDIA recommends the use of the following flags:
nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 ...
> Jarvis waiting for Triton server to load all models...retrying in 1 second
I0703 14:46:13.220407 72 metrics.cc:228] Collecting metrics for GPU 0: GeForce RTX 3090
W0703 14:46:13.313200 72 pinned_memory_manager.cc:202] Unable to allocate pinned system memory, pinned memory pool will not be available: the provided PTX was compiled with an unsupported toolchain.
I0703 14:46:13.313429 72 cuda_memory_manager.cc:103] CUDA memory pool is created on device 0 with size 1000000000
E0703 14:46:13.341144 72 server.cc:182] Failed to finalize CUDA memory manager: CNMEM_STATUS_CUDA_ERROR
I0703 14:46:13.346126 72 model_repository_manager.cc:1066] loading: citrinet-1024-asr-trt-ensemble-vad-streaming-feature-extractor-streaming:1
I0703 14:46:13.446381 72 model_repository_manager.cc:1066] loading: citrinet-1024-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming:1
I0703 14:46:13.446797 72 custom_backend.cc:201] Creating instance citrinet-1024-asr-trt-ensemble-vad-streaming-feature-extractor-streaming_0_0_gpu0 on GPU 0 (8.6) using libtriton_jarvis_asr_features.so
W0703 14:46:13.448981 72 pinned_memory_manager.cc:104] failed to allocate pinned system memory: no pinned memory pool, falling back to non-pinned system memory
I0703 14:46:13.546578 72 model_repository_manager.cc:1066] loading: jarvis_punctuation_gen_output:1
I0703 14:46:13.546790 72 custom_backend.cc:198] Creating instance citrinet-1024-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming_0_0_cpu on CPU using libtriton_jarvis_asr_decoder_cpu.so
I0703 14:46:13.647101 72 model_repository_manager.cc:1066] loading: jarvis_punctuation_label_tokens_cap:1
I0703 14:46:13.647289 72 custom_backend.cc:198] Creating instance jarvis_punctuation_gen_output_0_0_cpu on CPU using libtriton_jarvis_nlp_punctuation.so
I0703 14:46:13.647798 72 model_repository_manager.cc:1240] successfully loaded 'jarvis_punctuation_gen_output' version 1
I0703 14:46:13.747325 72 model_repository_manager.cc:1066] loading: jarvis_punctuation_label_tokens_punct:1
I0703 14:46:13.747738 72 custom_backend.cc:198] Creating instance jarvis_punctuation_label_tokens_cap_0_0_cpu on CPU using libtriton_jarvis_nlp_seqlabel.so
I0703 14:46:13.748296 72 model_repository_manager.cc:1240] successfully loaded 'jarvis_punctuation_label_tokens_cap' version 1
I0703 14:46:13.847544 72 model_repository_manager.cc:1066] loading: jarvis_punctuation_merge_labels:1
I0703 14:46:13.848058 72 custom_backend.cc:198] Creating instance jarvis_punctuation_label_tokens_punct_0_0_cpu on CPU using libtriton_jarvis_nlp_seqlabel.so
I0703 14:46:13.848273 72 model_repository_manager.cc:1240] successfully loaded 'jarvis_punctuation_label_tokens_punct' version 1
I0703 14:46:13.947753 72 model_repository_manager.cc:1066] loading: jarvis_tokenizer:1
I0703 14:46:13.947915 72 custom_backend.cc:198] Creating instance jarvis_punctuation_merge_labels_0_0_cpu on CPU using libtriton_jarvis_nlp_labels.so
I0703 14:46:13.948418 72 model_repository_manager.cc:1240] successfully loaded 'jarvis_punctuation_merge_labels' version 1
I0703 14:46:14.047990 72 model_repository_manager.cc:1066] loading: citrinet-1024-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline:1
I0703 14:46:14.048128 72 custom_backend.cc:198] Creating instance jarvis_tokenizer_0_0_cpu on CPU using libtriton_jarvis_nlp_tokenizer.so
I0703 14:46:14.059261 72 model_repository_manager.cc:1240] successfully loaded 'jarvis_tokenizer' version 1
I0703 14:46:14.076481 72 model_repository_manager.cc:1240] successfully loaded 'citrinet-1024-asr-trt-ensemble-vad-streaming-feature-extractor-streaming' version 1
I0703 14:46:14.148231 72 model_repository_manager.cc:1066] loading: citrinet-1024-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline:1
I0703 14:46:14.148430 72 custom_backend.cc:198] Creating instance citrinet-1024-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline_0_0_cpu on CPU using libtriton_jarvis_asr_decoder_cpu.so
> Jarvis waiting for Triton server to load all models...retrying in 1 second
I0703 14:46:14.248458 72 model_repository_manager.cc:1066] loading: citrinet-1024-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline:1
I0703 14:46:14.248746 72 custom_backend.cc:201] Creating instance citrinet-1024-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline_0_0_gpu0 on GPU 0 (8.6) using libtriton_jarvis_asr_features.so
I0703 14:46:14.255102 72 model_repository_manager.cc:1240] successfully loaded 'citrinet-1024-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline' version 1
I0703 14:46:14.261317 72 model_repository_manager.cc:1240] successfully loaded 'citrinet-1024-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming' version 1
I0703 14:46:14.348671 72 model_repository_manager.cc:1066] loading: citrinet-1024-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming:1
I0703 14:46:14.348855 72 custom_backend.cc:198] Creating instance citrinet-1024-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline_0_0_cpu on CPU using libtriton_jarvis_asr_vad.so
I0703 14:46:14.380700 72 model_repository_manager.cc:1240] successfully loaded 'citrinet-1024-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline' version 1
I0703 14:46:14.448871 72 model_repository_manager.cc:1066] loading: jarvis-trt-citrinet-1024:1
I0703 14:46:14.449069 72 custom_backend.cc:198] Creating instance citrinet-1024-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming_0_0_cpu on CPU using libtriton_jarvis_asr_vad.so
I0703 14:46:14.477412 72 model_repository_manager.cc:1240] successfully loaded 'citrinet-1024-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming' version 1
I0703 14:46:14.549086 72 model_repository_manager.cc:1066] loading: jarvis-trt-jarvis_punctuation-nn-bert-base-uncased:1
I0703 14:46:14.649328 72 model_repository_manager.cc:1066] loading: jarvis_detokenize:1
I0703 14:46:14.749956 72 custom_backend.cc:198] Creating instance jarvis_detokenize_0_0_cpu on CPU using libtriton_jarvis_nlp_detokenizer.so
I0703 14:46:14.750520 72 model_repository_manager.cc:1240] successfully loaded 'jarvis_detokenize' version 1
I0703 14:46:14.832637 72 model_repository_manager.cc:1240] successfully loaded 'citrinet-1024-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline' version 1
> Jarvis waiting for Triton server to load all models...retrying in 1 second
> Jarvis waiting for Triton server to load all models...retrying in 1 second
> Jarvis waiting for Triton server to load all models...retrying in 1 second
> Jarvis waiting for Triton server to load all models...retrying in 1 second
> Jarvis waiting for Triton server to load all models...retrying in 1 second
> Jarvis waiting for Triton server to load all models...retrying in 1 second
> Jarvis waiting for Triton server to load all models...retrying in 1 second
> Jarvis waiting for Triton server to load all models...retrying in 1 second
> Jarvis waiting for Triton server to load all models...retrying in 1 second
I0703 14:46:23.684403 72 plan_backend.cc:384] Creating instance jarvis-trt-citrinet-1024_0_0_gpu0 on GPU 0 (8.6) using model.plan
E0703 14:46:23.750561 72 model_repository_manager.cc:1243] failed to load 'jarvis-trt-citrinet-1024' version 1: Internal: unable to create stream for jarvis-trt-citrinet-1024_0_0_gpu0: the provided PTX was compiled with an unsupported toolchain.
> Jarvis waiting for Triton server to load all models...retrying in 1 second
I0703 14:46:24.577908 72 plan_backend.cc:384] Creating instance jarvis-trt-jarvis_punctuation-nn-bert-base-uncased_0_0_gpu0 on GPU 0 (8.6) using model.plan
E0703 14:46:24.640913 72 model_repository_manager.cc:1243] failed to load 'jarvis-trt-jarvis_punctuation-nn-bert-base-uncased' version 1: Internal: unable to create stream for jarvis-trt-jarvis_punctuation-nn-bert-base-uncased_0_0_gpu0: the provided PTX was compiled with an unsupported toolchain.
E0703 14:46:24.640978 72 model_repository_manager.cc:1431] Invalid argument: ensemble 'citrinet-1024-asr-trt-ensemble-vad-streaming' depends on 'jarvis-trt-citrinet-1024' which has no loaded version
E0703 14:46:24.640990 72 model_repository_manager.cc:1431] Invalid argument: ensemble 'jarvis_punctuation' depends on 'jarvis-trt-jarvis_punctuation-nn-bert-base-uncased' which has no loaded version
E0703 14:46:24.640993 72 model_repository_manager.cc:1431] Invalid argument: ensemble 'citrinet-1024-asr-trt-ensemble-vad-streaming-offline' depends on 'jarvis-trt-citrinet-1024' which has no loaded version
I0703 14:46:24.641021 72 server.cc:504]
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+
I0703 14:46:24.641035 72 server.cc:543]
+----------+------------+--------+
| Backend | Path | Config |
+----------+------------+--------+
| tensorrt | <built-in> | {} |
+----------+------------+--------+
I0703 14:46:24.641127 72 server.cc:586]
+----------------------------------------------------------------------------------------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Model | Version | Status |
+----------------------------------------------------------------------------------------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| citrinet-1024-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming | 1 | READY |
| citrinet-1024-asr-trt-ensemble-vad-streaming-feature-extractor-streaming | 1 | READY |
| citrinet-1024-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline | 1 | READY |
| citrinet-1024-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline | 1 | READY |
| citrinet-1024-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline | 1 | READY |
| citrinet-1024-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming | 1 | READY |
| jarvis-trt-citrinet-1024 | 1 | UNAVAILABLE: Internal: unable to create stream for jarvis-trt-citrinet-1024_0_0_gpu0: the provided PTX was compiled with an unsupported toolchain. |
| jarvis-trt-jarvis_punctuation-nn-bert-base-uncased | 1 | UNAVAILABLE: Internal: unable to create stream for jarvis-trt-jarvis_punctuation-nn-bert-base-uncased_0_0_gpu0: the provided PTX was compiled with an unsupported toolchain. |
| jarvis_detokenize | 1 | READY |
| jarvis_punctuation_gen_output | 1 | READY |
| jarvis_punctuation_label_tokens_cap | 1 | READY |
| jarvis_punctuation_label_tokens_punct | 1 | READY |
| jarvis_punctuation_merge_labels | 1 | READY |
| jarvis_tokenizer | 1 | READY |
+----------------------------------------------------------------------------------------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I0703 14:46:24.641204 72 tritonserver.cc:1658]
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Option | Value |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.9.0 |
| server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics |
| model_repository_path[0] | /data/models |
| model_control_mode | MODE_NONE |
| strict_model_config | 1 |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 1000000000 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I0703 14:46:24.641208 72 server.cc:234] Waiting for in-flight requests to complete.
I0703 14:46:24.641210 72 model_repository_manager.cc:1099] unloading: jarvis_tokenizer:1
I0703 14:46:24.641238 72 model_repository_manager.cc:1099] unloading: jarvis_punctuation_merge_labels:1
I0703 14:46:24.643197 72 model_repository_manager.cc:1099] unloading: jarvis_punctuation_label_tokens_cap:1
I0703 14:46:24.643245 72 model_repository_manager.cc:1099] unloading: citrinet-1024-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming:1
I0703 14:46:24.643288 72 model_repository_manager.cc:1099] unloading: citrinet-1024-asr-trt-ensemble-vad-streaming-feature-extractor-streaming:1
I0703 14:46:24.643367 72 model_repository_manager.cc:1099] unloading: citrinet-1024-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline:1
I0703 14:46:24.643418 72 model_repository_manager.cc:1223] successfully unloaded 'jarvis_punctuation_merge_labels' version 1
I0703 14:46:24.643422 72 model_repository_manager.cc:1223] successfully unloaded 'jarvis_punctuation_label_tokens_cap' version 1
I0703 14:46:24.643456 72 model_repository_manager.cc:1099] unloading: citrinet-1024-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline:1
I0703 14:46:24.643533 72 model_repository_manager.cc:1099] unloading: jarvis_punctuation_gen_output:1
I0703 14:46:24.643568 72 model_repository_manager.cc:1099] unloading: citrinet-1024-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline:1
I0703 14:46:24.643616 72 model_repository_manager.cc:1099] unloading: citrinet-1024-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming:1
I0703 14:46:24.643734 72 model_repository_manager.cc:1223] successfully unloaded 'jarvis_punctuation_gen_output' version 1
I0703 14:46:24.643771 72 model_repository_manager.cc:1099] unloading: jarvis_punctuation_label_tokens_punct:1
I0703 14:46:24.643831 72 model_repository_manager.cc:1099] unloading: jarvis_detokenize:1
I0703 14:46:24.643908 72 server.cc:249] Timeout 30: Found 9 live models and 0 in-flight non-inference requests
I0703 14:46:24.644003 72 model_repository_manager.cc:1223] successfully unloaded 'jarvis_punctuation_label_tokens_punct' version 1
I0703 14:46:24.644148 72 model_repository_manager.cc:1223] successfully unloaded 'jarvis_detokenize' version 1
I0703 14:46:24.647471 72 model_repository_manager.cc:1223] successfully unloaded 'jarvis_tokenizer' version 1
I0703 14:46:24.655247 72 model_repository_manager.cc:1223] successfully unloaded 'citrinet-1024-asr-trt-ensemble-vad-streaming-voice-activity-detector-ctc-streaming' version 1
I0703 14:46:24.659110 72 model_repository_manager.cc:1223] successfully unloaded 'citrinet-1024-asr-trt-ensemble-vad-streaming-offline-voice-activity-detector-ctc-streaming-offline' version 1
I0703 14:46:24.668775 72 model_repository_manager.cc:1223] successfully unloaded 'citrinet-1024-asr-trt-ensemble-vad-streaming-offline-feature-extractor-streaming-offline' version 1
I0703 14:46:24.716230 72 model_repository_manager.cc:1223] successfully unloaded 'citrinet-1024-asr-trt-ensemble-vad-streaming-feature-extractor-streaming' version 1
I0703 14:46:24.814660 72 model_repository_manager.cc:1223] successfully unloaded 'citrinet-1024-asr-trt-ensemble-vad-streaming-offline-ctc-decoder-cpu-streaming-offline' version 1
I0703 14:46:24.826918 72 model_repository_manager.cc:1223] successfully unloaded 'citrinet-1024-asr-trt-ensemble-vad-streaming-ctc-decoder-cpu-streaming' version 1
> Jarvis waiting for Triton server to load all models...retrying in 1 second
I0703 14:46:25.643998 72 server.cc:249] Timeout 29: Found 0 live models and 0 in-flight non-inference requests
error: creating server: Internal - failed to load all models
> Jarvis waiting for Triton server to load all models...retrying in 1 second
> Jarvis waiting for Triton server to load all models...retrying in 1 second
> Triton server died before reaching ready state. Terminating Jarvis startup.
Check Triton logs with: docker logs
/opt/jarvis/bin/start-jarvis: line 1: kill: (72) - No such process
(jarvis-venv) untow@fast-imapc010:~/jarvis/jarvis_quickstart_v1.2.1-beta$ ^C
(jarvis-venv) untow@fast-imapc010:~/jarvis/jarvis_quickstart_v1.2.1-beta$ Health ready check failed.
my config.sh file:
# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
#
# NVIDIA CORPORATION and its licensors retain all intellectual property
# and proprietary rights in and to this software, related documentation
# and any modifications thereto. Any use, reproduction, disclosure or
# distribution of this software and related documentation without an express
# license agreement from NVIDIA CORPORATION is strictly prohibited.
# Enable or Disable Jarvis Services
service_enabled_asr=true
service_enabled_nlp=false
service_enabled_tts=false
# Specify one or more GPUs to use
# specifying more than one GPU is currently an experimental feature, and may result in undefined behaviours.
gpus_to_use="device=0"
# Specify the encryption key to use to deploy models
MODEL_DEPLOY_KEY="tlt_encode"
# Locations to use for storing models artifacts
#
# If an absolute path is specified, the data will be written to that location
# Otherwise, a docker volume will be used (default).
#
# jarvis_init.sh will create a `jmir` and `models` directory in the volume or
# path specified.
#
# JMIR ($jarvis_model_loc/jmir)
# Jarvis uses an intermediate representation (JMIR) for models
# that are ready to deploy but not yet fully optimized for deployment. Pretrained
# versions can be obtained from NGC (by specifying NGC models below) and will be
# downloaded to $jarvis_model_loc/jmir by `jarvis_init.sh`
#
# Custom models produced by NeMo or TLT and prepared using jarvis-build
# may also be copied manually to this location $(jarvis_model_loc/jmir).
#
# Models ($jarvis_model_loc/models)
# During the jarvis_init process, the JMIR files in $jarvis_model_loc/jmir
# are inspected and optimized for deployment. The optimized versions are
# stored in $jarvis_model_loc/models. The jarvis server exclusively uses these
# optimized versions.
jarvis_model_loc="jarvis-model-repo"
# The default JMIRs are downloaded from NGC by default in the above $jarvis_jmir_loc directory
# If you'd like to skip the download from NGC and use the existing JMIRs in the $jarvis_jmir_loc
# then set the below $use_existing_jmirs flag to true. You can also deploy your set of custom
# JMIRs by keeping them in the jarvis_jmir_loc dir and use this quickstart script with the
# below flag to deploy them all together.
use_existing_jmirs=false
# Ports to expose for Jarvis services
jarvis_speech_api_port="50051"
jarvis_vision_api_port="60051"
# NGC orgs
jarvis_ngc_org="nvidia"
jarvis_ngc_team="jarvis"
jarvis_ngc_image_version="1.2.1-beta"
jarvis_ngc_model_version="1.2.0-beta"
# Pre-built models listed below will be downloaded from NGC. If models already exist in $jarvis-jmir
# then models can be commented out to skip download from NGC
########## ASR MODELS ##########
models_asr=(
### Punctuation model
"${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_punctuation:${jarvis_ngc_model_version}"
### Citrinet-1024 Streaming w/ CPU decoder, best latency configuration
"${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_citrinet_1024_asrset1p7_streaming:${jarvis_ngc_model_version}"
### Citrinet-1024 Streaming w/ CPU decoder, best throughput configuration
# "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_citrinet_1024_asrset1p7_streaming_throughput:${jarvis_ngc_model_version}"
### Citrinet-1024 Offline w/ CPU decoder,
"${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_citrinet_1024_asrset1p7_offline:${jarvis_ngc_model_version}"
### Jasper Streaming w/ CPU decoder, best latency configuration
# "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_jasper_english_streaming:${jarvis_ngc_model_version}"
### Jasper Streaming w/ CPU decoder, best throughput configuration
# "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_jasper_english_streaming_throughput:${jarvis_ngc_model_version}"
### Jasper Offline w/ CPU decoder
# "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_jasper_english_offline:${jarvis_ngc_model_version}"
### Quarztnet Streaming w/ CPU decoder, best latency configuration
# "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_quartznet_english_streaming:${jarvis_ngc_model_version}"
### Quarztnet Streaming w/ CPU decoder, best throughput configuration
# "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_quartznet_english_streaming_throughput:${jarvis_ngc_model_version}"
### Quarztnet Offline w/ CPU decoder
# "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_quartznet_english_offline:${jarvis_ngc_model_version}"
### Jasper Streaming w/ GPU decoder, best latency configuration
# "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_jasper_english_streaming_gpu_decoder:${jarvis_ngc_model_version}"
### Jasper Streaming w/ GPU decoder, best throughput configuration
# "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_jasper_english_streaming_throughput_gpu_decoder:${jarvis_ngc_model_version}"
### Jasper Offline w/ GPU decoder
# "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_jasper_english_offline_gpu_decoder:${jarvis_ngc_model_version}"
)
########## NLP MODELS ##########
models_nlp=(
### Bert base Punctuation model
"${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_punctuation:${jarvis_ngc_model_version}"
### BERT base Named Entity Recognition model fine-tuned on GMB dataset with class labels LOC, PER, ORG etc.
"${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_named_entity_recognition:${jarvis_ngc_model_version}"
### BERT Base Intent Slot model fine-tuned on weather dataset.
"${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_intent_slot:${jarvis_ngc_model_version}"
### BERT Base Question Answering model fine-tuned on Squad v2.
"${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_question_answering:${jarvis_ngc_model_version}"
### Megatron345M Question Answering model fine-tuned on Squad v2.
# "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_nlp_question_answering_megatron:${jarvis_ngc_model_version}"
### Bert base Text Classification model fine-tuned on 4class (weather, meteorology, personality, nomatch) domain model.
"${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_text_classification:${jarvis_ngc_model_version}"
)
########## TTS MODELS ##########
models_tts=(
"${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_tts_ljspeech:${jarvis_ngc_model_version}"
)
NGC_TARGET=${jarvis_ngc_org}
if [[ ! -z ${jarvis_ngc_team} ]]; then
NGC_TARGET="${NGC_TARGET}/${jarvis_ngc_team}"
else
team="\"\""
fi
# define docker images required to run Jarvis
image_client="nvcr.io/${NGC_TARGET}/jarvis-speech-client:${jarvis_ngc_image_version}"
image_speech_api="nvcr.io/${NGC_TARGET}/jarvis-speech:${jarvis_ngc_image_version}-server"
# define docker images required to setup Jarvis
image_init_speech="nvcr.io/${NGC_TARGET}/jarvis-speech:${jarvis_ngc_image_version}-servicemaker"
# daemon names
jarvis_daemon_speech="jarvis-speech"
jarvis_daemon_client="jarvis-client"
my setup:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.84 Driver Version: 460.84 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 3090 Off | 00000000:2D:00.0 Off | N/A |
| 0% 41C P8 23W / 420W | 72MiB / 24259MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 992 G /usr/lib/xorg/Xorg 56MiB |
| 0 N/A N/A 1160 G /usr/bin/gnome-shell 13MiB |
+-----------------------------------------------------------------------------+
Thanks a lot in advance!