Triton server died before reaching ready state. Terminating Jarvis startup

Hi, I want to set up the Jarvis server with jarvis_init.sh, but is facing a problem of:
Triton server died before reaching ready state. Terminating Jarvis startup.

I have tried ignoring this issue and run jarvis_start.sh, but it just loops Waiting for Jarvis server to load all models...retrying in 10 seconds, and ultimately printed out

Health ready check failed.
Check Jarvis logs with: docker logs jarvis-speech

I am not sure what I am doing wrong.

Below is the output of docker logs jarvis-speech
I did not make any changes to the config.sh file.

==========================
== Jarvis Speech Skills ==
==========================

NVIDIA Release 21.05 (build 23684531)

Copyright (c) 2018-2021, NVIDIA CORPORATION.  All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying
project or file.

WARNING: The NVIDIA Driver was not detected.  GPU functionality will not be available.
   Use 'nvidia-docker run' to start this container; see
   https://github.com/NVIDIA/nvidia-docker/wiki/nvidia-docker .

NOTE: The SHMEM allocation limit is set to the default of 64MB.  This may be
   insufficient for the inference server.  NVIDIA recommends the use of the following flags:
   nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 ...

  > Jarvis waiting for Triton server to load all models...retrying in 1 second
I0609 23:03:48.230563 54 pinned_memory_manager.cc:206] Pinned memory pool is created at '0x2035a0000' with size 268435456
I0609 23:03:48.230645 54 cuda_memory_manager.cc:103] CUDA memory pool is created on device 0 with size 1000000000
E0609 23:03:48.236969 54 model_repository_manager.cc:1946] Poll failed for model directory 'jarvis-trt-jarvis_qa-nn-bert-base-uncased': failed to open text file for read /data/models/jarvis-trt-jarvis_qa-nn-bert-base-uncased/config.pbtxt: No such file or directory
I0609 23:03:48.237594 54 model_repository_manager.cc:1066] loading: jarvis_qa_preprocessor:1
  > Jarvis waiting for Triton server to load all models...retrying in 1 second
I0609 23:03:48.338706 54 custom_backend.cc:198] Creating instance jarvis_qa_preprocessor_0_0_cpu on CPU using libtriton_jarvis_nlp_tokenizer.so
I0609 23:03:48.379554 54 model_repository_manager.cc:1240] successfully loaded 'jarvis_qa_preprocessor' version 1
I0609 23:03:48.379708 54 server.cc:504]
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+

I0609 23:03:48.379732 54 server.cc:543]
+----------+------------+--------+
| Backend  | Path       | Config |
+----------+------------+--------+
| tensorrt | <built-in> | {}     |
+----------+------------+--------+

I0609 23:03:48.379790 54 server.cc:586]
+------------------------+---------+--------+
| Model                  | Version | Status |
+------------------------+---------+--------+
| jarvis_qa_preprocessor | 1       | READY  |
+------------------------+---------+--------+

I0609 23:03:48.379897 54 tritonserver.cc:1658]
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Option                           | Value
                                          |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id                        | triton
                                          |
| server_version                   | 2.9.0
                                          |
| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics |
| model_repository_path[0]         | /data/models
                                          |
| model_control_mode               | MODE_NONE
                                          |
| strict_model_config              | 1
                                          |
| pinned_memory_pool_byte_size     | 268435456
                                          |
| cuda_memory_pool_byte_size{0}    | 1000000000
                                          |
| min_supported_compute_capability | 6.0
                                          |
| strict_readiness                 | 1
                                          |
| exit_timeout                     | 30
                                          |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I0609 23:03:48.379930 54 server.cc:234] Waiting for in-flight requests to complete.
I0609 23:03:48.379936 54 model_repository_manager.cc:1099] unloading: jarvis_qa_preprocessor:1
I0609 23:03:48.380021 54 server.cc:249] Timeout 30: Found 1 live models and 0 in-flight non-inference requests
I0609 23:03:48.381494 54 model_repository_manager.cc:1223] successfully unloaded 'jarvis_qa_preprocessor' version 1
  > Jarvis waiting for Triton server to load all models...retrying in 1 second
I0609 23:03:49.380324 54 server.cc:249] Timeout 29: Found 0 live models and 0 in-flight non-inference requests
error: creating server: Internal - failed to load all models
  > Triton server died before reaching ready state. Terminating Jarvis startup.
Check Triton logs with: docker logs
/opt/jarvis/bin/start-jarvis: line 1: kill: (54) - No such process

Not sure whether this is helpful or not, but below is the output of my nvidia-smi

Thu Jun 10 07:48:39 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.28       Driver Version: 470.76       CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA Quadro R...  Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   49C    P8     8W /  N/A |    164MiB /  6144MiB |    ERR!      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

I am using wsl2 with Ubuntu 18.04 on a Windows 10 device with NVIDIA Quadro RTX 3000 GPU.

Thank you!

Could you please run the script jarvis_clean.sh and then start afresh?
In case issue persist, please share the latest log, cmd output and system details so we can help better?

Thanks