Problems with Jarvis_Build

AI & Data Science Deep Learning (Training & Inference) Riva
1

Hi,
im trying to deploy the pretrained german Quartznet 15x5 nemo model.
I followed these steps:

1. Create the .enemo file from the .nemo file with:

python /home/ws/untow/jarvis/nemo_to_jarvis/convasr_to_enemo.py --nemo_file=/home/ws/untow/jarvis/jarvis-model-repo/models/germanQuartz.nemo --onnx_file=/home/ws/untow/jarvis/jarvis-model-repo/models/germanQuartz.onnx --enemo_file=/home/ws/untow/jarvis/jarvis-model-repo/models/germanQuartz.enemo

So my .enemo file is located in /jarvis-model-repo/models

2. Docker run

docker run --gpus all -it --rm -v /home/ws/untow/jarvis/jarvis-model-repo/models:/servicemaker-dev -v /home/ws/untow/jarvis:/data --entrypoint=“/bin/bash” nvcr.io/nvidia/jarvis/jarvis-speech:1.2.1-beta-servicemaker

3. Run jarvis-build with the --offline flag, because i want to create an offline-capable model

jarvis-build speech_recognition /servicemaker-dev/germanQuartz.jmir /servicemaker-dev/germanQuartz.enemo –offline

4. Run Jarvis
executing the following scripts in the /jarvis/jarvis_quickstart_v1.2.1-beta dir:
jarvis_clean.sh
jarvis_init.sh
jarvis_start.sh

i changed my config file to this:

# Copyright (c) 2021, NVIDIA CORPORATION.  All rights reserved.
#
# NVIDIA CORPORATION and its licensors retain all intellectual property
# and proprietary rights in and to this software, related documentation
# and any modifications thereto.  Any use, reproduction, disclosure or
# distribution of this software and related documentation without an express
# license agreement from NVIDIA CORPORATION is strictly prohibited.

# Enable or Disable Jarvis Services
service_enabled_asr=true
service_enabled_nlp=false
service_enabled_tts=false

# Specify one or more GPUs to use
# specifying more than one GPU is currently an experimental feature, and may result in undefined behaviours.
gpus_to_use="device=0"

# Specify the encryption key to use to deploy models
MODEL_DEPLOY_KEY="tlt_encode"

# Locations to use for storing models artifacts
#
# If an absolute path is specified, the data will be written to that location
# Otherwise, a docker volume will be used (default).
#
# jarvis_init.sh will create a `jmir` and `models` directory in the volume or
# path specified. 
#
# JMIR ($jarvis_model_loc/jmir)
# Jarvis uses an intermediate representation (JMIR) for models
# that are ready to deploy but not yet fully optimized for deployment. Pretrained
# versions can be obtained from NGC (by specifying NGC models below) and will be
# downloaded to $jarvis_model_loc/jmir by `jarvis_init.sh`
# 
# Custom models produced by NeMo or TLT and prepared using jarvis-build
# may also be copied manually to this location $(jarvis_model_loc/jmir).
#
# Models ($jarvis_model_loc/models)
# During the jarvis_init process, the JMIR files in $jarvis_model_loc/jmir
# are inspected and optimized for deployment. The optimized versions are
# stored in $jarvis_model_loc/models. The jarvis server exclusively uses these
# optimized versions.
jarvis_model_loc="/home/ws/untow/jarvis/jarvis-model-repo"

# The default JMIRs are downloaded from NGC by default in the above $jarvis_jmir_loc directory
# If you'd like to skip the download from NGC and use the existing JMIRs in the $jarvis_jmir_loc
# then set the below $use_existing_jmirs flag to true. You can also deploy your set of custom
# JMIRs by keeping them in the jarvis_jmir_loc dir and use this quickstart script with the
# below flag to deploy them all together.
use_existing_jmirs=true

# Ports to expose for Jarvis services
jarvis_speech_api_port="50051"
jarvis_vision_api_port="60051"

# NGC orgs
jarvis_ngc_org="nvidia"
jarvis_ngc_team="jarvis"
jarvis_ngc_image_version="1.2.1-beta"
jarvis_ngc_model_version="1.2.0-beta"

# Pre-built models listed below will be downloaded from NGC. If models already exist in $jarvis-jmir
# then models can be commented out to skip download from NGC

########## ASR MODELS ##########

models_asr=(
### Punctuation model
    "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_punctuation:${jarvis_ngc_model_version}"

### Citrinet-1024 Streaming w/ CPU decoder, best latency configuration
    "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_citrinet_1024_asrset1p7_streaming:${jarvis_ngc_model_version}"

### Citrinet-1024 Streaming w/ CPU decoder, best throughput configuration
#    "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_citrinet_1024_asrset1p7_streaming_throughput:${jarvis_ngc_model_version}"

### Citrinet-1024 Offline w/ CPU decoder, 
    "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_citrinet_1024_asrset1p7_offline:${jarvis_ngc_model_version}"

### Jasper Streaming w/ CPU decoder, best latency configuration
#    "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_jasper_english_streaming:${jarvis_ngc_model_version}"

### Jasper Streaming w/ CPU decoder, best throughput configuration
#    "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_jasper_english_streaming_throughput:${jarvis_ngc_model_version}"

###  Jasper Offline w/ CPU decoder
#    "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_jasper_english_offline:${jarvis_ngc_model_version}"
 
### Quarztnet Streaming w/ CPU decoder, best latency configuration
#    "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_quartznet_english_streaming:${jarvis_ngc_model_version}"

### Quarztnet Streaming w/ CPU decoder, best throughput configuration
#    "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_quartznet_english_streaming_throughput:${jarvis_ngc_model_version}"

### Quarztnet Offline w/ CPU decoder
#    "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_quartznet_english_offline:${jarvis_ngc_model_version}"

### Jasper Streaming w/ GPU decoder, best latency configuration
#    "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_jasper_english_streaming_gpu_decoder:${jarvis_ngc_model_version}"

### Jasper Streaming w/ GPU decoder, best throughput configuration
#    "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_jasper_english_streaming_throughput_gpu_decoder:${jarvis_ngc_model_version}"

### Jasper Offline w/ GPU decoder
#    "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_asr_jasper_english_offline_gpu_decoder:${jarvis_ngc_model_version}"
)

########## NLP MODELS ##########

models_nlp=(
### Bert base Punctuation model
    "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_punctuation:${jarvis_ngc_model_version}"

### BERT base Named Entity Recognition model fine-tuned on GMB dataset with class labels LOC, PER, ORG etc.
#    "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_named_entity_recognition:${jarvis_ngc_model_version}"

### BERT Base Intent Slot model fine-tuned on weather dataset.
 #   "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_intent_slot:${jarvis_ngc_model_version}"

### BERT Base Question Answering model fine-tuned on Squad v2.
#    "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_question_answering:${jarvis_ngc_model_version}"

### Megatron345M Question Answering model fine-tuned on Squad v2.
#    "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_nlp_question_answering_megatron:${jarvis_ngc_model_version}"

### Bert base Text Classification model fine-tuned on 4class (weather, meteorology, personality, nomatch) domain model.
#    "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_text_classification:${jarvis_ngc_model_version}"
)

########## TTS MODELS ##########

models_tts=(
   "${jarvis_ngc_org}/${jarvis_ngc_team}/jmir_jarvis_tts_ljspeech:${jarvis_ngc_model_version}"
)

NGC_TARGET=${jarvis_ngc_org}
if [[ ! -z ${jarvis_ngc_team} ]]; then
  NGC_TARGET="${NGC_TARGET}/${jarvis_ngc_team}"
else
  team="\"\""
fi

# define docker images required to run Jarvis
image_client="nvcr.io/${NGC_TARGET}/jarvis-speech-client:${jarvis_ngc_image_version}"
image_speech_api="nvcr.io/${NGC_TARGET}/jarvis-speech:${jarvis_ngc_image_version}-server"

# define docker images required to setup Jarvis
image_init_speech="nvcr.io/${NGC_TARGET}/jarvis-speech:${jarvis_ngc_image_version}-servicemaker"

# daemon names
jarvis_daemon_speech="jarvis-speech"
jarvis_daemon_client="jarvis-client"

I’m not exactly sure where my models directory, with the models created by jarvis_build, is?

Am i wrong with changing the jarvis_model_loc directory in line 43, or can you point me to the mistake i made?
Because im getting an error when running Inference with:

python3 transcribe_file_offline.py --audio /home/ws/untow/jarvis/audio/00002.wav

The error is:

Traceback (most recent call last):
  File "transcribe_file_offline.py", line 62, in <module>
    response = client.Recognize(request)
  File "/home/ws/untow/jarvis/jarvis-venv/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/ws/untow/jarvis/jarvis-venv/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "Error: Model is not available on server"
        debug_error_string = "{"created":"@1625496050.258926342","description":"Error received from peer ipv6:[::1]:50051","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"Error: Model is not available on server","grpc_status":3}"

Thanks alot for helping me out.
Greets

2

Hi @martin.waldschmidt ,
Please check the below link for complete build process.
https://docs.nvidia.com/deeplearning/jarvis/user-guide/docs/custom-model-deployment.html#build-process.
Quoting from the link

When you execute jarvis_start.sh , it starts with the jarvis-speech container by mounting this $jarvis_model_loc directory to /data inside the container.

Thanks!