JARVIS throwing errors for offline ASR when using own model

Description

I’m trying to transcribe a large number of audio files using my own ASR model.

I’ve trained a Jasper model using nemo. The model works well if I test it using a simple program I wrote that loads the checkpoint i created using EncDecCTCModel.restore_from then call transcribe, but it’s slow (as expected).

My next step was to deploy the model to NEMO for offline transcription, but I can’t get that working. I can get streaming ASR to work using the transcribe_file.py example app, but the results are worse than my test program. Since I don’t need streaming right now, I tried to deploy the model in offline mode to see if the results were any better. I did this by adding the --offline option to the javis-build command. This created a bunch of offline models, but when I try to run transcribe_file_offline.py I get an error “Error: Model is not available on server”. I can see there are offline models in my “models” directory so I am unsure how to fix this. Any help would be much appreciated.

Environment

TensorRT Version:
GPU Type: T4
Nvidia Driver Version: 450.119.03
CUDA Version: 11.0
CUDNN Version:
Operating System + Version: Ubuntu 18.04.5 LTS
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag): Docker nvcr.io/nvidia/jarvis/jarvis-speech:1.0.0-b.3-server

Relevant Files

The models are too large but the javis-build command created the following in my “models” directory.

jarvis-asr
jarvis-asr-ctc-decoder-cpu-streaming
jarvis-asr-ctc-decoder-cpu-streaming-offline
jarvis-asr-feature-extractor-streaming
jarvis-asr-feature-extractor-streaming-offline
jarvis-asr-voice-activity-detector-ctc-streaming
jarvis-asr-voice-activity-detector-ctc-streaming-offline
jarvis-trt-jarvis-asr-am-streaming
jarvis-trt-jarvis-asr-am-streaming-offline

Steps To Reproduce

Create the .enemo file from the nemo model

python ~/NeMo/scripts/export/convasr_to_enemo.py --nemo_file=/data/models/moneypenny.nemo --onnx_file=/data/models/moneypenny.onnx --enemo_file=/data/models/moneypenny.enemo

Start docker

docker run --gpus all -it --rm -v /data/models:/servicemaker-dev -v /data/jarvis:/data --entrypoint="/bin/bash" nvcr.io/nvidia/jarvis/jarvis-speech:1.0.0-b.3-servicemaker

Build the Jarvis models from the .enemo file

jarvis-build speech_recognition /servicemaker-dev/moneypenny-offline.jmir /servicemaker-dev/moneypenny.enemo --offline
jarvis-build speech_recognition /servicemaker-dev/moneypenny-streaming.jmir /servicemaker-dev/moneypenny.enemo

Run Jarvis

jarvis_init.sh
jarvis_start.sh

Test the streaming ASR (This works)

python3 ./transcribe_file.py --audio-file ~/test.wav

Test the offline ASR (This fails)

python3 ./transcribe_file_offline.py --audio-file ~/test.wav

Error output

Traceback (most recent call last):
File “./transcribe_file_offline.py”, line 62, in
response = client.Recognize(request)
File “/home/ubuntu/.local/lib/python3.6/site-packages/grpc/_channel.py”, line 946, in call
return _end_unary_response_blocking(state, call, False, None)
File “/home/ubuntu/.local/lib/python3.6/site-packages/grpc/_channel.py”, line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = “Error: Model is not available on server”
debug_error_string = “{“created”:”@1620636224.069156557",“description”:“Error received from peer ipv6:[::1]:50051”,“file”:“src/core/lib/surface/call.cc”,“file_line”:1067,“grpc_message”:“Error: Model is not available on server”,“grpc_status”:3}"

Hi @pete.hanlon
Could you please share the output of following command so we can help better

docker logs jarvis-speech

Meanwhile, could you please try just deploying the offline model instead of both versions?

Thanks

Hi Sunil,

Thanks for getting back to me so quickly. I tried deploying just the offline model and that fixed the problem! To be honest I thought I tried that already but clearly I hadn’t.

I don’t need the streaming model at the moment but I will in the future. Should the streaming and the offline models coexist in a single JARVIS server or would you expect to deploy them onto separate servers?

Hi @pete.hanlon,

They can coexist, the deployed models have different names for that reason.
The server will choose the appropriate model depending on what the client requests as long as the GPU has enough memory to load both models.

Thanks

Hi @SunilJB , Hi @pete.hanlon ,

i reproduced the steps from above. When running jarvis_init.sh i’m getting the following output:


Logging into NGC docker registry if necessary...
Pulling required docker images if necessary...
Note: This may take some time, depending on the speed of your Internet connection.
> Pulling Jarvis Speech Server images.
  > Image nvcr.io/nvidia/jarvis/jarvis-speech:1.2.1-beta-server exists. Skipping.
  > Image nvcr.io/nvidia/jarvis/jarvis-speech-client:1.2.1-beta exists. Skipping.
  > Image nvcr.io/nvidia/jarvis/jarvis-speech:1.2.1-beta-servicemaker exists. Skipping.

Converting JMIRs at /home/ws/untow/jarvis/jarvis-model-repo/jmir to Jarvis Model repository.
+ docker run --init -it --rm --gpus '"device=0"' -v /home/ws/untow/jarvis/jarvis-model-repo:/data -e MODEL_DEPLOY_KEY=tlt_encode --name jarvis-service-maker nvcr.io/nvidia/jarvis/jarvis-speech:1.2.1-beta-servicemaker deploy_all_models /data/jmir /data/models

==========================
== Jarvis Speech Skills ==
==========================

NVIDIA Release devel (build 22382700)

Copyright (c) 2018-2021, NVIDIA CORPORATION.  All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying
project or file.

NOTE: The SHMEM allocation limit is set to the default of 64MB.  This may be
   insufficient for the inference server.  NVIDIA recommends the use of the following flags:
   nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 ...

Traceback (most recent call last):
  File "/opt/conda/bin/jarvis-deploy", line 8, in <module>
    sys.exit(deploy_from_jmir())
  File "/opt/conda/lib/python3.8/site-packages/servicemaker/cli/deploy.py", line 73, in deploy_from_jmir
    raise FileExistsError(f"{args.target} exists. Use --force/-f to overwrite.")
FileExistsError: /data/models exists. Use --force/-f to overwrite.
+ echo

+ echo 'Jarvis initialization complete. Run ./jarvis_start.sh to launch services.'
Jarvis initialization complete. Run ./jarvis_start.sh to launch services.

Then, when running test-inference, im getting the same error from above:

Traceback (most recent call last):
  File "transcribe_file_offline.py", line 62, in <module>
    response = client.Recognize(request)
  File "/home/ws/untow/jarvis/jarvis-venv/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/ws/untow/jarvis/jarvis-venv/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "Error: Model is not available on server"
        debug_error_string = "{"created":"@1625496050.258926342","description":"Error received from peer ipv6:[::1]:50051","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"Error: Model is not available on server","grpc_status":3}"
>

Where do i find this “models” directory?

after running jarvis_build exactly like

jarvis-build speech_recognition /servicemaker-dev/germanQuartz.jmir /servicemaker-dev/germanQuartz.enemo --offline

assuming i started the docker like this

docker run --gpus all -it --rm -v /home/ws/untow/jarvis/jarvis-model-repo/models:/servicemaker-dev -v /home/ws/untow/jarvis:/data --entrypoint="/bin/bash" nvcr.io/nvidia/jarvis/jarvis-speech:1.2.1-beta-servicemaker

shouldnt the model-repository be in /home/ws/untow/jarvis?
i cant find the it there.

If i missed to give important data, hmu
Thanks for your help in advance :)

Hi @martin.waldschmidt ,
Can you please try the steps from the below link?
https://docs.nvidia.com/deeplearning/riva/user-guide/docs/custom-model-deployment.html#build-process
Quoting from there

When you execute riva_start.sh , it starts with the riva-speech container by mounting this $riva_model_loc directory to /data inside the container.

In case if the issue persist, please share your model and reproducible script with us.

Thanks!

Hi Martin,
can you let us know if the suggestion from AskanshaS resolves you concern.
Also to ensure a better response to a new concern its best to spin up a new topic - and reference the original post if its a related problem.
Thanks so much for sharing - your experiences helps so many others who many be seeing the same problems.
Thanks !

Hi @nadeemm , Hi @AakankshaS ,

sorry for replying that late. I had some major problems with my RTX3090. But they are fixed now.

So i see you changed from Jarvis to Riva…
My first Question is:
How can i change my .nemo file into an .riva file?
I found this:, but i dont understand how i setup an NeMo environment.
Can u please point out how this is done properly?
Or is there a script like the old convasr_to_enemo.py, which converted nemo to enemo, which then was used for jarvis_build ?

Im very confused at this point. Thanks for your help.

Martin

Hi @martin.waldschmidt,

You can use pip install approach to install nemo on your setup

One more way to avoid system dependency is to use NeMo NGC container.
https://ngc.nvidia.com/

Post installation step as mentioned in the doc, you can use nemo2riva to convert nemo models directly to .riva models

Please do let us know in case you get any error or issues.

Thanks

Hi @SunilJB ,

thanks for your fast reply.
so what i did is:

  • Create Nemo environment
    docker run --runtime=nvidia -it --rm -v --shm-size=16g -p 8888:8888 -p 6006:6006 --ulimit memlock=-1 --ulimit stack=67108864 nvcr.io/nvidia/nemo:v1.0.0b1

  • Followed Steps from the RIVA Docs from inside the container
    $pip3 install nvidia-pyindex
    worked fine. But then:
    pip3 install nemo2riva-1.3.0_beta-py3-none-any.whl
    ERROR!
    ERROR: Could not install packages due to an EnvironmentError: [Error 2] No such file or directory: '/workspace/nemo/nemo2riva-1.3.0_beta-py3-none-any.whl'

I found the nemo2riva files in the RIVA-Quickstart Directory. So where do i have to put it so that it workes? Or do i have to bind-mount my Quickstart Directory into the docker container?

im sorry for asking this many questions… im kind of new to this stuff

Thanks alot for your help!

The nemo2riva tool is currently packaged and available via Riva Quickstart.
So you can either mount/download the RIVA quickstart folder inside your nemo container to keep it separate environment
or you can use nemo pip install in order to setup NeMo to generate .riva model from same or virtual env.

Thanks

Hi @SunilJB ,
once again, thanks for your fast reply. I chose to download riva quickstart inside my Nemo Container.
After that i executed these:

pip3 install nvidia-pyindex
pip3 install nemo2riva-1.5.0_beta-py3-none-any.whl
nemo2riva --out /NeMo/german_quartznet.riva german_quartznet.nemo

But i get an Error, which u can look at at the end of this post, after the last command:

Im in a NeMo 1.1.0 environment. This should be no problem as the docs mention:

NeMo environment version 1.1.0 or greater.

And im using this german Quartznet STT Model from NGC.

Thanks for helping me out!
My Command and the full Error from above is:

root@d600818a1649:/workspace/nemo# nemo2riva --out /NeMo/german_quartznet.riva german_quartznet.nemo
INFO: Logging level set to 20
INFO: Restoring NeMo model from 'german_quartznet.nemo'
ERROR: Nemo2Jarvis: Failed to restore model from NeMo file : german_quartznet.nemo. Please make sure you have the latest NeMo package installed with [all] dependencies.
Traceback (most recent call last):
  File "/opt/conda/bin/nemo2riva", line 8, in <module>
    sys.exit(nemo2riva())
  File "/opt/conda/lib/python3.6/site-packages/nemo2riva/cli/nemo2riva.py", line 50, in nemo2riva
    Nemo2Riva(args)
  File "/opt/conda/lib/python3.6/site-packages/nemo2riva/convert.py", line 42, in Nemo2Riva
    raise e
  File "/opt/conda/lib/python3.6/site-packages/nemo2riva/convert.py", line 35, in Nemo2Riva
    model = ModelPT.restore_from(restore_path=nemo_in)
  File "/opt/conda/lib/python3.6/site-packages/nemo/core/classes/modelPT.py", line 271, in restore_from
    cls, restore_path, override_config_path, map_location, strict, return_config
  File "/opt/conda/lib/python3.6/site-packages/nemo/core/connectors/save_restore_connector.py", line 136, in restore_from
    instance = calling_cls.from_config_dict(config=conf)
  File "/opt/conda/lib/python3.6/site-packages/nemo/core/classes/common.py", line 473, in from_config_dict
    instance = cls(cfg=config)
TypeError: Can't instantiate abstract class ModelPT with abstract methods list_available_models, setup_training_data, setup_validation_data
INFO: Internal process exited

i found my mistake:
i followed the steps from NVIDIAs NeMo 1.2 Container page .

They mention to get started by running this command:

docker run --runtime=nvidia -it --rm -v --shm-size=16g -p 8888:8888 -p 6006:6006 --ulimit memlock=-1 --ulimit stack=67108864 nvcr.io/nvidia/nemo:v1.0.0b1

But this actually runs a 1.0 Container.

I leave this here, so others can learn from my mistakes.

THANKS TO THE NVIDIA TEAM FOR HELPING ME OUT!

1 Like