Problems when running ./riva_init.sh with custom Quartznet Model

AI & Data Science Deep Learning (Training & Inference) Riva
1

Hello,
im trying to deploy a .nemo Quartznet Model.
To reproduce, im listing the steps i have done sofar:

  1. I transformed my .nemo file into an .riva file by using the nemo2riva package.
    → german_quartznet.riva
  2. Riva-Build as follows:

docker run --gpus all -it --rm -v /home/ws/untow/riva2:/servicemaker-dev -v /home/ws/untow/riva2/models:/data --entrypoint=“/bin/bash” nvcr.io/nvidia/riva/riva-speech:1.5.0-beta-servicemaker

riva-build speech_recognition /servicemaker-dev/german_quartznet_greedy.rmir /servicemaker-dev/german_quartznet.riva --offline --decoder_type=greedy

  1. ./riva_init.sh output:
Logging into NGC docker registry if necessary...
Pulling required docker images if necessary...
Note: This may take some time, depending on the speed of your Internet connection.
> Pulling Riva Speech Server images.
  > Image nvcr.io/nvidia/riva/riva-speech:1.5.0-beta-server exists. Skipping.
  > Image nvcr.io/nvidia/riva/riva-speech-client:1.5.0-beta exists. Skipping.
  > Image nvcr.io/nvidia/riva/riva-speech:1.5.0-beta-servicemaker exists. Skipping.

Converting RMIRs at /home/ws/untow/riva2/riva_model_loc/rmir to Riva Model repository.
+ docker run --init -it --rm --gpus '"device=0"' -v /home/ws/untow/riva2/riva_model_loc:/data -e MODEL_DEPLOY_KEY=tlt_encode --name riva-service-maker nvcr.io/nvidia/riva/riva-speech:1.5.0-beta-servicemaker deploy_all_models /data/rmir /data/models

==========================
=== Riva Speech Skills ===
==========================

NVIDIA Release devel (build 22382700)

Copyright (c) 2018-2021, NVIDIA CORPORATION.  All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying
project or file.

NOTE: The SHMEM allocation limit is set to the default of 64MB.  This may be
   insufficient for the inference server.  NVIDIA recommends the use of the following flags:
   nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 ...

2021-09-07 12:01:16,375 [INFO] Writing Riva model repository to '/data/models'...
2021-09-07 12:01:16,375 [INFO] The riva model repo target directory is /data/models
2021-09-07 12:01:16,743 [INFO] Extract_binaries for featurizer -> /data/models/riva-asr-feature-extractor-streaming-offline/1
2021-09-07 12:01:16,744 [INFO] Extract_binaries for nn -> /data/models/riva-trt-riva-asr-am-streaming-offline/1
2021-09-07 12:01:16,776 [INFO] Printing copied artifacts:
2021-09-07 12:01:16,776 [INFO] {'onnx': '/data/models/riva-trt-riva-asr-am-streaming-offline/1/model_graph.onnx'}
2021-09-07 12:01:16,776 [INFO] Building TRT engine from ONNX file
[TensorRT] WARNING: /workspace/TensorRT/t/oss-cicd/oss/parsers/onnx/onnx2trt_utils.cpp:227: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[TensorRT] ERROR: safeContext.cpp (124) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
[TensorRT] ERROR: safeContext.cpp (124) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2021-09-07 12:01:23,164 [ERROR] Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/servicemaker/cli/deploy.py", line 86, in deploy_from_rmir
    generator.serialize_to_disk(
  File "/opt/conda/lib/python3.8/site-packages/servicemaker/triton/triton.py", line 356, in serialize_to_disk
    module.serialize_to_disk(repo_dir, rmir, config_only, verbose, overwrite)
  File "/opt/conda/lib/python3.8/site-packages/servicemaker/triton/triton.py", line 244, in serialize_to_disk
    self.update_binary(version_dir, rmir, verbose)
  File "/opt/conda/lib/python3.8/site-packages/servicemaker/triton/triton.py", line 603, in update_binary
    with self.build_trt_engine_from_onnx(model_weights) as engine, open(
AttributeError: __enter__

+ echo

+ echo 'Riva initialization complete. Run ./riva_start.sh to launch services.'
Riva initialization complete. Run ./riva_start.sh to launch services.
  1. ./riva_start.sh
    → health ready check failed

The weird thing is that i was able to initialize without an error yesterday. I already started from scratch 4 or 5 times now but the Error is consistent.

Thanks for your help in advance!

2

Hi @martin.waldschmidt
Could you please check if config.sh file is updated correctly

To deploy your own custom RMIR, or set of RMIRs, you would simply need to place them inside the $riva_model_loc/rmir directory. Ensure that you have defined a directory (that you have access to) in the $riva_model_loc variable in config.sh

Also for non-english please use additional options mentioned in below link:
https://docs.nvidia.com/deeplearning/riva/user-guide/docs/service-asr.html?highlight=speech_recognition#non-english-languages

In case issue persist, could you please share the config.sh file and actual model that is being used in this case so we can help better?

Thanks