Not able to run LM fine tuned qurtznet model

AI & Data Science Deep Learning (Training & Inference) Riva
1

Please provide the following information when requesting support.

Hardware - GPU (T4)
Hardware - CPU
Operating System - ubuntu
Riva Version = 1.0 Beta
TLT Version (if relevant)
How to reproduce the issue ? (This is for errors. Please share the command and the detailed log here)

Fine tune quartznet model with custom language model. it succeeds

riva-build speech_recognition
/servicemaker-dev/riva-custom-speech.rmir:tlt_encode
/servicemaker-dev/quartznet_asr_set_1pt2.riva:tlt_encode
–name riva-custom-speech-quartznet
–decoder_type=os2s
–decoding_language_model_binary=/lm/text.binary -f

2021-08-21 18:01:26,090 [INFO] Packing binaries for nn
2021-08-21 18:01:26,663 [INFO] Trying to extract from model quartznet_asr_set_1pt2.riva
2021-08-21 18:01:27,207 [INFO] Packing binaries for lm_decoder
2021-08-21 18:01:27,207 [INFO] Trying to copy model binary from /tmp/tmpd4t7f589/vocab.txt into rmir at /servicemaker-dev/riva-custom-speech.rmir.
2021-08-21 18:01:27,208 [INFO] Trying to copy model binary from /lm/text.binary into rmir at /servicemaker-dev/riva-custom-speech.rmir.
2021-08-21 18:01:27,209 [INFO] Packing binaries for rescorer
2021-08-21 18:01:27,210 [INFO] Trying to copy model binary from /tmp/tmpd4t7f589/vocab.txt into rmir at /servicemaker-dev/riva-custom-speech.rmir.
2021-08-21 18:01:27,210 [INFO] Packing binaries for vad
2021-08-21 18:01:27,211 [INFO] Trying to copy model binary from /tmp/tmpd4t7f589/vocab.txt into rmir at /servicemaker-dev/riva-custom-speech.rmir.

---------------------- SUCESS SO FAR--------

but when trying to do riva_init.sh i get following error

2021-08-21 18:22:08,719 [INFO] Building TRT engine from ONNX file
[libprotobuf WARNING /workspace/TensorRT/t/oss-cicd/oss/build/third_party.protobuf/src/third_party.protobuf/src/google/protobuf/io/coded_stream.cc:604] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING /workspace/TensorRT/t/oss-cicd/oss/build/third_party.protobuf/src/third_party.protobuf/src/google/protobuf/io/coded_stream.cc:81] The total number of bytes read was 564429124
[TensorRT] WARNING: /workspace/TensorRT/t/oss-cicd/oss/parsers/onnx/onnx2trt_utils.cpp:227: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.

2021-08-21 18:31:48,833 [INFO] Extract_binaries for vad → /data/models/riva-custom-speech-voice-activity-detector-ctc-streaming/1
2021-08-21 18:31:48,835 [INFO] Extract_binaries for lm_decoder → /data/models/riva-custom-speech-ctc-decoder-cpu-streaming/1
2021-08-21 18:31:48,837 [ERROR] Couldn’t extract /lm/vocab.txt to /data/models/riva-custom-speech-ctc-decoder-cpu-streaming/1
Traceback (most recent call last):
File “/opt/conda/lib/python3.8/site-packages/servicemaker/rmir/rmir.py”, line 314, in extract
raise ValueError(f"{target_name} not in artifacts")
ValueError: lm_decoder-vocab.txt not in artifacts
2021-08-21 18:31:48,838 [INFO] {‘vocab_file’: ‘/data/models/riva-custom-speech-ctc-decoder-cpu-streaming/1/vocab.txt’, ‘decoding_language_model_binary’: ‘/data/models/riva-custom-speech-ctc-decoder-cpu-streaming/1/text.binary’, ‘tokenizer_model’: ‘/data/models/riva-custom-speech-ctc-decoder-cpu-streaming/1/tokenizer.model’}
2021-08-21 18:31:48,838 [ERROR] Traceback (most recent call last):
File “/opt/conda/lib/python3.8/site-packages/servicemaker/cli/deploy.py”, line 85, in deploy_from_rmir
generator.serialize_to_disk(
File “/opt/conda/lib/python3.8/site-packages/servicemaker/triton/triton.py”, line 340, in serialize_to_disk
module.serialize_to_disk(repo_dir, rmir, config_only, verbose, overwrite)
File “/opt/conda/lib/python3.8/site-packages/servicemaker/triton/triton.py”, line 231, in serialize_to_disk
self.update_binary(version_dir, rmir, verbose)
File “/opt/conda/lib/python3.8/site-packages/servicemaker/triton/asr.py”, line 409, in update_binary
vocab_file = RivaConfigGenerator.get_binary_from_key(self, copied, ‘decoding_vocab’)
File “/opt/conda/lib/python3.8/site-packages/servicemaker/triton/triton.py”, line 294, in get_binary_from_key
raise Exception(binary_key + " not found in .rmir")
Exception: decoding_vocab not found in .rmir

  • echo

  • echo ‘Riva initialization complete. Run ./riva_start.sh to launch services.’
    Riva initialization complete. Run ./riva_start.sh to launch services.

2

Hi @ai.consultancy.prashant
Could you please share docker logs riva-speech log output so we can help better?

Thanks

3

This is the output riva_init.sh. I did not run riva_start.sh

4

I had another go with riva-build – and I get following error:

root@ce4e2c8bb794:/opt/riva# riva-build speech_recognition /data/rmir/riva-custom-speech.rmir:tlt_encode /servicemaker-dev/Citrinet-1024-Jarvis-ASRSet-1_7-encrypted.riva:tlt_encode --name=riva-custom-speech --decoder_type=flashlight --chunk_size=0.16 --padding_size=1.92 --ms_per_timestep=80 --flashlight_decoder.asr_model_delay=-1 --featurizer.use_utterance_norm_params=False --featurizer.precalc_norm_time_steps=0 --featurizer.precalc_norm_params=False --decoding_language_model_binary=/lm/text.binary --decoding_vocab=/lm/vocab.txt
2021-08-23 19:26:03,442 [INFO] Packing binaries for nn
2021-08-23 19:26:04,874 [INFO] Trying to extract from model Citrinet-1024-Jarvis-ASRSet-1_7-encrypted.riva
2021-08-23 19:26:10,689 [INFO] Packing binaries for lm_decoder
2021-08-23 19:26:10,689 [INFO] Trying to copy model binary from /tmp/tmpytcyamxc/vocab.txt into rmir at /data/rmir/riva-custom-speech.rmir.
2021-08-23 19:26:10,690 [INFO] Trying to copy model binary from /lm/text.binary into rmir at /data/rmir/riva-custom-speech.rmir.
2021-08-23 19:26:10,692 [INFO] Trying to copy model binary from /lm/vocab.txt into rmir at /data/rmir/riva-custom-speech.rmir.
2021-08-23 19:26:10,692 [ERROR] Could not create rmir eff handle
Traceback (most recent call last):
File “/opt/conda/lib/python3.8/site-packages/servicemaker/rmir/rmir.py”, line 202, in write
target_path = self._eff.create_file_handle(
File “”, line 524, in create_file_handle
KeyError: ‘File lm_decoder-vocab.txt already present in the archive’
2021-08-23 19:26:12,159 [INFO] Trying to extract from model Citrinet-1024-Jarvis-ASRSet-1_7-encrypted.riva
2021-08-23 19:26:13,519 [INFO] Packing binaries for rescorer
2021-08-23 19:26:13,519 [INFO] Trying to copy model binary from /tmp/tmpytcyamxc/vocab.txt into rmir at /data/rmir/riva-custom-speech.rmir.
2021-08-23 19:26:13,520 [INFO] Packing binaries for vad
2021-08-23 19:26:13,520 [INFO] Trying to copy model binary from /tmp/tmpytcyamxc/vocab.txt into rmir at /data/rmir/riva-custom-speech.rmir.
root@ce4e2c8bb794:/opt/riva#

Nonetheless it created .rmir file so tried riva_init.sh with it but got following error.

ubuntu@ip-172-31-12-53 : ~/storage/riva_quickstart_v1.4.0-beta $ bash riva_init.sh

Logging into NGC docker registry if necessary…

Pulling required docker images if necessary…

Note: This may take some time, depending on the speed of your Internet connection.

Pulling Riva Speech Server images.

Image nvcr.io/nvidia/riva/riva-speech:1.4.0-beta-server exists. Skipping.

Image nvcr.io/nvidia/riva/riva-speech-client:1.4.0-beta exists. Skipping.

Image nvcr.io/nvidia/riva/riva-speech:1.4.0-beta-servicemaker exists. Skipping.

Converting RMIRs at riva-model-repo/rmir to Riva Model repository.

==========================

=== Riva Speech Skills ===

==========================

NVIDIA Release devel (build 22382700)

Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.

NVIDIA modifications are covered by the license terms that apply to the underlying

project or file.

NOTE: Legacy NVIDIA Driver detected. Compatibility mode ENABLED.

NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be

insufficient for the inference server. NVIDIA recommends the use of the following flags:

nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 …

2021-08-23 19:46:23,061 [INFO] Writing Riva model repository to ‘/data/models’…

2021-08-23 19:46:23,061 [INFO] The riva model repo target directory is /data/models

2021-08-23 19:46:24,404 [INFO] Extract_binaries for featurizer → /data/models/riva-custom-speech-feature-extractor-streaming/1

2021-08-23 19:46:24,406 [INFO] Extract_binaries for nn → /data/models/riva-trt-riva-custom-speech-am-streaming/1

2021-08-23 19:46:24,406 [INFO] Extract_binaries for nn → /data/models/riva-trt-riva-custom-speech-am-streaming/1

2021-08-23 19:46:29,183 [INFO] Printing copied artifacts:

2021-08-23 19:46:29,183 [INFO] {‘onnx’: ‘/data/models/riva-trt-riva-custom-speech-am-streaming/1/model_graph.onnx’}

2021-08-23 19:46:29,183 [INFO] Building TRT engine from ONNX file

[libprotobuf WARNING /workspace/TensorRT/t/oss-cicd/oss/build/third_party.protobuf/src/third_party.protobuf/src/google/protobuf/io/coded_stream.cc:604] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.

[libprotobuf WARNING /workspace/TensorRT/t/oss-cicd/oss/build/third_party.protobuf/src/third_party.protobuf/src/google/protobuf/io/coded_stream.cc:81] The total number of bytes read was 564429124

[TensorRT] WARNING: /workspace/TensorRT/t/oss-cicd/oss/parsers/onnx/onnx2trt_utils.cpp:227: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.

2021-08-23 19:56:10,950 [INFO] Extract_binaries for vad → /data/models/riva-custom-speech-voice-activity-detector-ctc-streaming/1
2021-08-23 19:56:10,952 [INFO] Extract_binaries for lm_decoder → /data/models/riva-custom-speech-ctc-decoder-cpu-streaming/1
2021-08-23 19:56:10,953 [ERROR] Couldn’t extract /lm/vocab.txt to /data/models/riva-custom-speech-ctc-decoder-cpu-streaming/1
Traceback (most recent call last):
File “/opt/conda/lib/python3.8/site-packages/servicemaker/rmir/rmir.py”, line 314, in extract
raise ValueError(f"{target_name} not in artifacts")
ValueError: lm_decoder-vocab.txt not in artifacts
2021-08-23 19:56:10,954 [INFO] {‘vocab_file’: ‘/data/models/riva-custom-speech-ctc-decoder-cpu-streaming/1/vocab.txt’, ‘decoding_language_model_binary’: ‘/data/models/riva-custom-speech-ctc-decoder-cpu-streaming/1/text.binary’, ‘tokenizer_model’: ‘/data/models/riva-custom-speech-ctc-decoder-cpu-streaming/1/tokenizer.model’}
2021-08-23 19:56:10,955 [ERROR] Traceback (most recent call last):
File “/opt/conda/lib/python3.8/site-packages/servicemaker/cli/deploy.py”, line 85, in deploy_from_rmir
generator.serialize_to_disk(
File “/opt/conda/lib/python3.8/site-packages/servicemaker/triton/triton.py”, line 340, in serialize_to_disk
module.serialize_to_disk(repo_dir, rmir, config_only, verbose, overwrite)
File “/opt/conda/lib/python3.8/site-packages/servicemaker/triton/triton.py”, line 231, in serialize_to_disk
self.update_binary(version_dir, rmir, verbose)
File “/opt/conda/lib/python3.8/site-packages/servicemaker/triton/asr.py”, line 409, in update_binary
vocab_file = RivaConfigGenerator.get_binary_from_key(self, copied, ‘decoding_vocab’)
File “/opt/conda/lib/python3.8/site-packages/servicemaker/triton/triton.py”, line 294, in get_binary_from_key
raise Exception(binary_key + " not found in .rmir")
Exception: decoding_vocab not found in .rmir

  • echo

  • echo ‘Riva initialization complete. Run ./riva_start.sh to launch services.’
    Riva initialization complete. Run ./riva_start.sh to launch services.

5

Hi @ai.consultancy.prashant

Could you please try riva installation clean-up before retrying the build process.
https://docs.nvidia.com/deeplearning/riva/user-guide/docs/deploy-local.html?highlight=clean#clean-up

Thanks

6

I did not use riva_clean.sh but I had removed the existing rmir and models from the appropriate directories before running riva-build. I think the issue is with the code itself. If the .riva models already has a vocab.txt then it fails to add the custom vocab.txt provided in command line arguments.

7

I tried with bash riva_clean.sh …same result
bash riva_clean.sh

Cleaning up local Riva installation.

Deleting docker volume…

Found docker volume ‘riva-model-repo’. Delete? [y/N] y

riva-build speech_recognition
/servicemaker-dev/riva-custom-speech.rmir:tlt_encode
/servicemaker-dev/Citrinet-1024-Jarvis-ASRSet-1_7-encrypted.riva:tlt_encode
–name=riva-custom-speech
–decoder_type=flashlight
–chunk_size=0.16
–padding_size=1.92
–ms_per_timestep=80
–flashlight_decoder.asr_model_delay=-1
–featurizer.use_utterance_norm_params=False
–featurizer.precalc_norm_time_steps=0
–featurizer.precalc_norm_params=False
–decoding_language_model_binary=/lm/text.binary
–decoding_vocab=/lm/vocab.txt

2021-08-25 05:41:00,085 [INFO] Packing binaries for nn
2021-08-25 05:41:01,539 [INFO] Trying to extract from model Citrinet-1024-Jarvis-ASRSet-1_7-encrypted.riva
2021-08-25 05:41:07,490 [INFO] Packing binaries for lm_decoder
2021-08-25 05:41:07,491 [INFO] Trying to copy model binary from /tmp/tmpmy3pysra/vocab.txt into rmir at /servicemaker-dev/riva-custom-speech.rmir.
2021-08-25 05:41:07,494 [INFO] Trying to copy model binary from /lm/text.binary into rmir at /servicemaker-dev/riva-custom-speech.rmir.
2021-08-25 05:41:08,976 [ERROR] Could not create rmir eff handle
Traceback (most recent call last):
File “/opt/conda/lib/python3.8/site-packages/servicemaker/rmir/rmir.py”, line 202, in write
target_path = self._eff.create_file_handle(
File “”, line 524, in create_file_handle
KeyError: ‘File lm_decoder-vocab.txt already present in the archive’
2021-08-25 05:41:08,977 [ERROR] Could not find required binary for lm_decoder at location /lm/vocab.txt
NoneType: None
2021-08-25 05:41:10,440 [INFO] Trying to extract from model Citrinet-1024-Jarvis-ASRSet-1_7-encrypted.riva
2021-08-25 05:41:11,822 [INFO] Packing binaries for rescorer
2021-08-25 05:41:11,823 [INFO] Trying to copy model binary from /tmp/tmpmy3pysra/vocab.txt into rmir at /servicemaker-dev/riva-custom-speech.rmir.
2021-08-25 05:41:11,824 [INFO] Packing binaries for vad
2021-08-25 05:41:11,825 [INFO] Trying to copy model binary from /tmp/tmpmy3pysra/vocab.txt into rmir at /servicemaker-dev/riva-custom-speech.rmir.

8

Btw, greedy decoder_type is working fine.

riva-build speech_recognition /servicemaker-dev/riva-custom-speech.rmir:tlt_encode /servicemaker-dev/Citrinet-1024-Jarvis-ASRSet-1_7-encrypted.riva:tlt_encode --name=riva-custom-speech --decoder_type=greedy --chunk_size=0.16 --padding_size=1.92 --ms_per_timestep=80 --greedy_decoder.asr_model_delay=-1 --featurizer.use_utterance_norm_params=False --featurizer.precalc_norm_time_steps=0 --featurizer.precalc_norm_params=False -f
2021-08-25 05:46:05,202 [INFO] Packing binaries for nn
2021-08-25 05:46:06,654 [INFO] Trying to extract from model Citrinet-1024-Jarvis-ASRSet-1_7-encrypted.riva
2021-08-25 05:46:12,612 [INFO] Packing binaries for lm_decoder
2021-08-25 05:46:12,612 [INFO] Trying to copy model binary from /tmp/tmp9wrnn764/vocab.txt into rmir at /servicemaker-dev/riva-custom-speech.rmir.
2021-08-25 05:46:14,105 [INFO] Trying to extract from model Citrinet-1024-Jarvis-ASRSet-1_7-encrypted.riva
2021-08-25 05:46:15,495 [INFO] Packing binaries for rescorer
2021-08-25 05:46:15,496 [INFO] Trying to copy model binary from /tmp/tmp9wrnn764/vocab.txt into rmir at /servicemaker-dev/riva-custom-speech.rmir.
2021-08-25 05:46:15,496 [INFO] Packing binaries for vad
2021-08-25 05:46:15,497 [INFO] Trying to copy model binary from /tmp/tmp9wrnn764/vocab.txt into rmir at /servicemaker-dev/riva-custom-speech.rmir.

9

Kaldi decoder type also fails

---------- RIVA : BUILD ------

riva-build speech_recognition /servicemaker-dev/riva-custom-speech.rmir:tlt_encode /servicemaker-dev/Citrinet-1024-Jarvis-ASRSet-1_7-encrypted.riva:tlt_encode --name=riva-custom-kaldi --decoding_language_model_arpa=/lm/text.arpa --decoder_type=kaldi -f
2021-08-24 22:15:39,285 [INFO] Packing binaries for nn
2021-08-24 22:15:40,754 [INFO] Trying to extract from model Citrinet-1024-Jarvis-ASRSet-1_7-encrypted.riva
2021-08-24 22:15:46,781 [INFO] Packing binaries for lm_decoder
2021-08-24 22:15:46,782 [INFO] Trying to copy model binary from /tmp/tmpefvvn9fm/vocab.txt into rmir at /servicemaker-dev/riva-custom-speech.rmir.
2021-08-24 22:15:46,783 [INFO] Trying to copy model binary from /lm/text.arpa into rmir at /servicemaker-dev/riva-custom-speech.rmir.
2021-08-24 22:15:48,285 [INFO] Trying to extract from model Citrinet-1024-Jarvis-ASRSet-1_7-encrypted.riva
2021-08-24 22:15:49,681 [INFO] Packing binaries for rescorer
2021-08-24 22:15:49,681 [INFO] Trying to copy model binary from /tmp/tmpefvvn9fm/vocab.txt into rmir at /servicemaker-dev/riva-custom-speech.rmir.
2021-08-24 22:15:49,682 [INFO] Packing binaries for vad
2021-08-24 22:15:49,682 [INFO] Trying to copy model binary from /tmp/tmpefvvn9fm/vocab.txt into rmir at /servicemaker-dev/riva-custom-speech.rmir.

---------- RIVA - INIT --------

$ bash riva_init.sh
.
.
.
Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.

NVIDIA modifications are covered by the license terms that apply to the underlying

project or file.

NOTE: Legacy NVIDIA Driver detected. Compatibility mode ENABLED.

NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be

insufficient for the inference server. NVIDIA recommends the use of the following flags:

nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 …

2021-08-25 07:16:51,163 [INFO] Writing Riva model repository to ‘/data/models’…

2021-08-25 07:16:51,163 [INFO] The riva model repo target directory is /data/models

2021-08-25 07:16:52,523 [INFO] Extract_binaries for featurizer → /data/models/riva-custom-kaldi-feature-extractor-streaming/1

2021-08-25 07:16:52,525 [INFO] Extract_binaries for nn → /data/models/riva-trt-riva-custom-kaldi-am-streaming/1

2021-08-25 07:16:57,385 [INFO] Printing copied artifacts:

2021-08-25 07:16:57,385 [INFO] {‘onnx’: ‘/data/models/riva-trt-riva-custom-kaldi-am-streaming/1/model_graph.onnx’}

2021-08-25 07:16:57,385 [INFO] Building TRT engine from ONNX file

[libprotobuf WARNING /workspace/TensorRT/t/oss-cicd/oss/build/third_party.protobuf/src/third_party.protobuf/src/google/protobuf/io/coded_stream.cc:604] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.

[libprotobuf WARNING /workspace/TensorRT/t/oss-cicd/oss/build/third_party.protobuf/src/third_party.protobuf/src/google/protobuf/io/coded_stream.cc:81] The total number of bytes read was 564431804

[TensorRT] WARNING: /workspace/TensorRT/t/oss-cicd/oss/parsers/onnx/onnx2trt_utils.cpp:227: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.

2021-08-25 07:26:29,723 [INFO] Extract_binaries for vad → /data/models/riva-custom-kaldi-voice-activity-detector-ctc-streaming/1

2021-08-25 07:26:29,725 [INFO] Extract_binaries for lm_decoder → /data/models/riva-custom-kaldi-ctc-decoder-gpu-streaming/1

2021-08-25 07:26:29,727 [INFO] Building language model fst from arpa file: /data/models/riva-custom-kaldi-ctc-decoder-gpu-streaming/1/text.arpa

2021-08-25 07:26:29,727 [WARNING] This command can take a long time to execute depending on the size of the language model

2021-08-25 07:26:29,975 [INFO] Extract_binaries for rescorer → /data/models/riva-custom-kaldi-lattice-post-processor/1

2021-08-25 07:26:29,977 [INFO] Lattice rescorer will not do any rescoring

2021-08-25 07:26:29,978 [INFO] Extract_binaries for self → /data/models/riva-custom-kaldi/1

  • echo

  • echo ‘Riva initialization complete. Run ./riva_start.sh to launch services.’

Riva initialization complete. Run ./riva_start.sh to launch services.

---------------- RIVA - START —

Riva waiting for Triton server to load all models…retrying in 1 second

I0825 07:27:37.221656 73 metrics.cc:228] Collecting metrics for GPU 0: Tesla T4

I0825 07:27:37.224915 73 onnxruntime.cc:1722] TRITONBACKEND_Initialize: onnxruntime

I0825 07:27:37.224945 73 onnxruntime.cc:1732] Triton TRITONBACKEND API version: 1.0

I0825 07:27:37.224952 73 onnxruntime.cc:1738] ‘onnxruntime’ TRITONBACKEND API version: 1.0

I0825 07:27:37.408339 73 pinned_memory_manager.cc:206] Pinned memory pool is created at ‘0x7fd2c0000000’ with size 268435456

I0825 07:27:37.408747 73 cuda_memory_manager.cc:103] CUDA memory pool is created on device 0 with size 1000000000

I0825 07:27:37.414177 73 model_repository_manager.cc:1066] loading: riva-custom-kaldi-feature-extractor-streaming:1

I0825 07:27:37.514491 73 model_repository_manager.cc:1066] loading: riva-custom-kaldi-ctc-decoder-gpu-streaming:1

I0825 07:27:37.514927 73 custom_backend.cc:201] Creating instance riva-custom-kaldi-feature-extractor-streaming_0_0_gpu0 on GPU 0 (7.5) using libtriton_riva_asr_features.so

I0825 07:27:37.614825 73 model_repository_manager.cc:1066] loading: riva-custom-kaldi-lattice-post-processor:1

I0825 07:27:37.615270 73 custom_backend.cc:201] Creating instance riva-custom-kaldi-ctc-decoder-gpu-streaming_0_0_gpu0 on GPU 0 (7.5) using libtriton_riva_asr_decoder_gpu.so

W:parameter_parser.cc:106: Parameter forerunner_start_offset_ms could not be set from parameters

W:parameter_parser.cc:107: Default value will be used

W:parameter_parser.cc:106: Parameter voc_string could not be set from parameters

W:parameter_parser.cc:107: Default value will be used

E:ctc_gpu_decoder_context.cc:734: use_subword == True is not supported by Kaldi decoder

E0825 07:27:37.692320 73 dynamic_batch_scheduler.cc:253] Initialization failed for dynamic-batch scheduler thread 0: initialize error for ‘riva-custom-kaldi-ctc-decoder-gpu-streaming’: (13) Invalid parameters in model configuration

E0825 07:27:37.692451 73 sequence_batch_scheduler.cc:1297] failed creating dynamic sequence batcher for OldestFirst 0: Initialization failed for all dynamic-batch scheduler threads

E0825 07:27:37.701087 73 model_repository_manager.cc:1243] failed to load ‘riva-custom-kaldi-ctc-decoder-gpu-streaming’ version 1: Internal: Initialization failed for all sequence-batch scheduler threads

I0825 07:27:37.715132 73 model_repository_manager.cc:1066] loading: riva-custom-kaldi-voice-activity-detector-ctc-streaming:1

I0825 07:27:37.715375 73 custom_backend.cc:198] Creating instance riva-custom-kaldi-lattice-post-processor_0_0_cpu on CPU using libtriton_riva_asr_lattices.so

I0825 07:27:37.718043 73 custom_backend.cc:198] Creating instance riva-custom-kaldi-lattice-post-processor_0_1_cpu on CPU using libtriton_riva_asr_lattices.so

ldi-ctc-decoder-gpu-streaming/1/words.txt

E:lattice_post_processor_context.cc:74: Exception when initializing lattice post processor: Could not read symbol table from file /data/models/riva-custom-kaldi-ctc-decoder-gpu-streaming/1/words.txt

ERROR: SymbolTable::ReadText: Can’t open file /data/models/riva-custom-kaldi-ctc-decoder-gpu-streaming/1/words.txt

E:lattice_post_processor_context.cc:74: Exception when initializing lattice post processor: Could not read symbol table from file /data/models/riva-custom-kaldi-ctc-decoder-gpu-streaming/1/words.txt

ERROR: SymbolTable::ReadText: Can’t open file /data/models/riva-custom-kaldi-ctc-decoder-gpu-streaming/1/words.txt

ERROR: SymbolTable::ReadText: Can’t open file /data/models/riva-custom-kaldi-ctc-decoder-gpu-streaming/1/words.txt

E:lattice_post_processor_context.cc:74: Exception when initializing lattice post processor: Could not read symbol table from file /data/models/riva-custom-kaldi-ctc-decoder-gpu-streaming/1/words.txt

E:lattice_post_processor_context.cc:74: Exception when initializing lattice post processor: Could not read symbol table from file /data/models/riva-custom-kaldi-ctc-decoder-gpu-streaming/1/words.txt

ERROR: SymbolTable::ReadText: Can’t open file /data/models/riva-custom-kaldi-ctc-decoder-gpu-streaming/1/words.txt

I0825 07:27:37.749965 73 model_repository_manager.cc:1240] successfully loaded ‘riva-custom-kaldi-lattice-post-processor’ version 1

I0825 07:27:37.815420 73 model_repository_manager.cc:1066] loading: riva-trt-riva-custom-kaldi-am-streaming:1

I0825 07:27:37.815807 73 custom_backend.cc:198] Creating instance riva-custom-kaldi-voice-activity-detector-ctc-streaming_0_0_cpu on CPU using libtriton_riva_asr_vad.so

10

Any upates on this one?

11

Hi @ai.consultancy.prashant ,
Can you please share the language model you are using so that we can help better, Meanwhile Can you please try the following steps:

1. Download the language models:
ngc registry model download-version "nvidia/tao/speechtotext_english_lm:deployable_v1.0"

2. Download the quartznet model:
ngc registry model download-version "nvidia/tao/speechtotext_english_quartznet:deployable_v1.2"

3.Run servicemaker for riva-build & riva-deploy:
 docker run --gpus all -it --rm -v <riva models path>:/servicemaker-dev -v <deployment path where trt files gets generated>:/data --entrypoint=/bin/bash nvcr.io/nvidia/riva/riva-speech:1.4.0-beta-servicemaker

4. riva-build:
riva-build speech_recognition riva-custom-speech.rmir:tlt_encode quartznet_asr_set_1pt2.riva:tlt_encode --name riva-custom-speech-quartznet --decoder_type=os2s --decoding_language_model_binary=speechtotext_english_lm_vdeployable_v1.0/mixed-lower.binary  -f

5. riva-deploy:
riva-deploy riva-custom-speech.rmir:tlt_encode /data/models/

6. Edit config.sh & set model-repo location to <deployment path where trt files gets generated> in setsp 3 & run ./riva_start.sh.
12

I will try these. For the language model it is standard languge model generated with kenlm with lmplz and build_binary. I have tested the language model with other scripts and it works fine.

I opened up .riva bundle (since it is just .tar.gz ) and I see it already has .vocab.txt and that is somehow conflicting with the --decoding_vocab argument.

13

please try renaming the -decoding_vocab param.
Somethhing like --decoding_vocab=“vocab_lm.txt”

14

Changing the name of my vocabulary file solved this problem for me. The vocabulary file was named “vocab.txt” but I renamed it to “vocab_lm.txt”.