NVIDIA Riva ASR failed start with WFST decoders

Hi. I am trying to reproduce this tutorial (Tutorial page) and deploy a Conformer-CTC acoustic model with WFST decoders.
I downloaded the acoustic model from here RIVA Conformer ASR English - ASR set 6.0 | NVIDIA NGC
I immediately took the Conformer-CTC-L_spe128_en-US_6.0.riva model, so I started the tutorial from the build stage.
The build and deploy stages were successful. However, when executing the bash riva_start.sh i got this

Starting Riva Speech Services. This may take several minutes depending on the number of models deployed.
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Health ready check failed.
Check Riva logs with: docker logs riva-speech

I executed the command docker logs riva-speech and found the following error in the logs:

E0227 08:29:32.956880 102 model_lifecycle.cc:596] failed to load 'asr_offline_kaldi_wfst_Conformer-CTC-L_spe128_en-US_6.0.riva-ctc-decoder-gpu-streaming-offline' version 1: Invalid argument: riva/cbe/asr/ctc-decoder/ctc-decoder.cc:73] Error occured during factory initialization: Kaldi decoder does not support subword decoding

Does anyone have any ideas what this could be about and what I should do in this case?

I am experiencing the same issues, can anyone shed some light on this? @rvinobha
I have also opened an issue on Unable to deploy on RIVA · Issue #39 · nvidia-riva/riva-asrlib-decoder · GitHub.

The problem is, that the GitHub library and Riva pipeline instructions (Support for Class Based n-gram Language Models in Riva (WFST Decoder) — NVIDIA Riva) provide information on how to deploy or work only with subword (BPE) models. There are no scripts for a character based model. Once a model is deployed Riva/Triton crashes, with

Kaldi decoder does not support subword decoding

I found a way to avoid the mistake in my case. After executing the command riva_init.sh, configuration files with the extension .pbtxt appear. These files contain the use_subword parameter, which can be switched to False. After that, the error disappears and the RIVA pipeline is successfully started. However, another problem appears here - very low confidence. Does anyone know what could be the reason?

@avenkatesan @rvinobha can you help here?

@serhii-artemuk thank you for this. I can confirm it works in my case as well. As for the confidence levels I would assume they are low as they are utterance based ASR Advanced Details — NVIDIA Riva.

In riva logs I’ve noticed the following

I0329 16:46:20.022289  4122 lattice_post_processor.cc:267] Problem obtaining timestamps

Do you see that as well?

@ilb yes, i have the same message in logs.