Unable to load riva model build with --nn.use_trt_fp32 flag

AI & Data Science Deep Learning (Training & Inference) Riva
1

Following discussion here: Riva providing empty transcriptions for a few audios, but nemo does not for those audios - Deep Learning (Training & Inference) / Riva - NVIDIA Developer Forums

I tried to riva_build a model with fp32. But riva_start.sh will not work, citing
| riva-trt-asr-conformer_en_us-offline-flashlight-am-streaming-offline | 1 | UNAVAILABLE: Unavailable: unable to find ‘/data/models/riva-trt-asr-conformer_en_us-offline-flashlight-am-streaming-offline/1/model.plan’ for model instance 'riva-trt-asr-conformer_en_us-offline-flashlight- |

2

Hi @tim.xia

Thanks for your interest in Riva,

riva-build creates the .rmir
riva-deploy generates the model directories

After that we can deploy the model using riva_server

Was riva-deploy performed ?

Request to kindly share the

  1. Complete riva-build command used
  2. Complete riva-deploy command used
  3. config.sh used for starting the riva-sever

Thanks

3

I usually follow the route of riva_quickstart, with riva_init.sh then riva_start.sh.
It works for most models except when I add the nn.use_trt_fp32 flag. Then there is just no model.plan in the models dir. Is there other settings that are required for this –nn.use_trt_fp32 flag?

4

Hi @tim.xia

My Apologies, if i made wrong understanding,
I will reproduce the issue from my end
Request to kindly share the

  1. Complete riva-build command used
  2. Complete riva-deploy command used
  3. config.sh used for starting the riva-sever

If it is a custom trained conformer model, are we doing steps suggested in the below doc
https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/asr-python-advanced-finetune-am-conformer-ctc-tao-deployment.html?highlight=custom%20model

If not please let me know

Thanks