Please provide the following information when requesting support.
Hardware - GPU: Nvidia Tesla T4
Hardware - CPU: Intel(R) Xeon(R) CPU @ 2.30GHz
Operating System: Ubuntu 20.04.5
Riva Version: 2.14
TLT Version (if relevant)
How to reproduce the issue ? (This is for errors. Please share the command and the detailed log here)
Hello! I followed instructions in Kubernetes — NVIDIA Riva to deploy the rmir_asr_conformer_en_us_str:2.14.0 model to Google Kubernetes Engine (GKE) in Autopilot mode.
It successfully downloaded the model. However, when starting the riva-api and with triton on the same container, triton fails to load the models.
Please find attached logs below.
logs-riva-api.txt (30.8 KB)
logs-riva-init.txt (47.9 KB)
It shows errors:
error: creating server: Internal - failed to load all models
Riva waiting for Triton server to load all models…retrying in 1 second
Riva waiting for Triton server to load all models…retrying in 1 second
Triton server died before reaching ready state. Terminating Riva startup.
Check Triton logs with: docker logs
/opt/riva/bin/start-riva: line 1: kill: (21) - No such process
Looking at the model status, two models fail to load:
±----------------------------------------------------------±--------±--------------------------------------------------------------------------------------------------------------------------------------------+
| Model | Version | Status |
±----------------------------------------------------------±--------±--------------------------------------------------------------------------------------------------------------------------------------------+
| conformer-en-US-asr-streaming-ctc-decoder-cpu-streaming | 1 | UNAVAILABLE: Invalid argument: riva/cbe/asr/ctc-decoder/ctc-decoder.cc:104] Class logits tensor vocab size doesn’t match decoder vocab size |
| conformer-en-US-asr-streaming-endpointing-streaming | 1 | UNAVAILABLE: Invalid argument: riva/cbe/backend_schema.cc:55] Input/Output ‘CLASS_LOGITS’ in dimension 1 expected size 1, got 129 |
| conformer-en-US-asr-streaming-feature-extractor-streaming | 1 | READY |
| riva-trt-conformer-en-US-asr-streaming-am-streaming | 1 | READY |
±----------------------------------------------------------±--------±--------------------------------------------------------------------------------------------------------------------------------------------+
Driver Version: 470.199.02 CUDA Version: 11.8