Riva-speech container failed to start

AI & Data Science Deep Learning (Training & Inference) Riva
1

Hardware - GPU: Tesla P40
Hardware - CPU: Intel(R) Xeon(R) Gold 5118 @ 2.30GHz
Operating System: Linux 4.14.49 x86_64 GNU/Linux
Driver Version: 460.73.01
CUDA Version: 11.2
Riva Version: v2.9.0

When run the command bash riva_start.sh, riva-speech container exit abnormlly. and run the command docker logs xxxx(riva-speech’s containerId), following error message printed:
cudaError-t 209 : “no kernel image is available for execution on the device” returned from ‘cudaGetLastError()’ in fileexternal/cu-feat-extr/src/cudafeat/feature-online-batched-spectral-cuda.cc line 122
cudaError-t 209 : “no kernel image is available for execution on the device” returned from ‘cudaGetLastError()’ in fileexternal/cu-feat-extr/src/cudafeat/feature-online-batched-spectral-cuda.cc line 149
cudaError-t 209 : “no kernel image is available for execution on the device” returned from ‘cudaGetLastError()’ in fileexternal/cu-feat-extr/src/cudafeat/feature-online-batched-spectral-cuda.cc line 199
cudaError-t 209 : “no kernel image is available for execution on the device” returned from ‘cudaGetLastError()’ in fileexternal/cu-feat-extr/src/cudafeat/feature-online-batched-spectral-cuda.cc line 257

[logging.cc:43] 6: The engine plan file is generated on an incompatible device, expecting compute 6.1 got compute 7.5, please rebuild.
[logging.cc:43] 4: [runtime.cpp::deserializeCudaEngine::66] Error Code 4: Internal Error (Engine deserialization failed.)
[tensorrt.cc:5665] TRITONBAKEND_ModelInstanceFinalize: delete instance state
[tensorrt.cc:5604] TRITONBAKEND_ModelFinalize: delete model state
[tensorrt.cc:5627] TRITONBAKEND_ModelInstanceInitialize: riva-trt-confomer-e-S-a-offLne-an-streanino-off1ine A (GPU device 0)
failed to load ‘riva-trt-conformer-en-US-asr-streamino-am-streaming’ version 1: Internal: unable to create TensorRT engine

And following 3 models failed to load: riva-trt-conformer-en-US-asr-offline-am-streaming-offline, riva-trt-contormer-en-US-asr-streaming-am-streaming-am-streaming and riva-trt-riva-punctuation-en-US-nn-bert-base-uncased.

Could you please help to check the reason?

2

Hi @174362510

Thanks for your interest in Riva

Thanks for sharing the logs,

Apologies, we guess that your card Tesla P40 is of Pascal Architecture, we need Volta or higher for Riva

Riva is supported on any NVIDIA Volta or later GPU (NVIDIA Turing and NVIDIA Ampere GPU architecture) for development purposes. Care must be taken to not exceed the memory available when selecting models to deploy. 16+ GB VRAM is recommended.

Reference
https://docs.nvidia.com/deeplearning/riva/user-guide/docs/support-matrix.html#server-hardware

Thanks

1 Like
3

okay, got it, thank you very much for your help.

4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.