Hardware - GPU: Tesla P40
Hardware - CPU: Intel(R) Xeon(R) Gold 5118 @ 2.30GHz
Operating System: Linux 4.14.49 x86_64 GNU/Linux
Driver Version: 460.73.01
CUDA Version: 11.2
Riva Version: v2.9.0
When run the command bash riva_start.sh, riva-speech container exit abnormlly. and run the command docker logs xxxx(riva-speech’s containerId), following error message printed:
cudaError-t 209 : “no kernel image is available for execution on the device” returned from ‘cudaGetLastError()’ in fileexternal/cu-feat-extr/src/cudafeat/feature-online-batched-spectral-cuda.cc line 122
cudaError-t 209 : “no kernel image is available for execution on the device” returned from ‘cudaGetLastError()’ in fileexternal/cu-feat-extr/src/cudafeat/feature-online-batched-spectral-cuda.cc line 149
cudaError-t 209 : “no kernel image is available for execution on the device” returned from ‘cudaGetLastError()’ in fileexternal/cu-feat-extr/src/cudafeat/feature-online-batched-spectral-cuda.cc line 199
cudaError-t 209 : “no kernel image is available for execution on the device” returned from ‘cudaGetLastError()’ in fileexternal/cu-feat-extr/src/cudafeat/feature-online-batched-spectral-cuda.cc line 257
[logging.cc:43] 6: The engine plan file is generated on an incompatible device, expecting compute 6.1 got compute 7.5, please rebuild.
[logging.cc:43] 4: [runtime.cpp::deserializeCudaEngine::66] Error Code 4: Internal Error (Engine deserialization failed.)
[tensorrt.cc:5665] TRITONBAKEND_ModelInstanceFinalize: delete instance state
[tensorrt.cc:5604] TRITONBAKEND_ModelFinalize: delete model state
[tensorrt.cc:5627] TRITONBAKEND_ModelInstanceInitialize: riva-trt-confomer-e-S-a-offLne-an-streanino-off1ine A (GPU device 0)
failed to load ‘riva-trt-conformer-en-US-asr-streamino-am-streaming’ version 1: Internal: unable to create TensorRT engine
And following 3 models failed to load: riva-trt-conformer-en-US-asr-offline-am-streaming-offline, riva-trt-contormer-en-US-asr-streaming-am-streaming-am-streaming and riva-trt-riva-punctuation-en-US-nn-bert-base-uncased.
Could you please help to check the reason?