- Hardware - GPU A100
- Hardware - CPU Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz
- Operating System - Ubuntu 20.04.5 LTS
- Riva Version - 2.10.0
- NVidia driver version: 525.60.13
- CUDA version: 12.0
- Docker version: 23.0.1
Steps to reproduce:
bash riva_init.sh bash riva_start.sh
And start producing any ASR streaming requests from client (nvidia-riva-client==2.10.0)
Riva server can’t process any ASR requests and throwing a lot of errors like the following:
cudaError_t 700 : "an illegal memory access was encountered" returned from 'cudaMemset2DAsync( data_.get(), stride_ * sizeof(Real), 0, num_cols_ * sizeof(Real), num_rows_, cudaStreamPerThread)' in fileriva/utils/matrix/cu_matrix.cc line 122'
I also tried setting gpus_to_use=“all”, but nothing changed.
I want to note that this problem occurs with Riva versions starting from 2.8.0, while 2.7.0 works without any issues with the same configuration.
Hope these files could help during the investigation.
config.sh (12.7 KB)
nvidia-smi.txt (2.3 KB)
riva_init.log (150.2 KB)
riva_speech.log (2.0 MB)
riva_start.log (265 Bytes)
Any help on this would be much appreciated.
Thanks in advance!