Hardware - GPU T4 (AWS g4dn)
Hardware - CPU
Operating System ubuntu 20.04
Riva Version 2.7.1 - Conformer-en-US-asr-streaming - Latency configs - Normal quickstart
Every few days my Riva instance just stops sending back results, I can’t understand the cause, It may be CPU related as my CPU usage gets stuck at 100% but my GPU is at 50% usage. The problem also is that using the health checker returns “serving” even though the docker isn’t transcribing anything which is a problem as AWS can’t remove the instance as it’s still seen as healthy.
I usually only have to stop the docker and start again and then boom everything works for a few more days…
If any advice could be given on how to avoid this error and also would it be possible to “fix” the grpc health proto to pick up the error
Here are the docker logs when it stops working:
I0327 00:51:45.587536 1342393 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 00:51:45.587587 1342393 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 00:52:45.052994 1341894 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.055202 1342119 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.056403 1342289 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.057823 1342291 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.057969 1342292 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.059031 1342293 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.059175 1342295 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.059224 1342298 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.099135 1342313 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.099450 1342321 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.099563 1342327 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.099692 1342329 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.099833 1342333 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.099951 1342335 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.100075 1342305 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.099751 1342331 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.100109 1342337 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.099784 1342317 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.100878 1342307 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.100929 1342303 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.101022 1342309 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.101209 1342319 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.101382 1342311 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.101414 1342323 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.099149 1342315 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.100443 1342325 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.107815 1342339 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.108002 1342341 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.108042 1342345 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.108378 1342355 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.108394 1342357 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.108604 1342353 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.108107 1342349 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.108845 1342359 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.108943 1342361 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.109005 1342343 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.109165 1342347 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.109339 1342351 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.120488 1342367 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.120611 1342363 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.120692 1342365 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.120889 1342393 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.121027 1342371 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.121049 1342369 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.121284 1342375 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.121333 1342373 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.121440 1342377 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.121495 1342379 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.121883 1342381 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.122318 1342383 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.122480 1342385 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.122555 1342391 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.122610 1342387 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 00:52:45.122649 1342389 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 03:45:05.597268 1342395 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 03:45:05.597628 1341894 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 03:45:05.615300 1342395 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 430485317
I0327 03:45:05.615483 1342395 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 03:45:05.615711 1342395 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 03:45:05.615643 1341894 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 554202918
I0327 03:45:05.615911 1341894 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 03:45:05.616072 1341894 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 03:45:05.838603 1341894 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
I0327 03:45:19.118520 1356820 grpc_riva_asr.cc:709] Send silence buffer for EOS
I0327 03:46:00.158834 1356817 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 03:46:00.158939 1356817 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 1163263917
I0327 03:46:00.158993 1356817 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 03:46:00.159080 1356817 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 03:46:01.889643 1356899 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 03:46:01.889981 1356899 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 2122691523
I0327 03:46:01.890024 1356899 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 03:46:01.890089 1356899 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 03:46:01.899533 1356901 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 03:46:01.899574 1356901 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 1007182468
I0327 03:46:01.899605 1356901 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 03:46:01.899668 1356901 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 03:46:02.035912 1356901 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
I0327 03:46:05.597250 1342395 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 03:46:07.256565 1356905 grpc_riva_asr.cc:709] Send silence buffer for EOS
I0327 03:46:14.696384 1356903 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 03:46:14.705098 1356903 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 863164831
I0327 03:46:14.705286 1356903 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 03:46:14.705459 1356903 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 03:46:14.716058 1356903 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
I0327 03:46:14.717353 1342395 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 03:46:14.731281 1342395 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 1882651082
I0327 03:46:14.731338 1342395 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 03:46:14.731420 1342395 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 03:46:14.748783 1342395 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
I0327 03:46:17.636723 1356926 grpc_riva_asr.cc:709] Send silence buffer for EOS
I0327 03:46:19.486764 1356935 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 03:46:19.492204 1356935 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 1105644434
I0327 03:46:19.492372 1356935 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 03:46:19.492548 1356935 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 03:46:19.494937 1356935 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
W0327 03:46:59.118960 1356820 grpc_riva_asr.cc:803] Response timeout. requests sent: 84 received: 0
E0327 03:46:59.119297 1341894 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
I0327 03:47:00.159745 1356817 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 03:47:01.888411 1356899 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 03:47:14.694996 1356922 grpc_riva_asr.cc:709] Send silence buffer for EOS
W0327 03:47:47.256768 1356905 grpc_riva_asr.cc:803] Response timeout. requests sent: 33 received: 0
E0327 03:47:47.257000 1356901 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
W0327 03:47:57.636931 1356926 grpc_riva_asr.cc:803] Response timeout. requests sent: 19 received: 0
E0327 03:47:57.637065 1342395 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
W0327 03:48:01.068158 1356937 grpc_riva_asr.cc:803] Response timeout. requests sent: 84 received: 0
E0327 03:48:01.068280 1356935 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
W0327 03:48:54.695279 1356922 grpc_riva_asr.cc:803] Response timeout. requests sent: 1 received: 0
E0327 03:48:54.695591 1356903 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
I0327 04:25:31.699447 1356817 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 04:25:31.718576 1356923 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 04:25:31.731549 1360201 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 04:25:31.731703 1360201 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 368960121
I0327 04:25:31.731842 1360201 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 04:25:31.731985 1360201 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 04:25:31.739498 1360202 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 04:25:31.742751 1356923 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 695276432
I0327 04:25:31.742949 1356923 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 04:25:31.743161 1356923 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 04:25:31.755501 1360204 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 04:25:31.755657 1360204 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 1200129690
I0327 04:25:31.755776 1360204 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 04:25:31.755909 1360204 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 04:25:31.762933 1360206 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 04:25:31.762969 1360206 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 1529920193
I0327 04:25:31.762998 1360206 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 04:25:31.763057 1360206 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 04:25:31.763132 1360206 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
I0327 04:25:31.775501 1360208 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 04:25:31.775651 1360208 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 19707768
I0327 04:25:31.775758 1360208 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 04:25:31.775871 1360208 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 04:25:31.787504 1360211 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 04:25:31.787639 1360211 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 572855759
I0327 04:25:31.787707 1360211 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 04:25:31.787822 1360211 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 04:25:31.799511 1360214 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 04:25:31.799734 1360214 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 04:25:31.799857 1360214 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 535294208
I0327 04:25:31.799928 1360214 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 04:25:31.800051 1360214 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 04:26:31.717860 1360201 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 04:26:31.717919 1356923 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 04:26:31.720963 1360204 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 04:26:31.721200 1360208 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 04:26:31.721290 1360211 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 04:26:31.727156 1360214 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
W0327 04:28:11.721333 1360210 grpc_riva_asr.cc:803] Response timeout. requests sent: 1 received: 0
E0327 04:28:11.721658 1360206 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
I0327 19:36:54.419623 1360281 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 19:36:54.427616 1360281 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 1814640997
I0327 19:36:54.427814 1360281 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 19:36:54.428050 1360281 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 19:36:54.428670 1360281 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
I0327 19:36:54.428548 1360216 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 19:36:54.481520 1360216 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 1512618971
I0327 19:36:54.481818 1360216 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 19:36:54.481999 1360216 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 19:36:54.482209 1360216 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
I0327 19:37:02.516621 1436034 grpc_riva_asr.cc:709] Send silence buffer for EOS
I0327 19:37:21.583901 1436029 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 19:37:21.592458 1436029 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 73855759
I0327 19:37:21.592594 1436029 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 19:37:21.592768 1436029 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 19:37:21.592962 1436029 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
I0327 19:37:21.599514 1436071 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 19:37:21.599561 1436071 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 1391675200
I0327 19:37:21.599597 1436071 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 19:37:21.599658 1436071 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 19:37:21.642338 1436071 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
I0327 19:37:35.235005 1436077 grpc_riva_asr.cc:709] Send silence buffer for EOS
I0327 19:37:54.422303 1436031 grpc_riva_asr.cc:709] Send silence buffer for EOS
I0327 19:37:57.155050 1436074 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 19:37:57.163718 1436074 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 419870784
I0327 19:37:57.163993 1436074 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 19:37:57.164162 1436074 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 19:37:57.164466 1436074 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
I0327 19:37:57.164701 1436128 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 19:37:57.164780 1436128 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 2003442896
I0327 19:37:57.164881 1436128 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 19:37:57.165014 1436128 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 19:37:57.221837 1436128 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
I0327 19:38:02.631515 1436136 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 19:38:02.631788 1436136 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 955005476
I0327 19:38:02.632017 1436136 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 19:38:02.632179 1436136 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 19:38:02.636039 1436136 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
I0327 19:38:07.646932 1436134 grpc_riva_asr.cc:709] Send silence buffer for EOS
I0327 19:38:21.584863 1436073 grpc_riva_asr.cc:709] Send silence buffer for EOS
I0327 19:38:36.107769 1436189 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 19:38:36.107828 1436189 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 721428693
I0327 19:38:36.107863 1436189 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 19:38:36.107923 1436189 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 19:38:36.108043 1436189 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
W0327 19:38:42.517004 1436034 grpc_riva_asr.cc:803] Response timeout. requests sent: 51 received: 0
E0327 19:38:42.517136 1360216 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
I0327 19:38:57.154855 1436130 grpc_riva_asr.cc:709] Send silence buffer for EOS
I0327 19:39:04.817842 1436131 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 19:39:04.825748 1436131 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 789804963
I0327 19:39:04.825819 1436131 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 19:39:04.825881 1436131 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 19:39:04.825973 1436131 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
I0327 19:39:08.272218 1436235 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 19:39:08.272518 1436235 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 610357204
I0327 19:39:08.272668 1436235 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 19:39:08.272825 1436235 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 19:39:08.273026 1436235 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
W0327 19:39:15.235232 1436077 grpc_riva_asr.cc:803] Response timeout. requests sent: 86 received: 0
E0327 19:39:15.235564 1436071 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
W0327 19:39:34.422544 1436031 grpc_riva_asr.cc:803] Response timeout. requests sent: 5 received: 0
E0327 19:39:34.422847 1360281 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
I0327 19:39:38.006788 1436246 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 19:39:38.007083 1436246 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 503011856
I0327 19:39:38.007230 1436246 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 19:39:38.007390 1436246 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 19:39:38.007762 1436246 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
W0327 19:39:42.960772 1436145 grpc_riva_asr.cc:803] Response timeout. requests sent: 51 received: 0
E0327 19:39:42.960904 1436136 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
W0327 19:39:47.647145 1436134 grpc_riva_asr.cc:803] Response timeout. requests sent: 66 received: 0
E0327 19:39:47.647291 1436128 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
W0327 19:40:01.585296 1436073 grpc_riva_asr.cc:803] Response timeout. requests sent: 1 received: 0
E0327 19:40:01.585641 1436029 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
I0327 19:40:09.198720 1436292 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 19:40:09.199013 1436292 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 1370517358
I0327 19:40:09.199160 1436292 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 19:40:09.199317 1436292 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 19:40:09.207211 1436292 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
I0327 19:40:10.047799 1436338 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 19:40:10.048034 1436338 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 1640365538
I0327 19:40:10.048192 1436338 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 19:40:10.048343 1436338 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 19:40:10.048547 1436338 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
W0327 19:40:16.575634 1436198 grpc_riva_asr.cc:803] Response timeout. requests sent: 86 received: 0
E0327 19:40:16.575987 1436189 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
I0327 19:40:20.548349 1436342 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 19:40:20.556110 1436342 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 607823297
I0327 19:40:20.556285 1436342 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 19:40:20.556452 1436342 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 19:40:20.556679 1436342 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
I0327 19:40:20.559244 1436360 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 19:40:20.614410 1436360 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 1514567389
I0327 19:40:20.614647 1436360 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 19:40:20.614805 1436360 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 19:40:20.615031 1436360 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
I0327 19:40:25.611673 1436366 grpc_riva_asr.cc:709] Send silence buffer for EOS
W0327 19:40:37.155104 1436130 grpc_riva_asr.cc:803] Response timeout. requests sent: 2 received: 0
E0327 19:40:37.155460 1436074 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
I0327 19:40:40.849066 1436363 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 19:40:40.849366 1436363 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 375716318
I0327 19:40:40.849512 1436363 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 19:40:40.849666 1436363 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 19:40:40.849866 1436363 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
W0327 19:40:45.188427 1436237 grpc_riva_asr.cc:803] Response timeout. requests sent: 51 received: 0
E0327 19:40:45.188568 1436131 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
W0327 19:40:48.618125 1436248 grpc_riva_asr.cc:803] Response timeout. requests sent: 66 received: 0
E0327 19:40:48.618269 1436235 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
I0327 19:41:03.236411 1436362 grpc_riva_asr.cc:709] Send silence buffer for EOS
W0327 19:41:18.463075 1436294 grpc_riva_asr.cc:803] Response timeout. requests sent: 86 received: 0
E0327 19:41:18.463207 1436246 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
W0327 19:41:49.485517 1436340 grpc_riva_asr.cc:803] Response timeout. requests sent: 51 received: 0
E0327 19:41:49.485882 1436292 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
W0327 19:41:50.399289 1436344 grpc_riva_asr.cc:803] Response timeout. requests sent: 66 received: 0
E0327 19:41:50.399619 1436338 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
W0327 19:42:05.612094 1436366 grpc_riva_asr.cc:803] Response timeout. requests sent: 32 received: 0
E0327 19:42:05.612236 1436360 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
W0327 19:42:21.350667 1436398 grpc_riva_asr.cc:803] Response timeout. requests sent: 86 received: 0
E0327 19:42:21.350795 1436363 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
W0327 19:42:43.236801 1436362 grpc_riva_asr.cc:803] Response timeout. requests sent: 5 received: 0
E0327 19:42:43.237140 1436342 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
I0327 22:27:32.008563 1436456 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 22:27:32.017781 1436456 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 1824951222
I0327 22:27:32.018138 1436456 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 22:27:32.018559 1436396 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 22:27:32.018765 1436396 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 904017712
I0327 22:27:32.018915 1436396 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 22:27:32.018859 1436456 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 22:27:32.019385 1436396 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 22:27:32.097800 1436396 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
I0327 22:27:50.597599 1450285 grpc_riva_asr.cc:709] Send silence buffer for EOS
I0327 22:28:32.007946 1436456 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
W0327 22:29:30.597797 1450285 grpc_riva_asr.cc:803] Response timeout. requests sent: 117 received: 0
E0327 22:29:30.597934 1436396 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
I0327 22:32:35.683727 1450282 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 22:32:35.692714 1450329 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 22:32:35.702116 1450329 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 247270799
I0327 22:32:35.702297 1450329 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 22:32:35.702522 1450329 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 22:32:35.712162 1450282 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 676374236
I0327 22:32:35.712381 1450282 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 22:32:35.712536 1450282 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 22:32:35.932543 1450282 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 22:32:35.932724 1450329 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
I0327 22:32:52.174475 1450329 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 22:32:52.184103 1450708 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 22:32:52.184150 1450708 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 1729031448
I0327 22:32:52.184185 1450708 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 22:32:52.184249 1450708 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 22:32:52.193095 1450329 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 1593115303
I0327 22:32:52.193351 1450329 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 22:32:52.193562 1450329 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 22:32:52.403241 1450708 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
I0327 22:33:04.412986 1450735 grpc_riva_asr.cc:709] Send silence buffer for EOS
I0327 22:33:52.174809 1450329 grpc_riva_asr.cc:1136] ASRService.StreamingRecognize returning OK
W0327 22:34:44.413406 1450735 grpc_riva_asr.cc:803] Response timeout. requests sent: 76 received: 0
E0327 22:34:44.413794 1450708 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
I0327 23:22:09.258858 1450329 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 23:22:09.261608 1450732 grpc_riva_asr.cc:935] ASRService.StreamingRecognize called.
I0327 23:22:09.268189 1450732 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 178538026
I0327 23:22:09.268342 1450732 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 23:22:09.268503 1450732 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 23:22:09.282737 1450329 grpc_riva_asr.cc:962] ASRService.StreamingRecognize performing streaming recognition with sequence id: 98793444
I0327 23:22:09.282948 1450329 grpc_riva_asr.cc:1019] Using model conformer-en-US-asr-streaming for inference
I0327 23:22:09.283097 1450329 grpc_riva_asr.cc:1035] Model sample rate= 16000 for inference
I0327 23:22:09.283303 1450329 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
I0327 23:22:09.498375 1450732 riva_asr_stream.cc:214] Detected format: encoding = 1 RAW numchannels = 1 samplerate = 16000 bitspersample = 16
I0327 23:22:14.332911 1454844 grpc_riva_asr.cc:709] Send silence buffer for EOS
I0327 23:23:09.257745 1454842 grpc_riva_asr.cc:709] Send silence buffer for EOS
W0327 23:23:54.333321 1454844 grpc_riva_asr.cc:803] Response timeout. requests sent: 32 received: 0
E0327 23:23:54.333643 1450732 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure
W0327 23:24:49.258004 1454842 grpc_riva_asr.cc:803] Response timeout. requests sent: 4 received: 0
E0327 23:24:49.258141 1450329 grpc_riva_asr.cc:1132] ASRService.StreamingRecognize returning failure