Tritonserver - Segmentation fault when running diarization on long audio (1h)

AI & Data Science Deep Learning (Training & Inference) Riva
1

Please provide the following information when requesting support.

Hardware - NVIDIA GeForce RTX 2080
Operating System - WSL with Ubuntu 20.04
Riva Version - 2.10

How to reproduce the issue ? (This is for errors. Please share the command and the detailed log here)

I tried to run asr + diarization on long audio (1 hour long) and the triton server crashed.
I used a finetuned model based on Conformer and I also tried with Citrinet_1024.
The asr task works fine with both model on the long audio. The diarization succeed on short audio.

Here are the logs + config file :
config.sh (12.7 KB)
logs.txt (117.6 KB)

2

Hi @mel.adl

Thanks for your interest in Riva

Thanks for sharing the logs,

The Segmentation fault seems related to GPU and its Usage

Care must be taken to not exceed the memory available when selecting models to deploy. 16+ GB VRAM is recommended.

Please refer
https://docs.nvidia.com/deeplearning/riva/user-guide/docs/support-matrix.html#data-center

Thanks

3

Hi @rvinobha and thank you for your help.

In the meanwhile, I’ve switched to a RTX 3080 and I still encounter the same trouble.