GPU memory leak in nemo asr_model.transcribe()

AI & Data Science Deep Learning (Training & Inference) cuDNN
1

Hello, I’m getting a memory leak issue when trying to use the ASR model inference using asr_model.transcribe()

You can reproduce this issue using this notebook: Google Colab

Screenshot from 2023-01-08 23-10-52
Screenshot from 2023-01-08 23-10-521441×813 128 KB

As you can see in the image the GPU memory is increasing over time.

You can take any random WAV audio file and make sure you run it for 40 min to 1 hour. The Memory error should reproduce like mine.

2

Hi @mehadi.hasan ,
This doesnt look like a cudnn issue.
Maybe CUDA or Nemo Forums could be the right platform.

Thanks