Error Code 2: OutOfMemory (no further information)

AI & Data Science Deep Learning (Training & Inference) Riva
,
1

Please provide the following information when requesting support.

Hardware - GPU (A100/A30/T4/V100)
Hardware - CPU
Operating System - ubuntu 22.04.1 LTS
Riva Version - 2.5.0
TLT Version (if relevant)
How to reproduce the issue ? (This is for errors. Please share the command and the detailed log here)

I am attaching my output_riva_init.log file
output_riva_init.log (6.8 MB)

My error is about out of memory
but I still have a lot of memory left

Changes made in config.sh

service_enabled_asr=true
service_enabled_nlp=false
service_enabled_tts=false

  models_asr+=(
  ### Streaming w/ CPU decoder, best latency configuration
      "${riva_ngc_org}/${riva_ngc_team}/rmir_asr_${asr_acoustic_model}_${modified_lang_code}_str${decoder}:${riva_ngc_model_version}"

  ### Streaming w/ CPU decoder, best throughput configuration
  #    "${riva_ngc_org}/${riva_ngc_team}/rmir_asr_${asr_acoustic_model}_${modified_lang_code}_str_thr${decoder}:${riva_ngc_model_version}"

  ### Offline w/ CPU decoder,
      # "${riva_ngc_org}/${riva_ngc_team}/rmir_asr_${asr_acoustic_model}_${modified_lang_code}_ofl${decoder}:${riva_ngc_model_version}"
  )

What is the main cause of this error and What to do know ?

Note : Please see my first 2 comments to know about my cpu and gpu related information
Since new user can add one link per post I am unable to add them here
Thanks

2

My CPU realted information
cpuinfo.txt (13.8 KB)

3

My GPU realted information
gpuinfo.txt (1.0 KB)

4

Hi @sanath303

Thanks for your interest in Riva

Thanks for sharing the logs, cpu and gpu information

i guess the GPU present at your end is GeForce MX230
Apologies, I guess currently it is not supported
Please find the support matrix
https://docs.nvidia.com/deeplearning/riva/user-guide/docs/support-matrix.html#support-matrix

Thanks

5

Ohh yeah ,I want to practice ASR . Any solutions on what do to next ? I mean will you suggest any cloud servers which has Riva supported GPUs ?

6

@rvinobha any update regarding my query ?
I am very much enthuiastic to get hands on with riva so please spare few mintues
Thanks

7

Hi @sanath303

We can use any cloud service (AWS, Azure, GCP etc.) which has the following configuration as per support matrix

https://docs.nvidia.com/deeplearning/riva/user-guide/docs/support-matrix.html#server-hardware

Thanks

8

Hey @rvinobha from your suggestion I started using azure nvidia-gpu-optimized-vmi-a10 vm

But there are no nvidia drivers in that VM and I am unable to install them also

Here are some commands and there outputs which can help in debugging the problem

Command : nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Command : sudo apt install nvidia-driver-495

Reading package lists... Done

Building dependency tree

Reading state information... Done

Package nvidia-driver-495 is not available, but is referred to by another package.

This may mean that the package is missing, has been obsoleted, or

is only available from another source

**E:** Package 'nvidia-driver-495' has no installation candidate

Command : sudo apt install nvidia-495

Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package nvidia-495

and for lspci command I am not getting any output

Some of the commands I tried
sudo add-apt-repository ppa:graphics-drivers/pp
sudo apt update

output for uname -a
Linux sanath 5.15.0-1020-azure #25~20.04.1-Ubuntu SMP Thu Sep 1 19:20:56 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

9

any updates @rvinobha ?

10

Hi @sanath303

My Apologies for the delay

Since the issue is related to driver installation,

We request you to post the query in the below forum

Thanks