Bash riva_start.sh never complete on Jetson AGX Orin

AI & Data Science Deep Learning (Training & Inference) Riva
1

Please provide the following information when requesting support.

Hardware - Jetson AGX Orin
Hardware - CPU
Operating System: JetPack 5.0.2
Riva Version: riva_quickstart_arm64_v2.8.1
TLT Version (if relevant)
How to reproduce the issue ? (This is for errors. Please share the command and the detailed log here)

Today I have flashed the latest JetPack 5.0.2 into the Jetson AGX Orin unit. Then I have downloaded the riva_quickstart_arm64_v2.8.1

When I run the bash riva_init.sh, it finishes without any error except the bash riva_start.sh.

Can you please give me the suggestion how to solve the problem?

Thanks
Jimmy

Starting Riva Speech Services. This may take several minutes depending on the number of models deployed.
53b96b57a1999e4006324210317be0156a8a3bb9cb64bb476dc1e620468be53c
docker: Error response from daemon: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as ‘csv’
invoking the NVIDIA Container Runtime Hook directly (e.g. specifying the docker --gpus flag) is not supported. Please use the NVIDIA Container Runtime instead.: unknown.
Error response from daemon: Container 53b96b57a1999e4006324210317be0156a8a3bb9cb64bb476dc1e620468be53c is not running
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds

2

Hi @jimmyq

Thanks for your interest in Riva,

Please share your

  1. config.sh
  2. complete log output of bash riva_init.sh and riva_start.sh

Thanks for your jist of log, i also recommend to check the NGC setup
the ngc files would be present in $HOME/.ngc
open the config file (PLEASE DO NOT EDIT THIS FILE), (PLEASE DO NOT SHARE THIS FILE AS IT CONTAINS SENSITIVE INFORMATION)
and check for format_type, if it is set as csv and not json,
Run ngc config set and change the type as json (prompted on step 3 after API key and ORG), and retry the setup again, and let us know if it works

Thanks

3

init.log (18.3 KB)
start.log (2.8 KB)

Hi @rvinobha,

Thanks for you suggestion. After I have try your one, in the .ngc/config, I can see:
[CURRENT]
format_type = json

But I still have the same problem: run riva_start.sh has the error as seen in the attached the start.log

Thanks so much for help
Jimmy

4

Or it may be the docker version related problem?

5

Hi @jimmyq

Can you kindly verify Step 6 from the below document has been performed

You have set the default runtime to nvidia on the Jetson platform by adding the following line in the /etc/docker/daemon.json file. Restart the Docker service using sudo systemctl restart docker after editing the file.
"default-runtime": "nvidia"
and perhaps try again

6

Hi @rvinobha,

Here is my /etc/docker/daemon.json file.

{
“runtimes”: {
“nvidia”: {
“path”: “nvidia-container-runtime”,
“runtimeArgs”: ,
“default-runtime”: “nvidia”
}
}
}

After I have done: sudo systemctl restart docker, I have done:

bash riva_clean.sh
bash riva_init.sh
bash riva_start.sh

While first two commands has no problem, the riva_start.sh still face the same problem.

I am thinking to re-flash the JetPack5.0.2 image into my Jetson AGX Orin and then to try the Riva again. I guess that my docker may need to be downgraded?

If you have new suggestions, please let me know since I need to make the RIVA working on the Jetson Orin.

Thanks
Jimmy

7

Hi @jimmyq

What is the current version of your docker

Thanks

8

Hi Jimmy

Are you sure your daemon.json is correct? Pleas take a look onto Upgrading to the NVIDIA Container Runtime for Docker :: DGX Systems Documentation Chapter 4

{
“default-runtime”: “nvidia”,
“runtimes”: {
“nvidia”: {
“path”: “/usr/bin/nvidia-container-runtime”,
“runtimeArgs”:
}
},

}

9

Just got the Jetson Orin nano dev kit.
I am having the exact same issue. Was this resolved and how?

10

Hi @adventuredaisy and @er-er-ha

Thanks for your interest in Riva

The Problems could be because of

  1. Version of Jetpack version
  2. whether all Jetpack components installed successfully?

For Riva 2.9.0, Jetpack 5.0.2 should be used.
But if you are using latest version Riva 2.10.0 release Jetpack 5.1, should be used.
https://docs.nvidia.com/deeplearning/riva/user-guide/docs/release-notes.html#key-features-and-enhancements

Can you check your Jetpack version and provide us feedback

Thanks