Riva_start.sh fails on Jetson Orin

yanchaoliu · March 12, 2023, 6:23am

I bought a Jetson AGX Orin Development Kit recently and wanted to run some speech AI on it.

I followed the official Quick Start Guide [cannot put the link due to being a new user].

All steps went OK, until the “bas riva_start.sh” step. I got the following output on screen after issuing the command.

~/riva_quickstart_arm64_v2.9.0$ bash riva_start.sh
Starting Riva Speech Services. This may take several minutes depending on the number of models deployed.
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Health ready check failed.
Check Riva logs with: docker logs riva-speech

The output of the “docker logs riva-speech” command is attached here (logs.txt).
logs.txt (82.2 KB)

Hope to get help here. Many thanks!

rvinobha · March 21, 2023, 6:57am

Hi @yanchaoliu

Thanks for your interest in Riva

I will check regarding this error with the Riva Team, in meantime one quick suggestion

Can you try the setup again with a different port (i.e default port 50051), from logs i find you are using 8001 port,

Kindly also share the config.sh used

Thanks

rvinobha · March 21, 2023, 7:51am

HI @yanchaoliu

I have updates from Riva Team

The Team analyzed the logs, The errors are due to unrecognized symbols from the TRT library, which happens due to TRT version mismatch b/w container and host, e.g.

unable to load shared library: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8: undefined symbol: _ZN5nvdla8IProfile17setUseSoftMaxOptzEb

They told me to verify

which Jetpack version has been installed on the Jetson Orin ?
whether all Jetpack components installed successfully?

we strongly beleive it may be jetson related

For Riva 2.9.0 (which is your version I believe), Jetpack 5.0.2 should be used.

If you are using latest Jetpack 5.1, Please switch to Riva 2.10.0 release.

Please check and provide us the feedback

Thanks

ritthyk5050 · May 11, 2023, 7:09am

I’m using nvidia agx orin
bash ./riva_init.sh
Logging into NGC docker registry if necessary…
Pulling required docker images if necessary…
Note: This may take some time, depending on the speed of your Internet connection.

Pulling Riva Speech Server images.
Pulling nvcr.io/nvidia/riva/riva-speech:2.11.0. This may take some time…

Downloading models (RMIRs) from NGC…
Note: this may take some time, depending on the speed of your Internet connection.
To skip this process and use existing RMIRs set the location and corresponding flag in config.sh.
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as ‘csv’
invoking the NVIDIA Container Runtime Hook directly (e.g. specifying the docker --gpus flag) is not supported. Please use the NVIDIA Container Runtime instead.: unknown.
Error in downloading models.

bash ./riva_start.sh

Starting Riva Speech Services. This may take several minutes depending on the number of models deployed.
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Health ready check failed.
Check Riva logs with: docker logs riva-speech

config.sh file

GPU family of target platform. Supported values: tegra, non-tegra

riva_target_gpu_family=“non-tegra”

Name of tegra platform that is being used. Supported tegra platforms: orin, xavier

riva_tegra_platform=“orin”

Enable or Disable Riva Services

service_enabled_asr=true
service_enabled_nlp=true
service_enabled_tts=true
service_enabled_nmt=true

language_code=(“en-US”)

asr_acoustic_model=(“conformer”)
gpus_to_use=“device=0”
MODEL_DEPLOY_KEY=“tlt_encode”

riva_model_loc=“riva-model-repo”

if [[ $riva_target_gpu_family == “tegra” ]]; then
riva_model_loc=“pwd/model_repository”
fi

use_existing_rmirs=false

Ports to expose for Riva services

riva_speech_api_port=“50051”

NGC orgs

riva_ngc_org=“nvidia”
riva_ngc_team=“riva”
riva_ngc_image_version=“2.11.0”
riva_ngc_model_version=“2.11.0”

help me sort out my problem
Also it’s taking a lot more time to pull models while executing riva_init.sh file

rvinobha · May 29, 2023, 5:16am

Hi @ritthyk5050

Thanks for your interest in Riva

Apologies for the delay

Can you kindly share the Jetpack version used
Riva 2.11 required JetPack 5.1 or JetPack 5.1.1
kindly set the default runtime to nvidia on the Jetson platform if no already done

You have set the default runtime to nvidia on the Jetson platform by adding the following line in the /etc/docker/daemon.json file. Restart the Docker service using sudo systemctl restart docker after editing the file.

"default-runtime": "nvidia"

Please check the above and let us know if it helps

Thanks

hpetty_c2i · June 19, 2023, 2:54pm

Jetson Model: AGX Orin
Jetpack Version: 5.1 (L4T 35.2.1)
Riva Version: 2.11.0

I am having the same problem as OP. Nearly identical config. I have also done the "default-runtime": "nvidia" sugeestion. Running bash riva_start.sh produces same result. Runs until finally quitting and saying:

...
Waiting for Riva server to load all models…retrying in 10 seconds
Waiting for Riva server to load all models…retrying in 10 seconds
Health ready check failed.
Check Riva logs with: docker logs riva-speech

I have included my config.sh and the output of docker logs riva-speech in the log.txt file.

I am only trying to run the nlp models to test a simple sample script but can not get the Riva Service to work. Is Riva 2.10.0 more stable?

config.sh (13.3 KB)
log.txt (13.0 KB)

rvinobha · June 25, 2023, 2:57am

Hi @hpetty_c2i

Thanks for your interest in Riva

Can you run the following command and share with us the output

docker info | grep -i runtime

Also can you check if you are able to run any basic container with nvidia-runtime
2. docker run -it --rm --runtime nvidia ubuntu:20.04

Thanks

hpetty_c2i · June 26, 2023, 12:15pm

Runtimes: io.containerd.runtime.v1.linux nvidia runc io.containerd.runc.v2
Default Runtime: nvidia

And yes I am able to run the container