Issues with Riva-Speech Docker Logs and Unusual Memory Consumption Behavior

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin Nano
1

I am encountering an issue with my Riva setup, specifically related to Docker logs and memory consumption during the Riva server startup. I have been troubleshooting this problem, but I am unable to resolve it, so I would appreciate any assistance or insights.

Environment Details:

Riva Version : riva_quickstart_arm64_v2.19
Docker Version : 28.0.4.5 LTS
OS : ubuntu 22.04
Model : jetson orin nano developer kit

Issue:

  1. Docker Logs : When attempting to retrieve Docker logs for the riva-speech container, I am not getting any output, even though I have confirmed that Docker is running in the background. This behavior is similar to a past issue where the Docker service wasn’t running, but I have verified that it is operational now.
  2. Memory Consumption : Normally, when starting the Riva server, it consumes a large amount of memory (80-90%). However, this time, I am only seeing a modest increase of about 5-10%. This is unusual, and I suspect that it might indicate an underlying issue with the model or the server itself.

Troubleshooting Steps Taken:

  • Confirmed that Docker is running.
  • Verified that no issues were found with the Docker service.
  • Checked memory usage during Riva server startup.
  • Attempted to access the riva-speech Docker logs without success.

Questions:

Could the lack of memory consumption during startup indicate that the model hasn’t loaded properly into memory?
Could this behavior be related to an issue with the model itself?
What additional steps can I take to further diagnose and resolve the issue?

3

Hi,

Could you share the detailed steps you used?
Do you follow Jetson-container?

Thanks.

4
  1. Follow RIVA quick start guide from: Quick Start Guide — NVIDIA Riva
  2. Edited the docker daemon file to set default runtime to “nvidia” and restarted the docker service
  3. Installed the NGC CLI with API_key through :NVIDIA NGC CLI Install for ARM64 linux and followed all the steps

download the tool

verify download
set the tool to command path
set NGC credentials

  1. Download riva through Local Deployment Using Quick Start Scripts in: Quick Start Guide — NVIDIA Riva
  2. Edited the config.sh to enable only ASR and set gpu family to : Tegra and platform to : orin
  3. Ran riva_init through bash
  4. Riva runs in a docker container so added profile to docker group and set the user to pulse-access
  5. Installed the RIVA python client through :GitHub - nvidia-riva/python-clients
  6. Imported all the requirements and followed all the step and build the whl file
  7. Tried to Started the riva server with riva_start.sh but got this response
    Health ready check failed
    Check riva logs with docker logs riva-speech
5

Following up on the health check failure, I’ve identified some errors in the riva_start.sh script.


jetson@ubuntu:~/riva_quickstart_arm64_v2.19.0$ bash riva_start.sh

Starting Riva Speech Services. This may take several minutes depending on the number of models deployed.

Performing health check for Riva server on port 50051...

Attempt 1/30: Checking server health...

[WARN] Health probe failed on attempt 1. Exit code: 1

[DEBUG] grpc_health_probe output:

Error response from daemon: container 345ca4dafe96f805c1ddd88db1b9da9cb77fa6b900312e56e88cf7fde6f61bc3 is not running

This indicates that the Docker container isn’t running. When attempting to manually run the container, I encountered the following error:


jetson@ubuntu:~/riva_quickstart_arm64_v2.19.0$ docker ps -a

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

345ca4dafe96 [nvcr.io/nvidia/riva/riva-speech:2.19.0-l4t-aarch64](http://nvcr.io/nvidia/riva/riva-speech:2.19.0-l4t-aarch64) "start-riva --riva-u…" 11 minutes ago Created riva-speech

28684827e309 hello-world "/hello" 49 minutes ago Exited (0) 49 minutes ago elastic_sanderson

jetson@ubuntu:~/riva_quickstart_arm64_v2.19.0$ docker run 345ca4dafe96

Unable to find image '345ca4dafe96:latest' locally

docker: Error response from daemon: pull access denied for 345ca4dafe96, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.

See 'docker run --help'.

jetson@ubuntu:~/riva_quickstart_arm64_v2.19.0$

It seems like there might be an issue with pulling the Docker image or accessing the repository. Could you please provide any insights or suggestions on how to address this?