Riva ASR + Llamaspeak (via NanoLLM) on custom docker network

Goal: Get NanoLLM to work with Riva Server on a separate (non-Host) network called riva_network.

Situation: It seems that Riva Server will not talk with NanoLLM on any other network but “host”.

Steps to reproduce

  1. Ready three containers: IsaacROS + NanoLLM (w/ ROS Humble) + Riva Server
  2. In IsaacROS container, include --network ros_network as part of your docker run
  3. In NanoLLM container, do the following: include --network ros_network so that container starts connected to the ros_network. Once container starts, in different terminal connect container to an additional network via the docker network connect riva_network <nanoLLM container> command (apparently, based on this link, docker does not allows connection to two networks at startup.)
  4. Start Riva container by modifying docker run command in riva_start.sh with --network riva_network tag
  5. Back in NanoLLM container, run Llamaspeak: python3 -m nano_llm.agents.web_chat --api=mlc --model meta-llama/Meta-Llama-3-8B-Instruct --asr=riva --tts=piper

Expected Outcome: In NanoLLM, Llamaspeak should start and connect to the Riva server via the Riva_network.

Actual Outcome: In the NanoLLM container, I get the following errors:

23:19:17 | INFO | mounting webserver path /tmp/uploads to /uploads
23:19:17 | INFO | starting webserver @ https://0.0.0.0:8050
23:19:17 | SUCCESS | WebChat - system ready
Exception in thread Thread-5:
Traceback (most recent call last):
File “/usr/lib/python3.8/threading.py”, line 932, in _bootstrap_inner

  • Serving Flask app ‘nano_llm.web.server’
    • Debug mode: on
      self.run()
      File “/opt/NanoLLM/nano_llm/plugins/audio/riva_asr.py”, line 81, in run
      self.generate(self.audio_queue)
      File “/opt/NanoLLM/nano_llm/plugins/audio/riva_asr.py”, line 97, in generate
      for response in responses:
      File “/usr/local/lib/python3.8/dist-packages/riva/client/asr.py”, line 338, in streaming_response_generator
      for response in self.stub.StreamingRecognize(generator, metadata=self.auth.get_auth_metadata()):
      File “/usr/local/lib/python3.8/dist-packages/grpc/_channel.py”, line 543, in next
      return self._next()
      File “/usr/local/lib/python3.8/dist-packages/grpc/_channel.py”, line 969, in _next
      raise self
      grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
      status = StatusCode.UNAVAILABLE
      details = “failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: Connection refused”
      debug_error_string = “UNKNOWN:Error received from peer {created_time:“2024-05-18T23:19:17.726143924+00:00”, grpc_status:14, grpc_message:“failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:50051: Failed to connect to remote host: Connection refused”}”

@augustmille the different networks is beyond my personal experience but you could try the Riva client python samples from the NanoLLM container first until you get the connection to Riva server working:

There is also the --riva-server argument in NanoLLM that changes the IP/port of the Riva server if needed.

Thanks for the quick feedback.
I can’t seem to find a guide or tutorial around --riva-server argument. Short of scouring around the Python scripts to figure out how this works, can you point me to any material that shows me how to use this?

Quick updates:
–riva-server works as an argument in nano_llm.
You’d just need to be incredible careful about what your “Riva server URI” is, which is not straightforward when things are being run in docker containers.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.