Nemo-RT — sub-second ES/EN Voice Agent (VAD + STT + LLM + TTS) on one GPU, drop-in for the OpenAI Realtime API

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
1

Nemo-RT Community is a real-time bilingual Voice Agent: it hears you, thinks, and answers out loud in under a second. Every stage — VAD, STT, LLM, TTS — is co-located on a single NVIDIA GPU, and no audio or inference ever leaves the box.

audio → VAD → STT (NeMo Conformer, ES/EN language-adaptive)
→ LLM (Qwen3-8B-FP8, vLLM)
→ TTS (NeMo FastPitch + HiFi-GAN) → audio out

It speaks the OpenAI Realtime API protocol, so any existing Realtime client, SDK or SIP bridge points at your own box with a one-line change:

  • wss://api.openai.com/v1/realtime

  • ws://your-box:8000/v1/realtime

No per-minute billing, no account, no telemetry. One command brings it up on a fresh box.

Use cases. The repo ships an Asterisk/SIP bridge (integrations/asterisk/, via ARI + external-media RTP), validated on a live call — the same bridge that talks to OpenAI’s cloud talks to your box, only REALTIME_URL changes. That covers what people currently rent by the minute: contact-center front line, a 24/7 IVR that actually understands, appointment setting, order taking, support hotlines. Off the phone: a local voice assistant for a kiosk or browser — anything you would have pointed at the OpenAI Realtime API.

Measured:

  • H100 (SXM) — full end-to-end, including a live SIP phone call. TTFA ~0.12–0.16 s.
  • DGX Spark (GB10, 128 GB unified) — ~20 concurrent calls, sub-second TTFA.
  • GH200 — runs the arm64 build.
  • RTX 4090 (24 GB) — the whole stack fits in ~21.5 GB. Live voice TTFA 0.17–0.59 s; LLM 52 tok/s single-stream.

Why the Spark. 128 GB unified means memory stops being the ceiling on concurrent sessions, and GB10 has native FP8 — which is what the default model wants. The arm64 build means there’s nothing to cross-compile. One caveat: the Spark I measured on was already provisioned, so setup.sh — the one-command install — hasn’t been exercised against a fresh Spark OS. If you run it, I’d like to know what breaks.

Apache-2.0. (INFINITO CLOUD — NVIDIA Inception member.)

Repo: GitHub - infinitocloud/nemo-rt-community: Real-time, sub-second, bilingual (ES/EN) voice AI that runs entirely on your own NVIDIA GPU. One command, Apache-2.0. Truly on-premise — no cloud, no per-minute fees, no kill-switch. · GitHub
Demo (44 s): https://youtu.be/Ltr6j-ucodo

2

This is incredible, thanks for sharing, I’ll check it out later

3

Thanks! I’m following this thread — if you hit any snag, post it here and I’ll dig in.

4

Can you wire in a different LLM? 8B model is…well, not very smart.

5

Hi!

Yes, it is entirely possible to replace Qwen3 with any other model that supports FP8. I agree that it isn’t the “smartest” model; I chose it because I was looking for ultra-fast responses and good support for Spanish and English.

Thanks!

6

Can LLM allow users to configure it as a model that has already been deployed locally

7

Hi,
Hmm, I’m not sure what the question is, but this version only allows the use of the configured LLM—in this case, Qwen 3B—though you can easily change it; it doesn’t connect to other external models for now. Please let me know if I misunderstood the question.
Thanks.