Real-time ASR on 2×2 DGX Spark (GB10) stacks

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
,
1

Hi all,

I’d like to add real-time ASR to my AI stack, and I have a few concerns I’d like to share before committing to a design. Any guidance would be very welcome.

Current setup

  • 1 VM running the AI gateway / load balancer, using LiteLLM to issue dedicated API keys to users
  • 2 stacks of 2× DGX Spark (GB10) each. Each stack runs vLLM in a 2-node configuration (1 head + 1 worker)
  • Both stacks currently serve the same model: DeepSeek-V4-Flash-Vision-Exp
  • Clients: 8 dev engineers + 2 sysadmin/infra engineers behind the gateway. No rate limits or spend caps configured yet
  • Interconnect between nodes in a stack: 400GB cable nvidia recommended (overkill but only reference I can buy at work)

My questions

  1. Topology. For multi-user serving, is it better to keep 2 independent stacks with latency-based load balancing, or to merge them into a single 4×GB10 stack?
  2. ASR model. I’m considering the NVIDIA Nemotron real-time ASR modelor, if the available memory allows, Voxtral real-time ASR. Which would you pick for this hardware?
  3. Placement. If the 2-stack + load-balancing layout remains the right call for multi-user traffic, my plan is to run an additional vLLM instance on each stack dedicated to the real-time ASR model, and register it as a separate deployment in LiteLLM. Is that the right approach, or should ASR live somewhere else?
  4. Latency isolation (my main concern). DeepSeek users can open multiple concurrent sessions with heavy prefill and large input prompts that miss the prefix cache, unavoidable with several users. Memory bandwidth and interconnect are shared and limited on the GB10, so I’m worried about ASR latency degrading under LLM load. How would you isolate a latency-sensitive streaming workload from a throughput-bound prefill workload on the same nodes?

What I’ve considered so far

  • Two stacks + load-balancing rather than one larger stack, to keep blast radius per user small
  • A dedicated vLLM instance per stack for ASR, exposed through the gateway as a separate route, with latency- or least-busy-based routing
  • Capping the LLM instances (KV cache / GPU memory utilization / max concurrent sequences) so they can’t fully starve the ASR instance
  • Adding basic gateway-level guardrails (per-key rate limits, token budgets) before anything else, since I currently have none

Any tips on the design would be appreciated.

Thanks for your consideration.

Best,
William

2

We recently had another post about this. @martinB78 came up with a neat soln worth reviewing. I just grabbed a 16GB 5060 for aux models.

3

Is this thread the right topic?

I considered using a dedicated GPU for ASR, but that isn’t an option for us right now. Given the size of these ASR models, I assumed the GB10 would have enough headroom and bandwidth to handle it. I’ve previously set up NIM and vLLM-omni on a personal single-node GB10.

Now, I’m looking to deploy ASR for about a dozen users on my company’s GB10 setup. Since the GB10 is quite a unique architecture and feels more like a personal dev device than a multi-user team platform (even if it’s capable of both), I’m wondering what key factors or potential bottlenecks I should keep in mind.

4