so my little initial project with DGX is to have Home Assistant running on it, then have llama connect to it to control the house switches and what not. Is this a bad idea? I got the base working but absolutely got stumped on trying to get a gpu run speech to text like whisper/parakeet/etc… I’ve tried so many things Gemini recommended but none of them are setup for Spark. It had me try to get many different versions but kept failing. Even if I wanted to get speech to text to talk to Llama, I’ll need a good speech to text model right? I’m very new and feel I’m already in over my head. Please help.
You won’t get full support/all features of Home Assistant when trying to run it on a box with a regular OS like a Spark.
Nabu Casa decided to focus on their HaOS which makes the maintenance for them easier. You can run it with docker, but with limitations (see above).
I run my home assistant on a separate mini pc (currently still with an outdated installation method with supervised, but I will switch to HaOS). As far as I know Pipecat is popular among HA users. I didn’t dive into that yet. You should check the HA forum on working recipes for voice integration.
You could start exploring the ASR / TTS part with vLLM. vLLM supports ASR models like openai/whisper-large-v3 which serves an Open AI compatible endpoint as v1/audio/transcriptions. May be you start your experiments with that to find a model. To test you could use Open WebUI which can use vLLM endpoints. I played around with that.
I don’t recall any ready to use recipes in here. But it isn’t in my focus yet and there a some threads where people try to get Parakeet / NVIDIA ASR running on Spark.
Like this one:
I’ve been running my HomeAssistant for several years on an 8GB Raspberry Pi 4. Works great, less filling.
When I get around to it, I will try to have it call various tools on the Spark.
I stopped using Gemini and tried Claude to help me. For anyone looking for the same thing which is to run a GPU version of it so it’s faster on DGX:
“That was Voxtral — specifically mistralai/Voxtral-Mini-3B-2507, running in the voxtral-wyoming container on port 10300. It’s the STT engine that transcribes your voice after Hey Jarvis triggers.”
This project? Seems to be rather new. Also never heard of Wyoming before. Thanks for the hint!
It’s not as fast as I thought it would be. I’m trying trying to break down which process is making the response time more than 5 seconds…
I am still on this…since month…
honestly I am quite jealous how well gemini voice conversation work.
It replies within seconds and does not “think” to long on easy questions.
Task 1 Wakeword
I do not want the Spark to constantly analyse the noise and conversations in the room so I definitely need a Wakeword.
Something like “Computer” in Star Trek Next Generation
I think they were already Prompting in that Series.
rhasspy/wyoming-openwakeword:latest
Task 2 STT - Voice recognition
Have tried multiple STT Models, Whisper
they need to be fast, accurate and understand municipal languages like English, German, French, Spanish…
Alibaba/Qwen3-TTS-12Hz-1.7B-Base
fedirz/faster-whisper-server:latest-cuda
WHISPER__MODEL=Systran/faster-distil-whisper-large-v3
deepdml/faster-whisper-large-v3-turbo-ct2
openai/whisper-large-v3-turbo
Also tried
nvcr.io/nim/nvidia/parakeet-1-1b-rnnt-multilingual:latest
So far they are all to slow… there is this awkward silence for about 10sec until the text shows up.
Would love to have something that I can watch the words appear while talking.
Task 3 Super fast LLM
I got a Openclaw running in a docker container that can control Home-assistant. It works impressively but is super slow.
Using a 30B thinking model like Qwen3.5-35B-A3B or Nemotron-3-Nano-30B-A3B-NVFP4 is not helping much. Its rather ridiculous how much thinking that model is doing.
Simple commands like “turn the light on in the bedroom” can take up to 5min and “harder” commands like play Music in the Living room can take up to 15min.
So there needs to be an Operator or Orchestrate LLM that is small, fast and makes the decision on how to process the Prompt or to send it to a bigger LLM.
So far I need advice whats the best practise of doing so.
Task 4 TTS - reply
For hands free voice interaction with the Computer - or house - it would be nice to have a feedback. Friendly, short, precise and correct.
If spoken to in German it should reply in German…and if spoken to in English it should reply in English. Best would be if you can determine a nice voice that does not sound like a robot.
So far I tried
Alltalk
Alltalk2
F5-TTS
Alibaba/Qwen3-TTS-12Hz-1.7B-Base
kokoro-tts GitHub - remsky/Kokoro-FastAPI: Dockerized FastAPI wrapper for Kokoro-82M text-to-speech model w/CPU ONNX and NVIDIA GPU PyTorch support, handling, and auto-stitching · GitHub
nvcr.io/nim/nvidia/magpie-tts-multilingual:latest
oobabooga
and many versions of xTTS and xTTS2
xtts-streaming-server:arm64-cuda13
most of the times i am hitting this pytorch audio issue and if it works the processing ends up being done by the CPU - which is insanely slow.
it should be possible
or maybe with Nemotron-speech
or NVIDIA Nemotron ASR
Next thing I try is this
Ok I could not get Pipecat running on the Spark.
but I got
I have a quite different setup where i run multi-location HA (2 houses in different cities), each one on a dedicated locally deployed HAOS (with MASS). i have deployed several audio devices (nabu voice, atoms3r) in different rooms. all with custom firmware on board. the flow i use is the following:
- AtomS3R stream consistently the sound to a local sidecar to HAOS (docker) that implements wakeword “Hey Jarvis”, while Nabu devices run with local wakewords
- the wakeword docker acts as a proxy that send audio frames (after the wake) to a single centralized orchestrator (python) which run in a single location (on a Minisforum Atomman G7Pro - RTX5070 8gb). it does several things like speaker recognition and stt (parakeet) and uses a local small LLM (qwen2.5:7B instruct, 6Q_K) on llama.cpp to process intent routing (very fast, max 1sec latency end to end). If intent is home control it directly call HAOS executing the command and coming back with a reply (tone). if intent is simple chat (weather, time, math operations, simple stuff) it try to reply using the same model. in all other cases (or if i use specific keywords) it route the request to openclaw, which is installed on the same atomman as a different LXC. Openclaw according to the request can use cloud models (sonnet mostly) or a local model (abliterated mostly) running on a nearby DGX and reply to me using TTS (qwen3-tts with static prompt profile).
the LLM router and the orchestrator includes a specific keywork to trigger the intent LIVE_CHAT, which basically create an always on link with openclaw that skip most of the pipeline to make it more fast and like a ‘gemini-live’ experience.
i’m very happy with overall latencies: max 1sec for the HA command execution (unless is a bundle action): wakeword is 20ms, stt is 100ms, intent on average with proxy cache is 0.8ms. higher when openclaw is involved since frontier models includes cloud latency + not very fast so it’s usually 4-5 sec till the stt start to stream a reply.
DGX is used to run: stt (parakeet), tts (qwen3-tts), video pipeline (i2v, v2v, p2i, image mix, ACE, hallo4, reactor, all integrated as openclaw skill) and llama.cpp with qwopus 3.5:27B abliterated (at 12tok/s).
tried to document everything as much as possible: GitHub - croll83/jarvis: AI driven smart home · GitHub
Task 4 TTS
@martinB78, @monaco.marco83, that’s really cool—it’s kind of the dream of our
youth, the one we never really have time for.
It’s also cool that you’ve delved a bit deeper into the Spark + Audio topic.
Still, I have two general questions:
a) Have you also experimented with Audio > via BT > Spark > LLM?
b) Have you generally considered moving the Jarvis Orchestartor stack
into the Spark as well?
c) Would all LLMs also work with the Spark if the Gemini 3 Pro were replaced? Do you see that as a realistic possibility?
d) What is the Ontology Server used for, and in which areas does it provide
added value?
e) Ollama was replaced by llama.cpp, right?
thanx. .cj
Hello cjg,
You’re absolutely right—that “sci-fi” assistant vibe is something we’ve seen in movies for decades, but we’re only just now getting the hardware to actually build it.
To your questions:
a) Bluetooth & Microphones: I haven’t experimented with dedicated BT mic devices yet. My current approach is using the device I always have on me: my smartphone. I use a messenger app as the input interface to trigger the system. However, I agree that having small, localized mic nodes in every room (or a wearable) would be the ideal next step for a truly ambient system.
b) Orchestrator Placement: I’ll let @monaco.marco83 weigh in on the specifics there. Personally, I prefer the Spark to focus exclusively on what it does best: running the LLMs. I like keeping Home Assistant on separate, dedicated hardware. Since I tinker with the Spark quite often, I don’t want to risk the 24/7 reliability of my home automation.
c) LLM Fallbacks: You can use LiteLLM to define which model serves as your primary or fallback. While you could use a Gemini API key there, I’d actually recommend looking into Claude Opus 4.6. In my experience, it’s significantly more capable than Gemini 3 Pro for complex orchestration tasks.
d) I skip that question
e) Ollama vs. llama.cpp: Exactly. While both support GGUF formats, I’ve found that llama.cpp provides a noticeable speed advantage on this specific hardware stack.
Bt mic sucks for assistant. The connection isn’t live all the time and you have to “wake up” the bt connection which is not great for conversational or command stuff. U need one where it’s always on. I tried with the Anker conference microphone and the lag kills it.
My biggest issue right now is the mic not picking up the messages correctly hence needing a better STT which doesn’t lag. I’m aiming for quick responses so standards are 2 sec reaction time to command and talk to.
hey @_cjg indeed, youth dream coming to a reality :D
on your questions:
a) nope sir, no need for that. Audio mic with good quality for STT on long range (mic array + beamform) comes with WiFi so no need to use BT and manage connection persistence. Low quality audio device (like AtomM5S) are good in proximity (50cm) so i mostly use them on desk or with a battery, that makes BT less relevant as you move around with a Jarvis mic in your hand :D
b) i honestly did not because i need to use a VERY FAST intent router which require the real GPU to keep overall end2end latency of voice command → action under 1sec. so i run it on the G7 Pro which have a 5070 8Gb and Qwen2.5 7B instruct. but pretty sure if latency is not a big deal for you it will run flawlessly on the spark.
c) TL/DR: yes, for home assistant you don’t even need big intelligence, any LLM can serve it. And you can run quasi-intelligent model on the spark, it’s just slower. To be more precise (maybe you read the old documentation, i just refactored and pushed on the repo), i’m not using gemini anymore (except for nano-banana: ‘hey jarvis create a picture of a monkey sipping a drink on a beach and stream it to the TV in the living room’). I moved the entire ‘real AI stack’ on agent (first openclaw, now hermes), which use different LLMs according to the need. the agent is seen by the orchestrator as a black box: if the intent router tags the voice command as ‘AI_AGENT’, it just pass the prompt with speker id to the Hermes agent. It default on a shared agent that read the speaker id and trigger the specific agent of the user talking (so if i speak, my personal hermes agent will trigger, if my wife speak, her personal agent will trigger). Agent and orchestrato share the same mem0 memory, so they both keep the same context of the conversation. the agent then eval the prompt and trigger a different model according to the complexity: easy prompt or ‘abliterated’ prompts goes to Qwopus27B running on the spark. more complex prompt goest to either haiku or sonnet or opus. + if the prompt require an audio/video gen pipeline, the agent have the skill to call the comfyUI apis running on the spark. I hope it’s clear.
d) ontology server is also deprecated in my setup. i created it while on openclaw and it’s a way to create a structured memory for agents: rather than reading .md files (increasing latency) i built a server to operate a structured memory with ACL, so users can access only they part of knowledge (or the public shared one). but this is deprecated since i moved from openclaw to hermes which already provide full isolation among agents. i paried it with mem0 that provides fact extraction + relational graph, so the ontology server become useless.
e) i replaced with llama.cpp because i quantized the model KV and weight with TQ3, so i’m running the forked llama.cpp version by turbodan and it rocks. but that is only for the spark and the ‘abilterated’ model (qwopus27B). going to change this again with the today’s release of qwen3.6 which i was waiting for since a while. i just need jack to release the opus trained so i can abliterate and replace the current one, maybe i will not need to run the KV fork of llama.cpp anymore (as MoE stock already provide a great tps point).
"Thanks for the detailed response! I really admire your professional approach to this—it’s definitely made me rethink a few things 😉. My original plan for those long winter evenings (while my wife is busy knitting 😉) was to use my conference speakers. But after my experience with the Spark, Bluetooth has already worn out its welcome—and that’s even before mentioning the latency issues.
I’m going to go back to the drawing board. If I’ve understood you and your docs correctly, your setup is already—or at least nearly—parallel-ready (LLM context and latency aside), right? It seems relatively straightforward to route the output to a specific speaker ID, provided they aren’t in the same room. That just leaves the orchestrator used to share context with the agent. We’d still need a solution there (maybe a switch?).
I assume you’ve either got this running already or at least have a solid plan for it. Even if this remains purely conceptual for me due to time constraints, thanks a lot for sharing the project! Best, cj"
it depends from what you mean by parallel :D it can use any LLM exposed via ollama/llama.cpp yes.
Yessir, currently the orchestrator reads audio devices (MIC) discovered on the network and not initialized (AtomM5S and Nabu Voice) and let you add them. then it reads the speakers on HASS and for every mic configured, you can associate it to a speaker or use their own internal speaker.
The flow is the following:
- when you speak to one of the mic device, it get the request, process it and then retrive from conf what is the associated spekaer of that specific device for the response.
- if the MIC device is configured to use the internal speaker, orchestrator will use the local tts, build the response and play on the internal speaker
- else if the MIC device is configured to use an HASS speaker AND it’s a normal HASS speaker (media_player) → orchestrator will use the local tts, build the response and play on the internal speaker via HASS
- else if the MIC device is configured to use an HASS speaker AND it’s an Alexa Echo Device (with Alexa Media Player HACS), it does not build the TTS, but call Alexa TTS directly with text (properly formatted for tts) and the Alexa device stream the response with the original Alexa voice
- if the audio device have internal speaker, it’s always used as a fallback.
- as a final fallback, the response is sent via telegram if none of previous is working.
On top of this you’ll find other cherries like, the moment wakeword is triggered, it automatically find all HASS speaker device in the same room and if any, reduce the volume to 10% (to avoid background noise on the STT), or the DND mode or the ability to kill the response (sometime it could start talking because of an hallucination and go on for minutes. you have a panic button to stop it immediately and abort).
Last commit actually implements Redis for context. both hermes (with a custom plugin) and orchestrator read and write their context in the registry so context is fully shared. orchestrator context, becoming part of the agent context, is then persisted into the mem0 long term memory together with the hermes one.
Fully running and operational on a single HASS location. waiting to move in the summer house to configure the second location, too, and run both of them in parallel.
Whats your guys experience with the Model for Homeassistant. I tried GPT OSS 120b but it is not smart enough and Qwen 3.6 - overheats the spark.
what are you guys using?