Controls the right thing
The assistant knows which Voice PE heard me and which room it belongs to. “Turn off the light” can resolve locally, while ambiguous requests produce a real clarification instead of a creative guess.
Project
An actually useful Voice Assistant
I wanted something that could do more than set a timer, turn on a lamp and apologise when I asked a perfectly reasonable question. So I built a Swedish voice assistant around the things my family actually uses.
StatusRunning in our home, every day.
LanguageSwedish, with room-aware context and follow-up conversations.
HardwareHome Assistant Voice Preview Edition, a Mac mini and our existing Sonos speakers.
StackHome Assistant Assist, Hermes Agent and a deliberately narrow set of custom Python tools.
Why
The missing middle
Most voice assistants are decent at tiny commands and surprisingly bad at everything between those commands and a full conversation. I wanted to ask ordinary household questions in ordinary Swedish: when the next useful bus leaves, whether electricity gets cheaper this afternoon, how warm the kitchen is, what is on the shopping list, or if the car is actually ready to go.
The goal was never to put a chatbot in a speaker. The goal was to make the house easier to use without reaching for a phone. That distinction ended up shaping the whole architecture.
Capabilities
What it does
Answers live household questions
Weather from SMHI, 15-minute Tibber prices, Mercedes and Easee charging state, local buses, roadworks and nearby power or water disruptions. Every answer carries freshness rules, even if I never hear them.
Plays music where I am
Flexible Swedish music and radio requests go through Music Assistant. Unless I name another speaker, playback starts on the Sonos in the room that heard me.
Remembers things properly
Reminders, alarms and safe timed actions are stored in Home Assistant rather than living in a model conversation. They survive restarts and return to the speaker where they were created.
Uses our actual shopping list
It adds, reads and removes items in the Nya Matlistan account my family already shares. Quantities and item variants survive, and removals require an explicit confirmation on a later voice turn.
Handles the second sentence
After a useful answer it can listen briefly without another wake word. Short follow-ups such as “and tomorrow?” or “the second one” are tied to the previous context and cannot silently become unrelated commands.
Architecture
Fast path first, language model second
HearVoice PEroom + device context
UnderstandGoogle speech-to-textSwedish
RouteHome Assistant Assistnative fast path or Hermes
ReasonHermes Voiceisolated agent profile
ActConstrained toolsHome Assistant + fixed providers
SpeakAzure neural voiceSwedish response
Simple native intents stay inside Home Assistant and return quickly. Richer requests move to a dedicated Hermes profile with only household-specific tools enabled. It has no terminal, browser, files, messaging or arbitrary Home Assistant access.
The speech and language layers are cloud-assisted because Swedish quality matters more to me than winning a local-only purity contest. State, routing, permissions and execution remain under my control at home.
Safety
The model is not the authority
Provider hosts are fixed in code. The model cannot supply a URL, token, account or arbitrary entity ID.
Weather, prices, vehicle state and disruptions have explicit freshness limits. Stale or broken data is reported as unavailable, never filled in.
Clarifications and destructive actions use short-lived, session-bound, single-use tokens. The same model turn cannot approve itself.
Routine shopping responses are rendered from the structured tool result, so a confident model sentence cannot turn an error into “done”.
Reality
The hard parts were not the AI parts
Getting a model to understand a request was relatively easy. Getting an entire voice pipeline to behave predictably was not. Home Assistant could recognise a shopping-list sentence and then fail because its native shopping intent was unavailable. A perfectly good answer could be cut off by the wrong playback route. A reminder created at 14:03 could miss the minute boundary. “Raise the volume” could mean the assistant, the music or every speaker in the room.
Most of the work became routing, provenance, timeouts, confirmation semantics, stale-data handling and tests against the real Assist pipeline. The finished system is less magical than the first prototype and dramatically more useful.
Result
A household interface
I can ask it to play P4 Kronoberg in the kitchen, tell me when electricity is cheapest, add bearnaisesås to our shared list, remind me about something later and explain why the car is not charging. It answers in Swedish, in the room I am standing in, and usually without making me think about the machinery underneath. That is the closest I have come to an actually useful voice assistant.