Personal voice assistant built on Claude tool calling, with a custom trained wake word and a fully local speech stack.
Ash is a voice assistant built for one user, not a product. Off the shelf assistants match fixed commands and know nothing about my work. Ash reasons about each request with an LLM and looks up my context before every reply. The name comes from my own, Akash.
The project doubles as an end to end systems exercise: LLM tool calling, a speech pipeline, a custom trained model, documented architecture decisions, and a cloud deployment plan. Phase 1 of 4, the laptop agent, is nearing completion.
A local wake word model listens for "Hey Ash", Whisper transcribes the request, and the orchestrator injects personal context before entering a Claude tool calling loop. Tool results are fed back until the model produces a final answer, which is styled by the persona layer and spoken by Piper. In later phases the orchestrator moves to AWS Lambda behind API Gateway, with laptop, phone, and Raspberry Pi agents as thin clients that each expose only their own tools.
The execution layer parses the request, calls tools, and handles confirmations. It stays literal and correct however a command is phrased, and destructive actions such as delete or overwrite always require explicit confirmation. Only once the action is decided does the reply pass through Ash's voice, defined in a versioned persona spec loaded as the system prompt. Humour never changes what actually executes.
openWakeWord has no pretrained "Hey Ash", so I trained a custom model on synthetic speech: generated voice clips, adversarial negative phrases, background noise augmentation, and a small classifier exported to ONNX. Getting the community Colab pipeline running meant debugging missing wheels, a subprocess import path issue, and a pronunciation dictionary failure caused by a punctuated phrase. Short one syllable triggers like bare "Ash" proved unreliable in testing.
Each tool is a single module with a JSON schema and a run() function, registered in one line. Built tools cover time, weather via Open-Meteo with IP based location fallback, and file operations such as moving files and creating folders, with confirmation on anything destructive.
Python was chosen over Java, my stronger language, because the tool calling code changes weekly and Python's LLM tooling is the most mature. Lambda over EC2 means near zero cost while idle, and running Whisper, Piper, and openWakeWord locally keeps the whole voice stack free. Estimated running cost is AUD $10 to 20 a month, mostly Claude API usage. Every decision is logged in an ADR file.
| Service | Role | Status |
|---|---|---|
| Claude API | Reasoning and tool calling orchestration | Built |
| openWakeWord | Custom "Hey Ash" wake word detection | Built |
| Whisper | Local speech to text | Built |
| Piper | Local text to speech | Built |
| Open-Meteo | Weather tool data | Built |
| AWS Lambda + API Gateway | Cloud orchestrator for all devices | Planned |
| DynamoDB | Memory and state | Planned |