Build a Fully Offline Voice Assistant in 2026: Whisper + LLM + Piper (Step-by-Step)

Quick Facts

• STT layer: whisper.cpp (best for Apple Silicon and embedded), faster-whisper (best for NVIDIA GPU Python pipelines).
• LLM layer: Ollama with Llama 3.1 8B (recommended), Phi-4 (smaller, good quality), or Mistral 7B (comparable quality to Llama 3.1 8B).
• TTS layer: Piper (fastest, CPU-only, real-time on Pi), Coqui TTS (better quality, needs GPU).
• Wake word options: OpenWakeWord (MIT, fully offline), Porcupine (free tier, 1 custom wake word).
• Minimum hardware: Raspberry Pi 5 with 8 GB RAM (~$100) for 5–8 second latency.
• Recommended hardware: Mac Mini M5 24 GB (~$600) or desktop with RTX 3060 12 GB (~$800) for 1–2 second latency.
• Languages: Whisper supports 99 languages. Piper supports 20+ language voice packs. LLM performance varies by language.

Why Build a Local Voice Assistant?

• Privacy: No audio leaves your home. No wake word audio stored in the cloud. No conversation history on third-party servers. Critical for healthcare workers, lawyers, journalists, and anyone with sensitive work.
• Cost: No subscription. Alexa+ (formerly Alexa Premium) costs $4.99/month. Google One costs $1.99–$9.99/month. A local assistant is one-time hardware cost.
• Customization: Choose your wake word, personality, system prompt, and capabilities. Add custom commands, connect to local home automation systems, integrate with local APIs.
• Offline operation: Works without internet. Power outage (with a UPS) + internet outage: your local assistant still works. Useful for cabins, remote locations, and emergency preparedness.
• What you give up: Web search, smart-home integration with proprietary clouds, calendar sync with cloud services, the years of RLHF tuning that makes Alexa/Siri smooth at edge cases.

The Three-Layer Architecture

• Layer 1 — STT (Speech-to-Text): whisper.cpp or faster-whisper. Converts microphone audio to text. Runs offline, no network.
• Layer 2 — LLM (Reasoning): Ollama serving Llama 3.1 8B, Phi-4, or Mistral 7B. Takes the transcribed text + conversation history + system prompt and generates a response. Runs offline, no network.
• Layer 3 — TTS (Text-to-Speech): Piper or Coqui TTS. Converts the LLM response text to audio and plays it through the speaker. Runs offline, no network.
• Orchestrator: A Python script that wires the three together: captures audio from microphone → passes to STT → passes transcript to LLM → passes response to TTS → plays audio.
• Optional wake word: A lightweight always-on detector (OpenWakeWord, Porcupine) that triggers the full pipeline only when the wake phrase is detected. Without this, the orchestrator runs whisper.cpp continuously — consuming more CPU and generating more false positives.

Hardware Requirements

Step 1: Set Up Speech-to-Text

• Install whisper.cpp: git clone https://github.com/ggerganov/whisper.cpp && cd whisper.cpp && make -j4
• Download your model: bash ./models/download-ggml-model.sh small (small = 3.4% WER, good speed balance)
• Test transcription: ./main -m models/ggml-small.bin -f test.wav — should produce accurate text output.
• Enable Metal on Mac: make -j4 WHISPER_COREML=1 then bash models/generate-coreml-model.sh small
• Whisper model selection: base for Raspberry Pi (1 GB RAM, low latency), small for the sweet spot (2 GB RAM, 3.4% WER), large-v3 for highest accuracy (10 GB VRAM/RAM).
• Configure silence suppression: Add --no-speech-threshold 0.6 --suppress-blank flags to avoid transcribing silence as hallucinated text.
• Test with a 10-second recording: Record yourself saying a test phrase, verify Whisper transcribes it accurately. Check both noise conditions and quiet speech.

Step 2: Set Up the Local LLM

• Install Ollama: Download from ollama.com. Available for macOS, Linux, and Windows. Installs in under 2 minutes.
• Pull model: ollama pull llama3.1:8b (recommended) or ollama pull phi4 (lighter, good for 16 GB RAM systems).
• Test: ollama run llama3.1:8b "What is the capital of France?" — verify the response is accurate and fast.
• System prompt for voice: Use a short, directive system prompt: "You are a helpful voice assistant. Keep responses concise — 1–3 sentences maximum. Never use bullet points, markdown, or formatting. Speak naturally as if in conversation."
• Temperature: Set temperature to 0.3–0.5 for more predictable, factual responses. Lower temperature reduces hallucinations in voice responses.
• Max tokens: Limit response length with --num-predict 150 — long responses take more TTS time and feel unnatural in voice interaction.

Step 3: Set Up Text-to-Speech

• Install Piper: pip install piper-tts
• Download a voice: piper --download-dir voices --update-voices --voice en_US-lessac-medium (or any voice from the Piper voices page on Hugging Face).
• Test: echo "Hello, how can I help you today?" | piper --model voices/en_US-lessac-medium.onnx --output-raw | aplay -r 22050 -f S16_LE -c 1
• Audio output: Piper outputs raw PCM or WAV. Pipe to aplay (Linux), afplay (Mac), or use the sounddevice Python library for cross-platform playback.
• Alternative (better quality): Coqui VITS backend — install pip install TTS, use tts --model_name tts_models/en/vctk/vits. Requires ~2 GB VRAM; 2–3× slower than Piper but noticeably more natural.
• Voice selection: For voice assistants, choose a medium-quality voice rather than high — medium voices are faster and the difference is negligible over a speaker.

Step 4: Connect the Pipeline

#!/usr/bin/env python3
"""Minimal offline voice assistant: Whisper STT + Ollama LLM + Piper TTS."""

import subprocess
import tempfile
import sounddevice as sd
import soundfile as sf
import numpy as np
import requests
import json

SAMPLE_RATE = 16000
RECORD_SECONDS = 5
OLLAMA_URL = "http://localhost:11434/api/generate"
WHISPER_BIN = "./whisper.cpp/main"
WHISPER_MODEL = "./whisper.cpp/models/ggml-small.bin"
PIPER_BIN = "piper"
PIPER_VOICE = "voices/en_US-lessac-medium.onnx"
SYSTEM_PROMPT = (
    "You are a helpful voice assistant. Keep responses to 1-3 sentences. "
    "Never use markdown, bullet points, or formatting. Speak naturally."
)

conversation_history = []

def record_audio(seconds: int = RECORD_SECONDS) -> np.ndarray:
    print("Listening...")
    audio = sd.rec(int(seconds * SAMPLE_RATE), samplerate=SAMPLE_RATE, channels=1, dtype="int16")
    sd.wait()
    return audio

def transcribe(audio: np.ndarray) -> str:
    with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as f:
        sf.write(f.name, audio, SAMPLE_RATE)
        result = subprocess.run(
            [WHISPER_BIN, "-m", WHISPER_MODEL, "-f", f.name, "--no-timestamps", "--no-prints"],
            capture_output=True, text=True
        )
    return result.stdout.strip()

def ask_llm(text: str) -> str:
    conversation_history.append({"role": "user", "content": text})
    response = requests.post(OLLAMA_URL, json={
        "model": "llama3.1:8b",
        "system": SYSTEM_PROMPT,
        "messages": conversation_history,
        "stream": False,
    })
    reply = response.json()["message"]["content"]
    conversation_history.append({"role": "assistant", "content": reply})
    return reply

def speak(text: str) -> None:
    with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as f:
        subprocess.run(
            f'echo "{text}" | {PIPER_BIN} --model {PIPER_VOICE} --output_file {f.name}',
            shell=True, check=True
        )
        data, sr = sf.read(f.name)
        sd.play(data, sr)
        sd.wait()

def main():
    print("Voice assistant ready. Press Ctrl+C to stop.")
    while True:
        audio = record_audio()
        transcript = transcribe(audio)
        if not transcript or len(transcript) < 3:
            continue
        print(f"You: {transcript}")
        response = ask_llm(transcript)
        print(f"Assistant: {response}")
        speak(response)

if __name__ == "__main__":
    main()

Step 5: Wake Word Detection

• OpenWakeWord (MIT license): Fully open-source, runs on CPU, supports custom wake words via fine-tuning on your own phrase. Install: pip install openwakeword. Works on Raspberry Pi. Best for fully offline and open-source setups.
• Porcupine (Picovoice): Proprietary but with a free tier for personal use. Pre-built wake words include "Alexa", "Hey Siri", "Ok Google", and custom options like "Hey Jarvis". Very accurate and low false-positive rate. Install: pip install pvporcupine.
• Integration pattern: Run OpenWakeWord/Porcupine in a loop. When wake word is detected, play a "ding" sound (user feedback), then trigger the Whisper + LLM + TTS pipeline for one query. Return to wake word listening after TTS playback.
• Always-on power: Wake word detection uses ~2–5% CPU on a Raspberry Pi 5 — negligible. You can leave the assistant running 24/7 with minimal power draw.
• Custom wake words (OpenWakeWord): Generate 500 positive and 500 negative audio examples of your wake phrase using text-to-speech, then fine-tune OpenWakeWord for < 30 minutes on CPU. Accuracy comparable to Porcupine for common English words.

Latency Optimization

• STT optimization (~0.2–0.5 sec): Use Whisper small instead of large-v3. Use VAD to trim silence before passing audio to Whisper — shorter audio = faster transcription.
• LLM optimization (~0.5–1.5 sec first token): Pre-load the model at startup (Ollama does this automatically). Use Q4_K_M quantization for the best speed/quality balance. Set --num-predict 100–150 to limit response length.
• Streaming LLM → TTS: Stream the LLM output token-by-token. Start TTS on each completed sentence (end of sentence detected by period/question mark). This reduces perceived latency by 0.3–0.7 seconds — the user hears the start of the answer while the LLM is still generating the end.
• TTS optimization (~0.1–0.3 sec): Piper generates the first audio within 50 ms. Pre-initialize Piper at startup. Use --output-raw to stream audio as it generates rather than waiting for a full file.
• Keep models in memory: Ollama keeps models warm in VRAM automatically. whisper.cpp loaded in stream mode stays in memory. Avoid reloading models between queries.
• Target latency by hardware tier: Pi 5: 5–8 sec (acceptable for non-conversational). Mini PC CPU: 3–5 sec (borderline conversational). Desktop GPU: 1–2 sec (natural). Mac M5: 1–1.5 sec (excellent).

Privacy and Security

• Verify with Wireshark: Run Wireshark on your network interface during a conversation with the assistant. You should see zero packets from the assistant process. Any unexpected traffic indicates a misconfiguration — check that Ollama's external API is disabled if you have a public IP.
• No audio stored: Neither whisper.cpp nor faster-whisper write audio files by default — they process in memory. The Python orchestrator in this guide writes a temporary WAV file for whisper.cpp, which is deleted after transcription.
• No conversation history stored: The conversation history in the example script is in-memory only and resets when you restart. For persistent history, implement explicit storage with a local database and encryption at rest.
• GDPR compliance: Because all processing is local and no data leaves your network, a local voice assistant for internal use does not require a data processing agreement. There is no data controller/processor relationship with a third party.
• Network isolation: For maximum privacy, add a firewall rule blocking outbound traffic from the assistant process. Ollama and whisper.cpp will function normally — they require no network access after model download.

FAQ

Sources

• whisper.cpp on GitHub — Source, build instructions, stream mode documentation.
• Ollama on ollama.com — Installation, model library, API reference.
• Piper TTS on GitHub — Source, voice pack downloads, Raspberry Pi setup guide.
• OpenWakeWord on GitHub — Source, custom wake word training guide, supported architectures.
• Porcupine by Picovoice — Wake word SDK, free tier terms, Python SDK documentation.
• Picovoice Console — Create free-tier API key for Porcupine personal use.

Related Reading

• Local Speech-to-Text 2026: Whisper.cpp vs faster-whisper — Deep dive on the STT layer of this pipeline.
• Local TTS and Voice Cloning 2026: Piper vs Coqui vs XTTS v2 — Deep dive on the TTS layer and voice cloning options.
• Local Multimodal AI Pipeline 2026 — Extend this voice pipeline with vision model support.
• How to Install Ollama — Ollama installation and model management.
• Local LLM Hardware Guide 2026 — Hardware selection for the full voice + LLM pipeline.
• Build a Local Voice Assistant on Your Phone — The mobile version of this guide (iPhone + Android).