Key Takeaways
- Willow Inference Server is free and open-source; the official GitHub LICENSE is the Apache License 2.0
- Runs Whisper-based automatic speech recognition (ASR) and text-to-speech (TTS), reachable over WebRTC, REST, and WebSockets
- Deployed via Docker Compose on Linux (Windows via WSL); a CPU-only Docker Compose file is also provided, though CPU performance is documented as significantly slower than GPU
- Runs on NVIDIA GPUs from a GTX 1060 3 GB up to an RTX 4090; roughly 6 GB VRAM is recommended to run all default Whisper model sizes plus TTS together
- Supports custom TTS voice creation from a relatively small sample recording
- Built by Tovera as the backend for the open-source Willow voice assistant hardware project
- No longer supports general LLM/chat inference — that feature was explicitly removed in a February 2026 commit; the project's own maintainers direct users to run a separate tool such as Ollama alongside it for that need
- GitHub repository (github.com/toverainc/willow-inference-server) shows roughly 510 stars as of September 2026; commit activity is bursty rather than continuous — treat this as a smaller, slower-moving project than a heavily staffed commercial one
📍 In One Sentence
Willow Inference Server is a free, open-source, self-hosted server that runs Whisper-based speech recognition and text-to-speech on your own NVIDIA GPU, with no paid tier.
💬 In Plain Terms
Instead of sending your voice recordings to a cloud speech API, Willow Inference Server runs the speech-to-text and text-to-speech models on a computer you control, usually one with an NVIDIA graphics card. It was built to power the Willow voice assistant hardware project, but it can run on its own for any app that needs to turn speech into text or text into speech without a cloud dependency.
What Is Willow Inference Server?
Willow Inference Server is a self-hosted server that runs speech recognition and speech synthesis models on your own hardware, reachable over WebRTC, REST, and WebSockets. Its own GitHub description reads: "Open source, local, and self-hosted highly optimized language inference server supporting ASR/STT, TTS, and LLM across WebRTC, REST, and WS" — a description this review found is now partly out of date, since general LLM support was removed from the codebase in February 2026 (see the history section below). As of this review, WIS is accurately described as an ASR/TTS server, not a general LLM server.
- Core function: a speech inference server that accepts audio or text over WebRTC, REST, or WebSockets and returns transcribed text or synthesized audio
- Speech recognition engine: OpenAI's Whisper models, tuned by the project for low-latency streaming use
- Text-to-speech engine: a bundled TTS pipeline (the repository includes a dedicated
Dockerfile.xttsandxttsdirectory) supporting custom voice creation from short sample recordings - Deployment target: NVIDIA CUDA-capable GPUs, with a documented CPU-only fallback path for lower performance
- Developer: Tovera, the organization behind the Willow open-source voice assistant hardware project
- Canonical repository: github.com/toverainc/willow-inference-server — the project and organization name in current, active use for the server's source code, releases, and issue tracker
Willow Inference Server Project History
Willow Inference Server's GitHub repository was created in February 2023, and its commit history shows bursts of activity separated by long quiet periods rather than continuous weekly development — a pattern worth knowing before you depend on it for production use.
- 1February 2023 — Repository created
Why it matters: Willow Inference Server started as the backend companion to the Willow voice assistant hardware project, which the Tovera organization also maintains. - 2April 2024 — TTS engine updates
Why it matters: Commits from this period updated the bundled XTTS voice-cloning pipeline, per the repository's commit history. - 3June 2025 — Code-quality pass
Why it matters: A batch of commits replaced the project's flake8 linting with black code formatting and reformatted the Python codebase — a maintenance pass rather than a feature release, after roughly 14 months without a public commit. - 4February 2026 — LLM/chat support removed
Why it matters: A commit titled "README: drop LLM/chat references" states plainly: "We no longer support that. People can run ollama next to WIS." This is the most consequential change in the project's recent history — it narrows WIS's scope from a combined ASR/TTS/LLM server to an ASR/TTS-only server, and it is the most recent commit on the repository's main branch as of this review.
What Can You Do With Willow Inference Server?
Willow Inference Server's feature set centers on fast, self-hosted speech processing rather than general-purpose language model chat. Here is what each part actually does, per the project's own GitHub README.
- Automatic speech recognition (ASR) — transcribes spoken audio to text using Whisper-based models, tuned by the project toward "as-close-to-real-time-as-possible" streaming recognition rather than only batch transcription
- Multiple simultaneous Whisper model sizes — the README states all three default Whisper model sizes (base, medium, large-v2) can be loaded simultaneously within 6 GB of VRAM, letting a deployment trade accuracy for speed per-request instead of committing to one model size
- Text-to-speech (TTS) with voice cloning — synthesizes speech from text, with support for creating a custom voice from a relatively small sample audio recording, via the bundled XTTS-based pipeline
- Multi-protocol access — reachable over WebRTC (for real-time, low-bandwidth audio streaming), REST, and WebSockets, so a client can pick whichever transport fits its use case
- Hardware auto-detection — the server is documented as automatically detecting available hardware and adjusting behavior accordingly, rather than requiring manual configuration per GPU model
- Willow voice assistant backend — WIS is the inference backend the Willow open-source voice assistant hardware project connects to for turning spoken commands into text and, optionally, responses back into speech
Usage Examples: Three Ways to Use Willow Inference Server
These are concrete workflows built from Willow Inference Server's documented features above — not hypothetical use cases.
Run WIS as the backend for Willow voice assistant hardware
Deploy Willow Inference Server on a machine with a CUDA-capable NVIDIA GPU, then point a Willow hardware device or the Willow app at your self-hosted WIS instance instead of Tovera's hosted example server. Spoken commands captured by the device are streamed to WIS over WebRTC and transcribed with Whisper, keeping voice data on infrastructure you control.
Transcribe audio over REST without any voice hardware
Send a recorded audio file to WIS's REST endpoint from any script or application that can make an HTTP request, and receive back a text transcription. This works independently of the Willow hardware project — any app needing self-hosted, Whisper-quality speech-to-text can use WIS as a standalone transcription service.
Generate a custom cloned voice for TTS output
Provide WIS's TTS pipeline with a short sample recording of a target voice to create a custom voice profile, then send text to the TTS endpoint to have it synthesized in that voice — useful for accessibility tools or a voice assistant that should not sound like a generic default TTS voice.
Install Willow Inference Server
Willow Inference Server installs via Docker Compose, and its source code is on GitHub. Links and commands below are from the official GitHub repository — always verify against that page directly, since install instructions can change between commits.
GitHub repository (source code, Apache 2.0)
Clone the repository
- Link:
git clone https://github.com/toverainc/willow-inference-server.git
GPU deployment
- Link:
docker compose upusing the repository'sdocker-compose.ymlfile
CPU-only deployment
- Link:
docker compose -f docker-compose-cpu.yml up(documented as significantly slower than GPU)
Willow voice assistant project
- Link:
- github.com/toverainc — the parent organization's other repositories, including the Willow hardware/firmware project WIS was originally built to serve
Willow Inference Server needs an NVIDIA CUDA-capable GPU for its documented performance figures — a GTX 1060 3 GB is the stated minimum, with roughly 6 GB VRAM recommended to run ASR and TTS together. A CPU-only Docker Compose path exists and is documented in the repository, but the project states CPU performance is well behind any of the listed GPUs; treat CPU-only mode as a functional fallback for testing, not a production deployment target.
Willow Inference Server Pricing: Is It Really Free?
Yes — Willow Inference Server has no paid tier. The GitHub repository has no pricing page, and the LICENSE.md file applies to the whole application. The license text is the Apache License, Version 2.0 — permissive, with a patent grant and no copyleft obligation to release your own modifications.
- No subscription, no paid tier, no usage limits imposed by Willow Inference Server itself
- No account or sign-up required to deploy or use the software
- Running WIS still requires your own GPU hardware and the electricity/hosting cost of running it — the software itself is free, but self-hosting is not zero-cost the way a hosted API can be
- Tovera also operates a hosted example server for the separate Willow voice assistant hardware project — this review covers the self-hosted, free-to-run WIS software specifically; verify directly with Tovera for any details about a separate hosted offering and whether it carries its own cost
Willow Inference Server vs. whisper.cpp
Willow Inference Server and whisper.cpp both run OpenAI's Whisper speech recognition models locally, and they get compared often because both avoid sending audio to a cloud API. The clearest differences are in deployment model and scope.
Aspect | Willow Inference Server | |
|---|---|---|
| Scope | ASR + TTS server with WebRTC/REST/WS | ASR (transcription) only, no built-in server or TTS |
| Deployment | Docker Compose, GPU-focused | Compiled binary or library, CPU-friendly |
| Hardware target | NVIDIA CUDA GPU (3–6+ GB VRAM) | Runs on CPU; GPU acceleration optional |
| Real-time streaming | Built for it, via WebRTC | Possible with additional client-side work |
| Voice hardware integration | Built as Willow voice assistant backend | General-purpose library, no assistant hardware tie-in |
| Maintenance pace | Bursty; last commit February 2026 as of this review | Actively maintained with frequent commits |
If your priority is a ready-made server with WebRTC streaming and TTS built in, and you have a spare NVIDIA GPU, Willow Inference Server's feature set is the broader of the two. If your priority is the lightest possible CPU-friendly transcription library to embed in your own application, whisper.cpp is worth evaluating directly — see the whisper.cpp review for full details.
Who Should Use Willow Inference Server?
Whether Willow Inference Server fits depends on whether you have a compatible NVIDIA GPU, need both ASR and TTS in one server, and are comfortable with a project that ships updates in bursts rather than continuously.
Use Willow Inference Server if
- You are deploying or self-hosting the Willow open-source voice assistant hardware and want to run your own backend instead of Tovera's hosted example server
- You need both speech-to-text and text-to-speech from one self-hosted server, reachable over WebRTC for real-time use
- You have access to an NVIDIA CUDA-capable GPU with at least 3–6 GB of free VRAM
- You want custom voice cloning for TTS output from a short sample recording
- You are comfortable evaluating and, if needed, patching a smaller open-source project yourself, rather than depending on frequent upstream releases
Wait, or look elsewhere, if
- You need general LLM/chat inference from the same server — that feature was removed in February 2026; run a separate tool such as Ollama alongside WIS instead
- You do not have an NVIDIA GPU — WIS's documented CPU-only path exists but is stated to run well behind GPU performance, and there is no Apple Silicon or AMD-GPU-specific optimization documented
- You need guaranteed, continuous upstream maintenance — the project's commit history shows multi-month gaps between activity bursts, most recently February 2023 to April 2024, June 2025, and February 2026
- You only need simple offline transcription without a server, WebRTC, or TTS — a lighter library such as whisper.cpp may fit with less deployment overhead
Willow Inference Server vs. Other Speech Tools
Willow Inference Server is one of several self-hosted options for local speech recognition and synthesis. Here is how it sits next to other tools in that space — see the Local LLM Software Directory for the full catalog, and the dedicated Willow Inference Server vs. whisper.cpp comparison above for the closest head-to-head.
- whisper.cpp — a lightweight, CPU-friendly C/C++ port of OpenAI's Whisper for transcription only, with no built-in server, WebRTC, or TTS; see the dedicated comparison section above.
- Faster Whisper — a CTranslate2-based reimplementation of Whisper focused on inference speed; a library to embed rather than a ready-made ASR/TTS server like WIS.
- Piper TTS — a lightweight, CPU-friendly local text-to-speech engine; narrower in scope than WIS's combined ASR+TTS server but far lighter on hardware requirements.
- Coqui TTS — an open-source text-to-speech toolkit with voice cloning support, comparable in TTS scope to the XTTS pipeline WIS bundles, but distributed as a library rather than a WebRTC-ready server.
- Bark TTS — a generative text-to-speech model capable of non-speech audio (laughter, pauses); a different TTS engine choice than WIS's bundled XTTS pipeline.
- MacWhisper — a native macOS desktop app for Whisper transcription; a GUI desktop alternative for users who do not need WIS's server/WebRTC deployment model at all.
Common Mistakes When Evaluating Willow Inference Server
Most confusion about Willow Inference Server comes from outdated descriptions of its LLM support, unclear hardware expectations, or overestimating how actively it currently ships updates.
Mistake 1: Assuming Willow Inference Server still supports general LLM chat
The project's own GitHub description still mentions "LLM" as of this review, but a February 2026 commit titled "README: drop LLM/chat references" explicitly removed that support, stating: "We no longer support that. People can run ollama next to WIS." Treat WIS as an ASR/TTS server only, and pair it with a separate tool like Ollama for general text generation.
Mistake 2: Assuming any GPU will do
Willow Inference Server is built and documented around NVIDIA CUDA-capable GPUs. There is no AMD-GPU-specific optimization or Apple Silicon build documented — Apple Silicon users should look at Whisper-based alternatives designed for that hardware instead, such as MacWhisper.
Mistake 3: Treating audio processed by a self-hosted server as automatically private end-to-end
Running Willow Inference Server on your own hardware keeps your voice data off a third-party cloud API by default, since transcription and synthesis happen on infrastructure you control. That guarantee applies specifically to your own self-hosted deployment — if you instead point a Willow device at Tovera's hosted example server, or add any other network-connected integration, your audio does leave your own hardware at that point. Verify your specific deployment's data flow rather than assuming "self-hosted software" and "fully private" are automatically the same thing.
Mistake 4: Expecting a continuous release cadence
Willow Inference Server's commit history shows multi-month gaps between activity — for example, roughly 14 months of no public commits between April 2024 and June 2025. This is normal for a community-paced project, but plan around it if your deployment needs guaranteed prompt bug fixes.
Frequently Asked Questions
What is Willow Inference Server?
Willow Inference Server (WIS, github.com/toverainc/willow-inference-server) is a free, open-source, self-hosted server that runs Whisper-based speech recognition and text-to-speech, reachable over WebRTC, REST, and WebSockets. It was built by Tovera as the backend for the Willow voice assistant hardware project.
Is Willow Inference Server free?
Yes. The GitHub repository has no pricing page, and its LICENSE.md file is the Apache License 2.0, applying to the whole application with no paid tier. You still need to provide and pay for your own GPU hardware to run it.
Does Willow Inference Server support LLM/chat inference?
No, not as of this review. A February 2026 commit removed general LLM/chat support from the project, with the maintainers stating: "We no longer support that. People can run ollama next to WIS." Willow Inference Server is currently an ASR/TTS-only server.
What GPU does Willow Inference Server need?
An NVIDIA CUDA-capable GPU. The project documents a GTX 1060 3 GB as the practical minimum, with roughly 6 GB of VRAM recommended to run all three default Whisper model sizes (base, medium, large-v2) plus TTS together. A CPU-only Docker Compose path exists but is documented as significantly slower.
How do I install Willow Inference Server?
Clone the GitHub repository and run docker compose up using the provided docker-compose.yml for GPU deployment, or docker compose -f docker-compose-cpu.yml up for the CPU-only path. Check the repository directly for current instructions before installing, since they can change between commits.
Is Willow Inference Server actively maintained?
Its commit history shows bursts of activity separated by multi-month gaps rather than continuous development — for example, roughly 14 months of no public commits between April 2024 and June 2025. As of this review, the most recent commit on the main branch is from February 2026, and the repository has no tagged releases. It is not archived, but this is a smaller, community-paced project rather than one with a continuous release cadence.
What is the relationship between Willow Inference Server and Willow?
Willow is a separate open-source voice assistant hardware/firmware project, also maintained by Tovera. Willow Inference Server is the backend software that processes speech for Willow devices, but it can also run standalone for other speech-to-text or text-to-speech use cases.
Can Willow Inference Server clone a custom voice for TTS?
Yes. Its bundled XTTS-based TTS pipeline supports creating a custom voice profile from a relatively small sample audio recording, per the project's documentation.
What license does Willow Inference Server use?
The Apache License, Version 2.0, per the official LICENSE.md file — permissive, with a patent grant and no requirement to open-source your own modifications.
Does Willow Inference Server support real-time streaming?
Yes, over WebRTC, which the project built specifically for low-latency, real-time audio use cases such as a voice assistant listening continuously for commands. REST and WebSocket endpoints are also available for non-streaming or simpler integrations.