Key Takeaways
- Willow Inference Server is free and open-source; the official GitHub LICENSE is the Apache License 2.0
- Runs Whisper-based automatic speech recognition (ASR) and text-to-speech (TTS), reachable over WebRTC, REST, and WebSockets
- Deployed via Docker Compose on Linux (Windows via WSL); a CPU-only Docker Compose file is also provided, though CPU performance is documented as significantly slower than GPU
- Runs on NVIDIA GPUs from a GTX 1060 3 GB up to an RTX 4090; roughly 6 GB VRAM is recommended to run all default Whisper model sizes plus TTS together
- Supports custom TTS voice creation from a relatively small sample recording
- Built by Tovera as the backend for the open-source Willow voice assistant hardware project
- No longer supports general LLM/chat inference β that feature was explicitly removed in a February 2026 commit; the project's own maintainers direct users to run a separate tool such as Ollama alongside it for that need
- GitHub repository (github.com/toverainc/willow-inference-server) shows roughly 510 stars as of September 2026; commit activity is bursty rather than continuous β treat this as a smaller, slower-moving project than a heavily staffed commercial one
π In One Sentence
Willow Inference Server is a free, open-source, self-hosted server that runs Whisper-based speech recognition and text-to-speech on your own NVIDIA GPU, with no paid tier.
π¬ In Plain Terms
Instead of sending your voice recordings to a cloud speech API, Willow Inference Server runs the speech-to-text and text-to-speech models on a computer you control, usually one with an NVIDIA graphics card. It was built to power the Willow voice assistant hardware project, but it can run on its own for any app that needs to turn speech into text or text into speech without a cloud dependency.
πNote: This review is the deep-dive companion to Willow Inference Server's entry in the Local LLM Software Directory β see that page for how it compares at a glance to dozens of other local AI tools.
What Is Willow Inference Server?
Willow Inference Server is a self-hosted server that runs speech recognition and speech synthesis models on your own hardware, reachable over WebRTC, REST, and WebSockets. Its own GitHub description reads: "Open source, local, and self-hosted highly optimized language inference server supporting ASR/STT, TTS, and LLM across WebRTC, REST, and WS" β a description this review found is now partly out of date, since general LLM support was removed from the codebase in February 2026 (see the history section below). As of this review, WIS is accurately described as an ASR/TTS server, not a general LLM server.
- Core function: a speech inference server that accepts audio or text over WebRTC, REST, or WebSockets and returns transcribed text or synthesized audio
- Speech recognition engine: OpenAI's Whisper models, tuned by the project for low-latency streaming use
- Text-to-speech engine: a bundled TTS pipeline (the repository includes a dedicated
Dockerfile.xttsandxttsdirectory) supporting custom voice creation from short sample recordings - Deployment target: NVIDIA CUDA-capable GPUs, with a documented CPU-only fallback path for lower performance
- Developer: Tovera, the organization behind the Willow open-source voice assistant hardware project
- Canonical repository: github.com/toverainc/willow-inference-server β the project and organization name in current, active use for the server's source code, releases, and issue tracker
Willow Inference Server Project History
Willow Inference Server's GitHub repository was created in February 2023, and its commit history shows bursts of activity separated by long quiet periods rather than continuous weekly development β a pattern worth knowing before you depend on it for production use.
- 1February 2023 β Repository created
Why it matters: Willow Inference Server started as the backend companion to the Willow voice assistant hardware project, which the Tovera organization also maintains. - 2April 2024 β TTS engine updates
Why it matters: Commits from this period updated the bundled XTTS voice-cloning pipeline, per the repository's commit history. - 3June 2025 β Code-quality pass
Why it matters: A batch of commits replaced the project's flake8 linting with black code formatting and reformatted the Python codebase β a maintenance pass rather than a feature release, after roughly 14 months without a public commit. - 4February 2026 β LLM/chat support removed
Why it matters: A commit titled "README: drop LLM/chat references" states plainly: "We no longer support that. People can run ollama next to WIS." This is the most consequential change in the project's recent history β it narrows WIS's scope from a combined ASR/TTS/LLM server to an ASR/TTS-only server, and it is the most recent commit on the repository's main branch as of this review.
What Can You Do With Willow Inference Server?
Willow Inference Server's feature set centers on fast, self-hosted speech processing rather than general-purpose language model chat. Here is what each part actually does, per the project's own GitHub README.
- Automatic speech recognition (ASR) β transcribes spoken audio to text using Whisper-based models, tuned by the project toward "as-close-to-real-time-as-possible" streaming recognition rather than only batch transcription
- Multiple simultaneous Whisper model sizes β the README states all three default Whisper model sizes (base, medium, large-v2) can be loaded simultaneously within 6 GB of VRAM, letting a deployment trade accuracy for speed per-request instead of committing to one model size
- Text-to-speech (TTS) with voice cloning β synthesizes speech from text, with support for creating a custom voice from a relatively small sample audio recording, via the bundled XTTS-based pipeline
- Multi-protocol access β reachable over WebRTC (for real-time, low-bandwidth audio streaming), REST, and WebSockets, so a client can pick whichever transport fits its use case
- Hardware auto-detection β the server is documented as automatically detecting available hardware and adjusting behavior accordingly, rather than requiring manual configuration per GPU model
- Willow voice assistant backend β WIS is the inference backend the Willow open-source voice assistant hardware project connects to for turning spoken commands into text and, optionally, responses back into speech
Usage Examples: Three Ways to Use Willow Inference Server
These are concrete workflows built from Willow Inference Server's documented features above β not hypothetical use cases.
Install Willow Inference Server
Willow Inference Server installs via Docker Compose, and its source code is on GitHub. Links and commands below are from the official GitHub repository β always verify against that page directly, since install instructions can change between commits.
GitHub repository (source code, Apache 2.0)
Clone the repository
- Link:
git clone https://github.com/toverainc/willow-inference-server.git
GPU deployment
- Link:
docker compose upusing the repository'sdocker-compose.ymlfile
CPU-only deployment
- Link:
docker compose -f docker-compose-cpu.yml up(documented as significantly slower than GPU)
Willow voice assistant project
- Link:
- github.com/toverainc β the parent organization's other repositories, including the Willow hardware/firmware project WIS was originally built to serve
Willow Inference Server needs an NVIDIA CUDA-capable GPU for its documented performance figures β a GTX 1060 3 GB is the stated minimum, with roughly 6 GB VRAM recommended to run ASR and TTS together. A CPU-only Docker Compose path exists and is documented in the repository, but the project states CPU performance is well behind any of the listed GPUs; treat CPU-only mode as a functional fallback for testing, not a production deployment target.
Willow Inference Server Pricing: Is It Really Free?
Yes β Willow Inference Server has no paid tier. The GitHub repository has no pricing page, and the LICENSE.md file applies to the whole application. The license text is the Apache License, Version 2.0 β permissive, with a patent grant and no copyleft obligation to release your own modifications.
- No subscription, no paid tier, no usage limits imposed by Willow Inference Server itself
- No account or sign-up required to deploy or use the software
- Running WIS still requires your own GPU hardware and the electricity/hosting cost of running it β the software itself is free, but self-hosting is not zero-cost the way a hosted API can be
- Tovera also operates a hosted example server for the separate Willow voice assistant hardware project β this review covers the self-hosted, free-to-run WIS software specifically; verify directly with Tovera for any details about a separate hosted offering and whether it carries its own cost
Willow Inference Server vs. whisper.cpp
Willow Inference Server and whisper.cpp both run OpenAI's Whisper speech recognition models locally, and they get compared often because both avoid sending audio to a cloud API. The clearest differences are in deployment model and scope.
Aspect | Willow Inference Server | whisper.cpp |
|---|---|---|
| Scope | ASR + TTS server with WebRTC/REST/WS | ASR (transcription) only, no built-in server or TTS |
| Deployment | Docker Compose, GPU-focused | Compiled binary or library, CPU-friendly |
| Hardware target | NVIDIA CUDA GPU (3β6+ GB VRAM) | Runs on CPU; GPU acceleration optional |
| Real-time streaming | Built for it, via WebRTC | Possible with additional client-side work |
| Voice hardware integration | Built as Willow voice assistant backend | General-purpose library, no assistant hardware tie-in |
| Maintenance pace | Bursty; last commit February 2026 as of this review | Actively maintained with frequent commits |
If your priority is a ready-made server with WebRTC streaming and TTS built in, and you have a spare NVIDIA GPU, Willow Inference Server's feature set is the broader of the two. If your priority is the lightest possible CPU-friendly transcription library to embed in your own application, whisper.cpp is worth evaluating directly β see the whisper.cpp review for full details.
Who Should Use Willow Inference Server?
Whether Willow Inference Server fits depends on whether you have a compatible NVIDIA GPU, need both ASR and TTS in one server, and are comfortable with a project that ships updates in bursts rather than continuously.
Willow Inference Server vs. Other Speech Tools
Willow Inference Server is one of several self-hosted options for local speech recognition and synthesis. Here is how it sits next to other tools in that space β see the Local LLM Software Directory for the full catalog, and the dedicated Willow Inference Server vs. whisper.cpp comparison above for the closest head-to-head.
- whisper.cpp β a lightweight, CPU-friendly C/C++ port of OpenAI's Whisper for transcription only, with no built-in server, WebRTC, or TTS; see the dedicated comparison section above.
- Faster Whisper β a CTranslate2-based reimplementation of Whisper focused on inference speed; a library to embed rather than a ready-made ASR/TTS server like WIS.
- Piper TTS β a lightweight, CPU-friendly local text-to-speech engine; narrower in scope than WIS's combined ASR+TTS server but far lighter on hardware requirements.
- Coqui TTS β an open-source text-to-speech toolkit with voice cloning support, comparable in TTS scope to the XTTS pipeline WIS bundles, but distributed as a library rather than a WebRTC-ready server.
- Bark TTS β a generative text-to-speech model capable of non-speech audio (laughter, pauses); a different TTS engine choice than WIS's bundled XTTS pipeline.
- MacWhisper β a native macOS desktop app for Whisper transcription; a GUI desktop alternative for users who do not need WIS's server/WebRTC deployment model at all.
Common Mistakes When Evaluating Willow Inference Server
Most confusion about Willow Inference Server comes from outdated descriptions of its LLM support, unclear hardware expectations, or overestimating how actively it currently ships updates.
Frequently Asked Questions
What is Willow Inference Server?
Willow Inference Server (WIS, github.com/toverainc/willow-inference-server) is a free, open-source, self-hosted server that runs Whisper-based speech recognition and text-to-speech, reachable over WebRTC, REST, and WebSockets. It was built by Tovera as the backend for the Willow voice assistant hardware project.
Is Willow Inference Server free?
Yes. The GitHub repository has no pricing page, and its LICENSE.md file is the Apache License 2.0, applying to the whole application with no paid tier. You still need to provide and pay for your own GPU hardware to run it.
Does Willow Inference Server support LLM/chat inference?
No, not as of this review. A February 2026 commit removed general LLM/chat support from the project, with the maintainers stating: "We no longer support that. People can run ollama next to WIS." Willow Inference Server is currently an ASR/TTS-only server.
What GPU does Willow Inference Server need?
An NVIDIA CUDA-capable GPU. The project documents a GTX 1060 3 GB as the practical minimum, with roughly 6 GB of VRAM recommended to run all three default Whisper model sizes (base, medium, large-v2) plus TTS together. A CPU-only Docker Compose path exists but is documented as significantly slower.
How do I install Willow Inference Server?
Clone the GitHub repository and run docker compose up using the provided docker-compose.yml for GPU deployment, or docker compose -f docker-compose-cpu.yml up for the CPU-only path. Check the repository directly for current instructions before installing, since they can change between commits.
Is Willow Inference Server actively maintained?
Its commit history shows bursts of activity separated by multi-month gaps rather than continuous development β for example, roughly 14 months of no public commits between April 2024 and June 2025. As of this review, the most recent commit on the main branch is from February 2026, and the repository has no tagged releases. It is not archived, but this is a smaller, community-paced project rather than one with a continuous release cadence.
What is the relationship between Willow Inference Server and Willow?
Willow is a separate open-source voice assistant hardware/firmware project, also maintained by Tovera. Willow Inference Server is the backend software that processes speech for Willow devices, but it can also run standalone for other speech-to-text or text-to-speech use cases.
Can Willow Inference Server clone a custom voice for TTS?
Yes. Its bundled XTTS-based TTS pipeline supports creating a custom voice profile from a relatively small sample audio recording, per the project's documentation.
What license does Willow Inference Server use?
The Apache License, Version 2.0, per the official LICENSE.md file β permissive, with a patent grant and no requirement to open-source your own modifications.
Does Willow Inference Server support real-time streaming?
Yes, over WebRTC, which the project built specifically for low-latency, real-time audio use cases such as a voice assistant listening continuously for commands. REST and WebSocket endpoints are also available for non-streaming or simpler integrations.