Key Takeaways
- candle-vllm (github.com/EricLBuehler/candle-vllm) is a free, open-source, Rust-native inference and serving engine
- Built on Hugging Face's Candle machine learning framework by developer EricLBuehler
- MIT licensed, confirmed via the repository's license
- Draws its serving approach β continuous batching, efficient KV-cache handling β from the Python vLLM project's design, cited via the vLLM paper (arxiv.org/abs/2309.06180), but is a ground-up Rust reimplementation rather than a port of vLLM's Python code
- Runs an OpenAI-compatible API server on
http://localhost:2000by default, with an optional built-in web UI - Cross-platform on the same codebase: NVIDIA CUDA 11/12/13 on Linux, Apple Metal on macOS/Apple Silicon (16 GB+ unified memory recommended), and CPU fallback
- Roughly 728 GitHub stars as of this review
π In One Sentence
candle-vllm is a free, open-source (MIT) LLM serving engine written natively in Rust on top of Hugging Face's Candle framework, offering an OpenAI-compatible API server with vLLM-style serving behavior on both NVIDIA CUDA and Apple Metal.
π¬ In Plain Terms
candle-vllm is a program that runs an AI language model on your own computer or server and lets other apps talk to it using the same request format as OpenAI's API, so tools built for OpenAI's API can often point at it with a small config change. It is built entirely in Rust rather than Python, which its author frames as a safety and native-compilation advantage, and it copies the serving design of the well-known Python project vLLM without literally reusing vLLM's Python code.
πNote: This review is based on the project's own GitHub repository and README. It does not repeat the project's self-reported decode-speed benchmarks as independently verified figures, since PromptQuorum has not run its own hands-on performance testing of candle-vllm.
What Is candle-vllm?
candle-vllm is a free, open-source LLM inference and serving engine written entirely in Rust, built on top of Hugging Face's Candle machine learning framework, rather than on Python. It is developed by EricLBuehler and distributed on GitHub.
- Product type: a serving engine / API server you run yourself, not a hosted service or a desktop chat app
- Creator: EricLBuehler
- Underlying framework: Candle, Hugging Face's Rust-native machine learning framework β candle-vllm is built on top of it, not a fork or rename of Candle itself
- License: MIT
- Funding: no funding round, investor, or commercial backing was found for this review β treat it as an independently maintained, community open-source project
- Scale: roughly 728 GitHub stars, around 90 forks, and active recent commit history as of this review
What Does candle-vllm Actually Do?
candle-vllm loads an open LLM and serves it over an OpenAI-compatible HTTP API, applying serving optimizations similar in spirit to the Python vLLM project β continuous batching, efficient memory management for the attention cache, and quantization support β implemented natively in Rust.
- OpenAI-compatible API server: listens on
http://localhost:2000by default, so clients built against the OpenAI API format can often connect with a base-URL change - Optional built-in web UI: a ChatGPT-style chat interface can be launched alongside the API, on a port offset from the API port, per the project README
- Broad model-family support per the README: Qwen, Llama, Mistral, Phi3/Phi4, DeepSeek (including V3/R1), GLM, Yi, StableLM, Gemma, QwQ, and vision-language models
- Model format support: SafeTensors, GGUF, GPTQ, AWQ, Marlin, MXFP4, and NVFP4, per the project documentation
- Performance features described in the README: Flash Attention, a FlashInfer backend option, CUDA Graphs, continuous batching, and prefix caching (KV-cache reuse across requests)
- Memory-efficiency feature: "TurboQuant" KV-cache compression, which the README describes as achieving substantial context-length extension via 2β4 bit cache quantization variants
- Multi-GPU and multi-node support: tensor-parallel multi-GPU inference (multi-process mode recommended per the README) and TCP-based multi-node coordination without an MPI dependency
- Tool integration: documented Model Context Protocol (MCP) support and OpenAI-compatible tool/function calling
Platform, Pricing, and Licensing
Platform
- What candle-vllm states:
- A self-hosted server process, run from the command line or as a Rust library. Cross-platform: Linux (CUDA 11/12/13), macOS/Apple Silicon (Metal), and CPU fallback, same codebase across all three.
Cost
- What candle-vllm states:
- Free and open source, with no paid tier. You supply your own compute and download open model weights separately (e.g., from Hugging Face).
Licensing
- What candle-vllm states:
- MIT license.
Install method
- What candle-vllm states:
- Per the project README: a one-line shell installer for DEB/binary installs, a source build via
cargo installwith CUDA/Metal/CPU feature flags, or a Docker image with configurable CUDA/SM version.
Verify the current recommended install method and CUDA/driver compatibility on github.com/EricLBuehler/candle-vllm before running any command, since install instructions and supported CUDA versions can change between releases.
Install candle-vllm
candle-vllm installs free via a shell script, cargo (source build), or Docker, and its source code is on GitHub.
Source | Link |
|---|---|
| GitHub repository (source code, MIT) | github.com/EricLBuehler/candle-vllm |
| Candle framework (underlying ML framework) | github.com/huggingface/candle |
| Local LLM Software Directory entry | Local LLM Software Directory |
candle-vllm requires a terminal and either a prebuilt binary/Docker image or a Rust toolchain (via cargo) to build from source β there is no GUI installer, since it is a server/API component rather than an end-user desktop app.
candle-vllm vs. Python vLLM
candle-vllm is not a fork or port of the original Python vLLM project β it is a separate, ground-up Rust implementation that follows a similar serving design (continuous batching, efficient KV-cache management) and cites the vLLM paper as its reference approach.
Language/runtime
- candle-vllm:
- Rust, no Python runtime needed to run the server
- Python vLLM:
- Python, requires a Python environment
Underlying framework
- candle-vllm:
- Hugging Face Candle (Rust ML framework)
- Python vLLM:
- PyTorch
API compatibility
- candle-vllm:
- OpenAI-compatible HTTP API
- Python vLLM:
- OpenAI-compatible HTTP API
Platform support
- candle-vllm:
- CUDA, Apple Metal, CPU β same codebase
- Python vLLM:
- Primarily CUDA/ROCm-focused, no native Apple Metal backend
Ecosystem maturity
- candle-vllm:
- Smaller community (~728 stars), newer project
- Python vLLM:
- Large, widely deployed in production LLM-serving stacks
This comparison reflects each project's own documentation, not independent PromptQuorum benchmarking of either project. Verify current feature parity and performance directly with each project before choosing.
Who Should Use candle-vllm?
candle-vllm fits developers who want an OpenAI-compatible local serving engine without a Python dependency, and who value running the identical codebase across CUDA and Apple Metal.
What candle-vllm Is Not Good For
candle-vllm is not a good fit if you need the largest existing ecosystem of integrations, a no-terminal GUI installer, or independently verified performance numbers before you commit.
- Not the most mature ecosystem β Python vLLM has a much larger community, more third-party integrations, and more production track record as of this review
- Not a GUI or one-click desktop install β it is a server/API component set up via shell script, cargo build, or Docker
- Not a drop-in replacement for every Python-vLLM-specific plugin or workflow β some tooling built specifically around Python vLLM's internals has no direct Rust equivalent here
- Not independently benchmarked by PromptQuorum β this review does not confirm the project's self-reported decode-speed figures with its own hands-on testing
- Not backed by a funding round or company that this review could verify β treat it as an independently maintained, community-supported project
Common Mistakes When Evaluating candle-vllm
Most confusion about candle-vllm comes from assuming it is a Python-to-Rust port of vLLM, or assuming it is written in Python because of the "vllm" in its name.
Competitors and Alternatives
Within local LLM inference and serving engines, candle-vllm is most comparable to other self-hosted, OpenAI-compatible serving platforms β its main differentiator is being Rust-native rather than Python-based, with native Apple Metal support alongside CUDA.
Tool | Best known for | Link |
|---|---|---|
| LMDeploy | Python-based LLM serving toolkit from the InternLM team, with quantization and high-throughput inference | LMDeploy Review |
| NVIDIA Dynamo | Distributed inference-serving framework for large-scale, multi-node LLM deployments | Dynamo Review |
| LoRAX | Multi-LoRA-adapter serving framework for running many fine-tuned variants from one base model | LoRAX Review |
This list reflects tools commonly compared to candle-vllm within the inference/serving-engine segment, not an independent PromptQuorum ranking β verify each tool's current feature set before choosing.
Frequently Asked Questions
What is candle-vllm?
candle-vllm (github.com/EricLBuehler/candle-vllm) is a free, open-source, Rust-native LLM inference and serving engine built on Hugging Face's Candle framework, offering an OpenAI-compatible API server.
Is candle-vllm written in Python?
No. Despite the "vllm" in its name, candle-vllm is written entirely in Rust on top of Hugging Face's Candle machine learning framework. No Python runtime is required to run the server.
Is candle-vllm the same project as vLLM?
No. It is a separate, ground-up Rust reimplementation that follows a similar serving design (continuous batching, KV-cache management) to the Python vLLM project and cites the vLLM paper, but does not reuse vLLM's Python code.
Is candle-vllm free?
Yes, it is free and open source under the MIT license, with no paid tier.
What platforms does candle-vllm support?
The same codebase runs on NVIDIA CUDA (11/12/13) on Linux, Apple Metal on macOS/Apple Silicon (16 GB+ unified memory recommended), and CPU as a fallback.
How do I install candle-vllm?
Per the project README: a one-line shell installer for DEB/binary installs, a source build via cargo install with the appropriate CUDA/Metal/CPU feature flags, or a Docker image.
Does candle-vllm support quantized models?
Yes. It supports SafeTensors, GGUF, GPTQ, AWQ, Marlin, MXFP4, and NVFP4 model formats, per the project documentation.
Who created candle-vllm?
Developer EricLBuehler created and maintains candle-vllm, built on top of Hugging Face's Candle machine learning framework.
Does candle-vllm have a web interface?
It can optionally launch a built-in, ChatGPT-style web UI alongside its API server, on a port offset from the API port, per the project README.
Has PromptQuorum independently benchmarked candle-vllm's performance claims?
No. This review is based on the project's own GitHub repository and README, not hands-on performance testing by PromptQuorum.