Points clés
- llama.cpp: Fastest single-token latency (lowest ms/token). Best for interactive chat. Minimal dependencies.
- Ollama: Easiest to use. One command, auto-download models. Trade-off: 5–10% slower throughput than llama.cpp.
- vLLM: Highest throughput (tokens/sec) on batched requests. Best for production API servers. Steeper learning curve.
- Single-user chat: llama.cpp or Ollama (nearly identical speed).
- Multi-user API: vLLM (3–5× higher throughput).
- Casual use: Ollama (simplicity wins).
- All three produce identical model outputs—speed/feature differences only.
- Can run all three simultaneously on same machine (different ports). They don't conflict.
Speed Comparison Benchmarks
Benchmarked on RTX 4090, Llama 3 70B Q4, single request (batch=1), April 2026:
| Backend | Tokens/sec | ms/token | Memory |
|---|---|---|---|
| — | — | — | — |
| — | — | — | — |
| — | — | — | — |
| — | — | — | — |
Feature Comparison Table
| Feature | llama.cpp | Ollama | vLLM |
|---|---|---|---|
| — | — | — | — |
| — | — | — | — |
| — | — | — | — |
| — | — | — | — |
| — | — | — | — |
| — | — | — | — |
| — | — | — | — |
| — | — | — | — |
| — | — | — | — |
| — | — | — | — |
| — | — | — | — |
Batching & Throughput
This is where vLLM dominates:
- llama.cpp: No native batching. One request at a time. Latency: 27ms/token. Throughput: 36 tok/s.
- Ollama: Single-batch only. Cannot process 2+ requests in parallel. Same throughput as llama.cpp.
- vLLM: Native batching (batch=32 default). Processes 32 requests concurrently. Throughput: 250+ tok/s on same RTX 4090.
- vLLM's advantage multiplies with concurrent users. For API servers with 10+ users: vLLM is mandatory.
Setup Complexity
llama.cpp: Compile from source or download binary. Manual model file management. 30 min setup.
Ollama: `brew install ollama` or download installer. `ollama run mistral`. 5 min setup.
vLLM: `pip install vllm`, then `python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-2-70b-hf`. 15 min setup (Python + dependencies).
Winner for simplicity: Ollama.
API Compatibility
llama.cpp: Custom REST API (not OpenAI-compatible). Requires custom client code.
Ollama: OpenAI-compatible API (via `ollama serve` + client library). Works with most IDE extensions.
vLLM: OpenAI-compatible API (native `/v1/chat/completions`). Best compatibility.
For IDE integration (VS Code, Cursor): Ollama or vLLM. Skip llama.cpp.
When to Use Each
llama.cpp: Minimal dependencies, raw speed. Use if building custom inference engine.
Ollama: Everything-included simplicity. Use for chat UI + personal use.
vLLM: Production API server. Use for multi-user deployments, high throughput requirements.
Common Misconceptions
- llama.cpp is always faster. Only true for single-token latency. vLLM wins on throughput (batch requests).
- Ollama is slow. Not compared to llama.cpp—only 5–10% slower, a negligible difference for interactive chat.
- You must choose one. False. Can run all three simultaneously. Use Ollama for chat, vLLM for API.
FAQ
Which should I use as a beginner?
Ollama. One command, automatic model downloads, clean interface.
Which is fastest?
For single request: llama.cpp (~3% faster than Ollama). For 10 concurrent requests: vLLM (~7× faster).
Can I use llama.cpp instead of Ollama?
Yes, but more setup. Speed gain is negligible (3–5%) for most users.
Is vLLM production-ready?
Yes. Used in real deployments. Steeper learning curve, but worth it for high throughput.
Can I switch backends without retraining?
Yes. All three use GGUF models. Convert once, run on any backend.
Which backend is most stable?
Ollama (simple, fewer bugs). llama.cpp is stable too. vLLM updates frequently (more features, occasional breaking changes).
Sources
- llama.cpp official GitHub and benchmarks
- Ollama official documentation
- vLLM official documentation and GitHub
- PromptQuorum April 2026 inference benchmarks (RTX 4090)