Key Takeaways
- Free and Apache 2.0-licensed open-source, originated at the UC Berkeley Sky Computing Lab
- PagedAttention manages the KV cache in page-sized, non-contiguous blocks to reduce wasted GPU memory
- Continuous batching processes many concurrent requests instead of one fixed batch at a time
- Ships a built-in OpenAI-compatible API server, started with the
vllm servecommand - Supports quantization formats including AWQ, GPTQ, and FP8
- Supports tensor-parallel and pipeline-parallel serving across multiple GPUs
- Primary, best-supported hardware is NVIDIA GPUs; AMD, Intel, and other backends exist but with narrower coverage
- Not a single-user desktop app β no graphical installer, and not built around CPU-only or Apple Silicon hardware the way llama.cpp and Ollama are
π In One Sentence
vLLM is a free, Apache 2.0-licensed inference and serving library, originated at the UC Berkeley Sky Computing Lab, that uses PagedAttention and continuous batching to serve many concurrent LLM requests efficiently from a GPU, and ships a built-in OpenAI-compatible API server.
π¬ In Plain Terms
Instead of a desktop chat app, vLLM is server software: you point it at a model and it exposes an API that many people or applications can call at once, packing GPU memory more efficiently than a simple one-request-at-a-time setup.
πNote: This article is based on vLLM's official GitHub repository and public documentation, not independent benchmarking. Specific throughput or latency figures are not included because they were not independently measured for this article and vary heavily by GPU, model, batch size, and vLLM version.
What Is vLLM?
vLLM is a free, Apache 2.0-licensed library and server for running large language model inference at scale. It originated as a research project in the Sky Computing Lab at UC Berkeley and has since grown into one of the most widely used open-source LLM serving engines, with contributions from thousands of developers across academic and industry organizations. Unlike tools built primarily for a single user chatting with a model on their own machine, vLLM is designed to serve many concurrent requests β from multiple users or applications β off of shared GPU capacity as efficiently as possible.
- Originated in the Sky Computing Lab at UC Berkeley, now a community-governed open-source project
- Apache 2.0-licensed: source code is publicly available for use, modification, and redistribution under the license terms
- Loads models in the Hugging Face Transformers-compatible format, giving it broad architecture coverage β Llama, Mistral, Qwen, DeepSeek, and many other model families β without needing a separate model-conversion step for most models
- Built around serving many concurrent requests efficiently, not just running a single conversation quickly
- One of the most widely referenced open-source LLM serving projects on GitHub
What Is PagedAttention, and Why Does It Matter?
PagedAttention is the memory-management technique vLLM is best known for. During generation, a transformer model stores an attention key-value (KV) cache for every token in every active request β normally this cache is allocated as one large contiguous block per request, sized for the request's maximum possible length, which wastes GPU memory whenever a request finishes early or is shorter than the reserved maximum. PagedAttention instead splits the KV cache into small, fixed-size blocks (pages) that can be allocated non-contiguously and shared between requests, borrowing an idea from how operating systems manage virtual memory.
- Reduces memory waste from over-reserving KV-cache space for requests that turn out shorter than their maximum length
- Allows memory blocks to be shared between requests that share a common prefix, such as the same system prompt
- Frees the GPU to hold more concurrent requests' KV cache in the same amount of memory versus a naive contiguous-allocation approach
- Works together with continuous batching, which lets vLLM add and remove requests from an in-flight batch as they arrive and complete, rather than waiting for a fixed batch to fully finish before starting the next one
What Hardware Does vLLM Need?
vLLM's primary and best-supported target is NVIDIA GPUs with CUDA, and most production deployments run on NVIDIA hardware. The project also documents support for additional backends, but coverage and performance are not equal across all of them.
NVIDIA GPUs (CUDA)
- Details:
- The primary, most mature target. Tensor-parallel and pipeline-parallel serving across multiple NVIDIA GPUs is well documented and widely used in production.
AMD GPUs (ROCm)
- Details:
- Documented as a supported backend for AMD hardware via ROCm, with narrower real-world adoption and community coverage than the CUDA path.
Intel GPUs and Gaudi accelerators
- Details:
- Additional backends documented by the project for Intel hardware; treat as a smaller, less battle-tested deployment path than NVIDIA GPUs.
Google TPUs
- Details:
- A documented backend for Google Cloud TPU hardware, aimed at teams already running on that infrastructure.
CPU (x86 / ARM / PowerPC)
- Details:
- A CPU-only backend exists, but it is not vLLM's target use case β the project is built around GPU serving, and CPU execution is documented as substantially slower than GPU backends.
Apple Silicon (Mac)
- Details:
- Not a first-class, officially maintained path. Community-maintained projects (such as a Metal backend plugin) add partial Apple Silicon support, but coverage and maturity trail vLLM's NVIDIA GPU support by a wide margin.
If your goal is running a model on a single Mac or a CPU-only machine, vLLM is not the tool built for that β llama.cpp and tools built on it, like Ollama and LM Studio, target CPU and Apple Silicon hardware directly and are the better fit for that scenario.
What Quantization Formats Does vLLM Support?
vLLM supports serving models at reduced numeric precision to lower memory use and, in many cases, increase throughput, using several established quantization formats rather than a single proprietary one.
AWQ
- Details:
- Activation-aware Weight Quantization, a widely used 4-bit weight-quantization method with pre-quantized models published by the community on Hugging Face.
GPTQ
- Details:
- A post-training quantization method commonly distributed as pre-quantized model checkpoints, also typically run at 4-bit precision.
FP8
- Details:
- 8-bit floating-point precision, supported on newer NVIDIA GPU generations that include hardware FP8 support, trading some precision for lower memory use and faster execution than FP16/BF16.
INT8 / INT4
- Details:
- Lower-precision integer quantization paths documented by the project alongside AWQ and GPTQ for further memory reduction.
This article does not include independently measured quality-loss figures for each format β those vary by model architecture and task, so comparing outputs from a couple of formats on your own prompts is the most reliable way to judge the trade-off for your workload.
What Does the vLLM OpenAI-Compatible Server Provide?
Running vllm serve starts an HTTP server that implements the OpenAI API protocol, so applications and SDKs already built against the OpenAI API can often point at a self-hosted vLLM instance with only a base-URL and model-name change.
- OpenAI-compatible chat completions and completions endpoints, usable as a drop-in replacement for OpenAI-API-based client code
- Configurable host and port (the server listens on
http://localhost:8000by default) - Engine flags for tensor-parallel size, GPU memory utilization target, and quantization format, set at server startup
- Support for serving multiple LoRA adapters against a single loaded base model
- Structured-output and function/tool-calling support for compatible request formats
How Do You Install and Run vLLM?
vLLM is distributed as a Python package and typically installed with pip into a Python environment with an NVIDIA GPU and compatible CUDA drivers available.
- 1Confirm you have a supported NVIDIA GPU with current CUDA drivers installed (or check the project's docs for AMD/Intel/TPU-specific install instructions if you are targeting one of those backends).
- 2Create a Python virtual environment, then install vLLM:
pip install vllm. - 3Start the OpenAI-compatible server with a model from Hugging Face, for example:
vllm serve meta-llama/Llama-3.1-8B-Instruct. - 4For a pre-quantized model, pass the matching flag, for example:
vllm serve TheBloke/Llama-2-13B-AWQ --quantization awq. - 5For multi-GPU serving, add a tensor-parallel flag, for example:
vllm serve <model> --tensor-parallel-size 2to split the model across two GPUs. - 6By default the server listens on
http://localhost:8000; send a request to its/v1/chat/completionsendpoint with any OpenAI-API-compatible client library, orcurl. - 7Point existing OpenAI-API client code at your self-hosted server by changing only its base URL and model name.
Do I need a GPU to run vLLM?
For anything beyond testing, yes β vLLM's primary and best-supported target is NVIDIA GPUs. A CPU-only backend exists but is documented as substantially slower and is not the project's focus.
Can I run vLLM with a quantized model?
Yes β vLLM supports formats including AWQ, GPTQ, and FP8, and many pre-quantized models in these formats are published on Hugging Face and can be served with the matching --quantization flag.
How Does vLLM Compare to Ollama and LM Studio?
Ollama and LM Studio target a different problem than vLLM does: getting one person a model to chat with on their own machine, quickly and simply. vLLM targets serving many concurrent users or applications off shared GPU capacity as efficiently as possible. The two categories of tool are not close substitutes for most use cases.
- Ollama and LM Studio are commonly built around llama.cpp or similar engines and the GGUF model format, optimized for single-user use on consumer hardware including CPU-only machines and Apple Silicon
- vLLM is built around PagedAttention and continuous batching, optimized for high-concurrency GPU serving rather than single-user responsiveness on modest hardware
- Ollama installs in one command with no GPU required; vLLM expects a Python environment, an NVIDIA GPU in most deployments, and command-line configuration
- LM Studio adds a graphical desktop chat interface; vLLM has no graphical interface β it is accessed via its OpenAI-compatible API or command-line flags
- Both vLLM and llama.cpp-based tools can expose an OpenAI-compatible API, so front-end tooling built for that API can often work with either
How Does vLLM Compare to TGI and TensorRT-LLM?
vLLM, Hugging Face's Text Generation Inference (TGI), and NVIDIA's TensorRT-LLM all target the same broad job β production LLM serving at scale β but with different designs and trade-offs.
vLLM
- Details:
- Apache 2.0-licensed, Python-based, built around PagedAttention and continuous batching. Loads Hugging Face Transformers-compatible models directly, with broad architecture coverage and multi-vendor GPU backend support (NVIDIA primary; AMD, Intel, TPU documented).
TGI
- Details:
- Hugging Face's own serving engine, Apache 2.0-licensed, also supporting continuous batching and multiple quantization formats. Tightly integrated with the Hugging Face Hub and ecosystem.
TensorRT-LLM
- Details:
- NVIDIA's engine, built specifically for NVIDIA GPUs. Models are compiled ahead of time into an optimized TensorRT engine for the target GPU, which can yield strong performance on that specific hardware at the cost of a compilation step and less cross-hardware flexibility than vLLM or TGI.
This article has not independently benchmarked these three engines against each other and does not claim one is universally faster β throughput depends heavily on the model, hardware, batch characteristics, and each engine's version. See the enterprise inference server guide for a more detailed comparison covering deployment and licensing considerations for all three.
Who Should Use vLLM?
vLLM fits teams serving a model to many concurrent users or applications on GPU infrastructure, not people looking for the fastest way to chat with a model on their own computer.
vLLM vs. Alternatives at a Glance
These tools sit at different points on the single-user-versus-production-serving spectrum.
vLLM
- Interface & setup:
- Python package installed via pip; OpenAI-compatible API server started with
vllm serve. Expects an NVIDIA GPU and CUDA in most deployments. - Best for:
- High-throughput, multi-user GPU serving in production.
Ollama
- Interface & setup:
- CLI and REST API, commonly reported to run on llama.cpp as its backend on most platforms. One command installs it; one command pulls and runs a model.
- Best for:
- The fastest path to a running local model for a single user, no build step or GPU required.
LM Studio
- Interface & setup:
- Graphical desktop app for Mac, Windows, and Linux. Download, install, then browse and download a model from inside the app.
- Best for:
- Non-technical users who want a point-and-click local chat app.
llama.cpp
- Interface & setup:
- CLI, built-in web UI, and OpenAI-compatible API via llama-server. Build from source or use a pre-built binary; runs on CPU or GPU.
- Best for:
- Direct engine-level control, embedded/edge deployment, and CPU or Apple Silicon hardware.
This article has not independently benchmarked speed or output quality across these tools and does not claim one is technically superior β the comparison above covers documented architecture, setup, and access-model facts only. For per-hardware throughput numbers, see the llama.cpp vs. Ollama vs. vLLM comparison and the enterprise inference server guide.
What Does This Article Not Cover?
This is an explainer built from vLLM's public documentation and repository, not a hands-on benchmark report.
- No independently measured throughput, latency, or requests-per-second figures β these depend heavily on GPU, model, batch composition, and vLLM version
- No independently verified quality-loss percentages for specific quantization formats β these vary by model architecture and task
- No line-by-line security audit of the vLLM codebase β it is open-source and Apache 2.0-licensed, so the code itself is available for review
- No full coverage of every supported hardware backend, engine flag, or deployment orchestration option (Kubernetes, cloud-specific setups) β this article focuses on the concepts and flags most teams evaluate first
- No coverage of commercial support arrangements or managed vLLM hosting offerings, since vLLM itself is a community open-source project rather than a vendor product with a support contract
Common Mistakes When Trying vLLM
Most friction with vLLM comes from treating it like a single-user desktop tool rather than production server software.
Frequently Asked Questions
What is vLLM?
vLLM is a free, Apache 2.0-licensed library and server for high-throughput LLM inference, originated at the UC Berkeley Sky Computing Lab. It uses PagedAttention and continuous batching to serve many concurrent requests efficiently from a GPU.
Is vLLM free?
Yes. vLLM is free, open-source software released under the Apache 2.0 license, with no subscription or account requirement to run it yourself.
What is PagedAttention?
PagedAttention is vLLM's technique for managing the attention KV cache in small, page-sized, non-contiguous blocks instead of one large contiguous allocation per request, reducing wasted GPU memory and allowing memory to be shared between requests with a common prefix.
Does vLLM need a GPU?
For any real workload, yes β vLLM's primary and best-supported target is NVIDIA GPUs. A CPU-only backend exists but is documented as substantially slower and is not the project's focus, and Apple Silicon support is limited to community-maintained add-ons rather than a first-class path.
What quantization formats does vLLM support?
vLLM supports several formats including AWQ, GPTQ, FP8, and INT8/INT4, with many pre-quantized models in these formats published on Hugging Face.
Is vLLM better than Ollama?
"Better" depends on the job: vLLM is built for high-concurrency production GPU serving, while Ollama is built for the fastest path to a single-user local model with no GPU required. They are not close substitutes for most use cases β see the comparison table above.
Can vLLM serve models across multiple GPUs?
Yes. vLLM supports tensor-parallel and pipeline-parallel serving across multiple GPUs, configurable with flags like --tensor-parallel-size at server startup.
Does vLLM have an OpenAI-compatible API?
Yes. Running vllm serve starts a server implementing the OpenAI API protocol, so many applications built for the OpenAI API can point at a self-hosted vLLM instance with only a base-URL and model-name change.
How is vLLM different from TensorRT-LLM?
TensorRT-LLM is NVIDIA's engine, which compiles models ahead of time into an optimized engine for a specific NVIDIA GPU. vLLM loads Hugging Face Transformers-compatible models directly without an ahead-of-time compilation step and documents backend support beyond NVIDIA GPUs alone, trading some hardware-specific optimization for broader flexibility and faster iteration.