Skip to main content
PromptQuorumBuilt for humans. Structured for AI.
Home/Power Local LLM
Overview & Reference

vLLM Explained: High-Throughput LLM Serving with PagedAttention (2026)

·13 min readBy Hans Kuepper · Founder of PromptQuorum · Discovery engine for open-weight & open-source AI

vLLM is a free, open-source (Apache 2.0) library for high-throughput LLM inference and serving, originated by the Sky Computing Lab at UC Berkeley and now maintained by a large open-source community. Its key technical differentiator is PagedAttention, which manages the attention KV cache in non-contiguous, page-sized blocks — similar in spirit to how an operating system manages virtual memory — so a GPU can hold far more concurrent request state without over-allocating memory for each one. Combined with continuous batching, this lets vLLM serve many simultaneous users from one GPU (or a tensor-parallel multi-GPU setup) with less wasted memory than a naive serving loop. vLLM ships a built-in OpenAI-compatible API server (vllm serve), supports quantization formats including AWQ, GPTQ, and FP8, and is built for production and multi-tenant serving — not for the single-user, point-and-click local-chat use case that tools like Ollama and LM Studio target.

Key Takeaways

  • Free and Apache 2.0-licensed open-source, originated at the UC Berkeley Sky Computing Lab
  • PagedAttention manages the KV cache in page-sized, non-contiguous blocks to reduce wasted GPU memory
  • Continuous batching processes many concurrent requests instead of one fixed batch at a time
  • Ships a built-in OpenAI-compatible API server, started with the vllm serve command
  • Supports quantization formats including AWQ, GPTQ, and FP8
  • Supports tensor-parallel and pipeline-parallel serving across multiple GPUs
  • Primary, best-supported hardware is NVIDIA GPUs; AMD, Intel, and other backends exist but with narrower coverage
  • Not a single-user desktop app — no graphical installer, and not built around CPU-only or Apple Silicon hardware the way llama.cpp and Ollama are

📍 In One Sentence

vLLM is a free, Apache 2.0-licensed inference and serving library, originated at the UC Berkeley Sky Computing Lab, that uses PagedAttention and continuous batching to serve many concurrent LLM requests efficiently from a GPU, and ships a built-in OpenAI-compatible API server.

💬 In Plain Terms

Instead of a desktop chat app, vLLM is server software: you point it at a model and it exposes an API that many people or applications can call at once, packing GPU memory more efficiently than a simple one-request-at-a-time setup.

📌Note: This article is based on vLLM's official GitHub repository and public documentation, not independent benchmarking. Specific throughput or latency figures are not included because they were not independently measured for this article and vary heavily by GPU, model, batch size, and vLLM version.

Read in:
🇺🇸en🇩🇪de🇫🇷fr🇯🇵ja🇨🇳zh🇪🇸es🇧🇷pt🇸🇦ar🇰🇷ko

vLLM is a free, Apache 2.0-licensed library for LLM inference and serving that originated in the Sky Computing Lab at UC Berkeley and is now maintained by a large open-source community. Its core technical contribution is PagedAttention, a memory-management technique for the attention key-value (KV) cache that lets a GPU serve far more concurrent requests from the same memory than naive attention implementations, combined with continuous batching that keeps a GPU busy across many simultaneous requests instead of processing them one fixed batch at a time. vLLM ships a built-in OpenAI-compatible API server and targets production, multi-user GPU serving — it is a different kind of tool than single-user desktop apps like Ollama or LM Studio.

vLLM Explained: High-Throughput LLM Serving with PagedAttention (2026)

What Is vLLM?

vLLM is a free, Apache 2.0-licensed library and server for running large language model inference at scale. It originated as a research project in the Sky Computing Lab at UC Berkeley and has since grown into one of the most widely used open-source LLM serving engines, with contributions from thousands of developers across academic and industry organizations. Unlike tools built primarily for a single user chatting with a model on their own machine, vLLM is designed to serve many concurrent requests — from multiple users or applications — off of shared GPU capacity as efficiently as possible.

  • Originated in the Sky Computing Lab at UC Berkeley, now a community-governed open-source project
  • Apache 2.0-licensed: source code is publicly available for use, modification, and redistribution under the license terms
  • Loads models in the Hugging Face Transformers-compatible format, giving it broad architecture coverage — Llama, Mistral, Qwen, DeepSeek, and many other model families — without needing a separate model-conversion step for most models
  • Built around serving many concurrent requests efficiently, not just running a single conversation quickly
  • One of the most widely referenced open-source LLM serving projects on GitHub

What Is PagedAttention, and Why Does It Matter?

PagedAttention is the memory-management technique vLLM is best known for. During generation, a transformer model stores an attention key-value (KV) cache for every token in every active request — normally this cache is allocated as one large contiguous block per request, sized for the request's maximum possible length, which wastes GPU memory whenever a request finishes early or is shorter than the reserved maximum. PagedAttention instead splits the KV cache into small, fixed-size blocks (pages) that can be allocated non-contiguously and shared between requests, borrowing an idea from how operating systems manage virtual memory.

  • Reduces memory waste from over-reserving KV-cache space for requests that turn out shorter than their maximum length
  • Allows memory blocks to be shared between requests that share a common prefix, such as the same system prompt
  • Frees the GPU to hold more concurrent requests' KV cache in the same amount of memory versus a naive contiguous-allocation approach
  • Works together with continuous batching, which lets vLLM add and remove requests from an in-flight batch as they arrive and complete, rather than waiting for a fixed batch to fully finish before starting the next one

What Hardware Does vLLM Need?

vLLM's primary and best-supported target is NVIDIA GPUs with CUDA, and most production deployments run on NVIDIA hardware. The project also documents support for additional backends, but coverage and performance are not equal across all of them.

NVIDIA GPUs (CUDA)

Details:
The primary, most mature target. Tensor-parallel and pipeline-parallel serving across multiple NVIDIA GPUs is well documented and widely used in production.

AMD GPUs (ROCm)

Details:
Documented as a supported backend for AMD hardware via ROCm, with narrower real-world adoption and community coverage than the CUDA path.

Intel GPUs and Gaudi accelerators

Details:
Additional backends documented by the project for Intel hardware; treat as a smaller, less battle-tested deployment path than NVIDIA GPUs.

Google TPUs

Details:
A documented backend for Google Cloud TPU hardware, aimed at teams already running on that infrastructure.

CPU (x86 / ARM / PowerPC)

Details:
A CPU-only backend exists, but it is not vLLM's target use case — the project is built around GPU serving, and CPU execution is documented as substantially slower than GPU backends.

Apple Silicon (Mac)

Details:
Not a first-class, officially maintained path. Community-maintained projects (such as a Metal backend plugin) add partial Apple Silicon support, but coverage and maturity trail vLLM's NVIDIA GPU support by a wide margin.

If your goal is running a model on a single Mac or a CPU-only machine, vLLM is not the tool built for that — llama.cpp and tools built on it, like Ollama and LM Studio, target CPU and Apple Silicon hardware directly and are the better fit for that scenario.

What Quantization Formats Does vLLM Support?

vLLM supports serving models at reduced numeric precision to lower memory use and, in many cases, increase throughput, using several established quantization formats rather than a single proprietary one.

AWQ

Details:
Activation-aware Weight Quantization, a widely used 4-bit weight-quantization method with pre-quantized models published by the community on Hugging Face.

GPTQ

Details:
A post-training quantization method commonly distributed as pre-quantized model checkpoints, also typically run at 4-bit precision.

FP8

Details:
8-bit floating-point precision, supported on newer NVIDIA GPU generations that include hardware FP8 support, trading some precision for lower memory use and faster execution than FP16/BF16.

INT8 / INT4

Details:
Lower-precision integer quantization paths documented by the project alongside AWQ and GPTQ for further memory reduction.

This article does not include independently measured quality-loss figures for each format — those vary by model architecture and task, so comparing outputs from a couple of formats on your own prompts is the most reliable way to judge the trade-off for your workload.

What Does the vLLM OpenAI-Compatible Server Provide?

Running vllm serve starts an HTTP server that implements the OpenAI API protocol, so applications and SDKs already built against the OpenAI API can often point at a self-hosted vLLM instance with only a base-URL and model-name change.

  • OpenAI-compatible chat completions and completions endpoints, usable as a drop-in replacement for OpenAI-API-based client code
  • Configurable host and port (the server listens on http://localhost:8000 by default)
  • Engine flags for tensor-parallel size, GPU memory utilization target, and quantization format, set at server startup
  • Support for serving multiple LoRA adapters against a single loaded base model
  • Structured-output and function/tool-calling support for compatible request formats

How Do You Install and Run vLLM?

vLLM is distributed as a Python package and typically installed with pip into a Python environment with an NVIDIA GPU and compatible CUDA drivers available.

  1. 1
    Confirm you have a supported NVIDIA GPU with current CUDA drivers installed (or check the project's docs for AMD/Intel/TPU-specific install instructions if you are targeting one of those backends).
  2. 2
    Create a Python virtual environment, then install vLLM: pip install vllm.
  3. 3
    Start the OpenAI-compatible server with a model from Hugging Face, for example: vllm serve meta-llama/Llama-3.1-8B-Instruct.
  4. 4
    For a pre-quantized model, pass the matching flag, for example: vllm serve TheBloke/Llama-2-13B-AWQ --quantization awq.
  5. 5
    For multi-GPU serving, add a tensor-parallel flag, for example: vllm serve <model> --tensor-parallel-size 2 to split the model across two GPUs.
  6. 6
    By default the server listens on http://localhost:8000; send a request to its /v1/chat/completions endpoint with any OpenAI-API-compatible client library, or curl.
  7. 7
    Point existing OpenAI-API client code at your self-hosted server by changing only its base URL and model name.

Do I need a GPU to run vLLM?

For anything beyond testing, yes — vLLM's primary and best-supported target is NVIDIA GPUs. A CPU-only backend exists but is documented as substantially slower and is not the project's focus.

Can I run vLLM with a quantized model?

Yes — vLLM supports formats including AWQ, GPTQ, and FP8, and many pre-quantized models in these formats are published on Hugging Face and can be served with the matching --quantization flag.

How Does vLLM Compare to Ollama and LM Studio?

Ollama and LM Studio target a different problem than vLLM does: getting one person a model to chat with on their own machine, quickly and simply. vLLM targets serving many concurrent users or applications off shared GPU capacity as efficiently as possible. The two categories of tool are not close substitutes for most use cases.

  • Ollama and LM Studio are commonly built around llama.cpp or similar engines and the GGUF model format, optimized for single-user use on consumer hardware including CPU-only machines and Apple Silicon
  • vLLM is built around PagedAttention and continuous batching, optimized for high-concurrency GPU serving rather than single-user responsiveness on modest hardware
  • Ollama installs in one command with no GPU required; vLLM expects a Python environment, an NVIDIA GPU in most deployments, and command-line configuration
  • LM Studio adds a graphical desktop chat interface; vLLM has no graphical interface — it is accessed via its OpenAI-compatible API or command-line flags
  • Both vLLM and llama.cpp-based tools can expose an OpenAI-compatible API, so front-end tooling built for that API can often work with either

How Does vLLM Compare to TGI and TensorRT-LLM?

vLLM, Hugging Face's Text Generation Inference (TGI), and NVIDIA's TensorRT-LLM all target the same broad job — production LLM serving at scale — but with different designs and trade-offs.

vLLM

Details:
Apache 2.0-licensed, Python-based, built around PagedAttention and continuous batching. Loads Hugging Face Transformers-compatible models directly, with broad architecture coverage and multi-vendor GPU backend support (NVIDIA primary; AMD, Intel, TPU documented).
Articles about vLLM (9)

+85 more not shown

TGI

Details:
Hugging Face's own serving engine, Apache 2.0-licensed, also supporting continuous batching and multiple quantization formats. Tightly integrated with the Hugging Face Hub and ecosystem.

TensorRT-LLM

Details:
NVIDIA's engine, built specifically for NVIDIA GPUs. Models are compiled ahead of time into an optimized TensorRT engine for the target GPU, which can yield strong performance on that specific hardware at the cost of a compilation step and less cross-hardware flexibility than vLLM or TGI.
Articles about TensorRT-LLM (6)

Also mentioned in:

+1 more not shown

This article has not independently benchmarked these three engines against each other and does not claim one is universally faster — throughput depends heavily on the model, hardware, batch characteristics, and each engine's version. See the enterprise inference server guide for a more detailed comparison covering deployment and licensing considerations for all three.

Who Should Use vLLM?

vLLM fits teams serving a model to many concurrent users or applications on GPU infrastructure, not people looking for the fastest way to chat with a model on their own computer.

Use vLLM if

  • You are deploying an LLM behind an API that many users or downstream services will call concurrently
  • You have access to one or more NVIDIA GPUs (or a supported AMD/Intel/TPU deployment) and want to maximize throughput and concurrent-request capacity from that hardware
  • You want an OpenAI-compatible API server you self-host rather than depend on a third-party API provider
  • You need multi-GPU tensor-parallel serving for a model too large to fit on one GPU
  • You are comfortable with a Python environment, command-line configuration, and GPU/CUDA setup

Avoid using vLLM if

  • You want to run a model on your own laptop or desktop for personal, single-user chat — Ollama or LM Studio fit that need far better
  • You do not have a supported GPU — vLLM's CPU backend is not its focus, and it has no meaningful support for Apple Silicon the way llama.cpp or Ollama does
  • You want a graphical, point-and-click interface rather than an API server and command-line flags
  • You are not comfortable troubleshooting CUDA drivers, Python dependency conflicts, or GPU memory configuration

vLLM vs. Alternatives at a Glance

These tools sit at different points on the single-user-versus-production-serving spectrum.

vLLM

Interface & setup:
Python package installed via pip; OpenAI-compatible API server started with vllm serve. Expects an NVIDIA GPU and CUDA in most deployments.
Best for:
High-throughput, multi-user GPU serving in production.
Articles about vLLM (9)

+85 more not shown

Ollama

Interface & setup:
CLI and REST API, commonly reported to run on llama.cpp as its backend on most platforms. One command installs it; one command pulls and runs a model.
Best for:
The fastest path to a running local model for a single user, no build step or GPU required.
Articles about Ollama (10)

+313 more not shown

LM Studio

Interface & setup:
Graphical desktop app for Mac, Windows, and Linux. Download, install, then browse and download a model from inside the app.
Best for:
Non-technical users who want a point-and-click local chat app.
Articles about LM Studio (10)

+149 more not shown

llama.cpp

Interface & setup:
CLI, built-in web UI, and OpenAI-compatible API via llama-server. Build from source or use a pre-built binary; runs on CPU or GPU.
Best for:
Direct engine-level control, embedded/edge deployment, and CPU or Apple Silicon hardware.
Articles about llama.cpp (10)

+168 more not shown

This article has not independently benchmarked speed or output quality across these tools and does not claim one is technically superior — the comparison above covers documented architecture, setup, and access-model facts only. For per-hardware throughput numbers, see the llama.cpp vs. Ollama vs. vLLM comparison and the enterprise inference server guide.

Compare in directory

What Does This Article Not Cover?

This is an explainer built from vLLM's public documentation and repository, not a hands-on benchmark report.

  • No independently measured throughput, latency, or requests-per-second figures — these depend heavily on GPU, model, batch composition, and vLLM version
  • No independently verified quality-loss percentages for specific quantization formats — these vary by model architecture and task
  • No line-by-line security audit of the vLLM codebase — it is open-source and Apache 2.0-licensed, so the code itself is available for review
  • No full coverage of every supported hardware backend, engine flag, or deployment orchestration option (Kubernetes, cloud-specific setups) — this article focuses on the concepts and flags most teams evaluate first
  • No coverage of commercial support arrangements or managed vLLM hosting offerings, since vLLM itself is a community open-source project rather than a vendor product with a support contract

Common Mistakes When Trying vLLM

Most friction with vLLM comes from treating it like a single-user desktop tool rather than production server software.

Mistake 1: Trying to run it on a laptop with no dedicated GPU

vLLM's CPU backend exists but is not its focus, and it has no meaningful support for Apple Silicon the way llama.cpp or Ollama does. Attempting to run it on a CPU-only laptop or an Apple Silicon Mac typically leads to poor performance or setup friction that a purpose-built local tool would avoid entirely.

Mistake 2: Expecting a graphical installer or chat window

vLLM has no point-and-click installer and no built-in chat UI. It is installed with pip and accessed through its OpenAI-compatible API — a separate front end (self-built, or a third-party chat UI that speaks the OpenAI API) is needed for a chat window.

Mistake 3: Not sizing GPU memory for concurrent requests, only for the model itself

A GPU with just enough memory to load one copy of a model has little headroom for the KV cache of multiple concurrent requests. Sizing for expected concurrent load — not only for the model weights fitting in VRAM — avoids out-of-memory errors under real traffic.

Frequently Asked Questions

What is vLLM?

vLLM is a free, Apache 2.0-licensed library and server for high-throughput LLM inference, originated at the UC Berkeley Sky Computing Lab. It uses PagedAttention and continuous batching to serve many concurrent requests efficiently from a GPU.

Is vLLM free?

Yes. vLLM is free, open-source software released under the Apache 2.0 license, with no subscription or account requirement to run it yourself.

What is PagedAttention?

PagedAttention is vLLM's technique for managing the attention KV cache in small, page-sized, non-contiguous blocks instead of one large contiguous allocation per request, reducing wasted GPU memory and allowing memory to be shared between requests with a common prefix.

Does vLLM need a GPU?

For any real workload, yes — vLLM's primary and best-supported target is NVIDIA GPUs. A CPU-only backend exists but is documented as substantially slower and is not the project's focus, and Apple Silicon support is limited to community-maintained add-ons rather than a first-class path.

What quantization formats does vLLM support?

vLLM supports several formats including AWQ, GPTQ, FP8, and INT8/INT4, with many pre-quantized models in these formats published on Hugging Face.

Is vLLM better than Ollama?

"Better" depends on the job: vLLM is built for high-concurrency production GPU serving, while Ollama is built for the fastest path to a single-user local model with no GPU required. They are not close substitutes for most use cases — see the comparison table above.

Can vLLM serve models across multiple GPUs?

Yes. vLLM supports tensor-parallel and pipeline-parallel serving across multiple GPUs, configurable with flags like --tensor-parallel-size at server startup.

Does vLLM have an OpenAI-compatible API?

Yes. Running vllm serve starts a server implementing the OpenAI API protocol, so many applications built for the OpenAI API can point at a self-hosted vLLM instance with only a base-URL and model-name change.

How is vLLM different from TensorRT-LLM?

TensorRT-LLM is NVIDIA's engine, which compiles models ahead of time into an optimized engine for a specific NVIDIA GPU. vLLM loads Hugging Face Transformers-compatible models directly without an ahead-of-time compilation step and documents backend support beyond NVIDIA GPUs alone, trading some hardware-specific optimization for broader flexibility and faster iteration.

Sources

← Back to Power Local LLM

MCP