Skip to main content
PromptQuorumBuilt for humans. Structured for AI.
Home/Power Local LLM
Overview & Reference

SGLang Explained: RadixAttention and Structured LLM Serving (2026)

·13 min readBy Hans Kuepper · Founder of PromptQuorum · Discovery engine for open-weight & open-source AI

SGLang is a free, open-source (Apache 2.0) serving framework for large language models and vision-language models, originated by researchers connected to UC Berkeley, Stanford, and the LMSYS organization behind Chatbot Arena. Its key technical differentiator is RadixAttention, which stores the attention KV cache from completed and in-flight requests in a radix tree so that any new request sharing a prefix — a system prompt, few-shot examples, or an earlier turn in the same conversation — can automatically reuse the matching cache entries instead of recomputing them. SGLang also ships a Python-embedded frontend language (sgl.gen, sgl.select, and related primitives) for writing multi-call LLM programs, and enforces structured output — JSON schemas and regex patterns — directly inside the decoding loop rather than as a post-processing step. It ships an OpenAI-compatible server (python -m sglang.launch_server), documents NVIDIA GPUs as its primary and best-supported target with additional AMD, Intel, and other backends, and — like vLLM — is built for GPU-backed serving rather than the single-user, point-and-click local-chat use case that tools like Ollama and LM Studio target.

Key Takeaways

  • Free and Apache 2.0-licensed open-source, originated by researchers connected to UC Berkeley, Stanford, and LMSYS
  • RadixAttention automatically reuses KV-cache entries across requests that share a prefix, using a radix tree rather than per-request cache isolation
  • Structured output — JSON schemas and regex constraints — is enforced during decoding via a compressed finite-state machine, not as a post-processing filter
  • Ships a Python-embedded frontend DSL (sgl.gen, sgl.select, sgl.fork) for writing multi-call LLM programs
  • Built-in OpenAI-compatible API server, started with python -m sglang.launch_server
  • Supports continuous batching, tensor parallelism, and quantization formats including FP8, INT4, AWQ, and GPTQ
  • Primary, best-supported hardware is NVIDIA GPUs; the project also documents AMD, Intel, and other accelerator backends with narrower real-world coverage
  • Not a single-user desktop app — no graphical installer, and not built around CPU-only or Apple Silicon hardware the way llama.cpp and Ollama are

📍 In One Sentence

SGLang is a free, Apache 2.0-licensed serving framework for LLMs and vision-language models, originated by researchers connected to UC Berkeley, Stanford, and LMSYS, that uses RadixAttention to automatically reuse KV-cache state across requests sharing a common prefix and enforces structured output inside the decoding loop.

💬 In Plain Terms

Instead of a desktop chat app, SGLang is server software built for two things at once: serving many concurrent requests efficiently — especially ones that repeat a system prompt or conversation history — and guaranteeing the model's output actually matches a JSON schema or pattern you specify.

📌Note: This article is based on SGLang's official GitHub repository and public documentation, not independent benchmarking. SGLang's own materials cite specific speedup multipliers for RadixAttention and JSON decoding in particular release benchmarks; this article does not repeat those as universal figures, since they depend on the workload, hardware, and version tested, and both SGLang and vLLM publish benchmarks favorable to themselves.

Read in:
🇺🇸en🇩🇪de🇫🇷fr🇯🇵ja🇨🇳zh🇪🇸es🇧🇷pt🇸🇦ar🇰🇷ko

SGLang is a free, Apache 2.0-licensed serving framework for large language models and vision-language models, originated by researchers connected to UC Berkeley, Stanford, and the LMSYS organization behind Chatbot Arena. Its headline technical contribution is RadixAttention, a technique for automatic, fine-grained KV-cache reuse across multiple generation calls that share a common prefix — such as a repeated system prompt, few-shot examples, or a multi-turn conversation. SGLang pairs that backend with a Python-embedded frontend language for writing multi-call LLM programs and with engine-level structured-output support (JSON schemas and regex constraints enforced during decoding), positioning it as a serving engine built around structured generation and agentic workloads as much as raw throughput.

SGLang Explained: RadixAttention and Structured LLM Serving (2026)

What Is SGLang?

SGLang is a free, Apache 2.0-licensed serving framework for large language models and vision-language models. It originated from research connecting UC Berkeley, Stanford, and the LMSYS organization — the same community behind Chatbot Arena — and is now developed under the sgl-project GitHub organization. Unlike tools built primarily around a single user chatting with a model locally, SGLang targets two overlapping problems: serving many concurrent requests efficiently, and guaranteeing that a model's output conforms to a structured format such as JSON, which matters for function calling, agent pipelines, and other machine-consumed outputs.

  • Originated from research connecting UC Berkeley, Stanford, and LMSYS; developed today under the sgl-project open-source organization
  • Apache 2.0-licensed: source code is publicly available for use, modification, and redistribution under the license terms
  • Combines a Python-embedded frontend DSL for writing LLM programs with a co-designed backend runtime (the SGLang Runtime, often abbreviated SRT)
  • Loads Hugging Face Transformers-compatible model checkpoints, covering model families including Llama, Qwen, Mistral, and DeepSeek without a separate conversion step for most models
  • Documents production deployments generating large volumes of tokens daily, and lists a number of companies and research institutions as adopters in its own materials

What Is RadixAttention, and Why Does It Matter?

RadixAttention is the memory-management technique SGLang is best known for. Many real-world LLM workloads issue multiple generation calls that share a common prefix — the same system prompt across every request, the same few-shot examples, or earlier turns in an ongoing conversation. Recomputing the attention key-value (KV) cache for that shared prefix on every call wastes GPU compute and memory. RadixAttention instead stores KV-cache entries from both completed and currently running requests in a radix tree — a tree structure keyed on token sequences — so that a new request can automatically find and reuse the cache for any prefix it shares with earlier requests, without an application developer having to manually track or manage that reuse.

  • Automatically matches and reuses KV-cache entries across requests that share a token-sequence prefix, using a radix tree data structure
  • Covers prefixes from repeated system prompts, shared few-shot examples, and multi-turn conversation history — not just the exact same single request repeated verbatim
  • Applies a least-recently-used (LRU) eviction policy to the radix tree so cache memory can be reclaimed and reused as the tree grows
  • Works alongside continuous batching and paged, block-based KV-cache allocation, which lets SGLang add and remove requests from an in-flight batch as they arrive and complete

What Do the Frontend DSL and Structured Output Actually Do?

SGLang ships two related but distinct capabilities beyond core serving: a Python-embedded frontend language for writing LLM programs, and engine-level enforcement of structured output formats.

The frontend DSL

SGLang provides Python primitives — including sgl.gen for generation, sgl.select for choosing among a fixed set of options, and sgl.fork for branching into parallel generation paths — that let a developer express a multi-step LLM program (for example: generate a plan, then branch into several sub-generations in parallel, then merge results) as ordinary Python code, with the runtime handling batching and cache reuse across the branches automatically.

Engine-level structured output

When a request specifies a JSON schema or a regular expression the output must match, SGLang enforces that constraint inside the decoding loop by masking out logits for any token that would violate the schema or pattern at that step — rather than generating unconstrained text and validating or retrying afterward. The project documents an internal compressed finite-state-machine representation for these constraints, which is designed to let the engine decode multiple tokens in a single forward pass when the constraint uniquely determines what comes next.

What Hardware Does SGLang Need?

SGLang's primary and best-supported target is NVIDIA GPUs, and most production deployments described in the project's own materials run on NVIDIA hardware. The project also documents additional backends, though coverage and real-world adoption are not equal across all of them.

NVIDIA GPUs (CUDA)

Details:
The primary, most mature target, spanning data-center GPUs through recent consumer/workstation cards. Tensor-parallel serving across multiple NVIDIA GPUs is well documented.

AMD GPUs (ROCm)

Details:
Documented as a supported backend for AMD Instinct-class accelerators via ROCm, with narrower real-world adoption and community coverage than the CUDA path.

Intel Xeon CPUs and Gaudi accelerators

Details:
Additional backends documented by the project for Intel hardware; treat as a smaller, less battle-tested deployment path than NVIDIA GPUs.

Google TPUs and Ascend NPUs

Details:
Documented backends aimed at teams already running on Google Cloud TPU or Huawei Ascend infrastructure.

Apple Silicon (Mac)

Details:
Not a first-class, officially maintained path. SGLang is built around GPU-backed data-center and workstation hardware, not single-Mac local use.

If your goal is running a model on a single Mac or a CPU-only machine, SGLang is not the tool built for that — llama.cpp and tools built on it, like Ollama and LM Studio, target CPU and Apple Silicon hardware directly and are the better fit for that scenario.

What Quantization Formats Does SGLang Support?

SGLang supports serving models at reduced numeric precision to lower memory use and, in many cases, increase throughput, documenting several established quantization formats.

FP8

Details:
8-bit floating-point precision, supported on NVIDIA GPU generations with hardware FP8 support, trading some precision for lower memory use and faster execution than FP16/BF16.

FP4

Details:
A newer, lower-precision floating-point format documented by the project for the most recent-generation NVIDIA hardware that supports it.

AWQ

Details:
Activation-aware Weight Quantization, a widely used 4-bit weight-quantization method with pre-quantized models published by the community on Hugging Face.

GPTQ

Details:
A post-training quantization method commonly distributed as pre-quantized model checkpoints, also typically run at 4-bit precision.

INT4

Details:
A lower-precision integer quantization path documented by the project alongside AWQ and GPTQ for further memory reduction.

This article does not include independently measured quality-loss figures for each format — those vary by model architecture and task, so comparing outputs from a couple of formats on your own prompts is the most reliable way to judge the trade-off for your workload.

What Does the SGLang OpenAI-Compatible Server Provide?

Running python -m sglang.launch_server starts an HTTP server that implements the OpenAI API protocol, so applications and SDKs already built against the OpenAI API can often point at a self-hosted SGLang instance with only a base-URL and model-name change.

  • OpenAI-compatible chat completions and completions endpoints, usable as a drop-in replacement for OpenAI-API-based client code
  • Configurable host and port (commonly run on http://localhost:30000 in the project's own examples)
  • Request-level structured-output parameters for JSON-schema-constrained and regex-constrained generation, exposed through the API
  • Engine flags for tensor-parallel size, memory allocation, and quantization format, set at server startup
  • Support for serving multiple LoRA adapters against a single loaded base model

How Do You Install and Run SGLang?

SGLang is distributed as a Python package and typically installed into a Python environment with an NVIDIA GPU and compatible CUDA drivers available.

  1. 1
    Confirm you have a supported NVIDIA GPU with current CUDA drivers installed (or check the project's docs for AMD/Intel/TPU-specific install instructions if you are targeting one of those backends).
  2. 2
    Create a Python virtual environment, then install SGLang, for example: `pip install "sglang[all]"`.
  3. 3
    Start the OpenAI-compatible server with a model from Hugging Face, for example: python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 127.0.0.1 --port 30000.
  4. 4
    Send a basic chat request with any OpenAI-API-compatible client, for example the Python openai package pointed at base_url="http://127.0.0.1:30000/v1".
  5. 5
    For a JSON-constrained response, pass a JSON schema in the request's structured-output parameters so the server enforces the schema during decoding rather than only requesting JSON in the prompt.
  6. 6
    For multi-GPU serving, add a tensor-parallel flag, for example --tp-size 2 to split the model across two GPUs.
  7. 7
    Point existing OpenAI-API client code at your self-hosted server by changing only its base URL and model name.

Do I need a GPU to run SGLang?

For anything beyond testing, yes — SGLang's primary and best-supported target is NVIDIA GPUs. The project documents other accelerator backends, but they are not the main, best-supported deployment path.

Can I get guaranteed JSON output from SGLang?

Yes — pass a JSON schema in the structured-output parameters of your request, and SGLang enforces it during decoding by masking tokens that would violate the schema, rather than just asking the model to produce JSON in the prompt.

How Does SGLang Compare to vLLM?

SGLang and vLLM are the two most widely discussed open-source, GPU-backed LLM serving engines, and both are Apache 2.0-licensed and target production, multi-user serving rather than single-user desktop chat. Both projects publish benchmarks comparing themselves favorably to the other; this article does not adjudicate that comparison and instead describes each project's own documented design and claims.

Headline cache technique

SGLang:
RadixAttention — automatic, radix-tree-based KV-cache reuse across requests sharing any prefix.
vLLM:
PagedAttention — page-sized, non-contiguous KV-cache blocks that reduce memory waste from over-reserved allocations.

Structured output

SGLang:
Engine-level JSON-schema and regex enforcement is a core, heavily documented feature, built on a compressed finite-state machine.
vLLM:
Also supports structured/guided decoding through integrated grammar backends, documented as part of its broader feature set rather than the headline feature.

Programming model

SGLang:
Ships a Python-embedded frontend DSL (sgl.gen, sgl.select, sgl.fork) for multi-call LLM programs, in addition to the API server.
vLLM:
Primarily accessed as an API server or Python library call; does not ship a comparable program-authoring DSL.

Origin

SGLang:
Research connecting UC Berkeley, Stanford, and the LMSYS organization behind Chatbot Arena.
vLLM:
Originated at the UC Berkeley Sky Computing Lab.

Throughput claims

SGLang:
Publishes release benchmarks citing multipliers for RadixAttention and JSON decoding on specific workloads.
vLLM:
Publishes its own release benchmarks; describes PagedAttention's memory-efficiency rationale rather than a single universal speed number.

Neither engine's marketing benchmark should be taken as a neutral verdict — both are run by the project that built the faster-looking result, on workloads chosen by that project. If throughput is decision-critical, testing both engines against your own model, hardware, and traffic pattern is more reliable than any single article's number, including this one.

How Does SGLang Compare to llama.cpp and TensorRT-LLM?

SGLang, llama.cpp, and TensorRT-LLM sit at different points on the hardware-flexibility-versus-peak-optimization spectrum.

SGLang

Details:
Apache 2.0-licensed, Python-based, built around RadixAttention and engine-level structured output. Loads Hugging Face Transformers-compatible models directly; NVIDIA GPUs are the primary target, with additional documented backends.
Articles about SGLang (5)

Also mentioned in:

+1 more not shown

llama.cpp

Details:
MIT-licensed C/C++ inference engine built around the GGUF model format, running on CPU, Apple Silicon, and GPU. Targets single-machine and edge deployment rather than multi-GPU production clusters.
Articles about llama.cpp (10)

+168 more not shown

TensorRT-LLM

Details:
NVIDIA's engine, built specifically for NVIDIA GPUs. Models are compiled ahead of time into an optimized TensorRT engine for the target GPU, which can yield strong performance on that specific hardware at the cost of a compilation step and less cross-hardware flexibility than SGLang.
Articles about TensorRT-LLM (6)

Also mentioned in:

+1 more not shown

This article has not independently benchmarked these three engines against each other and does not claim one is universally faster — throughput depends heavily on the model, hardware, batch characteristics, and each engine's version. See the enterprise inference server guide for a deployment-focused comparison covering vLLM, TGI, and NVIDIA NIM as well.

Who Should Use SGLang?

SGLang fits teams serving a model to many concurrent users or applications on GPU infrastructure — particularly workloads with repeated prompt prefixes or a hard requirement on structured output — not people looking for the fastest way to chat with a model on their own computer.

Use SGLang if

  • You are deploying an LLM behind an API that many users or downstream services will call concurrently, especially with repeated system prompts, few-shot examples, or multi-turn conversations
  • Your application requires guaranteed JSON-schema-conformant or regex-conformant output — function calling, tool-use agents, or structured data extraction
  • You want to author multi-step LLM programs (branching generations, parallel sub-calls) using a Python-embedded DSL rather than orchestrating separate API calls yourself
  • You have access to one or more NVIDIA GPUs (or a supported AMD/Intel/TPU deployment) and want an OpenAI-compatible API server you self-host
  • You are comfortable with a Python environment, command-line configuration, and GPU/CUDA setup

Avoid using SGLang if

  • You want to run a model on your own laptop or desktop for personal, single-user chat — Ollama or LM Studio fit that need far better
  • You do not have a supported GPU — SGLang is not built around CPU-only or Apple Silicon hardware the way llama.cpp or Ollama is
  • You want a graphical, point-and-click interface rather than an API server, a Python DSL, and command-line flags
  • Your workload has no repeated prompt prefixes and no structured-output requirement — in that case, RadixAttention and constrained decoding offer little advantage over any other serving engine

SGLang vs. Alternatives at a Glance

These tools sit at different points on the single-user-versus-production-serving spectrum, and on the throughput-versus-structured-output-emphasis spectrum.

SGLang

Interface & setup:
Python package; OpenAI-compatible API server started with python -m sglang.launch_server. Expects an NVIDIA GPU and CUDA in most deployments.
Best for:
High-concurrency GPU serving with heavy prefix reuse and/or a hard requirement on structured (JSON/regex) output.
Articles about SGLang (5)

Also mentioned in:

+1 more not shown

vLLM

Interface & setup:
Python package; OpenAI-compatible API server started with vllm serve. Expects an NVIDIA GPU and CUDA in most deployments.
Best for:
High-throughput, multi-user GPU serving in production, broadly, without a structured-output-first design focus.
Articles about vLLM (10)

+85 more not shown

Ollama

Interface & setup:
CLI and REST API, commonly reported to run on llama.cpp as its backend on most platforms. One command installs it; one command pulls and runs a model.
Best for:
The fastest path to a running local model for a single user, no build step or GPU required.
Articles about Ollama (10)

+313 more not shown

llama.cpp

Interface & setup:
CLI, built-in web UI, and OpenAI-compatible API via llama-server. Build from source or use a pre-built binary; runs on CPU or GPU.
Best for:
Direct engine-level control, embedded/edge deployment, and CPU or Apple Silicon hardware.
Articles about llama.cpp (10)

+168 more not shown

This article has not independently benchmarked speed or output quality across these tools and does not claim one is technically superior — the comparison above covers documented architecture, setup, and access-model facts only. See the llama.cpp vs. Ollama vs. vLLM comparison for a dedicated throughput and setup-complexity comparison across those three, and the enterprise inference server guide for a deployment-focused look at vLLM, TGI, and NVIDIA NIM.

Compare in directory

What Does This Article Not Cover?

This is an explainer built from SGLang's public documentation and repository, not a hands-on benchmark report.

  • No independently measured throughput, latency, or requests-per-second figures for SGLang or its comparisons — these depend heavily on GPU, model, batch composition, and version
  • No independent adjudication of SGLang's own claimed speedup multipliers for RadixAttention or JSON decoding — those come from the project's release benchmarks, not third-party measurement
  • No line-by-line security audit of the SGLang codebase — it is open-source and Apache 2.0-licensed, so the code itself is available for review
  • No full coverage of every supported hardware backend, engine flag, or deployment orchestration option (Kubernetes, cloud-specific setups) — this article focuses on the concepts and flags most teams evaluate first
  • No coverage of commercial support arrangements or managed SGLang hosting offerings, since SGLang itself is a community open-source project rather than a vendor product with a support contract

Common Mistakes When Trying SGLang

Most friction with SGLang comes from treating it like a single-user desktop tool, or from expecting RadixAttention to help a workload that does not actually share prefixes.

Mistake 1: Trying to run it on a laptop with no dedicated GPU

SGLang is not built around CPU-only or Apple Silicon hardware the way llama.cpp or Ollama is. Attempting to run it on a CPU-only laptop or an Apple Silicon Mac typically leads to poor performance or setup friction that a purpose-built local tool would avoid entirely.

Mistake 2: Expecting RadixAttention to speed up traffic with no shared prefixes

RadixAttention's benefit comes specifically from reusing KV-cache state across requests that share a token-sequence prefix. If every request in your workload is fully independent — different system prompts, no few-shot examples, no conversation history — there is little or no shared prefix for RadixAttention to reuse, and the gain over a non-caching engine will be small.

Mistake 3: Asking for JSON in the prompt instead of using the structured-output parameters

Prompting a model to "respond in JSON" without using SGLang's schema-constrained decoding parameters still lets the model produce malformed or non-conformant output. The reliability benefit of engine-level structured output only applies when the JSON schema or regex constraint is actually passed through the API's structured-output fields.

Frequently Asked Questions

What is SGLang?

SGLang is a free, Apache 2.0-licensed serving framework for large language models and vision-language models, originated by researchers connected to UC Berkeley, Stanford, and the LMSYS organization behind Chatbot Arena. It is best known for RadixAttention, a technique for automatic KV-cache reuse across requests sharing a common prefix.

Is SGLang free?

Yes. SGLang is free, open-source software released under the Apache 2.0 license, with no subscription or account requirement to run it yourself.

What is RadixAttention?

RadixAttention is SGLang's technique for storing the attention KV cache from completed and in-flight requests in a radix tree, so that new requests sharing a token-sequence prefix — a system prompt, few-shot examples, or earlier conversation turns — can automatically reuse the matching cache instead of recomputing it.

Does SGLang guarantee valid JSON output?

When a request includes a JSON schema in SGLang's structured-output parameters, the engine masks out tokens that would violate the schema at each decoding step, which is designed to make the output conform to the schema by construction rather than by post-hoc validation. Simply asking for JSON in the prompt text without using these parameters does not get this guarantee.

Does SGLang need a GPU?

For any real workload, yes — SGLang's primary and best-supported target is NVIDIA GPUs. The project documents AMD, Intel, and other accelerator backends, but they are not the main deployment path, and there is no first-class Apple Silicon support.

What quantization formats does SGLang support?

SGLang supports several formats including FP8, FP4 on newer hardware, AWQ, GPTQ, and INT4, with many pre-quantized models in these formats published on Hugging Face.

Is SGLang better than vLLM?

Neither project's own benchmarks are a neutral verdict on this — both publish results favorable to themselves. SGLang emphasizes RadixAttention's prefix-sharing cache reuse and engine-level structured output as its headline features; vLLM emphasizes PagedAttention's memory efficiency. Which is a better fit depends on your workload's prefix-sharing pattern and whether structured output is a hard requirement — see the comparison table above.

Does SGLang have an OpenAI-compatible API?

Yes. Running python -m sglang.launch_server starts a server implementing the OpenAI API protocol, so many applications built for the OpenAI API can point at a self-hosted SGLang instance with only a base-URL and model-name change.

What is the SGLang frontend DSL used for?

It is a set of Python primitives — including sgl.gen, sgl.select, and sgl.fork — for writing multi-step LLM programs, such as branching into several parallel sub-generations and merging the results, as ordinary Python code rather than manually orchestrating separate API calls.

Who created SGLang, and what is RadixArk?

SGLang originated from research connecting UC Berkeley, Stanford, and the LMSYS organization behind Chatbot Arena. In 2026, SGLang co-creator Ying Sheng and Banghua Zhu founded RadixArk, an AI infrastructure startup that raised $100 million in seed funding led by Accel to commercialize services around SGLang while continuing its open-source development — the core framework remains Apache 2.0 and free.

Sources

← Back to Power Local LLM

MCP