Key Takeaways
- Free and Apache 2.0-licensed open-source, originated by researchers connected to UC Berkeley, Stanford, and LMSYS
- RadixAttention automatically reuses KV-cache entries across requests that share a prefix, using a radix tree rather than per-request cache isolation
- Structured output β JSON schemas and regex constraints β is enforced during decoding via a compressed finite-state machine, not as a post-processing filter
- Ships a Python-embedded frontend DSL (
sgl.gen,sgl.select,sgl.fork) for writing multi-call LLM programs - Built-in OpenAI-compatible API server, started with
python -m sglang.launch_server - Supports continuous batching, tensor parallelism, and quantization formats including FP8, INT4, AWQ, and GPTQ
- Primary, best-supported hardware is NVIDIA GPUs; the project also documents AMD, Intel, and other accelerator backends with narrower real-world coverage
- Not a single-user desktop app β no graphical installer, and not built around CPU-only or Apple Silicon hardware the way llama.cpp and Ollama are
π In One Sentence
SGLang is a free, Apache 2.0-licensed serving framework for LLMs and vision-language models, originated by researchers connected to UC Berkeley, Stanford, and LMSYS, that uses RadixAttention to automatically reuse KV-cache state across requests sharing a common prefix and enforces structured output inside the decoding loop.
π¬ In Plain Terms
Instead of a desktop chat app, SGLang is server software built for two things at once: serving many concurrent requests efficiently β especially ones that repeat a system prompt or conversation history β and guaranteeing the model's output actually matches a JSON schema or pattern you specify.
πNote: This article is based on SGLang's official GitHub repository and public documentation, not independent benchmarking. SGLang's own materials cite specific speedup multipliers for RadixAttention and JSON decoding in particular release benchmarks; this article does not repeat those as universal figures, since they depend on the workload, hardware, and version tested, and both SGLang and vLLM publish benchmarks favorable to themselves.
What Is SGLang?
SGLang is a free, Apache 2.0-licensed serving framework for large language models and vision-language models. It originated from research connecting UC Berkeley, Stanford, and the LMSYS organization β the same community behind Chatbot Arena β and is now developed under the sgl-project GitHub organization. Unlike tools built primarily around a single user chatting with a model locally, SGLang targets two overlapping problems: serving many concurrent requests efficiently, and guaranteeing that a model's output conforms to a structured format such as JSON, which matters for function calling, agent pipelines, and other machine-consumed outputs.
- Originated from research connecting UC Berkeley, Stanford, and LMSYS; developed today under the
sgl-projectopen-source organization - Apache 2.0-licensed: source code is publicly available for use, modification, and redistribution under the license terms
- Combines a Python-embedded frontend DSL for writing LLM programs with a co-designed backend runtime (the SGLang Runtime, often abbreviated SRT)
- Loads Hugging Face Transformers-compatible model checkpoints, covering model families including Llama, Qwen, Mistral, and DeepSeek without a separate conversion step for most models
- Documents production deployments generating large volumes of tokens daily, and lists a number of companies and research institutions as adopters in its own materials
What Is RadixAttention, and Why Does It Matter?
RadixAttention is the memory-management technique SGLang is best known for. Many real-world LLM workloads issue multiple generation calls that share a common prefix β the same system prompt across every request, the same few-shot examples, or earlier turns in an ongoing conversation. Recomputing the attention key-value (KV) cache for that shared prefix on every call wastes GPU compute and memory. RadixAttention instead stores KV-cache entries from both completed and currently running requests in a radix tree β a tree structure keyed on token sequences β so that a new request can automatically find and reuse the cache for any prefix it shares with earlier requests, without an application developer having to manually track or manage that reuse.
- Automatically matches and reuses KV-cache entries across requests that share a token-sequence prefix, using a radix tree data structure
- Covers prefixes from repeated system prompts, shared few-shot examples, and multi-turn conversation history β not just the exact same single request repeated verbatim
- Applies a least-recently-used (LRU) eviction policy to the radix tree so cache memory can be reclaimed and reused as the tree grows
- Works alongside continuous batching and paged, block-based KV-cache allocation, which lets SGLang add and remove requests from an in-flight batch as they arrive and complete
What Do the Frontend DSL and Structured Output Actually Do?
SGLang ships two related but distinct capabilities beyond core serving: a Python-embedded frontend language for writing LLM programs, and engine-level enforcement of structured output formats.
What Hardware Does SGLang Need?
SGLang's primary and best-supported target is NVIDIA GPUs, and most production deployments described in the project's own materials run on NVIDIA hardware. The project also documents additional backends, though coverage and real-world adoption are not equal across all of them.
NVIDIA GPUs (CUDA)
- Details:
- The primary, most mature target, spanning data-center GPUs through recent consumer/workstation cards. Tensor-parallel serving across multiple NVIDIA GPUs is well documented.
AMD GPUs (ROCm)
- Details:
- Documented as a supported backend for AMD Instinct-class accelerators via ROCm, with narrower real-world adoption and community coverage than the CUDA path.
Intel Xeon CPUs and Gaudi accelerators
- Details:
- Additional backends documented by the project for Intel hardware; treat as a smaller, less battle-tested deployment path than NVIDIA GPUs.
Google TPUs and Ascend NPUs
- Details:
- Documented backends aimed at teams already running on Google Cloud TPU or Huawei Ascend infrastructure.
Apple Silicon (Mac)
- Details:
- Not a first-class, officially maintained path. SGLang is built around GPU-backed data-center and workstation hardware, not single-Mac local use.
If your goal is running a model on a single Mac or a CPU-only machine, SGLang is not the tool built for that β llama.cpp and tools built on it, like Ollama and LM Studio, target CPU and Apple Silicon hardware directly and are the better fit for that scenario.
What Quantization Formats Does SGLang Support?
SGLang supports serving models at reduced numeric precision to lower memory use and, in many cases, increase throughput, documenting several established quantization formats.
FP8
- Details:
- 8-bit floating-point precision, supported on NVIDIA GPU generations with hardware FP8 support, trading some precision for lower memory use and faster execution than FP16/BF16.
FP4
- Details:
- A newer, lower-precision floating-point format documented by the project for the most recent-generation NVIDIA hardware that supports it.
AWQ
- Details:
- Activation-aware Weight Quantization, a widely used 4-bit weight-quantization method with pre-quantized models published by the community on Hugging Face.
GPTQ
- Details:
- A post-training quantization method commonly distributed as pre-quantized model checkpoints, also typically run at 4-bit precision.
INT4
- Details:
- A lower-precision integer quantization path documented by the project alongside AWQ and GPTQ for further memory reduction.
This article does not include independently measured quality-loss figures for each format β those vary by model architecture and task, so comparing outputs from a couple of formats on your own prompts is the most reliable way to judge the trade-off for your workload.
What Does the SGLang OpenAI-Compatible Server Provide?
Running python -m sglang.launch_server starts an HTTP server that implements the OpenAI API protocol, so applications and SDKs already built against the OpenAI API can often point at a self-hosted SGLang instance with only a base-URL and model-name change.
- OpenAI-compatible chat completions and completions endpoints, usable as a drop-in replacement for OpenAI-API-based client code
- Configurable host and port (commonly run on
http://localhost:30000in the project's own examples) - Request-level structured-output parameters for JSON-schema-constrained and regex-constrained generation, exposed through the API
- Engine flags for tensor-parallel size, memory allocation, and quantization format, set at server startup
- Support for serving multiple LoRA adapters against a single loaded base model
How Do You Install and Run SGLang?
SGLang is distributed as a Python package and typically installed into a Python environment with an NVIDIA GPU and compatible CUDA drivers available.
- 1Confirm you have a supported NVIDIA GPU with current CUDA drivers installed (or check the project's docs for AMD/Intel/TPU-specific install instructions if you are targeting one of those backends).
- 2Create a Python virtual environment, then install SGLang, for example: `pip install "sglang[all]"`.
- 3Start the OpenAI-compatible server with a model from Hugging Face, for example:
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 127.0.0.1 --port 30000. - 4Send a basic chat request with any OpenAI-API-compatible client, for example the Python
openaipackage pointed atbase_url="http://127.0.0.1:30000/v1". - 5For a JSON-constrained response, pass a JSON schema in the request's structured-output parameters so the server enforces the schema during decoding rather than only requesting JSON in the prompt.
- 6For multi-GPU serving, add a tensor-parallel flag, for example
--tp-size 2to split the model across two GPUs. - 7Point existing OpenAI-API client code at your self-hosted server by changing only its base URL and model name.
Do I need a GPU to run SGLang?
For anything beyond testing, yes β SGLang's primary and best-supported target is NVIDIA GPUs. The project documents other accelerator backends, but they are not the main, best-supported deployment path.
Can I get guaranteed JSON output from SGLang?
Yes β pass a JSON schema in the structured-output parameters of your request, and SGLang enforces it during decoding by masking tokens that would violate the schema, rather than just asking the model to produce JSON in the prompt.
How Does SGLang Compare to vLLM?
SGLang and vLLM are the two most widely discussed open-source, GPU-backed LLM serving engines, and both are Apache 2.0-licensed and target production, multi-user serving rather than single-user desktop chat. Both projects publish benchmarks comparing themselves favorably to the other; this article does not adjudicate that comparison and instead describes each project's own documented design and claims.
Headline cache technique
- SGLang:
- RadixAttention β automatic, radix-tree-based KV-cache reuse across requests sharing any prefix.
- vLLM:
- PagedAttention β page-sized, non-contiguous KV-cache blocks that reduce memory waste from over-reserved allocations.
Structured output
- SGLang:
- Engine-level JSON-schema and regex enforcement is a core, heavily documented feature, built on a compressed finite-state machine.
- vLLM:
- Also supports structured/guided decoding through integrated grammar backends, documented as part of its broader feature set rather than the headline feature.
Programming model
- SGLang:
- Ships a Python-embedded frontend DSL (
sgl.gen,sgl.select,sgl.fork) for multi-call LLM programs, in addition to the API server. - vLLM:
- Primarily accessed as an API server or Python library call; does not ship a comparable program-authoring DSL.
Origin
- SGLang:
- Research connecting UC Berkeley, Stanford, and the LMSYS organization behind Chatbot Arena.
- vLLM:
- Originated at the UC Berkeley Sky Computing Lab.
Throughput claims
- SGLang:
- Publishes release benchmarks citing multipliers for RadixAttention and JSON decoding on specific workloads.
- vLLM:
- Publishes its own release benchmarks; describes PagedAttention's memory-efficiency rationale rather than a single universal speed number.
Neither engine's marketing benchmark should be taken as a neutral verdict β both are run by the project that built the faster-looking result, on workloads chosen by that project. If throughput is decision-critical, testing both engines against your own model, hardware, and traffic pattern is more reliable than any single article's number, including this one.
How Does SGLang Compare to llama.cpp and TensorRT-LLM?
SGLang, llama.cpp, and TensorRT-LLM sit at different points on the hardware-flexibility-versus-peak-optimization spectrum.
SGLang
- Details:
- Apache 2.0-licensed, Python-based, built around RadixAttention and engine-level structured output. Loads Hugging Face Transformers-compatible models directly; NVIDIA GPUs are the primary target, with additional documented backends.
llama.cpp
- Details:
- MIT-licensed C/C++ inference engine built around the GGUF model format, running on CPU, Apple Silicon, and GPU. Targets single-machine and edge deployment rather than multi-GPU production clusters.
TensorRT-LLM
- Details:
- NVIDIA's engine, built specifically for NVIDIA GPUs. Models are compiled ahead of time into an optimized TensorRT engine for the target GPU, which can yield strong performance on that specific hardware at the cost of a compilation step and less cross-hardware flexibility than SGLang.
This article has not independently benchmarked these three engines against each other and does not claim one is universally faster β throughput depends heavily on the model, hardware, batch characteristics, and each engine's version. See the enterprise inference server guide for a deployment-focused comparison covering vLLM, TGI, and NVIDIA NIM as well.
Who Should Use SGLang?
SGLang fits teams serving a model to many concurrent users or applications on GPU infrastructure β particularly workloads with repeated prompt prefixes or a hard requirement on structured output β not people looking for the fastest way to chat with a model on their own computer.
SGLang vs. Alternatives at a Glance
These tools sit at different points on the single-user-versus-production-serving spectrum, and on the throughput-versus-structured-output-emphasis spectrum.
SGLang
- Interface & setup:
- Python package; OpenAI-compatible API server started with
python -m sglang.launch_server. Expects an NVIDIA GPU and CUDA in most deployments. - Best for:
- High-concurrency GPU serving with heavy prefix reuse and/or a hard requirement on structured (JSON/regex) output.
vLLM
- Interface & setup:
- Python package; OpenAI-compatible API server started with
vllm serve. Expects an NVIDIA GPU and CUDA in most deployments. - Best for:
- High-throughput, multi-user GPU serving in production, broadly, without a structured-output-first design focus.
Ollama
- Interface & setup:
- CLI and REST API, commonly reported to run on llama.cpp as its backend on most platforms. One command installs it; one command pulls and runs a model.
- Best for:
- The fastest path to a running local model for a single user, no build step or GPU required.
llama.cpp
- Interface & setup:
- CLI, built-in web UI, and OpenAI-compatible API via llama-server. Build from source or use a pre-built binary; runs on CPU or GPU.
- Best for:
- Direct engine-level control, embedded/edge deployment, and CPU or Apple Silicon hardware.
This article has not independently benchmarked speed or output quality across these tools and does not claim one is technically superior β the comparison above covers documented architecture, setup, and access-model facts only. See the llama.cpp vs. Ollama vs. vLLM comparison for a dedicated throughput and setup-complexity comparison across those three, and the enterprise inference server guide for a deployment-focused look at vLLM, TGI, and NVIDIA NIM.
What Does This Article Not Cover?
This is an explainer built from SGLang's public documentation and repository, not a hands-on benchmark report.
- No independently measured throughput, latency, or requests-per-second figures for SGLang or its comparisons β these depend heavily on GPU, model, batch composition, and version
- No independent adjudication of SGLang's own claimed speedup multipliers for RadixAttention or JSON decoding β those come from the project's release benchmarks, not third-party measurement
- No line-by-line security audit of the SGLang codebase β it is open-source and Apache 2.0-licensed, so the code itself is available for review
- No full coverage of every supported hardware backend, engine flag, or deployment orchestration option (Kubernetes, cloud-specific setups) β this article focuses on the concepts and flags most teams evaluate first
- No coverage of commercial support arrangements or managed SGLang hosting offerings, since SGLang itself is a community open-source project rather than a vendor product with a support contract
Common Mistakes When Trying SGLang
Most friction with SGLang comes from treating it like a single-user desktop tool, or from expecting RadixAttention to help a workload that does not actually share prefixes.
Frequently Asked Questions
What is SGLang?
SGLang is a free, Apache 2.0-licensed serving framework for large language models and vision-language models, originated by researchers connected to UC Berkeley, Stanford, and the LMSYS organization behind Chatbot Arena. It is best known for RadixAttention, a technique for automatic KV-cache reuse across requests sharing a common prefix.
Is SGLang free?
Yes. SGLang is free, open-source software released under the Apache 2.0 license, with no subscription or account requirement to run it yourself.
What is RadixAttention?
RadixAttention is SGLang's technique for storing the attention KV cache from completed and in-flight requests in a radix tree, so that new requests sharing a token-sequence prefix β a system prompt, few-shot examples, or earlier conversation turns β can automatically reuse the matching cache instead of recomputing it.
Does SGLang guarantee valid JSON output?
When a request includes a JSON schema in SGLang's structured-output parameters, the engine masks out tokens that would violate the schema at each decoding step, which is designed to make the output conform to the schema by construction rather than by post-hoc validation. Simply asking for JSON in the prompt text without using these parameters does not get this guarantee.
Does SGLang need a GPU?
For any real workload, yes β SGLang's primary and best-supported target is NVIDIA GPUs. The project documents AMD, Intel, and other accelerator backends, but they are not the main deployment path, and there is no first-class Apple Silicon support.
What quantization formats does SGLang support?
SGLang supports several formats including FP8, FP4 on newer hardware, AWQ, GPTQ, and INT4, with many pre-quantized models in these formats published on Hugging Face.
Is SGLang better than vLLM?
Neither project's own benchmarks are a neutral verdict on this β both publish results favorable to themselves. SGLang emphasizes RadixAttention's prefix-sharing cache reuse and engine-level structured output as its headline features; vLLM emphasizes PagedAttention's memory efficiency. Which is a better fit depends on your workload's prefix-sharing pattern and whether structured output is a hard requirement β see the comparison table above.
Does SGLang have an OpenAI-compatible API?
Yes. Running python -m sglang.launch_server starts a server implementing the OpenAI API protocol, so many applications built for the OpenAI API can point at a self-hosted SGLang instance with only a base-URL and model-name change.
What is the SGLang frontend DSL used for?
It is a set of Python primitives β including sgl.gen, sgl.select, and sgl.fork β for writing multi-step LLM programs, such as branching into several parallel sub-generations and merging the results, as ordinary Python code rather than manually orchestrating separate API calls.
Who created SGLang, and what is RadixArk?
SGLang originated from research connecting UC Berkeley, Stanford, and the LMSYS organization behind Chatbot Arena. In 2026, SGLang co-creator Ying Sheng and Banghua Zhu founded RadixArk, an AI infrastructure startup that raised $100 million in seed funding led by Accel to commercialize services around SGLang while continuing its open-source development β the core framework remains Apache 2.0 and free.