Key Takeaways
- MIT license, created by a developer known as turboderp
- Roughly 4,600 GitHub stars; repository at github.com/turboderp-org/exllamav2 is marked archived
- EXL2 quantization format: mixes 2, 3, 4, 5, 6, and 8-bit precision per layer to hit a target average bitrate, e.g. running a 70B model in 24 GB VRAM at roughly 2.55 bits per weight
- Successor project ExLlamaV3 (roughly 1,300 GitHub stars) introduces the EXL3 format, a streamlined variant of the QTIP quantization approach from Cornell RelaxML
- TabbyAPI, a FastAPI-based server, is the recommended OpenAI-compatible API layer for both versions
- Also integrated into text-generation-webui, lollms-webui, and the standalone ExUI web interface
📍 In One Sentence
ExLlamaV2 is a free, MIT-licensed inference library by turboderp for fast LLM inference on consumer NVIDIA GPUs, known for its flexible EXL2 quantization format, but its repository is now archived in favor of the actively developed successor, ExLlamaV3.
💬 In Plain Terms
ExLlamaV2 was built specifically to squeeze maximum speed out of a single consumer GPU using an unusually flexible quantization scheme; its own maintainer has since moved development to a new project, ExLlamaV3, rather than continuing to update V2.
📌Note: The ExLlamaV2 GitHub README itself states the project is archived and that development continues on ExLlamaV3 — anyone starting a new project today should evaluate ExLlamaV3 rather than building on the archived V2 codebase.
What Was ExLlamaV2?
ExLlamaV2 was a free, open-source inference library, released under the MIT license by a developer known as turboderp, built specifically to maximize inference speed on consumer-grade NVIDIA GPUs. It followed an earlier project, ExLlama, and was itself a more memory-efficient rewrite focused on running quantized models fast on a single GPU.
The repository, github.com/turboderp-org/exllamav2, reached roughly 4,600 GitHub stars before being marked archived, with its README pointing to ExLlamaV3 as the project where development now continues.
- Focused narrowly on NVIDIA consumer GPUs rather than the broad hardware-vendor coverage of engines like llama.cpp
- Introduced the EXL2 quantization format as its main technical contribution, allowing mixed-precision quantization within a single model
- Supported the same 4-bit GPTQ models as the original ExLlama, in addition to its own EXL2 format
- Installable via pip (
pip install exllamav2), prebuilt wheels, or building from source against the CUDA Toolkit
What Is EXL2 Quantization?
EXL2 is ExLlamaV2's quantization format, allowing different linear layers within the same model to use different bit-widths — from 2-bit to 8-bit — chosen automatically from calibration data to hit a target average bitrate. This is more flexible than a single fixed bit-width applied uniformly across an entire model.
- Supports 2, 3, 4, 5, 6, and 8-bit quantization levels, mixed per layer within one model
- More important weight columns can be quantized at higher precision while less important ones use fewer bits, similar in spirit to sparse quantization
- Enables extreme compression for large models on limited VRAM — documented testing shows a 70B-parameter model running in 24 GB VRAM at roughly 2.55 bits per weight with a 2048-token context
- Quantization parameters are chosen automatically based on calibration data rather than requiring manual per-layer configuration
Why Is ExLlamaV2 Archived, and What Does ExLlamaV3 Change?
The turboderp-org/exllamav2 repository is marked archived, with its own README stating development continues on ExLlamaV3. This is the single most important fact to know before evaluating ExLlamaV2 for a new project.
ExLlamaV3 introduces its own format, EXL3, described as a streamlined variant of the QTIP quantization approach from Cornell RelaxML. Unlike EXL2, EXL3 preserves original tensor structures rather than renaming them, intended to make future support from other frameworks easier. ExLlamaV3 documents CUDA 12.4 or later as a requirement and is explicitly flagged by its own maintainers as still maturing — expect some rough edges.
- ExLlamaV2: archived, MIT license, EXL2 format, still installable but receiving no further development
- ExLlamaV3: active, MIT license, EXL3 format (based on QTIP research), beta-stage stability per its own documentation
- Both are maintained by the same developer, turboderp, under the turboderp-org GitHub organization
📌Note: For any new deployment, evaluate ExLlamaV3 first — it is where active maintenance and improvements are happening. ExLlamaV2 remains relevant mainly for existing setups already built on it, or for EXL2-quantized model files already in circulation.
How Do You Install and Run ExLlamaV2 (or V3)?
ExLlamaV2 installs via pip, prebuilt wheels, or from source against the CUDA Toolkit; ExLlamaV3 follows a similar pattern. TabbyAPI is the recommended way to get an OpenAI-compatible server running on top of either.
- 1For ExLlamaV2: install via
pip install exllamav2, or download a prebuilt wheel matching your Python and CUDA version from the GitHub Releases page, or clone and build from source with the CUDA Toolkit installed. - 2For ExLlamaV3 (recommended for new setups): follow the equivalent installation paths documented at github.com/turboderp-org/exllamav3, noting CUDA 12.4 or later is required.
- 3Download a pre-quantized EXL2 (or EXL3) model, commonly published on Hugging Face by community quantizers.
- 4Install and run TabbyAPI, the recommended FastAPI-based server, to expose an OpenAI-compatible API endpoint with HF model downloading and Jinja2 chat template support.
- 5Alternatively, use ExLlamaV2/V3 through an existing integration — text-generation-webui, lollms-webui, or the standalone ExUI web interface — instead of running TabbyAPI directly.
Can I still use ExLlamaV2 today?
Yes, it remains installable and functional, but the repository is archived and will not receive further updates, bug fixes, or new features.
Do EXL2 model files work with ExLlamaV3?
No, not directly — ExLlamaV3 uses its own EXL3 quantization format. Existing EXL2 model files are made for ExLlamaV2/TabbyAPI setups, not V3.
What Hardware Does ExLlamaV2 Require?
ExLlamaV2 targets consumer-grade NVIDIA GPUs specifically — there is no AMD, Intel, or CPU-only support path. This narrow hardware focus is part of what let it optimize so heavily for single-GPU speed.
- NVIDIA GPUs only, with CUDA required — no documented AMD, Intel, or Apple Silicon support
- Designed around consumer-class cards rather than data-center GPUs, though it runs on those too
- VRAM requirements scale with model size and chosen EXL2 bitrate — the flexible quantization is specifically what allows large models to fit in comparatively little VRAM
- ExLlamaV3 documents CUDA 12.4 or later as a requirement, a newer baseline than ExLlamaV2 assumed
Who Should Use ExLlamaV2 (or ExLlamaV3)?
Use ExLlamaV3 for a new project if squeezing maximum speed and VRAM efficiency out of a single NVIDIA consumer GPU matters more than broad hardware support or long-term stability guarantees; there is little reason to start a brand-new project on the archived ExLlamaV2 today.
How Does ExLlamaV2 Compare to Alternatives?
ExLlamaV2's closest comparisons are its own successor and other engines with strong quantization or single-GPU speed focus.
Common Mistakes When Evaluating ExLlamaV2
Most confusion comes from not realizing the project is archived, or expecting cross-compatibility between EXL2 and EXL3 model files.
Frequently Asked Questions
Is ExLlamaV2 still maintained?
No. The turboderp-org/exllamav2 GitHub repository is marked archived, with its README stating development continues on ExLlamaV3 instead.
What is EXL2?
EXL2 is ExLlamaV2's quantization format, supporting mixed 2-to-8-bit precision per layer within a single model to hit a target average bitrate, allowing large models to fit in less VRAM.
What is ExLlamaV3, and how is it different?
ExLlamaV3 is the actively developed successor to ExLlamaV2, using a new EXL3 quantization format based on the QTIP approach from Cornell RelaxML. It is documented as beta-stage software by its own maintainers.
Who created ExLlamaV2 and ExLlamaV3?
Both were created by a developer known as turboderp, under the turboderp-org GitHub organization.
Is ExLlamaV2 free to use?
Yes. Both ExLlamaV2 and ExLlamaV3 are released under the MIT license, free for personal and commercial use.
Does ExLlamaV2 support AMD GPUs?
No. Both ExLlamaV2 and ExLlamaV3 require an NVIDIA GPU with CUDA — there is no AMD, Intel, or Apple Silicon support.
What is TabbyAPI?
TabbyAPI is a FastAPI-based server and the recommended way to expose an OpenAI-compatible API on top of ExLlamaV2 or ExLlamaV3, with added features like Hugging Face model downloading and Jinja2 chat template support.
Can EXL2 and EXL3 model files be used interchangeably?
No. EXL2 files are quantized for ExLlamaV2 and are not directly compatible with ExLlamaV3, which uses the separate EXL3 format.
Should I use ExLlamaV2 or ExLlamaV3 for a new project?
ExLlamaV3, since ExLlamaV2 is archived and receiving no further development. ExLlamaV3 is still beta-stage, so expect some rough edges compared to a mature, actively stabilized project.
How does ExLlamaV2 compare to llama.cpp?
llama.cpp supports a much wider range of hardware (NVIDIA, AMD, Apple Silicon, Intel, CPU-only) and is actively maintained; ExLlamaV2 was narrowly focused on NVIDIA consumer GPUs and is now archived, with ExLlamaV3 as its active successor.