Key Takeaways
- LoRAX (github.com/predibase/lorax) is a free, open-source, self-hosted multi-LoRA inference server
- Created by Predibase, an ML platform company founded in 2021 by Google and Uber alumni; Rubrik announced its acquisition of Predibase on June 25, 2025
- Apache 2.0 licensed, described by the project as free for commercial use
- Forked from Hugging Face's text-generation-inference (from v0.9.4), then extended with dynamic multi-LoRA adapter loading
- Requires an NVIDIA GPU, Ampere generation or newer, CUDA 11.8+, on Linux
- Supports adapters trained with PEFT or Ludwig; loads them from HuggingFace Hub, Predibase, or a local filesystem path
- Over 3,800 GitHub stars and 324+ forks as of this review
π In One Sentence
LoRAX is a free, open-source (Apache 2.0), self-hosted inference server, forked from Hugging Face's text-generation-inference, built by Predibase (acquired by Rubrik in 2025), that serves thousands of fine-tuned LoRA adapters on top of one shared base model on a single GPU, with over 3,800 GitHub stars.
π¬ In Plain Terms
Instead of running a separate GPU-hosted model for every fine-tuned variant of your base LLM, LoRAX loads one shared copy of the base model and dynamically swaps in small LoRA adapter weights per request β so one GPU can serve many different fine-tuned "personalities" of the same model at once, at a fraction of the cost of one GPU per model.
πNote: This review is based on LoRAX's own GitHub repository, README, and release notes. It does not claim PromptQuorum has independently benchmarked its throughput or latency against other inference servers β check the project's own documentation for current performance figures before a production sizing decision.
What Is LoRAX?
LoRAX ("LoRA eXchange") is a self-hosted inference server purpose-built to serve many fine-tuned LoRA adapters on top of one shared base model, rather than requiring a dedicated model deployment per fine-tune. Per its own positioning, it is "the open-source framework for serving hundreds of fine-tuned LLMs in production for the price of one."
- Product type: a self-hosted inference server (CLI + library), not a hosted API or a desktop app
- Creator: Predibase, founded in 2021 by Google and Uber alumni as an ML platform company
- Corporate status: Rubrik, a data-security company, announced its agreement to acquire Predibase on June 25, 2025; Predibase's commercial platform is now positioned under Rubrik, while the LoRAX open-source project remains published on GitHub under the Predibase organization
- Foundation: forked from Hugging Face's text-generation-inference (from v0.9.4), then extended specifically for dynamic multi-adapter serving
- License: Apache 2.0, confirmed via the GitHub repository
- Scale: over 3,800 GitHub stars and 324+ forks as of this review
Project History and Version Milestones
LoRAX has shipped continuously since its early 2024 releases, adding broader model support, quantization options, and serving features with each version. Version numbers and dates below come from the project's own GitHub release notes; check the releases page directly for anything shipped after this review's publish date.
- 1v0.6.0 β January 10, 2024: OpenAI-compatible API
Why it matters: Added OpenAI-compatible completions and chat-completions endpoints, letting existing OpenAI-client code point at a self-hosted LoRAX server. - 2v0.7.0 β February 1, 2024: Multi-adapter merging, EETQ/HQQ quantization
Why it matters: Enabled merging multiple LoRA adapters per request using linear, TIES, and DARE methods, and added two more quantization formats. - 3v0.8.0βv0.8.1 β February 2024: Structured output and Gemma support
Why it matters: Added JSON-schema-guided structured output via the Outlines library, plus support for Google's Gemma model family. - 4v0.9.0 β March 23, 2024: Dedicated adapter memory, Qwen2 support
Why it matters: Reserved dedicated GPU memory for adapters and added Qwen2 model support, per the release notes. - 5v0.10.0 β May 23, 2024: Medusa speculative decoding
Why it matters: Added Medusa speculative-decoding adapters and support for Phi-3, Command-R, and DBRX models, aimed at faster generation. - 6v0.11.0 β September 18, 2024: Prefix caching, vision-language support
Why it matters: Added prefix caching for repeated prompts, Llava-Next vision-language model support, and FP8 quantization for Mistral and Llama models. - 7v0.12.0 β November 6, 2024: Multi-LoRA prefix caching, function calling
Why it matters: Extended prefix caching to work across multiple adapters simultaneously, added FP8 KV-cache support, and added function-calling with schema enforcement, per the changelog β the most recent milestone this review could confirm from the public release notes.
What Does LoRAX Actually Do?
LoRAX loads one shared base model into GPU memory, then dynamically loads and swaps small LoRA adapter weights per incoming request, so many fine-tuned model variants can be served from a single deployment.
- Dynamic adapter loading β LoRA adapters are loaded on demand per request rather than requiring every adapter to be pre-loaded, with asynchronous prefetching and offloading between GPU and CPU memory to manage which adapters stay hot
- Heterogeneous continuous batching β packs requests targeting different adapters into the same batch, per its own architecture description, rather than requiring one batch per adapter
- Base model support β compatible with Llama, CodeLlama, Mistral, Zephyr, Qwen, Gemma, Phi-3, Command-R, DBRX, and Mllama/Llava-Next vision-language models, per its release notes
- Quantization options β fp16, or quantized with bitsandbytes, GPT-Q, AWQ, EETQ, or HQQ, plus FP8 KV-cache support added in later releases
- Adapter compatibility β works with adapters trained via PEFT or Ludwig, loaded from HuggingFace Hub, Predibase, or a local filesystem path
- OpenAI-compatible API β exposes chat-completions and completions endpoints in the OpenAI request/response format, so existing OpenAI-client code can point at a self-hosted LoRAX server
- Serving-infrastructure features β tensor parallelism, flash-attention, paged attention, token streaming, Prometheus metrics, and OpenTelemetry tracing, per its documentation
- Structured output β JSON-schema-guided generation via the Outlines library, plus function calling with schema enforcement in later releases
Usage Examples: Three Ways to Use LoRAX
These are concrete workflows built from LoRAX's own documented features, not hypothetical use cases.
Platform, Pricing, and Licensing
Platform
- What LoRAX states:
- Self-hosted, Linux only; requires an NVIDIA GPU, Ampere generation or newer, with CUDA 11.8+.
Cost
- What LoRAX states:
- Free and open source, described as free for commercial use. You pay only for your own GPU infrastructure β there is no separate paid LoRAX tier.
Licensing
- What LoRAX states:
- Apache 2.0, confirmed via the GitHub repository.
Install method
- What LoRAX states:
- Docker image (
ghcr.io/predibase/lorax:main), plus documented Kubernetes and SkyPilot deployment paths; the Python client installs separately via pip.
Predibase (LoRAX's creator, now positioned under Rubrik after its 2025 acquisition) also sells a separate managed commercial platform built on related technology β verify current commercial pricing directly with Predibase/Rubrik if a managed offering, rather than self-hosting, is what you need.
Install LoRAX
LoRAX runs as a self-hosted server via Docker, Kubernetes, or SkyPilot, with a separate Python client installable via pip.
Source | Link |
|---|---|
| GitHub repository (source code, Apache 2.0) | github.com/predibase/lorax |
| Docker image | docker pull ghcr.io/predibase/lorax:main |
| Python client | pip install lorax-client |
| Documentation | loraexchange.ai |
LoRAX requires an NVIDIA GPU (Ampere generation or newer) with CUDA 11.8+ on Linux β there is no CPU-only mode and no macOS/Windows-native install path. Always check the GitHub README for the current recommended deployment method before running any command.
LoRAX vs. a General-Purpose Inference Engine
LoRAX and a general-purpose inference engine such as LMDeploy or NVIDIA Dynamo both serve LLMs at scale, but LoRAX is purpose-built around one specific problem: serving many LoRA adapters on a single base model cheaply.
Aspect | LoRAX | General-purpose inference engine |
|---|---|---|
| Core job | Serve many LoRA adapters on one base model | Serve one or more full models at high throughput |
| Multi-adapter batching | Built-in heterogeneous continuous batching | Typically not a core focus |
| Base | Fork of Hugging Face text-generation-inference | Varies by project |
| OpenAI-compatible API | Yes | Often yes |
| Best fit | Many fine-tuned variants of one base model | Fewer models, maximum raw throughput |
If your workload is serving many fine-tuned variants of the same base model (per-customer or per-task adapters), LoRAX's adapter-focused batching is purpose-built for that. If you are serving a small number of distinct full models at maximum raw throughput with no adapter-swapping need, a general-purpose engine may fit better β verify current benchmarks on both projects' own sites before choosing, since both ship performance improvements frequently.
Who Should Use LoRAX?
LoRAX fits teams that have, or plan to have, many fine-tuned LoRA adapters of the same base model and want to serve them cost-effectively from shared GPU capacity.
What LoRAX Is Not Good For
LoRAX is not a good fit if you need CPU-only or non-Linux deployment, adapter training rather than serving, or a fully managed hosted platform.
- Not CPU-only β requires an NVIDIA GPU, Ampere generation or newer, with CUDA 11.8+; there is no CPU fallback mode
- Not cross-platform for the server itself β LoRAX's server runs on Linux only, per its own documentation
- Not a training tool β LoRAX serves LoRA adapters that were already trained elsewhere (via PEFT, Ludwig, or similar); it does not fine-tune models itself
- Not a managed hosted service on its own β it is self-hosted open-source software; Predibase's separate commercial platform (now positioned under Rubrik after its 2025 acquisition) is the managed option if you want one
- Not independently benchmarked by PromptQuorum for throughput or latency β this review is based on LoRAX's own documentation and release notes, not hands-on testing
Common Mistakes When Evaluating LoRAX
Most confusion about LoRAX comes from expecting it to train adapters, run without a GPU, or assuming its corporate backing is unchanged after Predibase's acquisition.
Competitors and Alternatives
LoRAX is most often compared to other self-hosted inference and fine-tuning tools, since it sits at the intersection of the inference-serving and LoRA fine-tuning segments. Companion to LoRAX's own entry in the Local LLM Software Directory.
Tool | Best known for | Link |
|---|---|---|
| LMDeploy | Open-source inference engine with quantization and serving toolkit | LMDeploy review |
| NVIDIA Dynamo | High-throughput inference-serving framework with API-server features | NVIDIA Dynamo review |
| Unsloth | Fast, memory-efficient LoRA/QLoRA fine-tuning library | Unsloth review |
| LLaMA-Factory | Unified fine-tuning framework supporting LoRA and other PEFT methods | LLaMA-Factory review |
This list reflects tools in the same inference-serving and LoRA fine-tuning segment as LoRAX, not an independent PromptQuorum ranking β verify each tool's current feature set before choosing, since LoRAX's multi-adapter-serving focus is not identical to a general-purpose inference engine or a training-only tool.
Frequently Asked Questions
What is LoRAX?
LoRAX ("LoRA eXchange," github.com/predibase/lorax) is a free, open-source, self-hosted inference server that serves many fine-tuned LoRA adapters on top of one shared base model on a single GPU.
Is LoRAX free?
Yes, LoRAX itself is free, open source (Apache 2.0), and described by the project as free for commercial use. You pay only for your own GPU infrastructure to run it.
Who created LoRAX?
LoRAX was created by Predibase, an ML platform company founded in 2021 by Google and Uber alumni. Rubrik, a data-security company, announced its agreement to acquire Predibase on June 25, 2025.
What GPU does LoRAX require?
An NVIDIA GPU, Ampere generation or newer, with CUDA 11.8+, on Linux. There is no CPU-only mode and no macOS/Windows-native server deployment.
How do I install LoRAX?
As a self-hosted server via the Docker image (ghcr.io/predibase/lorax:main), with Kubernetes and SkyPilot deployment paths also documented. The Python client installs separately via pip install lorax-client.
What base models does LoRAX support?
Per its release notes: Llama, CodeLlama, Mistral, Zephyr, Qwen, Gemma, Phi-3, Command-R, DBRX, and Mllama/Llava-Next vision-language models, among others β check the current documentation for the complete, up-to-date list.
Does LoRAX train LoRA adapters?
No. LoRAX is an inference server that serves adapters already trained elsewhere via PEFT, Ludwig, or similar tools. Pair it with a dedicated fine-tuning tool such as Unsloth or LLaMA-Factory if you need to produce the adapters first.
Does LoRAX have an OpenAI-compatible API?
Yes. LoRAX exposes chat-completions and completions endpoints in the OpenAI request/response format, so existing OpenAI-client code can point at a self-hosted LoRAX server.
What quantization formats does LoRAX support?
fp16, plus quantization via bitsandbytes, GPT-Q, AWQ, EETQ, or HQQ, and FP8 KV-cache support added in later releases, per its release notes.
Has PromptQuorum independently tested LoRAX's performance claims?
This review is based on LoRAX's own GitHub repository, README, and release notes, not hands-on benchmarking by PromptQuorum. Verify current throughput and latency figures directly against the project's own documentation before a production sizing decision.