Key Takeaways
- mlxcel (github.com/lablup/mlxcel) is a free, open-source inference CLI and server, installed via Homebrew or a source build
- Built by Lablup, the company behind the Backend.AI AI infrastructure platform
- Apache 2.0 licensed, confirmed via the GitHub repository's LICENSE file
- Fully native execution: Rust plus MLX C++ bindings, no Python interpreter in the request path, no checkpoint-conversion step for MLX SafeTensors weights
- Serves OpenAI, Anthropic, and Vertex AI-compatible routes, plus a documented, version-pinned
llama-server-compatible route and flag surface - Primary target is Apple Silicon (Metal); NVIDIA CUDA on Linux is a secondary target; AMD ROCm on Linux is experimental and source-build only
- Handles text generation, vision-language input, embeddings, reranking, and audio (transcription and text-to-speech) from one runtime
- Over 467 GitHub stars as of this review
π In One Sentence
mlxcel is a free, open-source (Apache 2.0), Rust-native inference CLI and server for LLMs, vision-language models, embeddings, rerankers, and audio, built by Lablup for Apple Silicon and NVIDIA CUDA, with experimental AMD ROCm support.
π¬ In Plain Terms
mlxcel is a command-line tool and server, written in Rust rather than Python, that loads AI models straight from their downloaded files and runs them on your Mac or a CUDA-equipped Linux machine β no separate Python environment to set up, and no conversion step before a model will load. It speaks the same request format as OpenAI and Anthropic's APIs, so existing client code can point at it directly.
πNote: This review is based on the mlxcel GitHub repository, its README, and its linked documentation. It does not repeat mlxcel's own published performance comparisons against mlx-lm and mlx-vlm as independently verified facts β those figures come from the project's own benchmark methodology and should be treated as vendor-reported, not PromptQuorum-tested. This review is the deep-dive companion to mlxcel's entry in the Local LLM Software Directory.
What Is mlxcel?
mlxcel is a native inference runtime β both a command-line tool and a server β for running AI models locally without a Python interpreter in the request path. Its own GitHub README describes it as delivering "high-performance LLM, VLM, embedding, reranking, and audio inference for Apple Silicon and NVIDIA CUDA systems," executed through "native MLX C++ bindings."
- Product type: a Rust-native CLI (
mlxcel) and server (mlxcel-server), distributed as compiled binaries via Homebrew or built from source β not a Python package and not a graphical desktop app - Creator: Lablup Inc., a company also known for Backend.AI, an open-source AI infrastructure and MLOps platform
- Repository: github.com/lablup/mlxcel, the original project β several forks of the name exist under other GitHub usernames, which list this repository as their parent
- License: Apache 2.0, confirmed via the repository's LICENSE file
- Companion product: Backend.AI Go, described in mlxcel's own README as an optional companion graphical interface for local chat and model management that can sit on top of
mlxcel-server
Project History
mlxcel's README documents its version history through its CHANGELOG and describes its current main branch as v0.7.0 plus unreleased work. The project has moved through a series of tagged releases, adding platform targets and serving features along the way, per its own changelog and README.
- 1Early releases (through v0.0.28) β Apple Silicon-first inference
Why it matters: The project's README references an earlier benchmark campaign (dated May 2026 in the README's own text) run on mlxcel 0.0.28, indicating the project already had working text-decode and prefill benchmarking in its early released versions. - 2v0.6.0 β Speculative decoding with an exactness policy
Why it matters: Per the README, this release introduced a policy that probes greedy exactness before enabling faster speculative-decoding paths, rather than always using the fastest available kernel regardless of output correctness. - 3v0.7.0 β Current release baseline
Why it matters: The README describes this as the latest tagged release as of this review, with DeepSeek-V4 architecture support, expanded llama-server route compatibility, and dtype fixes that improved half-precision performance on several model families. - 4Unreleased
mainβ Experimental AMD ROCm support
Why it matters: As of this review, AMD GPU support via `--features rocm` is present on the `main` branch but not yet part of a tagged release, per the README's own "Current main highlights" section.
What Can You Do With mlxcel?
mlxcel's feature set centers on running a broad range of model types natively, with production-oriented serving controls. Here is what each part actually does, per the project's own README.
- Direct checkpoint loading β loads MLX SafeTensors checkpoints from Hugging Face, including the
mlx-communityorganization, directly, with many standard SafeTensors embedding checkpoints also loading without a conversion step - Multi-format API compatibility β serves OpenAI (Chat Completions, Completions, Responses, Embeddings, Reranking, Audio), Anthropic Messages, and Vertex AI-compatible routes, alongside a documented
llama-server-compatible native route and flag surface pinned to a specific upstream release - Broad architecture coverage β dense transformers, sparse Mixture-of-Experts models, hybrid state-space models, vision-language/OCR models, embeddings, rerankers, speech-to-text, and text-to-speech, queryable via the
mlxcel archcommand - Multi-model routing β router mode discovers checkpoints from a model store or directory, loads them on demand, and bounds the resident set with least-recently-used eviction
- Live LoRA adapters β multiple LoRA adapters can stay unfused for per-request switching, or be fused for zero decode overhead, per the README
- Speculative decoding β measured, exactness-checked speculative decoding for several model families (including Gemma, Qwen, and GLM-4.7-Flash MTP paths), exposed at a
/v1/internal/mtp-policyendpoint - Operational controls β API-key and CORS/TLS configuration, idle model sleep/wake, GBNF grammars, expanded sampling controls, and prompt/cache observability
- Reproducible model surgery β YAML-defined load-time weight edits (
scale,add,prune,replace,interpolate) via a--surgeryflag, for reproducible checkpoint modification without hand-editing weights
Usage Examples: Three Ways to Use mlxcel
These are concrete workflows built from mlxcel's documented features above, using commands from the project's own README.
Install mlxcel
mlxcel installs free via Homebrew on macOS and Linux, and its source code is on GitHub for a from-source build. As a CLI and server tool for developers, there is no graphical installer.
Source | Link |
|---|---|
| GitHub repository (source code, Apache 2.0) | github.com/lablup/mlxcel |
| Homebrew install | brew tap lablup/tap && brew install mlxcel |
| Source build (Apple Silicon) | cargo build --release --features metal,accelerate |
| Source build (NVIDIA CUDA) | cargo build --release --features cuda |
| Documentation | docs/installation.md |
The Homebrew formula installs the latest tagged release. Building from source is required for the newest unreleased features (such as experimental AMD ROCm support) and for CUDA or ROCm targets, per the project README. Apple Silicon source builds require Rust, Xcode Command Line Tools, CMake, and the Metal toolchain component.
mlxcel Pricing: Is mlxcel Really Free?
Yes β mlxcel has no paid tier. It is distributed as a free, Apache 2.0-licensed open-source project with no account, license key, or usage cap. The GitHub repository has no pricing page.
- No cost to install or run mlxcel itself β the Homebrew formula and source build are both free
- Apache 2.0 license permits commercial use, modification, and redistribution, subject to the license's standard terms (attribution and license-notice preservation)
- The only real cost is the hardware required to run it (an Apple Silicon Mac or CUDA-equipped GPU) and the disk space for any models you download separately
- Lablup, the company behind mlxcel, also sells Backend.AI as a commercial AI infrastructure product, but this review found no evidence that any mlxcel feature is gated behind a Backend.AI purchase β the optional Backend.AI Go companion app is described in mlxcel's README as a separate, optional GUI, not a requirement
mlxcel vs. llama.cpp
mlxcel and llama.cpp are both native (non-Python) inference runtimes, but they differ in language, primary platform focus, and checkpoint format. llama.cpp is a C/C++ project supporting the widest range of hardware and quantized GGUF-format models; mlxcel is a Rust project focused on MLX SafeTensors checkpoints with Apple Silicon as its primary target and documented llama-server-route compatibility for easier drop-in use.
Language
- mlxcel:
- Rust
- llama.cpp:
- C/C++
Primary checkpoint format
- mlxcel:
- MLX SafeTensors
- llama.cpp:
- GGUF
Primary platform
- mlxcel:
- Apple Silicon (Metal), CUDA secondary
- llama.cpp:
- Broadest hardware support: CPU, CUDA, Metal, Vulkan, and more
Maintainer
- mlxcel:
- Lablup (Backend.AI company)
- llama.cpp:
- Community-maintained, originally by Georgi Gerganov
API compatibility
- mlxcel:
- OpenAI, Anthropic, Vertex, plus llama-server-compatible routes
- llama.cpp:
- Its own
llama-serveris the reference implementation mlxcel targets compatibility with
This is a factual feature comparison based on each project's own documentation, not a PromptQuorum-run benchmark of either tool. mlxcel explicitly targets compatibility with llama.cpp's llama-server route surface, per its own documented b10621 compatibility manifest, rather than positioning itself as an unrelated alternative.
Who Should Use mlxcel?
mlxcel fits developers who want a native, production-oriented inference runtime without a Python dependency in the serving path.
Common Mistakes When Evaluating mlxcel
Most confusion about mlxcel comes from assuming its platform support or checkpoint format is broader than documented.
Competitors and Alternatives
mlxcel sits in the same Apple Silicon and native-runtime inference-server segment as vllm-mlx, oMLX, and Rapid-MLX β independent projects that run local models with an OpenAI-compatible or similar API surface, differing mainly in language (Rust vs. Python), platform breadth, and feature emphasis.
Tool | Best known for | Link |
|---|---|---|
| vllm-mlx | vLLM-style continuous batching for Apple Silicon, Python-based | vllm-mlx review |
| oMLX | Apple Silicon inference server with SSD-backed prompt caching | oMLX review |
| Rapid-MLX | Native MLX inference server focused on serving speed | Rapid-MLX review |
| LoRAX | Serving thousands of LoRA adapters from one base model on a GPU | LoRAX review |
This list reflects tools commonly compared in the Apple Silicon and native-runtime inference-engine space, not an independent PromptQuorum ranking β verify each tool's current feature set before choosing.
Frequently Asked Questions
What is mlxcel?
mlxcel (github.com/lablup/mlxcel) is a free, open-source (Apache 2.0) Rust-native inference CLI and server for LLMs, vision-language models, embeddings, rerankers, and audio, built by Lablup.
Who makes mlxcel?
Lablup Inc., the company also known for the Backend.AI AI infrastructure and MLOps platform.
Is mlxcel free?
Yes. It installs free via Homebrew or a source build, is Apache 2.0 licensed, and has no paid tier, account, or usage cap.
How do I install mlxcel?
Run brew tap lablup/tap && brew install mlxcel on macOS or Linux, or build from source with Cargo using the appropriate feature flag (metal, cuda, or the experimental rocm), per the project's README.
What hardware does mlxcel support?
Apple Silicon (Metal) is the primary target. NVIDIA CUDA on Linux is a secondary, supported target. AMD ROCm on Linux is experimental and source-build-only. A plain CPU-only Linux build works but is not a validated release target.
Does mlxcel require Python?
No. mlxcel is written in Rust and executes MLX checkpoints through native MLX C++ bindings, with no Python interpreter in the request path.
What checkpoint format does mlxcel use?
MLX SafeTensors checkpoints, including those published under the mlx-community Hugging Face organization β not the GGUF format used by llama.cpp.
Is mlxcel compatible with llama.cpp's API?
Yes, partially and by design: mlxcel documents a frozen, version-pinned llama-server route and flag compatibility manifest alongside its own OpenAI, Anthropic, and Vertex-compatible endpoints.
Does mlxcel support audio and vision models?
Yes. It handles vision-language input, speech-to-text transcription, and text-to-speech synthesis (via Kokoro-family models) alongside text generation, embeddings, and reranking.
Has PromptQuorum independently tested mlxcel's benchmark claims?
No. This review is based on mlxcel's own GitHub repository and README, rather than hands-on benchmarking by PromptQuorum. Performance comparisons in the project's README are self-measured.