Key Takeaways
- vllm-mlx (github.com/waybarrios/vllm-mlx) is a free, open-source inference server, installed via
pip install vllm-mlx - Built by independent developer Way Barrios; not the official vLLM project (github.com/vllm-project/vllm) and not affiliated with the vLLM team
- Apache 2.0 licensed, confirmed via the GitHub repository's LICENSE file
- Exposes OpenAI-compatible (
/v1/chat/completions,/v1/embeddings,/v1/rerank,/v1/responses) and Anthropic-compatible (/v1/messages) endpoints from one server process - Serving features adapted from vLLM: continuous batching, paged KV cache, prefix caching, and optional SSD-tiered cache for long-context workloads
- Handles text, vision (image/video), audio (TTS/STT), and embedding/reranking models, all through native MLX on Apple Silicon
- Requires an Apple Silicon Mac (M1 through M5) β there is no Windows, Linux, or Intel Mac support
- Over 1,580 GitHub stars as of this review
π In One Sentence
vllm-mlx is a free, open-source (Apache 2.0), Python-based inference server that brings vLLM-style continuous batching and paged KV caching to Apple Silicon Macs via native MLX, exposing both OpenAI- and Anthropic-compatible API endpoints.
π¬ In Plain Terms
vllm-mlx lets you run AI models on your own Mac and talk to them the same way you would talk to OpenAI's or Anthropic's cloud API β just point your existing code at localhost instead. It is built to handle several requests at once efficiently, which is the "vLLM-style" part of its name, but it is a separate, unofficial project rather than vLLM itself running on a Mac.
πNote: This review is based on the vllm-mlx GitHub repository, its README, its documentation site, and its PyPI package listing. It does not repeat vllm-mlx's own published throughput benchmarks as independently verified facts β those figures come from the project's own README and should be treated as vendor-reported, not PromptQuorum-tested. This review is the deep-dive companion to vllm-mlx's entry in the Local LLM Software Directory.
What Is vllm-mlx?
vllm-mlx is a command-line inference server for Apple Silicon Macs that exposes locally running AI models through the same API shapes used by OpenAI and Anthropic's cloud services. Its own GitHub README describes it as bringing "continuous batching + OpenAI + Anthropic APIs in one server" with "native Apple Silicon inference." It is not a graphical desktop application β it is a Python package you run from the terminal, which then listens on a local port for API requests.
- Product type: a Python-based CLI inference server, installed as a pip/uv package, not a downloadable GUI app
- Creator: independent developer Way Barrios (GitHub handle
waybarrios); this review found no evidence of a company, funding round, or commercial entity behind the project - Repository: github.com/waybarrios/vllm-mlx, the earliest and most-starred of several same-named repositories on GitHub β other repos with the same name (e.g. under different usernames) describe this one as the project they mirror
- License: Apache 2.0, confirmed via the repository's LICENSE file
- Naming: "vllm-mlx" signals that the project follows vLLM's serving API design and terminology (continuous batching, paged KV cache) while running on MLX instead of vLLM's own CUDA/ROCm-based engine β it does not contain vLLM's original codebase
Project History
vllm-mlx is a relatively young project. Its PyPI package page and GitHub repository indicate active, ongoing development rather than a long multi-year version history, and its README documents a broad feature set already in place β OpenAI and Anthropic API compatibility, continuous batching, multimodal support, and MCP tool calling β as part of its current state rather than a staged rollout.
- Distributed on PyPI as the
vllm-mlxpackage, with published version tags visible viapip index versions vllm-mlxor the PyPI project page - Actively maintained: the GitHub repository shows ongoing commits and a documentation site at vllm-mlx.is-a.dev
- README is translated into multiple languages (English, Spanish, French, Chinese) by the project itself, unusual for a project of this size and a signal of an internationally distributed user base
- Feature surface already includes reasoning-model support (Qwen3, DeepSeek-R1 style reasoning extraction), speculative decoding, and Mixture-of-Experts optimizations as of this review
What Can You Do With vllm-mlx?
vllm-mlx's feature set centers on serving multiple concurrent requests efficiently on a single Mac, while covering more than plain text generation. Here is what each part actually does, per the project's own README and documentation site.
- Dual API compatibility β serves both OpenAI-style (
/v1/chat/completions,/v1/completions,/v1/embeddings,/v1/rerank,/v1/responses) and Anthropic-style (/v1/messages, with streaming, tool use, and system prompts) endpoints from one running server - Continuous batching β processes multiple concurrent requests together rather than one at a time, the same technique vLLM popularized for GPU serving, adapted here for MLX/Metal
- Paged and prefix KV caching β a memory-efficient, trie-based cache that shares repeated prompt prefixes across requests, with an optional
--ssd-cache-dirflag to spill cache to disk for long-context agent workloads - Multimodal support β text, image, video, and audio input from one server; documented compatible vision model families include Gemma, Qwen3-VL, Pixtral, and Llama vision variants
- Built-in audio β text-to-speech (multiple voices and languages per its README) and speech-to-text via Whisper-family models, run through the same server process
- MCP tool calling β Model Context Protocol support with parsers for multiple model families (the README lists OpenAI, Anthropic, Gemini, Qwen, DeepSeek, and Gemma-style tool-call formats among them), plus explicit Claude Code compatibility via the Anthropic-compatible endpoint
- Reasoning and MoE features β reasoning-token extraction for models like Qwen3 and DeepSeek-R1, plus Mixture-of-Experts routing options and speculative decoding for supported model families
- Observability and benchmarking β an optional Prometheus
/metricsendpoint and a built-invllm-mlx bench-servecommand for running and logging throughput sweeps
Usage Examples: Three Ways to Use vllm-mlx
These are concrete workflows built from vllm-mlx's documented features above, using commands from the project's own README.
Install vllm-mlx
vllm-mlx installs free via pip or uv, and its source code is on GitHub. As a CLI tool for developers, there is no per-OS downloadable installer β installation always goes through Python's package tooling.
Source | Link |
|---|---|
| Official documentation | vllm-mlx.is-a.dev |
| GitHub repository (source code, Apache 2.0) | github.com/waybarrios/vllm-mlx |
| PyPI package | pypi.org/project/vllm-mlx |
| Quick install (uv) | uv tool install vllm-mlx |
| Quick install (pip) | pip install vllm-mlx |
vllm-mlx requires an Apple Silicon Mac (M1, M2, M3, M4, or M5) running macOS with the MLX framework β it will not run on Intel Macs, Windows, or Linux. Audio features require an additional `pip install vllm-mlx[audio] step plus brew install espeak-ng` for non-English text-to-speech, per the project README.
vllm-mlx Pricing: Is vllm-mlx Really Free?
Yes β vllm-mlx has no paid tier. It is distributed as a free, Apache 2.0-licensed PyPI package with no account, license key, or usage cap. The GitHub repository has no pricing page, and no part of the README or documentation describes a commercial or enterprise edition.
- No cost to install or run vllm-mlx itself β
pip install vllm-mlxis the entire transaction - Apache 2.0 license permits commercial use, modification, and redistribution, subject to the license's standard terms (attribution and license-notice preservation)
- The only real cost is the Apple Silicon Mac hardware required to run it, and the disk space for any models you download separately from Hugging Face or the
mlx-communityorganization - No developer-hosted cloud version, hosted API, or managed offering was found for this review β vllm-mlx runs only on hardware you control
vllm-mlx vs. mlx-lm
vllm-mlx and mlx-lm both run models via Apple's MLX framework, but they solve different problems. mlx-lm is Apple's own lower-level Python library and CLI for running and fine-tuning MLX models one request at a time; vllm-mlx is a higher-level server built on top of libraries like mlx-lm, mlx-vlm, and mlx-audio (per its own architecture diagram) that adds concurrent-request serving features vLLM is known for.
Maintainer
- vllm-mlx:
- Independent developer (Way Barrios)
- mlx-lm:
- Apple's MLX team
Primary use case
- vllm-mlx:
- Concurrent-request API server
- mlx-lm:
- Single-request generation and fine-tuning library
Concurrent batching
- vllm-mlx:
- Continuous batching, paged KV cache
- mlx-lm:
- Not a core focus; typically one request at a time
API surface
- vllm-mlx:
- OpenAI + Anthropic compatible endpoints
- mlx-lm:
- Python API and CLI, no built-in Anthropic-format server
Relationship
- vllm-mlx:
- Builds on mlx-lm, mlx-vlm, mlx-audio internally
- mlx-lm:
- A foundational dependency other MLX tools build on
This is a factual feature comparison based on each project's own documentation, not a PromptQuorum-run benchmark of either tool. The two are not strictly competitors β vllm-mlx depends on mlx-lm rather than replacing it.
Who Should Use vllm-mlx?
vllm-mlx fits developers on Apple Silicon who want production-style concurrent serving rather than a single-user chat app.
Common Mistakes When Evaluating vllm-mlx
Most confusion about vllm-mlx comes from its name, which invites assumptions the project itself does not make.
Competitors and Alternatives
vllm-mlx sits in the same Apple Silicon inference-server segment as oMLX, Rapid-MLX, and mlxcel β all independent projects that run local models through MLX with an OpenAI-compatible or similar API surface, differing mainly in language (Python vs. Rust), feature emphasis, and caching strategy.
Tool | Best known for | Link |
|---|---|---|
| oMLX | Apple Silicon inference server with SSD-backed prompt caching | oMLX review |
| Rapid-MLX | Native MLX inference server focused on serving speed | Rapid-MLX review |
| mlxcel | Rust-native MLX runtime for LLMs, VLMs, embeddings, and audio | mlxcel review |
| LoRAX | Serving thousands of LoRA adapters from one base model on a GPU | LoRAX review |
This list reflects tools commonly compared in the Apple Silicon and adapter-serving inference-engine space, not an independent PromptQuorum ranking β verify each tool's current feature set before choosing.
Frequently Asked Questions
What is vllm-mlx?
vllm-mlx (github.com/waybarrios/vllm-mlx) is a free, open-source (Apache 2.0) inference server that runs AI models on Apple Silicon Macs via native MLX, exposing both OpenAI- and Anthropic-compatible API endpoints.
Is vllm-mlx the same as vLLM?
No. vllm-mlx is an independent, unofficial project that adapts vLLM's serving concepts (continuous batching, paged KV cache) for Apple's MLX framework. It is not affiliated with the official vLLM project or team, and does not contain vLLM's original codebase.
Is vllm-mlx free?
Yes. It installs free via pip install vllm-mlx, is Apache 2.0 licensed, and has no paid tier, account, or usage cap.
How do I install vllm-mlx?
Run pip install vllm-mlx or uv tool install vllm-mlx, per the project's own README. It requires an Apple Silicon Mac; no Windows, Linux, or Intel Mac support exists.
Does vllm-mlx work with Claude Code?
Yes. The project documents explicit Claude Code support by setting ANTHROPIC_BASE_URL to point at the locally served vllm-mlx endpoint, since it exposes an Anthropic-compatible /v1/messages endpoint.
What hardware does vllm-mlx require?
An Apple Silicon Mac (M1, M2, M3, M4, or M5). It uses Metal kernels via MLX and has no CPU-only fallback or non-Apple GPU support.
Who created vllm-mlx?
Independent developer Way Barrios, GitHub handle waybarrios. This review found no evidence of a company or funding round behind the project.
Does vllm-mlx support vision and audio models, or just text?
It supports text, vision (image/video), and audio (text-to-speech and speech-to-text) models, plus embeddings and reranking, all served from the same process, per the project README.
What is continuous batching?
A serving technique, popularized by vLLM, that processes multiple concurrent requests together rather than one at a time, improving throughput when a server handles several requests at once. vllm-mlx adapts this concept for MLX on Apple Silicon.
Has PromptQuorum independently tested vllm-mlx's benchmark claims?
No. This review is based on vllm-mlx's own GitHub repository, README, and documentation site, rather than hands-on benchmarking by PromptQuorum. Throughput numbers in the project's README are self-reported.