Key Takeaways
- Built into Docker Desktop (macOS, Windows) and Docker Engine (Linux) β not a separate download
- Local inference engine is llama.cpp for CPU/Apple Silicon or vLLM for NVIDIA-GPU-backed serving
- Runs as a native host process outside the container runtime, for direct GPU access
- CLI:
docker model pull <model>,docker model run <model>,docker model list - OpenAI-compatible API at
http://localhost:12434/engines/v1from the host, orhttp://model-runner.docker.internal/engines/v1from inside another container - Models come from Docker Hub's AI model catalog, other OCI-compliant registries, or Hugging Face
- No separate cost β the feature ships with Docker Desktop; whether Docker Desktop itself needs a paid subscription depends on company size
π In One Sentence
Docker Model Runner is a built-in Docker Desktop and Docker Engine feature that runs LLMs locally through docker model CLI commands, using llama.cpp or vLLM as a native host-process inference engine and exposing an OpenAI-compatible API.
π¬ In Plain Terms
If you already use Docker to run containers, Docker Model Runner lets you pull and run an AI model the same way β docker model pull and docker model run β without installing a separate chat app, and without the model actually running inside a container.
πNote: This review is the deep-dive companion to Docker Model Runner's entry in the Local LLM Software Directory β see that page for how it compares at a glance to dozens of other local AI tools.
What Is Docker Model Runner?
Docker Model Runner is a feature of Docker Desktop and Docker Engine for pulling and running large language models locally, using the same docker CLI developers already use for containers. Per Docker's own product page, it provides local-first inference with an OpenAI-compatible API, pulling models from Docker Hub, other OCI-compliant registries, or Hugging Face. It is not a chat application with a graphical window β it is a CLI feature plus a local inference server, meant to be used from a terminal, from Docker Compose, or from code that calls its API.
- Ships inside Docker Desktop (macOS, Windows) and as the
docker-model-pluginpackage for Docker Engine on Linux β not a separate installer - Local inference engines: llama.cpp for CPU and Apple Silicon, vLLM for NVIDIA-GPU-backed serving, per Docker's documentation
- Model sources: Docker Hub's curated AI model catalog, any OCI-compliant registry, or Hugging Face
- Integrates with tools developers already use around Docker: Docker Compose, Testcontainers, and frameworks such as LangChain, Spring AI, and Open WebUI via the OpenAI-compatible API
- Developer: Docker, Inc.
Install and Get Started
Docker Model Runner is a CLI feature bundled with Docker Desktop and Docker Engine, not a standalone downloadable app, so there is no separate installer or download button β you install Docker Desktop (or, on Linux, Docker Engine plus the model-runner plugin) and the docker model commands become available.
- 1Install Docker Desktop for macOS or Windows, or install Docker Engine plus the model-runner plugin on Linux (
sudo apt-get install docker-model-pluginon Debian/Ubuntu, orsudo dnf install docker-model-pluginon Fedora/RHEL). - 2Enable Model Runner if it is not already on:
docker desktop enable model-runner(Docker Desktop), or ensure the plugin is active on Docker Engine. - 3Pull a model from Docker Hub's AI catalog:
docker model pull ai/smollm2:360M-Q4_K_M. - 4Run it directly from the command line:
docker model run ai/smollm2:360M-Q4_K_M "Give me a fact about whales." - 5List locally pulled models at any time with
docker model list. - 6To reach the model over HTTP instead of the CLI, use the OpenAI-compatible API β see The OpenAI-Compatible API below for the exact endpoint.
Why It Runs as a Host Process, Not in a Container
Despite the docker command prefix, Docker Model Runner does not run model inference inside a container. Docker's own blog post introducing the feature is explicit about this: running docker model run "won't spin up any kind of container." Instead, it calls an Inference Server API hosted by Model Runner through Docker Desktop, and that Inference Server runs its inference engine (llama.cpp or vLLM) as a native host process β a normal process running directly on your operating system, the same as any other locally installed program.
- The stated reason is GPU access: a native host process can use your machine's GPU (Apple Silicon's integrated GPU, or an NVIDIA GPU) directly, without the extra indirection of container-to-host GPU passthrough
- This is a deliberate architecture choice for performance, not a limitation of Docker's container runtime β Docker Model Runner uses containers elsewhere in its workflow (for example, some model packaging is OCI-based) but not for the inference process itself
- Practically, this means a running model behaves more like a background service on your machine than like a container you could inspect with
docker ps - Models stay loaded in memory until either a different model is requested or a period of inactivity passes, so the first request after startup is slower than subsequent ones
llama.cpp vs. vLLM: Which Engine Runs Your Model
Docker Model Runner does not implement its own inference engine β it wraps two existing open-source engines and picks between them based on your hardware and the model format, per Docker's documentation.
llama.cpp
- Used for:
- Local development, resource-efficient inference
- Model format:
- GGUF (quantized)
- Platforms:
- CPU and Apple Silicon; also Windows/Linux GPU backends
vLLM
- Used for:
- Production-oriented, higher-throughput serving
- Model format:
- Safetensors
- Platforms:
- Linux and Windows (WSL2) with an NVIDIA GPU
This is the same llama.cpp engine reviewed in depth in this site's llama.cpp explainer, and the same vLLM engine covered in the vLLM explainer β Docker Model Runner is a packaging and CLI layer over these engines, not a from-scratch alternative to either one. Docker also documents experimental Diffusers support for image-generation models (Stable Diffusion) on Linux with NVIDIA GPUs, which is outside the scope of this LLM-focused review.
The OpenAI-Compatible API
Docker Model Runner exposes an OpenAI-compatible HTTP API so existing OpenAI-client code can point at a local model instead of a cloud endpoint, per Docker's blog post on the feature.
- From the host machine:
http://localhost:12434/engines/v1(TCP access must be enabled once withdocker desktop enable model-runner --tcp 12434) - From inside another container on the same Docker network:
http://model-runner.docker.internal/engines/v1 - Compatible with tools and frameworks already built for the OpenAI API format, including LangChain, Spring AI, and Open WebUI, per Docker's own integration examples
- Models load on demand when a request arrives and stay resident in memory until another model is requested or an inactivity timeout passes
Supported Platforms
Docker Model Runner's hardware support differs by operating system and which inference engine is in use, per Docker's documentation.
macOS
- Details:
- Apple Silicon, via llama.cpp with native GPU acceleration
Windows
- Details:
- AMD64 with NVIDIA GPU (driver 576.57 or newer) via llama.cpp or vLLM (WSL2); ARM64 with Qualcomm Adreno 6xx-series GPUs via llama.cpp
Linux
- Details:
- CPU-only, NVIDIA CUDA, AMD ROCm, or Vulkan backends via llama.cpp; NVIDIA GPU via vLLM (driver 575.57.08 or newer)
Exact minimum driver versions and supported GPU families change as Docker updates the feature β verify current requirements against docs.docker.com/ai/model-runner before planning a deployment around a specific GPU.
Docker Model Runner Pricing
Docker Model Runner itself adds no separate charge β it is a feature included with Docker Desktop and Docker Engine. Whether you owe anything depends entirely on whether your use of Docker Desktop qualifies as free, under Docker's own Subscription Service Agreement.
Personal (free)
- Who it is for:
- Individuals, education, non-commercial open source, and businesses under 250 employees and $10M annual revenue
- Docker Model Runner included?:
- Yes
Pro / Team / Business (paid)
- Who it is for:
- Larger companies, or anyone needing team features beyond the free tier's scope
- Docker Model Runner included?:
- Yes
Docker Hub's curated AI models are free to pull regardless of subscription tier. Paid Docker plans (Pro, Team, Business) add features unrelated to Model Runner itself, such as team seats and centralized management β check docker.com/pricing for current prices, since these change independently of the Model Runner feature.
Who Should Use Docker Model Runner?
Docker Model Runner fits a specific workflow: developers who already live in Docker and want local LLM inference without adding a second tool to their stack.
Competitors and Alternatives
Docker Model Runner sits in the local-inference-runtime segment alongside CLI-first engines and desktop apps that wrap the same underlying engines. See the Local LLM Software Directory for the full catalog.
- llama.cpp β the inference engine Docker Model Runner itself uses for CPU/Apple Silicon; running it directly instead gives you control over build flags and quantization that Docker's wrapper does not expose.
- vLLM β the same GPU-serving engine Docker Model Runner uses for production workloads; running it directly is the better fit if you need vLLM's full configuration surface rather than Docker's simplified CLI on top of it.
- Ollama β the closest one-command comparison: like Docker Model Runner, Ollama wraps llama.cpp behind a simple pull-and-run CLI and an OpenAI-compatible API, but as a standalone tool rather than a Docker Desktop feature β a better fit if you do not already use Docker.
- Jan β a free, open-source desktop app with a graphical chat window built on the same llama.cpp engine, for developers who want a point-and-click interface instead of a CLI-and-API workflow.
- GPT4All β another open-source, llama.cpp-based local chat app, for the same "I want a window, not a terminal" use case as Jan.
Common Mistakes
Most confusion about Docker Model Runner comes from assuming it behaves like an ordinary Docker container workload, or missing that it is a Docker Desktop feature rather than a separate product to install.
Frequently Asked Questions
What is Docker Model Runner?
Docker Model Runner is a feature built into Docker Desktop and Docker Engine that runs large language models locally through docker model CLI commands, using llama.cpp or vLLM as the inference engine and exposing an OpenAI-compatible API.
Is Docker Model Runner free?
The feature itself adds no separate charge β it ships with Docker Desktop and Docker Engine. Whether you need a paid Docker subscription depends on your organization's size: Docker Desktop is free for individuals, non-commercial open source projects, and businesses under 250 employees and $10 million in annual revenue, per Docker's Subscription Service Agreement.
Does Docker Model Runner run models inside a container?
No. Despite the docker model command prefix, the inference engine (llama.cpp or vLLM) runs as a native host process outside the container runtime, per Docker's own blog post introducing the feature β this gives it direct access to GPU hardware rather than going through container isolation.
What CLI commands does Docker Model Runner use?
The core commands are docker model pull <model> to download a model, docker model run <model> to run inference from the command line, and docker model list to see models already pulled locally.
What is the Docker Model Runner API endpoint?
An OpenAI-compatible API is exposed at http://localhost:12434/engines/v1 from the host machine (after enabling TCP access), or http://model-runner.docker.internal/engines/v1 from inside another container on the same Docker network.
Does Docker Model Runner use llama.cpp or vLLM?
Both, depending on the situation. llama.cpp handles CPU and Apple Silicon inference for local development; vLLM handles NVIDIA-GPU-backed serving on Linux and Windows WSL2, aimed at higher-throughput production use.
What platforms does Docker Model Runner support?
macOS on Apple Silicon; Windows on AMD64 with an NVIDIA GPU or ARM64 with a Qualcomm Adreno 6xx-series GPU; and Linux with CPU-only, NVIDIA CUDA, AMD ROCm, or Vulkan backends. See the Supported Platforms section above for details.
How is Docker Model Runner different from Ollama?
Both wrap llama.cpp behind a simple pull-and-run CLI and an OpenAI-compatible API. The main difference is packaging: Ollama is a standalone tool you install on its own, while Docker Model Runner is a feature of Docker Desktop/Engine β a better fit if you already use Docker daily, since it does not add a second tool to your workflow.
Where do Docker Model Runner's models come from?
Models can be pulled from Docker Hub's curated AI model catalog, any other OCI-compliant registry, or Hugging Face, per Docker's product page.