Skip to main content
PromptQuorumBuilt for humans. Structured for AI.
Home/Power Local LLM/Docker Model Runner Review: Local LLMs via the Docker CLI
Overview & Reference

Docker Model Runner Review: Local LLMs via the Docker CLI

·11 min read·By Hans Kuepper · Founder of PromptQuorum · Discovery engine for open-weight & open-source AI
Read in:
🇺🇸en🇩🇪de🇫🇷fr🇯🇵ja🇨🇳zh🇪🇸es🇧🇷pt🇸🇦ar🇰🇷ko

Docker Model Runner is a feature of Docker Desktop and Docker Engine, not a separate downloadable app, that lets you pull and run large language models with docker model CLI commands. Local inference runs through llama.cpp (CPU and Apple Silicon) or vLLM (NVIDIA GPU, for higher-throughput serving), and — the detail most reviews miss — that inference engine runs as a native host process, not inside a container, so it can access GPU hardware directly instead of going through container isolation. Docker Model Runner exposes an OpenAI-compatible API and ships free as part of Docker Desktop; whether Docker Desktop itself is free depends on your organization's size (see Pricing below).

Docker Model Runner is a feature built into Docker Desktop and Docker Engine that runs large language models locally using docker model CLI commands, backed by llama.cpp or vLLM, and exposed through an OpenAI-compatible API. It is not a standalone app — it is a capability added to Docker Desktop, so it ships free with the same Docker Desktop most developers already have installed. This review covers what Docker Model Runner actually does, how it runs models without putting inference inside a container, how to install and use it, and where it fits next to other local-runtime tools like Ollama and llama.cpp.

Docker Model Runner Review: Local LLMs via the Docker CLI

Key Takeaways

  • Built into Docker Desktop (macOS, Windows) and Docker Engine (Linux) — not a separate download
  • Local inference engine is llama.cpp for CPU/Apple Silicon or vLLM for NVIDIA-GPU-backed serving
  • Runs as a native host process outside the container runtime, for direct GPU access
  • CLI: docker model pull <model>, docker model run <model>, docker model list
  • OpenAI-compatible API at http://localhost:12434/engines/v1 from the host, or http://model-runner.docker.internal/engines/v1 from inside another container
  • Models come from Docker Hub's AI model catalog, other OCI-compliant registries, or Hugging Face
  • No separate cost — the feature ships with Docker Desktop; whether Docker Desktop itself needs a paid subscription depends on company size

📍 In One Sentence

Docker Model Runner is a built-in Docker Desktop and Docker Engine feature that runs LLMs locally through docker model CLI commands, using llama.cpp or vLLM as a native host-process inference engine and exposing an OpenAI-compatible API.

💬 In Plain Terms

If you already use Docker to run containers, Docker Model Runner lets you pull and run an AI model the same way — docker model pull and docker model run — without installing a separate chat app, and without the model actually running inside a container.

What Is Docker Model Runner?

Docker Model Runner is a feature of Docker Desktop and Docker Engine for pulling and running large language models locally, using the same docker CLI developers already use for containers. Per Docker's own product page, it provides local-first inference with an OpenAI-compatible API, pulling models from Docker Hub, other OCI-compliant registries, or Hugging Face. It is not a chat application with a graphical window — it is a CLI feature plus a local inference server, meant to be used from a terminal, from Docker Compose, or from code that calls its API.

  • Ships inside Docker Desktop (macOS, Windows) and as the docker-model-plugin package for Docker Engine on Linux — not a separate installer
  • Local inference engines: llama.cpp for CPU and Apple Silicon, vLLM for NVIDIA-GPU-backed serving, per Docker's documentation
  • Model sources: Docker Hub's curated AI model catalog, any OCI-compliant registry, or Hugging Face
  • Integrates with tools developers already use around Docker: Docker Compose, Testcontainers, and frameworks such as LangChain, Spring AI, and Open WebUI via the OpenAI-compatible API
  • Developer: Docker, Inc.

Install and Get Started

Docker Model Runner is a CLI feature bundled with Docker Desktop and Docker Engine, not a standalone downloadable app, so there is no separate installer or download button — you install Docker Desktop (or, on Linux, Docker Engine plus the model-runner plugin) and the docker model commands become available.

  1. 1
    Install Docker Desktop for macOS or Windows, or install Docker Engine plus the model-runner plugin on Linux (sudo apt-get install docker-model-plugin on Debian/Ubuntu, or sudo dnf install docker-model-plugin on Fedora/RHEL).
  2. 2
    Enable Model Runner if it is not already on: docker desktop enable model-runner (Docker Desktop), or ensure the plugin is active on Docker Engine.
  3. 3
    Pull a model from Docker Hub's AI catalog: docker model pull ai/smollm2:360M-Q4_K_M.
  4. 4
    Run it directly from the command line: docker model run ai/smollm2:360M-Q4_K_M "Give me a fact about whales."
  5. 5
    List locally pulled models at any time with docker model list.
  6. 6
    To reach the model over HTTP instead of the CLI, use the OpenAI-compatible API — see The OpenAI-Compatible API below for the exact endpoint.

Why It Runs as a Host Process, Not in a Container

Despite the docker command prefix, Docker Model Runner does not run model inference inside a container. Docker's own blog post introducing the feature is explicit about this: running docker model run "won't spin up any kind of container." Instead, it calls an Inference Server API hosted by Model Runner through Docker Desktop, and that Inference Server runs its inference engine (llama.cpp or vLLM) as a native host process — a normal process running directly on your operating system, the same as any other locally installed program.

  • The stated reason is GPU access: a native host process can use your machine's GPU (Apple Silicon's integrated GPU, or an NVIDIA GPU) directly, without the extra indirection of container-to-host GPU passthrough
  • This is a deliberate architecture choice for performance, not a limitation of Docker's container runtime — Docker Model Runner uses containers elsewhere in its workflow (for example, some model packaging is OCI-based) but not for the inference process itself
  • Practically, this means a running model behaves more like a background service on your machine than like a container you could inspect with docker ps
  • Models stay loaded in memory until either a different model is requested or a period of inactivity passes, so the first request after startup is slower than subsequent ones

llama.cpp vs. vLLM: Which Engine Runs Your Model

Docker Model Runner does not implement its own inference engine — it wraps two existing open-source engines and picks between them based on your hardware and the model format, per Docker's documentation.

This is the same llama.cpp engine reviewed in depth in this site's llama.cpp explainer, and the same vLLM engine covered in the vLLM explainer — Docker Model Runner is a packaging and CLI layer over these engines, not a from-scratch alternative to either one. Docker also documents experimental Diffusers support for image-generation models (Stable Diffusion) on Linux with NVIDIA GPUs, which is outside the scope of this LLM-focused review.

The OpenAI-Compatible API

Docker Model Runner exposes an OpenAI-compatible HTTP API so existing OpenAI-client code can point at a local model instead of a cloud endpoint, per Docker's blog post on the feature.

  • From the host machine: http://localhost:12434/engines/v1 (TCP access must be enabled once with docker desktop enable model-runner --tcp 12434)
  • From inside another container on the same Docker network: http://model-runner.docker.internal/engines/v1
  • Compatible with tools and frameworks already built for the OpenAI API format, including LangChain, Spring AI, and Open WebUI, per Docker's own integration examples
  • Models load on demand when a request arrives and stay resident in memory until another model is requested or an inactivity timeout passes

Supported Platforms

Docker Model Runner's hardware support differs by operating system and which inference engine is in use, per Docker's documentation.

macOS

Details:
Apple Silicon, via llama.cpp with native GPU acceleration

Windows

Details:
AMD64 with NVIDIA GPU (driver 576.57 or newer) via llama.cpp or vLLM (WSL2); ARM64 with Qualcomm Adreno 6xx-series GPUs via llama.cpp

Linux

Details:
CPU-only, NVIDIA CUDA, AMD ROCm, or Vulkan backends via llama.cpp; NVIDIA GPU via vLLM (driver 575.57.08 or newer)

Exact minimum driver versions and supported GPU families change as Docker updates the feature — verify current requirements against docs.docker.com/ai/model-runner before planning a deployment around a specific GPU.

Docker Model Runner Pricing

Docker Model Runner itself adds no separate charge — it is a feature included with Docker Desktop and Docker Engine. Whether you owe anything depends entirely on whether your use of Docker Desktop qualifies as free, under Docker's own Subscription Service Agreement.

Personal (free)

Who it is for:
Individuals, education, non-commercial open source, and businesses under 250 employees and $10M annual revenue
Docker Model Runner included?:
Yes

Pro / Team / Business (paid)

Who it is for:
Larger companies, or anyone needing team features beyond the free tier's scope
Docker Model Runner included?:
Yes

Docker Hub's curated AI models are free to pull regardless of subscription tier. Paid Docker plans (Pro, Team, Business) add features unrelated to Model Runner itself, such as team seats and centralized management — check docker.com/pricing for current prices, since these change independently of the Model Runner feature.

Who Should Use Docker Model Runner?

Docker Model Runner fits a specific workflow: developers who already live in Docker and want local LLM inference without adding a second tool to their stack.

Use Docker Model Runner if

  • You already use Docker Desktop or Docker Engine daily and want to add local LLM inference to the same CLI, without installing a separate app
  • Your application or service is already built around Docker Compose or containers, and you want the model endpoint reachable from other containers via model-runner.docker.internal
  • You want the option to move from CPU/Apple-Silicon development (llama.cpp) to NVIDIA-GPU production serving (vLLM) without switching tools
  • You are comfortable working from a terminal and an HTTP API rather than a graphical chat window

Wait, or look elsewhere, if

  • You want a graphical, point-and-click chat interface as your primary way of talking to a model — Docker Model Runner has no built-in chat window; pair it with a front end like Open WebUI if you want one
  • You are not already using Docker and do not want to install Docker Desktop just to run local models — a dedicated app such as Jan or GPT4All has a smaller footprint for that case
  • You need the inference process to run strictly inside a container for isolation reasons — Docker Model Runner deliberately runs its engine as a host process, not a container, for GPU performance (see Why It Runs as a Host Process above)
  • Your organization is above the 250-employee / $10M-revenue free-use threshold and has not budgeted for a paid Docker subscription — confirm your Docker licensing status first

Competitors and Alternatives

Docker Model Runner sits in the local-inference-runtime segment alongside CLI-first engines and desktop apps that wrap the same underlying engines. See the Local LLM Software Directory for the full catalog.

  • llama.cpp — the inference engine Docker Model Runner itself uses for CPU/Apple Silicon; running it directly instead gives you control over build flags and quantization that Docker's wrapper does not expose.
  • vLLM — the same GPU-serving engine Docker Model Runner uses for production workloads; running it directly is the better fit if you need vLLM's full configuration surface rather than Docker's simplified CLI on top of it.
  • Ollama — the closest one-command comparison: like Docker Model Runner, Ollama wraps llama.cpp behind a simple pull-and-run CLI and an OpenAI-compatible API, but as a standalone tool rather than a Docker Desktop feature — a better fit if you do not already use Docker.
  • Jan — a free, open-source desktop app with a graphical chat window built on the same llama.cpp engine, for developers who want a point-and-click interface instead of a CLI-and-API workflow.
  • GPT4All — another open-source, llama.cpp-based local chat app, for the same "I want a window, not a terminal" use case as Jan.

Compare in directory

Common Mistakes

Most confusion about Docker Model Runner comes from assuming it behaves like an ordinary Docker container workload, or missing that it is a Docker Desktop feature rather than a separate product to install.

Mistake 1: Assuming the model runs inside a container

Because the command is docker model run, it is easy to assume inference happens inside a container you could inspect with docker ps. It does not — Docker's own documentation confirms the inference engine runs as a native host process for direct GPU access, not inside a container.

Mistake 2: Looking for a separate installer or download page

Docker Model Runner has no standalone installer. It is a feature of Docker Desktop (macOS, Windows) and Docker Engine plus the model-runner plugin (Linux) — you install Docker itself, not "Docker Model Runner" as its own product.

Mistake 3: Assuming Docker Model Runner adds its own separate fee

Model Runner itself has no additional price tag. The only cost question is whether your existing Docker Desktop usage falls inside the free Personal tier (under 250 employees and $10M annual revenue, per Docker's Subscription Service Agreement) or requires a paid Pro/Team/Business plan — a licensing question about Docker Desktop generally, not about this feature specifically.

Mistake 4: Expecting one engine to cover every platform and GPU

llama.cpp and vLLM cover different hardware: llama.cpp handles CPU and Apple Silicon (plus some Windows/Linux GPU backends), while vLLM is limited to NVIDIA GPUs on Linux and Windows WSL2. Check the Supported Platforms table above before assuming a given model/engine combination will run on your machine.

Frequently Asked Questions

What is Docker Model Runner?

Docker Model Runner is a feature built into Docker Desktop and Docker Engine that runs large language models locally through docker model CLI commands, using llama.cpp or vLLM as the inference engine and exposing an OpenAI-compatible API.

Is Docker Model Runner free?

The feature itself adds no separate charge — it ships with Docker Desktop and Docker Engine. Whether you need a paid Docker subscription depends on your organization's size: Docker Desktop is free for individuals, non-commercial open source projects, and businesses under 250 employees and $10 million in annual revenue, per Docker's Subscription Service Agreement.

Does Docker Model Runner run models inside a container?

No. Despite the docker model command prefix, the inference engine (llama.cpp or vLLM) runs as a native host process outside the container runtime, per Docker's own blog post introducing the feature — this gives it direct access to GPU hardware rather than going through container isolation.

What CLI commands does Docker Model Runner use?

The core commands are docker model pull <model> to download a model, docker model run <model> to run inference from the command line, and docker model list to see models already pulled locally.

What is the Docker Model Runner API endpoint?

An OpenAI-compatible API is exposed at http://localhost:12434/engines/v1 from the host machine (after enabling TCP access), or http://model-runner.docker.internal/engines/v1 from inside another container on the same Docker network.

Does Docker Model Runner use llama.cpp or vLLM?

Both, depending on the situation. llama.cpp handles CPU and Apple Silicon inference for local development; vLLM handles NVIDIA-GPU-backed serving on Linux and Windows WSL2, aimed at higher-throughput production use.

What platforms does Docker Model Runner support?

macOS on Apple Silicon; Windows on AMD64 with an NVIDIA GPU or ARM64 with a Qualcomm Adreno 6xx-series GPU; and Linux with CPU-only, NVIDIA CUDA, AMD ROCm, or Vulkan backends. See the Supported Platforms section above for details.

How is Docker Model Runner different from Ollama?

Both wrap llama.cpp behind a simple pull-and-run CLI and an OpenAI-compatible API. The main difference is packaging: Ollama is a standalone tool you install on its own, while Docker Model Runner is a feature of Docker Desktop/Engine — a better fit if you already use Docker daily, since it does not add a second tool to your workflow.

Where do Docker Model Runner's models come from?

Models can be pulled from Docker Hub's curated AI model catalog, any other OCI-compliant registry, or Hugging Face, per Docker's product page.

Sources

← Back to Power Local LLM