Key Takeaways
- Free and MIT-licensed open-source, maintained by ggml-org on GitHub
- Written in C/C++, built on the ggml tensor library
- Defined the GGUF (.gguf) model file format used across the local-AI ecosystem
- Hardware backends: CUDA, Metal, Vulkan, ROCm/HIP, SYCL, and CPU (AVX/AVX2/AVX512)
- Quantization support from 8-bit down to roughly 1.5-bit per weight
- Ships llama-server, a built-in OpenAI-compatible HTTP server with a web UI
- Ollama uses llama.cpp as its inference engine on most platforms, adding a model registry and CLI on top
- No graphical installer β used via command line, a built-in web UI, or language bindings
π In One Sentence
llama.cpp is a free, MIT-licensed C/C++ inference engine created by Georgi Gerganov that runs LLMs locally on CPU or GPU, defined the GGUF model format, and serves as the inference backend several local-AI tools β including Ollama, on most platforms β build on top of.
π¬ In Plain Terms
Instead of a point-and-click app, llama.cpp is the actual engine code that loads a model file and generates text β Ollama and several other local-AI apps run llama.cpp under the hood and add a friendlier interface on top of it.
πNote: This article is based on llama.cpp's official GitHub repository and public documentation, not independent benchmarking. Specific tokens-per-second figures are not included because they were not independently measured for this article and vary heavily by hardware, model, and quantization format.
What Is llama.cpp?
llama.cpp is a free, MIT-licensed inference engine written in C/C++ that runs large language models locally with minimal external dependencies. It was created by Georgi Gerganov in 2023 as a from-scratch C/C++ port of Meta's LLaMA inference code, built to run on ordinary consumer hardware β including CPU-only machines β rather than requiring a data-center GPU. The project is now maintained by the ggml-org organization on GitHub, with contributions from a large open-source community, and has grown to support vision-language models in addition to text-only LLMs.
- Created by Georgi Gerganov, now maintained by the ggml-org organization
- Written in C/C++ and built on ggml, a tensor library the same team maintains
- MIT-licensed: the source code is publicly available for use, modification, and redistribution under the license terms
- Runs on CPU-only hardware as well as GPUs, unlike engines that require a CUDA-capable GPU to function at all
- Defined the GGUF model format, a single-file format storing model weights, quantization data, and metadata together
- One of the most widely referenced open-source LLM inference projects, with well over 100,000 GitHub stars
What Hardware Acceleration Does llama.cpp Support?
llama.cpp supports a wide range of hardware acceleration backends, which is one reason it runs on everything from a CPU-only laptop to a multi-GPU workstation.
CUDA
- Details:
- For NVIDIA GPUs. Offloads model layers to GPU memory for faster inference than CPU-only execution, with support for multi-GPU setups.
Metal
- Details:
- For Apple Silicon Macs (M1 and later). Uses Apple's Metal API to run inference on the GPU cores of M-series chips.
Vulkan
- Details:
- A cross-vendor GPU API supported on Windows and Linux, usable on GPUs from multiple manufacturers without a vendor-specific SDK.
ROCm / HIP
- Details:
- For AMD GPUs, via AMD's ROCm compute stack, as an alternative path to CUDA-only acceleration.
SYCL
- Details:
- For Intel GPUs, including integrated graphics on some Intel CPUs, via Intel's oneAPI SYCL implementation.
CPU (AVX / AVX2 / AVX512)
- Details:
- Optimized CPU-only execution using vectorized instructions on modern x86 processors β the fallback path that lets llama.cpp run with no GPU at all.
The project also documents additional and experimental backends (including BLAS, CANN, and WebGPU) in its build documentation on GitHub; coverage here focuses on the backends most local-LLM users will actually choose between.
What Is GGUF, and Which Quantization Format Should You Use?
GGUF is the model file format llama.cpp defined to store quantized model weights together with the metadata (tokenizer, architecture, context length) needed to run them, replacing the project's earlier GGML format. Quantization shrinks a model's memory footprint by storing its weights at lower numeric precision than the original training format, trading some output quality for lower VRAM/RAM use and faster inference.
Q8_0
- Details:
- 8-bit quantization, the closest to the original unquantized model among llama.cpp's common formats, at the largest file size of the group.
Q5_K_M
- Details:
- A 5-bit "K-quant" format that allocates precision unevenly across a model's layers, favoring quality over the smallest possible file size.
Q4_K_M
- Details:
- A 4-bit K-quant format commonly used as a balance point between file size and output quality β a frequent default recommendation for running models on limited VRAM.
Sub-4-bit (Q3, Q2, and ~1.5-bit formats)
- Details:
- Formats for fitting larger models into very limited memory. Quality loss becomes more noticeable as bit-width drops, so these are typically a last resort rather than a default choice.
The right format depends on your hardware and the specific model β this article does not include independently measured quality-loss percentages for specific formats, since those vary by model architecture and task. Downloading a model in more than one quantization and comparing outputs on your own prompts is the most reliable way to judge the trade-off for your use case.
What Does llama-server Provide?
llama-server is the HTTP server binary that ships with llama.cpp. A single binary exposes llama.cpp's own native API, an OpenAI-compatible API, an Ollama-compatible API shim, and a built-in browser-based chat UI, all backed by the same loaded model and KV cache.
- OpenAI-compatible endpoints under /v1, so tools already built for the OpenAI API can often point at a local llama-server instance with only a base-URL change
- A built-in web UI served directly by the binary, usable without installing a separate front end
- An Ollama-compatible API shim, letting some Ollama-oriented client tools connect to llama-server instead
- Introspection endpoints such as /props and /slots for inspecting server and request state
- Support for multiple concurrent request "slots" against one loaded model, rather than one request at a time
How Do You Build and Run llama.cpp?
llama.cpp is typically built from source with CMake, though the project also publishes pre-built release binaries for several platforms on its GitHub releases page if you want to skip compiling it yourself.
- 1Install a C++ toolchain and CMake for your operating system if you do not already have them.
- 2Clone the repository:
git clone https://github.com/ggml-org/llama.cpp. - 3Configure the build for your hardware backend β for example, enabling CUDA, Metal, or Vulkan support as a CMake option, or leaving it CPU-only.
- 4Build the project with CMake:
cmake -B buildfollowed bycmake --build build --config Release. - 5Download a GGUF-format model, for example from Hugging Face, choosing a quantization format that fits your available VRAM or RAM.
- 6Run the model from the command line with the
llama-clibinary for direct terminal chat, or startllama-serverto expose an OpenAI-compatible HTTP API and web UI. - 7If using llama-server, connect to it from a browser at its local address, or point any OpenAI-API-compatible client at its
/v1endpoint.
Do I have to build llama.cpp from source?
No β the project also publishes pre-built binaries on its GitHub releases page for several platforms, though building from source lets you enable the exact hardware backend (CUDA, Metal, Vulkan, and similar) for your machine.
Where do I get GGUF models to run?
Many GGUF-format models are published on Hugging Face and other model-sharing sites, typically in several quantization formats per model so you can pick one that fits your available memory.
How Does llama.cpp Relate to Ollama?
Ollama is one of the most widely used ways people run local LLMs without touching a build system, and it is commonly reported β in community and third-party technical write-ups β to use llama.cpp as its underlying inference engine on most platforms. Ollama itself does not prominently document this dependency in detail, so treat it as the commonly reported architecture rather than an official specification, though it is consistent with Ollama's own open-source codebase.
- llama.cpp is the low-level inference engine; Ollama is a packaging layer that adds a model registry, one-command pulls, a simpler CLI, and its own Modelfile configuration system on top
- Choosing llama.cpp directly gives you control over exact build flags, quantization format, and server configuration that Ollama's simplified interface does not expose
- Choosing Ollama trades that control for a faster path to a running model, at the cost of some flexibility over how the underlying engine is configured
- Both can serve an OpenAI-compatible API β llama.cpp via llama-server, Ollama via its own API layer
Who Should Use llama.cpp Directly?
llama.cpp fits people who want direct control over how a model runs, rather than the fastest path to a working chat window.
llama.cpp vs. Ollama vs. LM Studio vs. vLLM
These four tools sit at different points on the control-versus-convenience spectrum, and two of them (Ollama and, on Apple Silicon, parts of the ecosystem around LM Studio) are themselves built on inference engines rather than being from-scratch alternatives to one.
llama.cpp
- Interface & setup:
- CLI, built-in web UI, and OpenAI-compatible API via llama-server. Build from source with CMake or use a pre-built release binary; you choose the quantization format yourself.
- Best for:
- Maximum control, embedded/edge deployment, and custom pipelines built directly on the engine.
Ollama
- Interface & setup:
- CLI and REST API, commonly reported to run on llama.cpp as its backend on most platforms. One command installs it; one command pulls and runs a model.
- Best for:
- The fastest path to a running local model without a build step, for people who do not need engine-level configuration.
LM Studio
- Interface & setup:
- Graphical desktop app for Mac, Windows, and Linux. Download, install, then browse and download a model from inside the app.
- Best for:
- Non-technical users who want a point-and-click chat app instead of a command line.
vLLM
- Interface & setup:
- Python server with an OpenAI-compatible API, installed via pip in a Python/CUDA environment. Uses its own PagedAttention serving engine rather than GGUF or llama.cpp.
- Best for:
- High-throughput, multi-user GPU serving in production, not single-user local chat.
This article has not independently benchmarked speed or output quality across these four tools and does not claim one is technically superior β the comparison above covers documented architecture, setup, and access-model facts only. For per-hardware throughput numbers, see the dedicated llama.cpp vs. Ollama vs. vLLM comparison and the enterprise inference server guide covering vLLM in more depth.
What Does This Article Not Cover?
This is an explainer built from llama.cpp's public documentation and repository, not a hands-on benchmark report.
- No independently measured tokens-per-second or latency figures β throughput depends heavily on your specific hardware, model, quantization format, and build flags
- No independently verified quality-loss percentages for specific quantization formats β these vary by model architecture and task
- No line-by-line security audit of the llama.cpp codebase β it is open-source and MIT-licensed, so the code itself is available for review
- No coverage of every backend or build flag llama.cpp supports β this article focuses on the backends and formats most local-LLM users choose between
- No coverage of commercial support arrangements, since llama.cpp is a community open-source project rather than a vendor product with a support contract
Common Mistakes When Trying llama.cpp
Most friction with llama.cpp comes from treating it like a consumer app rather than a build-it-yourself engine.
Frequently Asked Questions
What is llama.cpp?
llama.cpp is a free, MIT-licensed C/C++ inference engine created by Georgi Gerganov that runs large language models locally on CPU or GPU. It is maintained by the ggml-org organization on GitHub.
Is llama.cpp free?
Yes. llama.cpp is free, open-source software released under the MIT license, with no subscription or account requirement.
What is GGUF?
GGUF is the model file format llama.cpp defined to store quantized model weights along with the metadata needed to run them, such as the tokenizer and architecture details. It is used across much of the local-AI ecosystem, not only by llama.cpp itself.
Does Ollama use llama.cpp?
Ollama is commonly reported, in community and third-party technical write-ups, to use llama.cpp as its inference engine on most platforms, adding its own model registry, CLI, and Modelfile system on top. Ollama does not prominently document this dependency itself.
What hardware does llama.cpp support?
llama.cpp supports CPU-only execution (with AVX/AVX2/AVX512 optimizations) as well as GPU acceleration via CUDA (NVIDIA), Metal (Apple Silicon), Vulkan, ROCm/HIP (AMD), and SYCL (Intel).
What quantization formats does llama.cpp support?
llama.cpp supports quantization from 8-bit formats like Q8_0 down to roughly 1.5-bit formats, with widely used middle-ground formats like Q5_K_M and Q4_K_M balancing file size against output quality.
Do I need to know C++ to use llama.cpp?
No β running models through the llama-cli or llama-server binaries does not require writing code. Building the project from source requires a C++ toolchain and CMake, but not writing C++ yourself, unless you want to modify the engine.
Does llama.cpp have a graphical user interface?
llama-server includes a built-in browser-based web UI, but there is no separate graphical installer or desktop app comparable to LM Studio. Several third-party apps, including Ollama-compatible and OpenAI-API-compatible front ends, can connect to llama-server instead.
Is llama.cpp better than Ollama?
"Better" depends on what you need: llama.cpp gives more direct control over build flags, quantization, and server configuration, while Ollama gives a faster, simpler path to a running model. This article does not claim one is technically superior; see the comparison table above and the dedicated benchmark comparison for more detail.
Can llama.cpp run without a GPU?
Yes. llama.cpp is designed to run on CPU-only hardware using vectorized instructions (AVX/AVX2/AVX512), in addition to supporting GPU acceleration when available.