Can CPUs Run LLMs?
Yes, modern CPUs (for example Intel Core 10th gen or newer, AMD Ryzen 5000 or newer, Apple M-series) can run 3–8B models at an estimated 8–30 tokens/second on dual-channel RAM. That is typically 5–10× slower than a discrete GPU or more, but it needs no VRAM. A PC with 8–32 GB of system RAM can run models that would otherwise call for a dedicated graphics card.
CPU inference trades speed for accessibility: no GPU driver setup and no VRAM limit, only system RAM. For casual use cases (a chatbot for one user, offline document processing), CPU-only is practical.
Speed is set by memory bandwidth, not clock speed: generating each token reads the whole model file from RAM once, so tokens per second ≈ memory bandwidth ÷ file size (a ceiling; real speed is lower). llama.cpp, which Ollama builds on, also uses CPU vector instructions such as AVX2 or ARM NEON where the processor supports them.
📍 In One Sentence
Modern CPUs can run 3B–8B LLMs at an estimated 8–30 tokens per second using Q4_K_M quantization and llama.cpp or Ollama, depending on RAM speed and model size.
💬 In Plain Terms
Yes. A typical laptop or desktop with enough RAM can run a small AI model locally without a GPU. It will be slower than a GPU, but fast enough for tasks where you are not waiting in real time.
Which CPU-Only LLMs Are Best in 2026?
Among the five models compared here, Phi-4 Mini (3.8B, Q4_K_M) is the best all-round CPU-only pick in 2026: an estimated 15–25 tok/s on dual-channel RAM from a 2.5 GB file. The table lists the five models by file size, RAM need and estimated speed. Speeds are PromptQuorum's own estimates, derived from the memory-bandwidth ceiling (dual-channel DDR4-3200 = 51.2 GB/s, DDR5-5600 = 89.6 GB/s, divided by the Ollama file size), for a 16 GB dual-channel desktop; they are not lab measurements. Single-channel RAM halves the result. Gemma 4 E2B is the least certain estimate because part of its 4.6 GB file is embedding tables; check your own speed with `ollama run <model> --verbose`.
Model | Params | File (Ollama) | Min RAM | Est. tok/s | Best For |
|---|---|---|---|---|---|
| Phi-4 Mini | 3.8B | 2.5 GB | 3 GB | 15–25 | General chat, code assist |
| Llama 3.2 3B | 3B | 2.0 GB | 2.5–3 GB | 18–30 | Smallest file, balanced |
| Gemma 4 E2B | 2.3B eff. / 5.1B total | 4.6 GB | 5–6 GB | 8–14 | 128K context, image input |
| Qwen3 8B | 8B | 5.2 GB | 6 GB | 8–15 | Higher quality, 16 GB RAM |
| DeepSeek-R1 7B | 7B (Qwen distill) | 4.7 GB | 6 GB | 8–14 | Reasoning, math |
How Fast Is CPU vs GPU Inference?
CPU inference is typically 5–10× slower than a discrete GPU or more, because the GPU has 7–20× the memory bandwidth: dual-channel DDR4-3200 delivers 51.2 GB/s, an RTX 3060 12 GB 360 GB/s and an RTX 4090 1,008 GB/s. The table applies the bandwidth rule (ceiling = bandwidth ÷ file size) to the 2.5 GB Phi-4 Mini file. Ceilings are upper limits; the estimated speeds are PromptQuorum approximations (roughly 75–100% of the ceiling on a healthy install), not lab measurements. GPU bandwidth figures come from NVIDIA's published specifications. Apple Silicon is not listed because Ollama uses the GPU on a Mac by default, so the CPU-only rule does not apply the same way.
Hardware | Bandwidth | Model | Ceiling tok/s | Est. tok/s |
|---|---|---|---|---|
| Dual-channel DDR4-3200 desktop (CPU) | 51.2 GB/s | Phi-4 Mini 2.5 GB | ~20 | 15–20 |
| Dual-channel DDR5-5600 desktop (CPU) | 89.6 GB/s | Phi-4 Mini 2.5 GB | ~36 | 25–32 |
| Single-channel DDR4-3200 laptop (CPU) | 25.6 GB/s | Phi-4 Mini 2.5 GB | ~10 | 7–10 |
| RTX 3060 12 GB (GPU) | 360 GB/s | Phi-4 Mini 2.5 GB | ~144 | Typically 5–10× CPU |
| RTX 4090 24 GB (GPU) | 1,008 GB/s | Phi-4 Mini 2.5 GB | ~403 | Up to ~20× DDR4 ceiling |
How Much RAM Does Each CPU-Only Model Need?
Rule of thumb: model file size + about 500 MB overhead = minimum RAM, plus more for a long context. A 2.0 GB file needs roughly 2.5–3 GB of free system RAM. File sizes are the Ollama downloads checked on ollama.com; context lengths are the maximums Ollama lists, while Ollama's default context is 4096 tokens unless you set `OLLAMA_CONTEXT_LENGTH`:
Model | File (Ollama) | Min RAM | Comfortable | Context Length |
|---|---|---|---|---|
| Llama 3.2 3B | 2.0 GB | 2.5–3 GB | 6 GB | 128K |
| Phi-4 Mini 3.8B | 2.5 GB | 3 GB | 6 GB | 128K |
| Gemma 4 E2B | 4.6 GB | 5–6 GB | 8 GB | 128K |
| DeepSeek-R1 7B | 4.7 GB | 6 GB | 8 GB | 128K |
| Qwen3 8B | 5.2 GB | 6 GB | 8–16 GB | 40K |
How Do You Enable CPU-Only Inference?
Ollama (simplest): run `ollama run phi4-mini`. Ollama uses the CPU when it finds no supported GPU. To force CPU on a machine that has a GPU, set `CUDA_VISIBLE_DEVICES=-1` (NVIDIA) or `OLLAMA_VULKAN=0` (Vulkan GPUs, including some integrated GPUs) before starting Ollama, then check the PROCESSOR column of `ollama ps`. LM Studio: set the GPU offload slider to 0 layers when loading the model. llama.cpp: use `--n-gpu-layers 0` to disable GPU offloading.
ollama run phi4-mini
# in a second terminal, while the model is loaded:
ollama ps # PROCESSOR column should read 100% CPUHow Do You Maximize CPU Inference Speed?
Memory bandwidth and file size set CPU speed, so dual-channel RAM and a Q4_K_M file give the biggest gains. Specific tips:
- Fill both memory channels — two matching RAM sticks double peak memory bandwidth versus one stick, the biggest single speed-up on a CPU-only PC.
- Use Q4_K_M quantization — the file is roughly 70% smaller than FP16 with a small quality loss, and because CPU decoding is bandwidth-limited a smaller file generates faster in about the same proportion.
- Reduce the context window — a longer context grows the KV cache and slows prompt processing. Set `OLLAMA_CONTEXT_LENGTH=2048` (the Ollama default is 4096) or `/set parameter num_ctx 2048` inside `ollama run`.
- Quantize the KV cache — `OLLAMA_FLASH_ATTENTION=1` with `OLLAMA_KV_CACHE_TYPE=q8_0` roughly halves KV-cache memory compared with the f16 default, which helps most on 8 GB machines.
- Keep the model loaded — Ollama unloads a model after 5 minutes by default. `OLLAMA_KEEP_ALIVE=30m` avoids reloading the file for every session.
- Check the thread count — Ollama chooses threads automatically; in llama.cpp, set `-t` to your number of physical cores. Close memory-hungry apps so the model does not swap.
When Should You Use CPU Instead of GPU?
Use Case | CPU | GPU |
|---|---|---|
| Real-time chat (sub-1-sec latency) | ⚠️ Usable at 15+ tok/s (est.); slower on 1 channel | ✅ Typically 5–10× faster |
| Batch processing (documents, logs) | ✅ Fine (speed matters less) | ⚠️ Overkill |
| Production API (cost-constrained) | ✅ Uses existing hardware | ⚠️ Extra GPU cost and power |
| Edge device (Raspberry Pi) | ✅ Often the only option | ❌ Limited GPU options |
| Development / local testing | ✅ Lower power, quieter | ⚠️ Overkill |
| LLM fine-tuning | ❌ Impractical (far slower) | ✅ Much faster |
Frequently Asked Questions About CPU-Only LLMs
How fast is CPU-only inference compared to a GPU?
For Phi-4 Mini (2.5 GB file), the CPU ceiling is about 20 tok/s on dual-channel DDR4-3200 (51.2 GB/s) and about 36 tok/s on dual-channel DDR5-5600 (89.6 GB/s); PromptQuorum's estimate is 15–25 tok/s in practice. A discrete GPU has far more memory bandwidth (RTX 3060: 360 GB/s, RTX 4090: 1,008 GB/s), so it is typically 5–10× faster or more. These are estimates, not lab measurements.
What's the smallest model that still produces coherent output on CPU?
Quality drops quickly below about 2B parameters. Llama 3.2 3B (2.0 GB file) and Phi-4 Mini (3.8B, 2.5 GB) are the smallest picks in this guide and give usable chat on 8 GB RAM. For even smaller files, see the fastest local LLMs for low-end PCs guide.
Can I run a 13B model on CPU?
Yes, but it is slow. A 13B model at Q4_K_M is a file of roughly 8 GB and needs 12 GB or more of system RAM. The bandwidth ceiling is about 6 tok/s on dual-channel DDR4-3200 (51.2 ÷ 8) and about 11 tok/s on DDR5-5600, and real speed is lower. That suits batch processing rather than live chat.
Does CPU inference use the GPU at all?
Not if you force it off. With `CUDA_VISIBLE_DEVICES=-1` (NVIDIA), `OLLAMA_VULKAN=0` (Vulkan GPUs) or llama.cpp `--n-gpu-layers 0`, inference runs on the CPU and system RAM only. Check the PROCESSOR column of `ollama ps` to confirm.
Is CPU-only inference stable?
It avoids GPU-specific failure modes such as driver crashes and GPU out-of-memory errors. The main remaining risk is system RAM running out, which pushes the model into swap and slows generation sharply; you control that by model choice and context length.
Do I need to adjust settings for Apple Silicon CPUs?
Usually not. Ollama uses the Apple GPU through Metal by default and the chip's unified memory, so a Mac is not a CPU-only case. Memory bandwidth varies by chip, so the same rule applies: tokens per second ≈ memory bandwidth ÷ model file size.
How much VRAM does a 7B model need in CPU-only mode?
Zero dedicated VRAM. A 7B model at Q4_K_M (the Ollama deepseek-r1:7b file is 4.7 GB) needs roughly 6 GB of system RAM to run comfortably. This is the core advantage of CPU-only inference — no GPU or VRAM required.
Can I fine-tune an LLM on CPU?
Technically yes, but it is impractical: training is far slower on a CPU than on a GPU. Use CPU-only mode for inference and switch to a GPU for any fine-tuning or training workload.
Can I run multiple models simultaneously on CPU?
Technically possible if RAM allows, but not practical. Models loaded together compete for the same memory bandwidth, so each one generates more slowly. Run one model at a time for the best speed, and see `OLLAMA_NUM_PARALLEL` and `OLLAMA_KEEP_ALIVE` in the Ollama FAQ if you serve several requests.
What are the security risks of CPU-only inference?
Running locally means prompts do not have to leave your machine, which removes the transmission risk of a cloud API. The remaining risks are those of any local software: keep the operating system patched and the machine physically secured, because sensitive data can still sit unencrypted on disk.
Is there a speed difference between Ollama and llama.cpp for CPU inference?
PromptQuorum has not measured it. Ollama builds on llama.cpp, so both rely on the same kind of CPU optimizations, and any gap comes mainly from thread handling and default settings. Start with Ollama's defaults, and use llama.cpp if you need finer-grained flags such as `-t`. Compare the two on your own machine with `ollama run <model> --verbose`.
Is CPU-only inference practical on an old laptop with 8–10 GB of RAM?
Yes, with a small model. Llama 3.2 3B (2.0 GB) or Phi-4 Mini (2.5 GB) fit within 8–10 GB of RAM. On a single-channel DDR4-3200 laptop the ceiling is about 10 tok/s for Phi-4 Mini, so expect an estimated 5–10 tok/s: fine for a lightweight chatbot or batch work, slow for fast back-and-forth chat.
Can CPUs run LLMs?
Yes. Modern CPUs (for example Intel Core 10th gen or newer, AMD Ryzen 5000 or newer) can run 3–8B models at an estimated 8–30 tok/s on dual-channel RAM, depending on model size. That is typically 5–10× slower than a discrete GPU or more, but it requires no dedicated VRAM.
What is the best CPU-only LLM?
Among the models compared here, Phi-4 Mini (3.8B, 2.5 GB file) is the best all-round pick at an estimated 15–25 tok/s. For the smallest file, Llama 3.2 3B (2.0 GB, est. 18–30 tok/s). For quality on 16 GB RAM, Qwen3 8B (5.2 GB, est. 8–15 tok/s). Gemma 4 E2B has a 4.6 GB download, so it is not the speed pick.
How much RAM do I need for CPU-only inference?
Use the rule: model file size + about 500 MB overhead, plus more for a long context. Phi-4 Mini (2.5 GB) needs about 3 GB, Llama 3.2 3B (2.0 GB) about 2.5–3 GB, Gemma 4 E2B (4.6 GB) about 5–6 GB, and Qwen3 8B (5.2 GB) about 6 GB.
How do I enable CPU-only mode?
In Ollama, run `ollama run phi4-mini`; it uses the CPU when no supported GPU is found. To force CPU on a machine with a GPU, set `CUDA_VISIBLE_DEVICES=-1` or `OLLAMA_VULKAN=0`. In llama.cpp, use `--n-gpu-layers 0`. In LM Studio, set GPU offload to 0 layers. Confirm with `ollama ps`.
Is CPU inference practical for production?
Yes, if you do not need real-time latency. Batch processing, asynchronous APIs and offline workflows work on a CPU. A CPU serves one user at low speed, so for interactive chat with sub-1-second latency or many concurrent users, use a GPU.
Which CPU-only models handle long context best?
On Ollama, Gemma 4 E2B, Phi-4 Mini, Llama 3.2 3B and DeepSeek-R1 7B list a 128K context window and Qwen3 8B lists 40K. Ollama's default context is only 4096 tokens unless you set `OLLAMA_CONTEXT_LENGTH`. A long context is not free on a CPU: prompt processing scales with the number of tokens you load, and the KV cache grows, so a very long window can take minutes to fill before the first token appears. If you do not need the whole window, cap it with `OLLAMA_CONTEXT_LENGTH` and answers come back noticeably faster.
Next steps
- Fastest Local LLMs for Low-End PCs — Old or low-end PC? Estimated speeds by RAM tier →
- LLM Quantization Explained — Why Q4_K_M matters for CPU inference speed →
- Best Local LLMs for Coding — Lightweight coding models that can run on CPU →
Why CPU-Only LLMs Matter for Privacy-Sensitive Deployments
EU / GDPR: when a model runs offline on your own CPU, prompts and documents do not have to leave the device, which for many small teams is a technically straightforward way to avoid third-country transfer questions under Art. 44 GDPR. Whether GDPR, the EU AI Act or other obligations apply depends on who deploys the system and for what purpose; this guide is not legal advice, and overall compliance depends on your full setup, not the inference architecture alone.
Offline and low-connectivity environments: CPU models work without a reliable internet connection. A model file such as Phi-4 Mini that you download once can run without further network access, which helps in regions with unstable connectivity or metered bandwidth.
What Are the Common CPU Inference Mistakes?
- Running FP16 instead of Q4_K_M. An FP16 Phi-4 Mini file is about 7.6 GB versus 2.5 GB at Q4_K_M, so the bandwidth ceiling drops from about 20 to about 7 tok/s on dual-channel DDR4-3200. Use quantized GGUF models for CPU inference.
- Forgetting to force CPU-only in llama.cpp. Without explicit flags, a GPU build may offload layers. Set `--n-gpu-layers 0` for pure CPU mode.
- Running one RAM stick. A single memory channel halves peak bandwidth, so the same model generates at roughly half the speed. Install two matching sticks before buying a faster CPU.
- Choosing too large a model. Phi-4 Mini (2.5 GB, ceiling about 20 tok/s on dual-channel DDR4-3200) is far more comfortable for interactive use than DeepSeek-R1 7B (4.7 GB, ceiling about 11 tok/s). Match model size to memory bandwidth, not just to free RAM.
- Using the wrong Ollama tag. `ollama run phi` loads Phi-2, not Phi-4 Mini. Use `ollama run phi4-mini`, and check ollama.com/library for exact model tags.
Sources
- llama.cpp on GitHub — CPU inference, thread and GPU-offload options
- Ollama FAQ — context length, KV cache type, keep-alive and the CPU/GPU split in ollama ps
- Ollama GPU documentation — forcing CPU-only, OLLAMA_VULKAN and supported GPUs
- Ollama library: phi4-mini — 2.5 GB, 128K context
- Ollama library: llama3.2 — 3B file 2.0 GB, 128K context
- Ollama library: gemma4 — e2b download 4.6–7.5 GB, 128K context, 2.3B effective / 5.1B total parameters
- Ollama library: qwen3 — 8B file 5.2 GB, 40K context
- Ollama library: deepseek-r1 — 7B file 4.7 GB, distilled from Qwen2.5
- Microsoft Phi-4-mini-instruct model card — Hugging Face
- NVIDIA GeForce RTX 3060 specifications — memory bandwidth
- NVIDIA GeForce RTX 4090 specifications — memory bandwidth
- Gemma 4 E2B summary — llm-stats (third-party)