Local LLM Hardware in 2026: GPU vs Mini PC vs Mac Compared

Choosing a GPU depends on your budget and the model size you want to run. The NVIDIA RTX 40-series (4060, 4070 Ti, 4090) and RTX 50-series (5060 Ti, 5080) dominate for local LLMs in 2026. Here are the top recommendations by use case:

VRAM requirements depend on three factors: model size (parameters), quantization (bits per weight), and inference mode. Use this formula to determine if your GPU has enough memory. For an interactive calculator, see the VRAM calculator for local LLMs.

Formula:

```text VRAM (GB) = (Model Size × Quantization Bits) ÷ 8 ```

Quantization values: FP16 = 16 bits, Q8_0 = 8 bits, Q5_K_M = 5 bits, Q4_K_M = 4 bits. The practical sweet spot is Q4_K_M -- it uses 4-bit weights with K-quantization, which NVIDIA GPUs accelerate more efficiently than the older Q4_0 format.

The VRAM formula (Model Size × Bits ÷ 8) covers model weights only -- KV cache adds significant additional VRAM that most guides ignore.

The KV cache stores attention state for every token in your context window. It grows linearly with context length and stays in VRAM throughout the session.

KV cache VRAM formula: `KV cache ≈ layers × heads × head_dim × 2 × context_length × 2 bytes`

As of May 2026, NVIDIA GPUs deliver the highest tokens/sec for local LLM inference across all price points. The sections below each tier give specific model recommendations. For a detailed benchmark comparison, see the best GPUs for local LLM guide.

Use this as a quick lookup by your GPU's VRAM tier:

All models listed below are open-weights — downloadable, fine-tunable, and free to run locally. If you're choosing between open-weights and proprietary APIs, see our open-source vs proprietary LLMs comparison for cost and performance trade-offs at different token volumes.

Hardware determines which models you can run; prompt engineering determines how well they perform. A well-structured prompt on a 7B model often outperforms a lazy prompt on a 70B model. See the complete prompt engineering guide for techniques that maximise output quality at any parameter count.

On a 16 GB VRAM GPU (NVIDIA RTX 4080, RTX 5080, or RTX 4090 laptop), the practical ceiling is 14-24B models. Mistral Small 3.1 24B at Q4_K_M is the best overall choice: it uses 13 GB VRAM, runs at 55 tok/sec, and is EU-origin with Apache 2.0 license.

Devstral Small 24B Q4_K_M fits at ~16 GB and is optimized for agentic coding workflows. The table below shows which models fit and which do not. "Does NOT fit" rows are included intentionally -- this is the most common mistake 16 GB owners make.

On a 12 GB VRAM GPU (NVIDIA RTX 4070 Ti, RTX 5070, or RTX 5060 Ti), you can run 7-8B models at Q8, 14B at Q4_K_M, or the new Llama 4 Scout 17B at Q4_K_M (MoE). Llama 4 Scout uses a Mixture-of-Experts architecture with 17B active parameters out of 109B total -- this makes Scout significantly more VRAM-efficient than its parameter count suggests.

Llama 3.1 8B at Q8_0 is the most reliable choice for conservative setups: 9 GB VRAM, 80 tok/sec, and full instruction-following quality. Qwen2.5 14B at Q4_K_M also fits at ~8.5 GB and delivers notably better reasoning than the 8B tier.

The RTX 4090 has 24 GB VRAM -- not enough for most 70B models at acceptable quality. Llama 3.3 70B at Q4_K_M requires approximately 39 GB. The common misconception is that "Q4 is small" -- at 70B parameters, even Q4 is large.

On a single RTX 4090, the best strategy is 27-32B models, which deliver strong quality and fit comfortably. Qwen 3.6 27B at Q4_K_M is the best dense coding model (77.2% SWE-bench). For true 70B at Q4+, you need 2× RTX 4090 or a 48 GB server GPU. See how to run 70B models on 24 GB VRAM for advanced techniques.

With a dedicated GPU, CPU and RAM are secondary components. The GPU handles matrix math; CPU/RAM manage context preparation. For a full comparison of GPU vs CPU vs Apple Silicon inference speeds, see the GPU vs CPU vs Apple Silicon guide.

Minimum CPU: 8-core processor (Intel Core i7 14th gen, AMD Ryzen 7 7700X, or newer). Older CPUs add 20%+ latency.

RAM: 16 GB minimum (with GPU). If running without GPU, 32+ GB recommended. RAM does not directly limit model size when GPU is present.

Storage: 500 GB SSD for model files and OS. M.2 NVMe is preferred (faster model loading).

Without a GPU, a machine with 16 GB system RAM can run 3B-7B models at 8-20 tokens/sec using CPU inference. The bottleneck is memory bandwidth, not RAM capacity -- CPUs have far lower bandwidth than GPUs, which is why inference is 5-10× slower.

On 16 GB system RAM, the practical rule is: model file size + 4 GB OS overhead ≤ 16 GB. A 7B model at Q4_K_M (4.9 GB) fits, but leaves little headroom for long contexts. The table below shows realistic options as of May 2026.

For a complete speed-optimized model guide covering CPU-only, 4 GB, 6 GB, and 8 GB VRAM tiers with real benchmarks, see **Fastest Local LLMs for Low-End PCs**.

Model files are large: a 7B model at 4-bit quantization is 4-5 GB. Plan storage around the number and size of models you want to keep locally.

Building a local LLM machine from scratch means prioritizing GPU first, then CPU and RAM. Here are three realistic configurations. For multi-GPU builds, see the multi-GPU local LLM guide.

If a $250–400 GPU is outside your budget, or your laptop is too old to support modern inference engines, local LLMs may not be cost-effective for you in 2026.

Calculate the real cost:

- Local: $800–2,000 upfront hardware + electricity + maintenance over 2–3 years

- Cloud: $5–50/month for typical developer use (Llama API or GPT-4o mini)

For light users (< 100,000 tokens/month), cloud APIs cost $5–10/month and require zero hardware. For heavy users (> 10M tokens/month), local breaks even in 6–12 months.

Compare full local vs cloud cost and performance trade-offs** to find your break-even point. Many developers discover cloud is cheaper for their actual usage pattern.

Already shopping below the recommended VRAM tiers? See Best Local AI App for a Low-End PC for which model and app combinations actually run on 8 GB or less.

With correct settings, llama.cpp on an RTX 4070 Ti achieves 85-95 tokens/sec on Llama 3.1 8B Q4_K_M -- more than double the default out-of-box speed. The single most impactful flag is `--n-gpu-layers 99`, which offloads all model layers to the GPU. Without it, layers fall back to CPU, creating a severe bottleneck.

These settings apply to llama.cpp directly and to Ollama (which uses llama.cpp internally). Ollama sets `--n-gpu-layers 99` automatically on NVIDIA hardware if drivers are installed correctly.

Apple Silicon (M-series) runs local LLMs efficiently using unified memory shared between CPU and GPU. M5 introduced since October 2025 offers a significant upgrade for local inference. Apple claims 4× faster LLM prompt processing vs M4.

The M5 Max with 128 GB unified memory is the first Apple Silicon chip that comfortably runs 70B models at Q4_K_M -- comparable to dual RTX 4090 desktops but in a laptop or Mac Studio form factor. The M5 Pro with 64 GB unified memory handles 32B models with generous headroom for KV cache and multitasking.

For production deployment (24/7 operation, multiple users), server-grade hardware is recommended over consumer GPUs. Consumer hardware is optimized for gaming, not sustained inference.

The NVIDIA DGX Spark ($3,999) is the only consumer desktop as of May 2026 that fits Llama 3.3 70B at Q8_0 entirely in unified memory.

Built on the GB10 Grace Blackwell Superchip, the DGX Spark launched in late 2025 as a compact desktop AI computer with 128 GB LPDDR5x unified memory. As of May 2026, the DGX Spark also runs Llama 4 Scout and Maverick entirely in memory, as well as Kimi K2.6 (quantized), making it suitable for multi-GPU setups at this tier.

EU (GDPR + EU AI Act): Running LLMs locally keeps all inference data within your infrastructure, eliminating cross-border data transfer concerns under GDPR Article 44. As of May 2026, EU enterprises deploying LLMs for customer data processing must ensure models never phone home -- local hardware removes this risk entirely. EU AI Act high-risk system obligations apply from August 2, 2026 (pending Digital Omnibus which may delay to December 2027). Local hardware satisfies data residency requirements by default.

Japan (APPI): Japan's Act on the Protection of Personal Information (APPI) revision (2022) requires data minimization for AI processing. On-premises LLM hardware on a RTX 4090 workstation satisfies this requirement for document processing and customer support automation.

China: China's Cyberspace Administration of China (CAC) Generative AI Regulations (2023) require domestically-deployed AI models to undergo registration. Running local hardware with open-weight models avoids API-based compliance exposure for internal enterprise use.