Skip to main content
PromptQuorumBuilt for humans. Structured for AI.
Home/Local LLMs
Hardware & Performance

Local LLM Hardware: GPU vs Mini PC vs Mac Compared

·13 min readBy · Founder of PromptQuorum · Discovery engine for open-weight & open-source AI

VRAM is the main hardware limit for local AI: at Q4_K_M, 7B models need 8 GB, 14B 12 GB, 24B 16 GB, 32B 24 GB, 70B ~40 GB. A model that does not fit spills onto the CPU and runs 5-10× slower; long context adds 25-100%.

Key Takeaways

  • 7B Q4 rule of thumb: a 7B model at Q4_K_M quantization needs about 4-5 GB of VRAM (or system RAM for CPU inference) — roughly 0.6 GB per billion parameters at 4-bit.
  • VRAM math: VRAM (GB) ≈ parameters in billions × bits per weight ÷ 8. Example: 70B at 4 bits = 70 × 4 ÷ 8 = 35 GB; real Q4_K_M files average about 4.6 bits per weight, so plan for ~40 GB.
  • 12 GB VRAM (RTX 4070 Ti): Best model: Llama 3.1 8B Q8 (~9 GB, 80 tok/sec). Also: Qwen3 8B (~8 GB, best multilingual + coding). Note: Llama 4 Scout (17B active / 109B total MoE) needs ~55 GB at Q4 and does NOT fit 12 GB.
  • 16 GB VRAM (RTX 5080 / RTX 5070 Ti): Best model: Mistral Small 3.1 24B Q4_K_M (~13 GB, 55 tok/sec). Also: Devstral Small 24B Q4_K_M for agentic coding. Mistral Small 4 (March 2026) is the newer one-model successor that folds in reasoning, vision, and coding.
  • 24 GB VRAM (RTX 4090 / RTX 5090): Most 70B models at Q4_K_M (~40 GB) do NOT fit. Best option: Qwen3.6 27B Q4_K_M (~16 GB, 77.2% SWE-bench Verified (developer-reported), strong dense coder) or DeepSeek-R1 32B Q4_K_M (~19 GB, 60 tok/sec).
  • CPU-only (16 GB system RAM): Llama 3.2 3B Q8 (20 tok/sec) or Phi-4 Mini Q4_K_M (25 tok/sec). A used RTX 4060 8 GB (~$280) or new RTX 5060 Ti 16 GB (~$800) is 5-10× faster.
  • MacBook on 8 GB RAM: run 3-4B models only — Phi-4 Mini, Llama 3.2 3B, or Gemma 3 4B at Q4_K_M via llama.cpp/Ollama (Metal). macOS needs at least 8 GB itself, so even these leave almost no room and 7B does not fit; 16 GB (about 8 GB left after macOS) is the practical Mac minimum.
  • Apple M5 Max (128 GB unified): runs 70B models at Q4_K_M comfortably (~12-15 tok/sec) in a laptop or Mac Studio — alongside Mac Studio and 128 GB AMD Strix Halo systems that also hold a 70B model. None of these fit GLM-5.3 (needs well over 128 GB even at 2-bit).
  • Current prices: the GDDR7 shortage has worsened, not eased — RTX 5090 street pricing has climbed past $4,300–$5,000+, and the RTX 4090 remains discontinued. Budget-conscious buyers should also check AMD's RX 9070 XT (16 GB GDDR6, ~$690–740) — GDDR6 has dodged most of the shortage, making it noticeably cheaper than the RTX 5070 Ti at similar VRAM. Buy from in-stock listings; check live prices before purchase.
  • llama.cpp speed tip: Always set `--n-gpu-layers 99`. This alone can roughly double speed on an RTX 4070 Ti (from ~40 to ~85 tok/sec; approximate, not a PromptQuorum measurement).
  • Quick reference: 7B@Q4_K_M = 4-5 GB | 70B@Q4_K_M = 40 GB | GLM-5.3@2-bit = well over 128 GB | RTX 4070 Ti = ~80 tok/s | RTX 4090 = ~150 tok/s | CPU-only 16 GB = 12-28 tok/s

📍 In One Sentence

Local LLM hardware is determined by VRAM: 7B models need 8 GB, 13B–14B need 12–16 GB, and 70B models need 35–48 GB — a used RTX 4060 8 GB (~$280) is a practical entry-level GPU in 2026.

💬 In Plain Terms

VRAM is the dedicated memory on your graphics card. The bigger the AI model, the more VRAM it needs. Rule of thumb: divide the model's size in gigabytes by its compression level (Q4 = divide by 8) to estimate VRAM. More VRAM means bigger models and faster responses.

This page contains links to third-party products for reference. PromptQuorum is not enrolled in any affiliate program — these are plain links that earn no commission. Clicking links and your next steps are entirely your own responsibility. Any recommendation on this page reflects our own reading of the vendors' public pages, not a vendor endorsement or a test.

Running a local LLM means matching the model to your GPU's VRAM. A 7B model needs about 4-5 GB VRAM at Q4_K_M (the standard rule of thumb: roughly 0.6 GB per billion parameters at 4-bit) or 8-9 GB at Q8, a 14B model needs ~9 GB at Q4_K_M, and most 70B models need ~40 GB -- more than a single RTX 4090 (24 GB) holds. This guide gives the exact hardware requirement per model size, then the best model for 8 GB, 12 GB, 16 GB, and 24 GB VRAM tiers, what it really takes to run 70B (and the far larger GLM-5.3) locally, CPU-only inference on 16 GB system RAM, MacBook 8 GB options, and current GPU prices after this year's memory shortage.

Slide Deck: Local LLM Hardware: GPU vs Mini PC vs Mac Compared

The slide deck below covers: GPU VRAM tiers for 12/16/24 GB, best models per tier with VRAM usage and speed benchmarks, CPU-only inference on 16 GB RAM, and llama.cpp speed flags for RTX 4070 Ti. Download the PDF as a Local LLM Hardware Guide 2026 reference card.

Browse the slides below or download as PDF for offline reference. Download Reference Card (PDF)

Local LLM Hardware: GPU vs Mini PC vs Mac Compared

Local LLM Hardware Requirements

The minimum hardware to run a local LLM in 2026 is an 8 GB VRAM GPU — or an Apple Silicon Mac with 16 GB unified memory — for 7B-class models. Requirements then scale with model size: 14B needs 12 GB, 24B needs 16 GB, 32B needs 24 GB, and a 70B model needs ~40 GB at Q4_K_M. GPU VRAM is the hard limit: it decides which models load at all. CPU and system RAM affect load time and CPU-only fallback speed, but not which model fits on the GPU.

Use this table as the direct answer to "what hardware do I need" — find your model size or VRAM tier, then jump to the per-tier model picks below. Speeds are approximate figures that vary by setup and are not PromptQuorum measurements.

Model size
VRAM at Q4_K_M
GPU example (2026)
Best model
Speed
3-4B4-5 GBAny 8 GB / Mac 8 GBPhi-4 Mini, Gemma 3 4B60-90 tok/s
7-8B5-9 GBRTX 5060 Ti, RTX 4060 (8 GB)Llama 3.1 8B, Qwen3 8B50-80 tok/s
14B~9 GBRTX 5070 (12 GB)Qwen3 14B~80 tok/s
24B~14 GBRTX 5070 Ti / 5080 (16 GB)Mistral Small 3.1 24B~55 tok/s
27-32B16-19 GBRTX 4090 / 5090 (24-32 GB)Qwen3.6 27B, DeepSeek-R1 32B55-60 tok/s
70B~40 GBDual RTX 5090, A100, Mac M5 Max 128 GBLlama 3.3 70B10-60 tok/s

•KeyPoint: In one sentence: match the model to your VRAM — 8 GB runs 7B, 12 GB runs 14B, 16 GB runs 24B, 24 GB runs 32B, and only 40 GB+ runs a 70B model at usable Q4_K_M quality.

•ProTip: Add headroom for the KV cache (conversation context): budget 25% on top of model weights for 8K context and up to 100% for 32K. See the KV cache section below.

Hardware Limitations of Local AI Models

The hardware limit of a local AI model is memory, not processing power: the model weights plus the context cache must fit in GPU VRAM (or in unified memory on a Mac), and whatever spills over runs on the CPU and system RAM, typically 5-10× slower. Four limits decide what you can run: VRAM capacity, memory bandwidth, context length and, without a GPU, system RAM.

The table gives the minimum memory per model size at Q4_K_M, using the figures from this guide. VRAM covers the weights only; add 25-100% for the KV cache depending on context length. The CPU-only column is an estimate: model file size plus about 4 GB for the operating system. For the quality, speed and real-time limits compared with cloud models, see local LLM vs cloud API trade-offs; for more size-by-size charts, see how much VRAM a local LLM needs.

  • VRAM capacity decides which models load at all. On a Mac, unified memory plays this role, but macOS keeps about 8 GB for itself (see Mac hardware).
  • Memory bandwidth decides speed once a model fits. CPUs have far lower bandwidth than GPUs, which is why CPU-only inference is 5-10× slower.
  • Context length is the hidden limit: the KV cache grows with every token, so a model that loads can still run out of VRAM at 32K context (see KV cache).
  • MoE and frontier models need memory for all experts, not only the active ones, so total parameters, not active parameters, set the requirement.
Model size
Min. VRAM (Q4_K_M)
CPU-only RAM (est.)
If you have less
3-4B4-5 GB~8-9 GBRarely a limit: 8 GB cards and Macs cope
7-8B5-9 GB~9-13 GBUse a 3-4B model or accept CPU offload
14B~9 GB~13 GBDrop to 7-8B or accept CPU offload
24B~14 GB~18 GB12 GB cards cannot hold it; drop to 14B
27-32B16-19 GB~20-23 GB16 GB cards need CPU offload; 24 GB fits
70B~40 GB~44 GBA 24 GB card holds only Q2_K; quality drops
MoE, 109B total (Llama 4 Scout)~55 GB~59 GBAll experts load; the 17B active size does not help
GLM-5.3 (~750B MoE, 2-bit)well over 128 GB combinedwell over 128 GBBeyond any single GPU or 128 GB machine

•KeyPoint: In one sentence: a local model is limited by the smallest memory pool it must fit in, and whatever does not fit in VRAM falls back to the CPU at a fraction of the speed.

12 GB, 16 GB and 24 GB VRAM GPUs: Models, Quantization and Context Limits

A 12 GB GPU runs 7-14B models, a 16 GB GPU runs up to 24B at Q4_K_M, and a 24 GB GPU runs 27-32B models; in every tier, context length is what you give up first. The table shows the largest model per tier, the quantization to use and how much context fits next to it, estimated from the KV cache table further down this guide.

  • 12 GB (RTX 5070, RTX 3060 12 GB): runs 7-8B at Q8_0 or 14B at Q4_K_M; 24B and larger do not fit. The 14B leaves only about 3 GB for context. Details: best models for 12 GB VRAM.
  • 16 GB (RTX 5070 Ti, RTX 5080, RX 9070 XT): runs 24B at Q4_K_M (~14 GB) with little context headroom, or 14B at Q8_0; from 30B up, CPU offload is needed. Details: best models for 16 GB VRAM.
  • 24 GB (RTX 4090 used, RTX 3090): runs 27-32B at Q4_K_M or Q5_K_M; 70B only at Q2_K. Details: which 70B models fit in 24 GB.
VRAM tier
Largest model that fits
Quantization
Context that fits (est.)
12 GB (RTX 5070)Qwen3 14B (~9 GB)Q4_K_M; Q8_0 for 8B8B Q8_0: ~16K; 8B Q4_K_M: ~32K
16 GB (RTX 5070 Ti, 5080)Mistral Small 3.1 24B (~14 GB)Q4_K_M; Q8_0 for 14B24B: ~2 GB spare; 8B Q8_0: ~32K
24 GB (RTX 4090, used)Qwen 3.6 27B (~16 GB), Qwen3 32B (~19 GB)Q4_K_M; Q5_K_M for 32B32B Q4_K_M: ~16K, ~32K with q8_0 KV

Context figures are PromptQuorum estimates scaled linearly from the KV cache table (fp16 K/V; Llama 3.1 8B about 0.5 GB per 4K tokens, Qwen3 32B about 1 GB per 4K). A quantized KV cache (`--cache-type-k q8_0`) roughly halves the cost. Real limits vary with runtime overhead and model architecture.

Best GPUs to Buy — Recommendations

The in-stock choice for local LLMs is the NVIDIA RTX 50-series (Blackwell): 5060 Ti, 5070, 5070 Ti, 5080, 5090. The RTX 40-series (4060, 4070 Ti, 4090) is discontinued and now sells scarce and above its old prices on the used market. The 2026 GDDR7/memory shortage has continued to worsen, pushing even 50-series cards further above MSRP, so treat every figure below as a typical street price and check live listings before buying. Recommendations by use case:

  • For 7B Models (Mistral, Phi-4, Llama 3.1) — Budget: RTX 5060 Ti 16 GB (~$800, well above its $429 MSRP) or a used RTX 4060 8 GB (~$280). Runs any 7B model at Q4_K_M (~4-5 GB). Speed: 50–70 tok/sec. Tier: Budget enthusiasts.
  • For 14B Models (Qwen3 14B, DeepSeek-R1) — Mainstream: RTX 5070 (12 GB, ~$880). A strong price-to-performance pick among new cards. Qwen3 14B Q4_K_M runs well with headroom. Speed: 85–110 tok/sec. Tier: Most popular.
  • For 24-32B Models (Qwen3.6, Mistral Small) — Mid-Range: RTX 5070 Ti (16 GB, ~$979) or RTX 5080 (16 GB, ~$1,450). Runs Mistral Small 3.1 24B and Devstral Small 24B Q4_K_M. Speed: 110–150 tok/sec. Tier: Professional developers. Budget alternative: AMD RX 9070 XT (16 GB GDDR6, ~$690–740) — GDDR6 has largely dodged the shortage, so it now undercuts the RTX 5070 Ti by $150–250 at similar VRAM, though ROCm's local-LLM tool support is narrower than CUDA's at the time of the last refresh (check the ROCm documentation).
  • For 70B Models (Llama 3.3) — High-End: RTX 5090 (32 GB, ~$2,000 MSRP but ~$4,300–5,000+ street and still climbing) fits 70B at Q4_K_M with light CPU offload. A used RTX 4090 (24 GB, ~$2,300) runs 70B only at Q2_K. For full Q4_K_M, use dual RTX 5090. Speed: ~200 tok/sec (5090, smaller models). Tier: Research + production.
  • For GLM-5.3 (~750B MoE) — Extreme: no single consumer GPU is enough. The 2-bit dynamic GGUF still needs well over 128 GB combined VRAM/RAM. Realistic local paths: a 4× RTX 3090/4090 rig with 192 GB+ system RAM, or a 512 GB Mac Studio (M5 Ultra), both relying on slow CPU/GPU hybrid offload (speed not verified by PromptQuorum).
  • A strong value pick in 2026: a single RTX 5070 Ti or 5080 (16 GB) is the sweet spot — it runs everything up to 32B at Q4_K_M without the street-price premium that currently applies to the RTX 5090.
  • For Apple Users: Mac M5 Max (128 GB unified memory, ~$6,000) runs 70B at Q4_K_M at ~12-15 tok/sec — slower than a multi-GPU desktop, but silent, power-efficient, and portable.
GPU
Best For
Price
Speed
Tier
RTX 5060 Ti (16 GB)7-13B models~$80050–70 tok/sBudget
RTX 5070 (12 GB)14B models~$88085–110 tok/sMainstream
RX 9070 XT (16 GB)24-32B models~$690–74090–130 tok/sBudget alternative
RTX 5070 Ti / 5080 (16 GB)24-32B models~$979–1,450110–150 tok/sProfessional
RTX 4090 (24 GB, used)32B, 70B (Q2)~$2,300150–180 tok/sEOL / used
RTX 5090 (32 GB)70B (Q4, light offload)~$2,000 MSRP (~$4,300–5,000+ street)~200 tok/sHigh-end
Dual RTX 509070B (Q4) full~$8,600–10,000300+ tok/sEnterprise
Mac M5 Max 128GB70B (Q4)~$6,000~12–15 tok/s (70B)Pro laptop
4× RTX 3090/4090 + 192GB RAMGLM-5.3 (2-bit)~$6,000–9,000slow (not verified)Extreme MoE

⚠️Warning: GPU pricing is volatile and still worsening. The GDDR7 shortage has pushed the RTX 5090 street price past $4,300–$5,000+ — far above its $1,999 MSRP — and the discontinued RTX 4090 is also reported at elevated used prices (see the Tom's Hardware price tracker in Sources). The prices above are typical street figures — always check current listings before buying. AMD's GDDR6-based RX 9070 XT is a notable exception: it has largely avoided the price spike. The RTX 5070 Ti's future is unclear: card makers reportedly told Hardware Unboxed in January 2026 that NVIDIA plans to discontinue it (heise, January 15, 2026), so check stock before planning a build around it. Speeds are approximate figures that vary by setup and are not PromptQuorum measurements.

How Do You Calculate VRAM Requirements?

VRAM requirements depend on three factors: model size (parameters), quantization (bits per weight), and inference mode. Use this formula to determine if your GPU has enough memory. For an interactive calculator, see the VRAM calculator for local LLMs.

Formula:

```text VRAM (GB) = (Model Size × Quantization Bits) ÷ 8 ```

Quantization values: FP16 = 16 bits, Q8_0 = 8 bits, Q5_K_M = 5 bits, Q4_K_M = 4 bits. The practical sweet spot is Q4_K_M -- it uses 4-bit weights with K-quantization, which NVIDIA GPUs accelerate more efficiently than the older Q4_0 format.

Model
FP16
Q8_0
Q5_K_M
Q4_K_M
Llama 4 Scout (109B total MoE)~218 GB~109 GB~68 GB~55 GB
Llama 3.1 8B16 GB8.5 GB5.7 GB4.7 GB
Qwen 3.6 27B~54 GB~28 GB~19 GB~16 GB
Qwen3 8B~16 GB~8.5 GB~5.7 GB~5 GB
Llama 3.3 70B140 GB70 GB48 GB40 GB
Qwen3 32B64 GB33 GB22 GB19 GB
Mistral Small 3.1 24B48 GB25 GB17 GB14 GB
Phi-4 Mini 3.8B7.6 GB4.1 GB2.7 GB2.3 GB

Q4_K_M is a common default for consumer hardware -- it needs roughly a quarter of the FP16 VRAM, and the quality loss is usually small (setup-dependent, not a PromptQuorum measurement). Llama 4 Scout uses MoE architecture with 17B active parameters out of 109B total. All 109B experts must be loaded into memory, so Scout needs ~55 GB at Q4 (fits 24 GB only at 1.78-bit). MoE reduces compute per token, not VRAM footprint.

Local LLM hardware requirements: VRAM calculator formula (model size × bits ÷ 8) for GPU and Mac
VRAM calculator showing the formula (Model Size × Bits) ÷ 8, with examples: 8B Q4_K_M = 4.7 GB, 13B Q5_K_M = 9.1 GB, 70B Q4_K_M = 40 GB. Q4_K_M is the recommended sweet spot for most hardware.

•KeyPoint: In one sentence: VRAM is the GPU's dedicated memory pool -- one of the key numbers that determine which AI models you can run locally and at what quality.

KV Cache: The Hidden VRAM Cost

The VRAM formula (Model Size × Bits ÷ 8) covers model weights only -- KV cache adds significant additional VRAM that most guides ignore.

The KV cache stores attention state for every token in your context window. It grows linearly with context length and stays in VRAM throughout the session.

KV cache VRAM formula: `KV cache ≈ layers × heads × head_dim × 2 × context_length × 2 bytes`

These figures assume fp16 (2 bytes per element) for the K and V tensors. Quantized KV cache (e.g. `--cache-type-k q8_0` in llama.cpp) uses roughly half the VRAM shown in the table.

Model
4K context
32K context
128K context
Llama 3.1 8B0.5 GB4 GB16 GB
Llama 3.3 70B1.25 GB10 GB40 GB
Qwen3 32B1 GB8 GB32 GB

•KeyPoint: In one sentence: KV cache is temporary VRAM used to store conversation context -- it grows with every token you generate and is separate from model weight storage.

⚠️Warning: A Llama 3.1 8B at Q4_K_M needs 4.7 GB for weights -- but add a 32K context window and total VRAM rises to ~8.7 GB. On an 8 GB card, this causes OOM errors.

•KeyPoint: Rule of thumb: this percentage shrinks as model size grows -- KV cache scales with layer count, while model weight size scales with total parameters, and layers grow more slowly than parameters as models get larger. Llama 3.1 8B (32 layers): a 32K-context cache is ~4 GB against ~4.7 GB of Q4_K_M weights (~85%). Llama 3.3 70B (80 layers): a 32K-context cache is ~10 GB against ~40 GB of Q4_K_M weights (~25%). Ollama default context is 2,048 tokens. To set higher: PARAMETER num_ctx 32768 in your Modelfile.

Which GPU Tier Matches Your Workload?

In published community benchmarks, NVIDIA GPUs typically deliver more tokens/sec than AMD cards at comparable prices for local LLM inference. The sections below each tier give specific model recommendations. For a detailed benchmark comparison, see the best GPUs for local LLM guide.

Tier
GPU
VRAM
Best For
Speed
Budget (~$800)RTX 5060 Ti16 GB7-13B models~60 tok/s
Mainstream (~$880)RTX 507012 GB7-14B models~90 tok/s
Mid (~$979)RTX 5070 Ti16 GB14-32B models~110 tok/s
High (~$1,450)RTX 508016 GB14-32B models~130 tok/s
Top (~$4,300–5,000+ street)RTX 509032 GB70B (Q4, light offload)~200 tok/s
Server ($7,000+)RTX 6000 Ada / A10048-80 GBMulti-user, 70B+Production
Desktop AI ($6,950)NVIDIA DGX Spark128 GBLarge MoE models (not GLM-5.3)~3 tok/s (dense 70B)
Local LLM GPU tier chart: RTX 5060 Ti to RTX 5090 and DGX Spark by VRAM, street price and speed
GPU tier recommendations (street prices): ~$800 RTX 5060 Ti (16GB, 7-13B, 60 tok/s), ~$880 RTX 5070 (12GB, 14B, 90 tok/s), ~$1,450 RTX 5080 (16GB, 14-32B, 130 tok/s), ~$4,300–5,000+ RTX 5090 (32GB, 70B, 200 tok/s), $6,950 DGX Spark (128GB, large MoE). GPU choice matters 10× more than CPU.

•KeyPoint: The RTX 50-series (Blackwell) is the current NVIDIA consumer generation — the RTX 40-series is discontinued. The RTX 5090 (32 GB) is the one consumer card in this guide that fits a 70B model at Q4_K_M with light offload, though the worsening memory shortage has pushed street prices far above its $1,999 MSRP.

Best Local LLMs by VRAM Tier

Use this as a quick lookup by your GPU's VRAM tier (speeds are approximate figures, not PromptQuorum measurements):

All models listed below are open-weights and can be downloaded and run locally under their own licences; check each licence before commercial use. Note: the Meta licences for its multimodal Llama models (Llama 3.2 Vision, Llama 4) do not grant usage rights to individuals domiciled in the EU or to companies with their principal place of business in the EU (end users of products that incorporate these models are exempt). If you're choosing between open-weights and proprietary APIs, see our open-source vs proprietary LLMs comparison for cost and performance trade-offs at different token volumes.

Hardware determines which models you can run; prompt engineering determines how well they perform. A well-structured prompt on a 7B model often outperforms a lazy prompt on a 70B model. See the complete prompt engineering guide for techniques that maximise output quality at any parameter count.

  • 8 GB VRAM (RTX 5060 Ti (8 GB variant), RTX 4060, Intel B580): Llama 3.1 8B Q4_K_M (4.7 GB, ~70 tok/s) -- recommended. Qwen3 8B (5 GB, best multilingual + coding). Phi-4 Mini 3.8B (2.3 GB, fastest). Gemma 3 4B (~3 GB, current-gen Google small model, multimodal). Avoid 13B+ models.
  • 12 GB VRAM (RTX 4070 Ti, RTX 5070, Intel B770): Llama 3.1 8B (4.7 GB, fast with headroom). Qwen3 14B Q4_K_M (8.5 GB, better reasoning on budget). Qwen3 8B (5 GB, best multilingual + coding). DeepSeek-R1 8B (5 GB, strong reasoning). Avoid 30B+ and MoE models like Llama 4 Scout (~55 GB at Q4).
  • 16 GB VRAM (RTX 4080, RTX 5070 Ti, RTX 5080): Mistral Small 3.1 24B Q4_K_M (14 GB, strong quality at the tier). Devstral Small 24B Q4_K_M (~16 GB) for agentic coding. Qwen3 14B (9 GB, fast with context headroom). Llama 3.3 70B at Q2_K (17 GB, possible but degraded quality).
  • 24 GB VRAM (RTX 5090, RTX 4090, Tesla L40): Qwen 3.6 27B Q4_K_M (~16 GB, 77.2% SWE-bench Verified (developer-reported), strong dense coding model). DeepSeek-R1 32B Q4_K_M (~19 GB, strong reasoning). Qwen3 32B Q5_K_M (~21 GB). Llama 3.3 70B needs 2× 24 GB GPUs at Q4_K_M. Newer: Qwen3.8 27B (18 GB download via Ollama, 256K context).
  • 32 GB VRAM (RTX 5090): Llama 3.3 70B Q4_K_M (40 GB -- needs minimal CPU offload for last layers). Qwen3 32B (19 GB, fits entirely with 13 GB spare). For agentic coding, the Kimi K2 line (MoE, 1T total / 32B active, Modified MIT) is the heavyweight pick -- Kimi K2.7 Code (June 2026) is the latest, with K2.6 the prior general release; both need quantization and heavy offload at this tier. RTX 5090 is the first single consumer GPU that fits a dense 70B with minimal offload.
  • 48+ GB VRAM (RTX 6000 Ada, A100, DGX Spark): Llama 3.3 70B Q4_K_M (40 GB, fits entirely). Llama 4 Scout (17B active / 109B total MoE, ~55 GB at Q4 -- best long-context 10M-token / multimodal pick). Llama 4 Maverick (17B active, 400B total, MoE). Llama 3.3 70B Q8_0 (70 GB -- needs 80 GB A100). NVIDIA DGX Spark (128 GB unified) fits every open-weight model up to and including 70B at Q8_0 with 58 GB to spare -- but not the newest frontier MoE models: GLM-5.3 (~750B total) needs well over 128 GB even at aggressive 2-bit quantization, well beyond the DGX Spark or a single 128 GB Mac Studio.

Interactive VRAM Calculator

Use this calculator to compute exact VRAM requirements for any combination of model, quantization, context, and batch size. Select your configuration and see which GPUs fit.

Popular Models

Base Model

6.50 GB

Context OH

1.50 GB

Batch OH

0.00 GB

System OH

1.00 GB

Total Minimum

9.00 GB

Recommended (with 25% safety margin)

11.25 GB

👉 Look for a GPU with at least 11.25 GB VRAM

Compatible GPUs

RTX 3060 (12 GB)

0.8 GB headroom

⚠️ Tight

RTX 4060 Ti (16 GB)

4.8 GB headroom

✅ Fits

RTX 4070 Ti Super (16 GB)

4.8 GB headroom

✅ Fits

RTX 4080 Super (16 GB)

4.8 GB headroom

✅ Fits

RTX 4090 (24 GB)

12.8 GB headroom

✅ Fits

RTX 5090 (32 GB)

20.8 GB headroom

✅ Fits

Mac Mini M6 (32 GB) (−8 GB macOS = 24 GB)

12.8 GB headroom

✅ Fits

Mac Mini M5 Pro (64 GB) (−8 GB macOS = 56 GB)

44.8 GB headroom

✅ Fits

Mac Studio M5 Max (128 GB) (−8 GB macOS = 120 GB)

108.8 GB headroom

✅ Fits

Mac Studio M5 Ultra (96 GB+) (−8 GB macOS = 88 GB)

76.8 GB headroom

✅ Fits

💡 Pro Tips:

  • Always use the "with safety margin" figure when buying a GPU
  • Q4 gives 90-95% quality with 25% size reduction. Q5 is better if you have room
  • Context overhead grows with conversation length. Budget 1-3 GB for typical usage
  • Batch size matters for multi-user APIs. Single-user chat can ignore batch overhead

📋 Share this configuration:

Loading...

Best Local LLMs for 16 GB VRAM

A strong local LLM for a 16 GB VRAM GPU in 2026 is Mistral Small 3.1 24B at Q4_K_M: it uses ~13 GB, runs at 55 tok/sec, and is among the strongest general models that fit with context headroom. 16 GB cards (NVIDIA RTX 5080, RTX 5070 Ti, RTX 4080 used, or an RTX 4090 laptop) top out at 14-24B models — a 70B model needs ~40 GB and does not fit.

For agentic coding, Devstral Small 24B Q4_K_M fits at ~16 GB; for reasoning, DeepSeek-R1 14B Q8_0 is the pick. The newer Mistral Small 4 (March 2026) is a single model that folds reasoning, vision, and coding together and is the natural successor as 16 GB-class default. The table below shows what fits and what does not — the "Does NOT fit" rows are the most common mistake 16 GB owners make.

Model
Quantization
VRAM Used
Speed (RTX 4080)
Best For
Fits 16 GB?
Mistral Small 3.1 24BQ4_K_M~13 GB55 tok/secGeneral chat✅ Yes
Devstral Small 24BQ4_K_M~16 GB45 tok/secAgentic coding✅ Tight
Qwen3 14BQ8_0~15 GB45 tok/secCoding + reasoning✅ Yes
DeepSeek-R1 14BQ8_0~15 GB40 tok/secMath + analysis✅ Yes
Llama 3.1 8BFP16~16 GB70 tok/secFastest responses✅ Tight
Llama 3.3 70BQ4_K_M~39 GB----❌ No (needs 39 GB)
Bar chart of which local LLM models fit in 16 GB of GPU VRAM and which 70B models do not
Bar chart showing which models fit in 16 GB VRAM: Mistral Small 3.1 24B Q4_K_M (13 GB ✅), Devstral Small 24B Q4_K_M (16 GB ✅), Qwen3 14B Q8_0 (15 GB ✅), Llama 3.3 70B Q4_K_M (39 GB ❌). Best choice: Mistral Small 3.1 24B for 55 tok/sec.

•ProTip: 🏆 A top pick for 16 GB: Mistral Small 3.1 24B Q4_K_M at ~13 GB, 55 tok/sec. For agentic coding, use Devstral Small 24B (Mistral AI, France) at 45 tok/sec. strong reasoning: DeepSeek-R1 14B Q8_0 at 40 tok/sec. Speeds are approximate figures that vary by setup and are not PromptQuorum measurements.

⚠️Warning: RTX 4090 laptop GPUs have 16 GB VRAM (not 24 GB). They share the same model ceiling as the RTX 4080 desktop.

•KeyPoint: When to upgrade to 24 GB (RTX 4090 desktop): only if you need 32B+ models at Q8, or want to run two models simultaneously without reloading.

Which Local LLMs Run Best on 12 GB VRAM?

On a 12 GB VRAM GPU (NVIDIA RTX 5070, RTX 4070 Ti, or RTX 3060 12 GB), you can run 7-8B models at Q8 or 14B at Q4_K_M. Note: MoE models like Llama 4 Scout do NOT fit here -- although Scout activates only 17B parameters per token, all 109B total experts must be loaded into memory, requiring ~55 GB at Q4.

Llama 3.1 8B at Q8_0 is the most reliable choice for conservative setups: 9 GB VRAM, 80 tok/sec, and full instruction-following quality. Qwen3 14B at Q4_K_M also fits at ~8.5 GB and delivers notably better reasoning than the 8B tier.

Model
Quantization
VRAM Used
Speed (RTX 4070 Ti)
Best For
Fits 12 GB?
Llama 3.1 8BQ8_0~9 GB80 tok/secBest overall, general chat + coding✅ Yes
Qwen3 14BQ4_K_M~8.5 GB65 tok/secBetter reasoning on budget✅ Yes
Llama 3.2 11B VisionQ5_K_M~8 GB65 tok/secImage + text tasks✅ Yes
Qwen3 8BQ8_0~8 GB85 tok/secBest multilingual + coding✅ Yes
Mistral Small v0.3FP16~14 GB----❌ No (needs 14 GB at FP16)
Llama 4 Scout (109B total MoE)Q4_K_M~55 GB----❌ No (all 109B experts must load)

•ProTip: 🏆 A top pick for 12 GB: Llama 3.1 8B Q8_0 at ~9 GB, 80 tok/sec. For better reasoning on the same card, use Qwen3 14B Q4_K_M at ~8.5 GB. Llama 4 Scout does not fit -- its 109B total MoE experts need ~55 GB at Q4. Speeds are approximate figures that vary by setup and are not PromptQuorum measurements.

•KeyPoint: RTX 3060 12GB is the budget entry point (~$285 used). It runs all 12 GB models but at ~60-70 tok/sec vs ~80-90 tok/sec on RTX 4070 Ti due to older memory architecture.

Which 70B Models Actually Fit in 24 GB VRAM (RTX 4090)?

The hardware requirement to run a 70B model locally at usable Q4_K_M quality is ~40 GB of VRAM — so a single 24 GB RTX 4090 is not enough. Your real options for 70B in 2026 are: 2× RTX 5090 (64 GB combined), an RTX 5090 (32 GB) with light CPU offload, a 48-80 GB server GPU (RTX 6000 Ada / A100), or an Apple M5 Max / 128 GB unified-memory system. The common misconception is that "Q4 is small" — at 70B parameters, even Q4 needs ~40 GB.

On a single 24 GB card, the better strategy is a 27-32B model, which delivers strong quality and fits comfortably with context headroom. Qwen3.6 27B at Q4_K_M is a strong dense coding model (77.2% SWE-bench Verified (developer-reported)); DeepSeek-R1 32B is a strong reasoning pick. A 24 GB GPU can only hold 70B at Q2_K, where quality drops noticeably. See how to run 70B models on 24 GB VRAM for offload and dual-GPU techniques. Newer still, Qwen3.8 27B (Ollama `qwen3.8:27b`, an 18 GB download with a 256K context window) is the 27B-class pick in our best local LLMs ranking and fits a 24 GB card.

If you want 64 GB without building a dual-card rig, single GPUs with 64 GB of VRAM exist and are worth comparing. At that capacity, a 70B model at Q4_K_M (~40 GB) or Q5 (45 GB+) fits on one card with room left for KV cache and a longer context window. Check each card's memory bandwidth, software support (CUDA, ROCm or other), power draw and warranty before buying, because speed and compatibility vary and can differ from a dual RTX 5090 setup. One listing of cards in this class is Orange Hardwares' 64 GB graphics card collection (vendor-suggested link).

Disclosure: Orange Hardwares suggested this link to us. PromptQuorum has no commercial relationship with Orange Hardwares and receives no payment, commission or other compensation for it. The link is labelled "vendor-suggested link" and carries the rel="sponsored" attribute, is not an endorsement, and we have not tested or verified the cards, prices or availability listed there.

Model
Quantization
VRAM Required
Fits 24 GB?
Speed (RTX 4090)
Notes
Qwen 3.6 27BQ4_K_M~16 GB✅ Yes55 tok/secStrong dense coding model, 77.2% SWE-bench Verified (developer-reported)
DeepSeek-R1 32BQ4_K_M~19 GB✅ Yes60 tok/secstrong reasoning, strong overall quality
Qwen3 32BQ5_K_M~21 GB✅ Yes55 tok/secHigh quality, excellent coding + instruction
Qwen3 32BQ8_0~34 GB❌ No--Requires 48 GB GPU
Llama 3.3 70BQ2_K~24 GB⚠️ Barely30 tok/secFits but Q2 quality is noticeably degraded
Llama 3.3 70BQ4_K_M~39 GB❌ No--Needs 2× RTX 4090 or A100 80 GB
Local LLM VRAM needs against the RTX 4090 24 GB limit: 27–32B models fit, 70B Q4_K_M does not
VRAM requirements vs RTX 4090 24 GB limit: Qwen 3.6 27B Q4_K_M (16 GB ✅), DeepSeek-R1 32B Q4_K_M (19 GB ✅), Qwen3 32B Q5_K_M (21 GB ✅), Llama 3.3 70B Q4_K_M (39 GB ❌ -- exceeds 24 GB by 63%). Sweet spot: 27-32B models at Q4-Q5.

•KeyPoint: 🏆 Best for RTX 4090 (24 GB): Qwen 3.6 27B Q4_K_M (~16 GB, 77.2% SWE-bench Verified (developer-reported)) as a strong dense coding model. For reasoning: DeepSeek-R1 32B Q4_K_M (~19 GB, 60 tok/sec). Better than Llama 3.3 70B Q2_K at far less VRAM. Speeds are approximate figures that vary by setup and are not PromptQuorum measurements.

⚠️Warning: If you specifically need 70B quality at Q4+, the RTX 4090 is not the right GPU. You need 2× RTX 4090 (48 GB combined via tensor parallelism) or an RTX 6000 Ada (48 GB). Running 70B at Q2_K on a single 4090 noticeably hurts output quality.

What CPU and RAM Do You Need?

With a dedicated GPU, CPU and RAM are secondary components. The GPU handles matrix math; CPU/RAM manage context preparation. For a full comparison of GPU vs CPU vs Apple Silicon inference speeds, see the GPU vs CPU vs Apple Silicon guide.

Minimum CPU: 8-core processor (Intel Core i7 14th gen, AMD Ryzen 7 7700X, or newer). Older CPUs add 20%+ latency.

RAM: 16 GB minimum (with GPU). If running without GPU, 32+ GB recommended. RAM does not directly limit model size when GPU is present.

Storage: 500 GB SSD for model files and OS. M.2 NVMe is preferred (faster model loading).

Which Models Run Well on 16 GB System RAM Without a GPU?

Without a GPU, a machine with 16 GB system RAM can run 3B-7B models at 8-20 tokens/sec using CPU inference. The bottleneck is memory bandwidth, not RAM capacity -- CPUs have far lower bandwidth than GPUs, which is why inference is 5-10× slower.

On 16 GB system RAM, the practical rule is: model file size + 4 GB OS overhead ≤ 16 GB. A 7B model at Q4_K_M (4.9 GB) fits, but leaves little headroom for long contexts. The table below shows realistic options.

For a complete speed-optimized model guide covering CPU-only, 4 GB, 6 GB, and 8 GB VRAM tiers with real benchmarks, see Fastest Local LLMs for Low-End PCs.

Model
Quantization
RAM Used
Speed (Ryzen 9 7950X)
Best For
Notes
Gemma 2 2BQ8_0~2.7 GB28 tok/secFastest, minimal RAMLeaves 13 GB free for OS
Phi-4 Mini 3.8BQ4_K_M~2.5 GB25 tok/secCoding on CPUBest quality-per-RAM ratio
Llama 3.2 3BQ8_0~3.8 GB20 tok/secGeneral chat, low RAMReliable, widely supported
Llama 3.1 8BQ4_K_M~4.9 GB12 tok/secBest CPU quality12 tok/sec is slow but usable for batch tasks
Llama 3.1 8BQ8_0~9 GB8 tok/secMax quality on CPUToo slow for interactive use on most CPUs
Local LLM CPU-only inference speeds on a Ryzen 9 7950X compared with a used RTX 3060 GPU
CPU-only inference speeds on Ryzen 9 7950X: Gemma 2 2B Q8_0 (28 tok/sec fastest), Phi-4 Mini Q4_K_M (25 tok/sec best choice), Llama 3.1 8B Q8_0 (8 tok/sec). A used RTX 3060 ($285) achieves 5-8× faster.

•ProTip: 🏆 Best for 16 GB RAM, no GPU: Phi-4 Mini 3.8B Q4_K_M (2.5 GB, 25 tok/sec). Delivers surprisingly strong coding and reasoning for its size. Speeds are approximate figures that vary by setup and are not PromptQuorum measurements.

•KeyPoint: CPU vs GPU speed reality: A used NVIDIA RTX 3060 12 GB (~$285) runs Llama 3.1 8B at 70+ tok/sec -- 5-8× faster than the Ryzen 9 7950X at CPU-only inference. If speed matters, buy a GPU before adding RAM.

⚠️Warning: Running a 7B model on 16 GB RAM with CPU-only leaves fewer than 7 GB for the OS and browser. With long conversation contexts (32k+ tokens), the model file grows beyond its base size and can cause RAM exhaustion. Keep context size under 4096 on 16 GB CPU-only machines.

How Much Storage Do You Need?

Model files are large: a 7B model at 4-bit quantization is 4-5 GB. Plan storage around the number and size of models you want to keep locally.

  • 500 GB SSD: OS + 1-2 small models (3B, 7B)
  • 1 TB SSD: OS + 3-5 models (mix of 7B and 13B)
  • 2 TB SSD: OS + 10+ models (various sizes)
  • 4 TB NVMe RAID: Production setup, fast model loading

What Hardware Build Should You Buy?

Building a local LLM machine from scratch means prioritizing GPU first, then CPU and RAM. Here are three realistic configurations. For multi-GPU builds, see the multi-GPU local LLM guide. For home automation setups, compact mini PCs are often a better fit than full desktop builds — see the best Mini PC for Home Assistant with local AI →.

Budget
GPU
CPU
RAM
Models
Cost
$1500 (entry)RTX 4070 Tii7 1370016 GB7-13BRealistic
$2500 (solid)RTX 4080i7 14700K32 GB13-30BRecommended
$6,500+ (high-end)2× RTX 4090 (used)Ryzen 9 7950X128 GBAny (70B+)Overkill for personal
Three local LLM hardware builds from an entry-level 16 GB GPU PC to a dual-GPU high-end workstation
Three build configurations: $1500 entry-level (RTX 4070 Ti, i7 13700, 16GB) for 7-13B models, $2500 solid build (RTX 4080, i7 14700K, 32GB) for 13-30B, $6,500+ high-end (2× RTX 4090, Ryzen 9, 128GB) for any model. The mid-level build is a balanced option.

What If You Can't Afford the Hardware?

If a $280–400 GPU is outside your budget, or your laptop is too old to support modern inference engines, local LLMs may not be cost-effective for you in 2026.

Calculate the real cost:

  • Local: $800–2,000 upfront hardware + electricity + maintenance over 2–3 years
  • Cloud: $5–50/month for typical developer use (Llama API or GPT-5.5 mini)

For light users (< 100,000 tokens/month), cloud APIs cost $5–10/month and require zero hardware. For heavy users (> 10M tokens/month), local breaks even in 6–12 months.

Compare full local vs cloud cost and performance trade-offs to find your break-even point. Many developers discover cloud is cheaper for their actual usage pattern.

Already shopping below the recommended VRAM tiers? See Best Local AI App for a Low-End PC for which model and app combinations actually run on 8 GB or less.

How Do You Maximize llama.cpp Speed on RTX 4070 Ti?

With correct settings, llama.cpp on an RTX 4070 Ti can reach roughly 85-95 tokens/sec on Llama 3.1 8B Q4_K_M -- often more than double the default out-of-box speed (approximate, not a PromptQuorum measurement). Typically the most impactful flag is `--n-gpu-layers 99`, which offloads all model layers to the GPU. Without it, layers fall back to CPU, creating a severe bottleneck.

These settings apply to llama.cpp directly and to Ollama (which uses llama.cpp internally). Ollama sets `--n-gpu-layers 99` automatically on NVIDIA hardware if drivers are installed correctly.

  • Q4_K_M can run roughly 15-20% faster than Q4_0 on an RTX 4070 Ti (approximate, not a PromptQuorum measurement). The K_M variant uses mixed quantization that NVIDIA tensor cores accelerate more efficiently. Always choose Q4_K_M over Q4_0 when both are available.
  • IQ4_XS is the smallest format (~8% smaller than Q4_K_M) with minimal quality loss. Useful for fitting Qwen3 14B into 12 GB VRAM when Q4_K_M is borderline.
  • Q5_K_M runs at nearly the same speed as Q4_K_M on NVIDIA GPUs (< 5% slower) while providing noticeably better output quality. Worth using when you have 20% VRAM headroom.
Flag
What It Does
Impact
Default
Notes
--n-gpu-layers 99Offloads all layers to GPU+100-150% speed0 (CPU only)Most important flag -- always set this first
--threads [cores]CPU threads for prompt processing+10-15% speedAll threads (including HT)Set to physical core count only. Hyperthreading hurts inference.
--ctx-size 2048KV cache / context window sizeSaves 0.5-8 GB VRAM40962048 = ~0.5 GB extra VRAM. 32768 = ~8 GB extra. Only increase if needed.
--n-batch 512Prompt processing batch size+5-10% throughput512Good default. Increase to 1024 for batch workloads if VRAM allows.
--flash-attnFlash Attention 2 kernel-20-30% VRAM at long ctxDisabledAvailable since llama.cpp b2900. Reduces VRAM for contexts > 8k tokens.
Local LLM speed on an RTX 4070 Ti: default llama.cpp settings against optimized GPU offload flags
Default llama.cpp config: ~40 tok/sec. Optimized (--n-gpu-layers 99 + --ctx-size 2048 + --flash-attn): ~90 tok/sec -- a 125% speed improvement on RTX 4070 Ti running Llama 3.1 8B Q4_K_M.

•ProTip: Run `ollama ps` to confirm your model is loaded on GPU. If GPU utilization shows 0% in `nvidia-smi` while generating, drivers are not correctly routing to CUDA. Reinstall NVIDIA CUDA Toolkit and restart Ollama.

•KeyPoint: RTX 4070 Ti speed reference (approximate, not a PromptQuorum measurement): Llama 3.1 8B Q4_K_M = 85-95 tok/sec. Qwen3 8B Q8_0 = 90-95 tok/sec. These assume --n-gpu-layers 99 and --ctx-size 2048.

⚠️Warning: Increasing --ctx-size beyond 8192 on a 12 GB GPU will cause model layer offloading back to CPU if the KV cache exhausts remaining VRAM. If speed drops suddenly on long conversations, reduce context size or use --flash-attn.

Can Mac Hardware Run Local LLMs?

Apple Silicon (M-series) runs local LLMs efficiently using unified memory shared between CPU and GPU. The base M5 launched October 2025; the M5 Pro and M5 Max followed in March 2026. Apple measures up to 4× faster LLM prompt processing (time-to-first-token) on M5 Pro/Max vs the M4 generation, though token-generation gains are more modest.

The M5 Max with 128 GB unified memory (up to 614 GB/s) runs 70B models at Q4_K_M comfortably — roughly 12-15 tok/sec — in a laptop or Mac Studio form factor. The M5 Pro (up to 64 GB unified, 307 GB/s) handles 32B models with generous headroom for KV cache and multitasking. Note the M5 Max still falls far short of what GLM-5.3 needs even at 2-bit -- an area where Apple's unified memory ceiling matters more than raw speed.

Apple refreshed the Mac mini and Mac Studio lineup on August 25, 2026, shipping September 22, 2026. The Mac mini now comes in M6 (from $899 with 16 GB, up to 32 GB unified memory, up to 170 GB/s) and M5 Pro (from $1,699, 64 GB max, 307 GB/s) configurations. The Mac Studio steps up to M5 Max (from $2,499, 128 GB max) and M5 Ultra (from $5,499 at 96 GB, up to 256 GB or 512 GB in the top configuration -- the 512 GB build became orderable on October 28, 2026, from $22,299). The outgoing M4 Mac mini (from $599) and M4 Pro Mac mini (up to $2,299, 64 GB max), plus the previous Mac Studio M4 Max / M3 Ultra, are now the prior generation.

The first independent M5 Ultra result we verified (BGR, September 21, 2026; 256 GB model, 4-bit Qwen 3.8 27B in LM Studio) measured about 55 tok/s against about 40 on an M3 Ultra; we found no independent inference figures for the M6, and Apple has not published third-party-verified figures of its own. For the first time, a single Mac can theoretically hold GLM-5.3 (~750B MoE, well over 128 GB at 2-bit) in unified memory: the top-configuration 512 GB Mac Studio M5 Ultra exceeds that footprint with room to spare. Until real-world benchmarks land, a 4× RTX 3090/4090 rig remains the better-documented local path for GLM-5.3-class models.

On a MacBook with 8 GB RAM, stick to 3-4B models. With unified memory shared between the OS and the model, 8 GB realistically fits Phi-4 Mini 3.8B, Llama 3.2 3B, or Gemma 3 4B at Q4_K_M via Ollama or llama.cpp (both use the Metal GPU backend automatically). macOS needs at least 8 GB of it, so even these leave almost no room; a 7B model does not fit at 8 GB and will swap under load. 16 GB (about 8 GB left after macOS) is the practical minimum for 7-8B models on a Mac.

Mac
GPU Memory
Best For
Limitation
M-series 8 GB (Air / base)8 GB unified3-4B models (Phi-4 Mini, Gemma 3 4B)7B does not fit; macOS needs 8 GB itself
M3 Pro MacBook Pro 16"18 GB unified7-8B models (fast)Can run 14B slowly
Mac mini M6 (from $899)16–32 GB unified, up to 170 GB/s7-8B models (16 GB), 14B (32 GB)No 32B+ headroom
M5 Pro (MacBook Pro / Mac mini, from $1,699)64 GB unified, 307 GB/s32B models comfortablyLlama 4 Scout runs well
M5 Max (MacBook Pro / Studio, from $2,499)128 GB unified, up to 614 GB/s70B models at Q4_K_M~12-15 tok/sec on 70B
Mac Studio M5 Ultra (from $5,499)96-512 GB unified70B+ and large MoE capacity512 GB from $22,299 (orderable since Oct 28)
Local LLM hardware on Mac: unified memory tiers from the 8 GB M-series to the Mac Studio M5 Ultra
Mac hardware comparison: 8 GB M-series (3-4B models), M3 Pro 16" (18GB, 7-8B), Mac mini M6 (16–32GB, 7-8B at 16GB / 14B at 32GB, $899), M5 Pro (64GB, 32B, $1,699), M5 Max (128GB, 70B at Q4_K_M ~12-15 tok/sec, from $2,499), Mac Studio M5 Ultra (up to 512GB, 70B+/GLM-5.3 capacity, $5,499). 16 GB unified (about 8 GB left after macOS) is the practical minimum for 7B models on a Mac.

When Should You Use Server vs Consumer Hardware?

For production deployment (24/7 operation, multiple users), server-grade hardware is recommended over consumer GPUs. Consumer hardware is optimized for gaming, not sustained inference.

  • Consumer (RTX 5090): ~$2,000 MSRP (~$4,300–5,000+ street), 32 GB VRAM, designed for single-user workloads; sustained 24/7 load may need extra cooling (check the manufacturer's specification).
  • Server (RTX 6000 Ada): ~$7,000, 48 GB VRAM, designed for 24/7 use, better cooling, error correction.
  • Recommendation: Start with an RTX 5090. If running 70B models 24/7 for multiple users, upgrade to dual A100 or RTX 6000 Ada.
Local LLM hardware compared: consumer RTX 5090 against server-grade RTX 6000 Ada for single-user or 24/7 use
Consumer vs server hardware: RTX 5090 (~$4,300–5,000+ street, 32GB, single-user, part-time) vs RTX 6000 Ada ($7,000+, 48GB, multi-user, 24/7 duty). Start with consumer hardware; upgrade to server-grade only if running production services.

NVIDIA DGX Spark: 128 GB Desktop AI Computer

The NVIDIA DGX Spark ($6,950 since October 2, 2026, up from its $3,999 launch price) is a compact 128 GB desktop AI computer that can hold Llama 3.3 70B at Q8_0 entirely in unified memory. Apple Mac Studio / MacBook Pro with 128 GB and AMD Strix Halo 128 GB systems can do the same, so it is not unique — but it ships with NVIDIA's CUDA software stack.

Built on the GB10 Grace Blackwell Superchip, the DGX Spark launched in October 2025 with 128 GB LPDDR5x unified memory. Note: its real memory bandwidth is ~273 GB/s, so dense 70B token generation is slow — an independent review by LMSYS reported roughly 3 tok/sec decode on Llama 3.1 70B (FP8, SGLang). The headline FP4 compute figure does not translate into fast single-stream decoding. The DGX Spark is best suited to large mixture-of-experts models (Llama 4 Scout/Maverick, Kimi K2) where only a fraction of parameters activate per token -- but its 128 GB ceiling is not unlimited: GLM-5.3 (~750B total) needs well over 128 GB even at 2-bit quantization and does not fit a single DGX Spark.

Spec
Value
Unified memory128 GB LPDDR5x
Llama 3.3 70B at Q4_K_M✅ fits (40 GB)
Llama 3.3 70B at Q8_0✅ fits (70 GB)
GLM-5.3 at 2-bit (well over 128 GB)❌ does not fit
Inference speed (70B)~3 tok/s
Price$6,950
OSDGX OS (Ubuntu), Ollama pre-installed
Memory bandwidth~273 GB/s (real)
vs RTX 50904× more memory, but far lower bandwidth

•KeyPoint: A discrete GPU (RTX 5090, or dual 5090) generates tokens much faster than the DGX Spark on dense models because of far higher memory bandwidth. Choose the DGX Spark for capacity — holding very large MoE models in one box — not for single-stream 70B speed.

What Are the Most Common Hardware Mistakes?

  • Buying CPU-only when GPU is available. A $600 RTX 4070 Ti will outperform a $2000 CPU. GPU dominates LLM speed.
  • Not accounting for VRAM overhead. Model file size + system overhead + context = total VRAM used. Always buy 25% more than the model size.
  • Assuming all 70B models fit in 40GB VRAM. They do, barely, in Q4 (4-bit) quantization only. Q5 requires 45+ GB.
  • Ignoring power supply and cooling. An RTX 4090 is rated at around 450W (an RTX 5090 around 575W). Check the PSU rating the card maker recommends, leave headroom and ensure good case airflow.
  • Thinking an old GPU will work. RTX 2080 is roughly 10× slower than RTX 4070 Ti (approximate, not a PromptQuorum measurement). Modern GPU architecture significantly outperforms prior generations.
  • Not accounting for KV cache VRAM on top of model weights: A 7B model at Q4_K_M is 4.7 GB of weights -- but with a 32K context window, the KV cache adds ~4 GB more, totalling ~8.7 GB. On an 8 GB card this causes OOM errors. Always add 25-100% to model size depending on context length.
  • Treating hardware cost as the only cost: If you cannot afford 16+ GB RAM or a dedicated GPU, cloud APIs cost less for low-volume use ($0.01–0.05 per 1K tokens). See Local LLM vs Cloud: Cost Analysis for the full trade-off.

What Regional Compliance Rules Apply to Local LLM Hardware?

EU (GDPR + EU AI Act): Running LLMs locally keeps all inference data within your infrastructure, which can remove the cross-border transfer question under GDPR Article 44 for the inference step itself. The EU AI Act's obligations for stand-alone high-risk AI systems (Annex III) were originally set to apply from August 2, 2026, but a proposed "Digital Omnibus on AI" would move that date to December 2, 2027 (with high-risk AI embedded in regulated products moved to August 2, 2028); the legislative status was still being reported as unsettled at the last refresh, so check the current text on EUR-Lex before relying on either date. The AI Act's Article 50 transparency duties still apply on the original schedule. Running inference locally keeps data inside your own infrastructure, which helps with data-residency obligations; whether a deployment is compliant still depends on the overall setup.

Japan (APPI): Japan's 2022 APPI amendment tightened breach-notification and cross-border-transfer rules but does not impose an AI-specific data-minimization requirement (it relies on general purpose-limitation duties). More relevant to AI are Japan's 2025 APPI reform package and its first AI law — the AI Promotion Act (in force since June 2025), an innovation-first framework with no penalties. On-premises LLM hardware keeps personal data inside your infrastructure for document processing and customer-support automation.

China: China's Cyberspace Administration of China (CAC) Interim Measures for Generative AI Services (effective August 2023) require providers with public-opinion influence to complete a CAC security assessment and algorithm filing. Since September 1, 2025, China also mandates labeling of AI-generated content under the CAC labeling Measures and national standard GB 45438-2025. Running local hardware with open-weight models can reduce reliance on API providers for internal enterprise use; whether a deployment meets the Chinese requirements still depends on the overall setup. This section is a general overview, not legal advice; have your specific deployment reviewed by qualified counsel.

Common Questions About Local LLM Hardware

Can I run a 70B model on a laptop?

Only with heavy quantization (Q2, 2-bit) and CPU fallback. Impractical. Laptops are suited for 7B models — see how to run a local LLM on a laptop for RAM tiers and thermals. For 70B, use a desktop with RTX 4090+.

Is RTX 4090 overkill for personal use?

Not if you run 70B models or multiple models simultaneously. For just 7B chat, RTX 4070 Ti suffices. RTX 4090 is a capable option if you want flexibility; it is discontinued, so check stock and used prices.

Should I buy RTX 5090 or wait for RTX 6090?

RTX 5090 is available (early 2026). RTX 6000 Ada server GPUs are also solid. Unless you have unlimited budget, an RTX 5090 or a used RTX 4090 are strong choices.

How does quantization affect quality?

As a rough community rule of thumb (no PromptQuorum measurement; it varies by model and task): Q8 loses little quality versus FP16, and Q4_K_M typically keeps most of it. For many everyday tasks the difference between Q4 and FP16 is small. Per-model figures are in the perplexity tables of the llama.cpp repository.

Can I upgrade GPU later?

Yes. Start with RTX 4070 Ti now, upgrade to RTX 5090 in 2 years if needed. GPU is the most replaceable component.

How much RAM do I need to run a 7B model locally?

8 GB RAM is the absolute minimum for a 7B model. 16 GB is recommended for comfortable use alongside browser and OS. 32 GB gives headroom for larger context windows and multitasking.

Can I run local LLMs on Apple Silicon (M1/M2/M3/M4/M5)?

Yes. Apple Silicon uses unified memory shared between CPU and GPU. M5 Pro (64 GB, 307 GB/s) runs 32B models well. M5 Max (128 GB, up to 614 GB/s) runs 70B at Q4_K_M at roughly 12-15 tok/sec. On an 8 GB Mac, stick to 3-4B models.

What are the best llama.cpp models for a MacBook with M3 and 8 GB RAM?

On a MacBook M3 with 8 GB RAM, run 3-4B models at Q4_K_M: Phi-4 Mini 3.8B, Llama 3.2 3B, or Gemma 3 4B. Use Ollama or llama.cpp — both use the Metal GPU backend automatically. A 7B model is borderline and will swap under load; keep context under 4096 tokens. For 7-8B use on a Mac, 16 GB unified memory (about 8 GB left after macOS) is the practical minimum.

What CPU is best for local LLMs without a GPU?

High-core-count CPUs with large L3 cache: AMD Ryzen 9 7950X or Intel Core i9-14900K. Expect 5-15 tokens/sec for 7B models. CPU inference is 5-10× slower than GPU.

Does storage speed affect local LLM performance?

Yes, at model load time. NVMe SSD (3-7 GB/s) loads a 7B model in 2-5 seconds vs. 20-60 seconds on HDD. Inference speed after loading is unaffected by storage.

Can I use multiple GPUs to run larger models?

Yes, via tensor parallelism. Two RTX 5090s (32 GB each) provide 64 GB VRAM, enough for a 70B model at Q4_K_M. Ollama and llama.cpp support multi-GPU via --n-gpu-layers split across cards. If you prefer a single card, GPUs with 64 GB of VRAM can also hold a 70B model at Q4_K_M or Q5 with room for KV cache; compare memory bandwidth and software support first. The 70B section of this guide links to a vendor-suggested listing, with our disclosure.

What are the best local LLMs for 16 GB VRAM in 2026?

Mistral Small 3.1 24B Q4_K_M (13 GB, 55 tok/sec) is a top pick for RTX 5080 / RTX 5070 Ti / RTX 4090 laptop. For agentic coding: Devstral Small 24B Q4_K_M (16 GB, 45 tok/sec). For reasoning: DeepSeek-R1 14B (15 GB, 40 tok/sec). The newer Mistral Small 4 (March 2026) is the one-model successor. Llama 3.3 70B does not fit -- it requires ~40 GB at Q4_K_M. Speeds are approximate figures that vary by setup and are not PromptQuorum measurements.

Can a single RTX 4090 run a 70B model at good quality?

No -- not at Q4_K_M quality. Llama 3.3 70B at Q4_K_M requires ~39 GB VRAM. The RTX 4090 has 24 GB. You can run it at Q2_K (~24 GB) but quality drops noticeably. Better options: Qwen 3.6 27B Q4_K_M (~16 GB, 77.2% SWE-bench Verified (developer-reported), strong dense coding) or DeepSeek-R1 32B Q4_K_M (~19 GB, strong reasoning).

What is the best local LLM for 16 GB system RAM without a GPU?

Phi-4 Mini 3.8B Q4_K_M (2.5 GB RAM, ~25 tok/sec on Ryzen 9 7950X) is a good option for CPU-only inference on 16 GB system RAM. Gemma 2 2B Q8 is the fastest at ~28 tok/sec. Llama 3.1 8B Q4_K_M (4.9 GB) also fits but runs at ~12 tok/sec -- slow for interactive use. Speeds are approximate figures that vary by setup and are not PromptQuorum measurements.

What is the memory rule of thumb for a 7B model at Q4 quantization?

A 7B model at Q4_K_M quantization needs about 4-5 GB of VRAM (or system RAM for CPU-only inference) -- roughly 0.6 GB per billion parameters at 4-bit precision. This scales linearly: a 14B model needs ~9 GB, a 32B model needs ~19 GB, and a 70B model needs ~40 GB, all at Q4_K_M.

What hardware do I need to run GLM-5.3 locally?

GLM-5.3 (Z.ai; reported figures, not verified by PromptQuorum) is a ~750B-parameter MoE model. Even the most aggressive 2-bit dynamic GGUF needs well over 128 GB of combined VRAM/RAM -- too large for a single RTX 5090 (32 GB), a 128 GB DGX Spark, or a 128 GB Mac Studio. Realistic local paths are a 4× RTX 3090/4090 rig with 192 GB+ system RAM, or a 512 GB Mac Studio (M5 Ultra), both relying on CPU/GPU hybrid offload at slow speeds (not verified by PromptQuorum). For most users, GLM-5.3 is effectively cloud-only.

Does more RAM help local LLMs beyond VRAM?

System RAM supports OS and multitasking but does not increase model capacity beyond GPU VRAM (for GPU-accelerated inference). With a GPU, 16 GB system RAM is sufficient. Without a GPU, 32+ GB RAM helps, but inference speed will be 5-10× slower than GPU-based.

Can you run a 30B parameter model on an RTX 5080 vs Mac Mini M4 Pro?

RTX 5080 (16 GB VRAM): 30B needs about 19 GB at Q4_K_M, so it does not fully fit and needs partial CPU offload (much slower) or a lower quantization; a 24B model (~14 GB at Q4_K_M) fits at roughly 80-120 tokens/sec. Mac Mini M4 Pro (48 GB unified): 30B runs at Q4_K_M (~19 GB) within the ~40 GB left after macOS, at 20-30 tokens/sec; Q8 (28 GB) also fits but leaves little room for context. RTX 5080 is 4-6× faster but less portable; Mac is energy-efficient but slower. Speeds are approximate figures that vary by setup and are not PromptQuorum measurements.

What are the hardware requirements for running a local coding LLM in 2026?

For good coding performance: RTX 4080+ (16 GB VRAM) with DeepSeek-Coder 33B Q4 or Mistral Small 3.1 24B Q4 for code generation. Minimum: RTX 4070 Ti (12 GB) with Mistral Small 3.1 24B Q4. CPU: 8+ cores. RAM: 16 GB system RAM. 500 GB SSD.

Is an RTX 3060 12GB still worth it for local LLMs in 2026?

RTX 3060 (12 GB) is dated (2021 architecture). It handles 7B-13B models at Q4 but produces 40-60 tokens/sec. A used one runs ~$285; a new RTX 5070 (~$880) or RTX 5060 Ti 16 GB (~$800) runs 2-3× faster. The RTX 3060 is only worth keeping as a secondary GPU for multi-card setups. Speeds are approximate figures that vary by setup and are not PromptQuorum measurements.

How much VRAM do you need for 7B, 14B, and 32B models?

7-8B models: about 4.7-5 GB at Q4_K_M, 5.7 GB at Q5_K_M, 16 GB at FP16. 14B models: about 9 GB at Q4_K_M and about 28 GB at FP16. 30-32B models: about 19 GB at Q4_K_M, 22 GB at Q5_K_M, 64 GB at FP16. Add 25-100% on top of the weights for the KV cache, depending on context length. Q4_K_M is the recommended quantization level for most hardware.

What is the best GPU configuration for enterprise LLM deployment in 2026?

For enterprise: 2× RTX 5090 (64 GB total VRAM) for redundancy and load distribution, or A100 (80 GB) for multi-tenant systems. RTX 5090 has a $1,999 MSRP (street ~$4,300–5,000+); A100 is $10,000+. Docker-based orchestration (vLLM, Ollama Serve) enables multi-model serving and concurrent user handling.

Does an RTX 4070 laptop support LLM quantization?

Yes. RTX 4070 laptops (8 GB VRAM) support Q4 and Q5 quantization for 7-13B models, typically at roughly 50-70 tokens/sec (approximate, not a PromptQuorum measurement). Higher-end laptops with the RTX 4090 mobile GPU (16 GB) handle up to 24B models. Quantization is essential for laptop inference—without it, only 3-7B models fit in 8 GB VRAM.

What specs should an AI PC have to run local LLMs fully offline?

Start from VRAM, then system RAM, then CPU cores — in that order, because that is the order in which they stop you. A 16GB-VRAM card covers most models people actually want; 24GB is where 32B-class models stop needing awkward compromises. System RAM matters second, and 32GB is a sensible floor for an AI PC that also does other work. CPU cores matter least for inference, though they decide how fast a model loads. Running fully offline is a software question rather than a hardware one: once the weights are on disk, Ollama and LM Studio can generally both run with networking off (model downloads and update checks still need a connection, and settings differ by version), so this build should not need an internet connection to answer a prompt. The tables above give exact system requirements per model size.

Which local LLM runs on an RTX 4060 (8 GB) with 16 GB of system RAM?

An 8 GB card such as the RTX 4060 runs 7-8B models and smaller: Llama 3.1 8B at Q4_K_M (4.7 GB, about 70 tok/sec) and Qwen3 8B (about 5 GB) fit with headroom, while Phi-4 Mini 3.8B (2.3 GB) and Gemma 3 4B (about 3 GB) are lighter and faster. Avoid 13B and larger models. Keep the context short: a 32K context adds about 4 GB of KV cache to Llama 3.1 8B and can cause out-of-memory errors on 8 GB. With 16 GB of system RAM the GPU does the work; the RAM only has to cover the OS and any CPU fallback. Speeds are approximate figures that vary by setup and are not PromptQuorum measurements.

Sources

Know your hardware needs? Find the best budget GPU for local LLMs.

Best Budget GPUs for Local LLMs →

A Note on Third-Party Facts

This article references third-party AI models, benchmarks, prices, and licenses. The AI landscape changes rapidly. Benchmark scores, license terms, model names, and API prices can shift between the time of writing and the time you read this. Before making deployment or compliance decisions based on this article, verify current figures on each provider’s official source: Hugging Face model cards for licenses and benchmarks, provider websites for API pricing, and EUR-Lex for current GDPR and EU AI Act text.

Run PromptQuorum with a local LLM, your own API keys, or both — you pick the backend.

Download the PromptQuorum Beta →

← Back to Local LLMs

MCP