Key Takeaways
- 7B models: 8GB minimum (Q4), 10GB comfortable (Q5), 14GB for Q8 full precision.
- 13B models: 12GB minimum (Q4), 14-16GB comfortable (Q5), 16GB for Q8.
- 70B models: 35-40GB minimum (Q4), 44GB+ for Q5/Q8 or multi-user setup.
- Quantization (Q4, Q5, Q8) reduces VRAM by 75-87% vs. full precision (FP32).
- Always over-allocate by 1-2GB for overhead (KV cache, optimizer state, system OS).
- Batch size ≠ VRAM per inference. Single inference uses same VRAM regardless of batch (batch processes sequentially).
- More VRAM doesn't speed up single-prompt inference. It only helps with multi-user/multi-request setups.
VRAM Rule of Thumb — Quick Reference
Don't have time for the formula? Use these simple rules. The numbers above (Quick Answer) are the bare compute minimum for the weights alone. The tiers below are the *comfortable GPU to actually buy* — they add headroom for KV cache and system overhead and round up to a real consumer VRAM size (6/8/12/16/24/32/48 GB).
Once you know your VRAM budget, see which GPUs fit each tier →
📍 In One Sentence
For 4-bit quantization, budget 0.6 GB VRAM per billion parameters — a 7B model fits in 5 GB, a 70B model needs 42 GB.
💬 In Plain Terms
Divide your model size in billions by roughly 1.7 to get the minimum VRAM in GB at Q4 (4-bit). A 7B model needs about 4–5 GB, a 13B about 8–9 GB, and a 70B about 40–48 GB. For CPU-only setups without a GPU, double those numbers since system RAM is slower and you need headroom.
- 3B models (Phi, StableLM): 4 GB VRAM minimum
- 7B models (Llama, Mistral, Qwen): 8 GB VRAM (Q4), 10 GB (Q5)
- 13B models (Llama 3.3, Mistral): 12 GB VRAM minimum (Q4)
- 22B models (Qwen3, Gemma): 16 GB VRAM (Q4)
- 70B models (Llama 3.3, Qwen 3.6): 35–40 GB VRAM (Q4), 44 GB+ (Q5)
- MoE models: VRAM scales with weights you must hold in memory. Example: Qwen 3.6 35B-A3B (3B active) fits a tiny ~2 GB footprint, while Llama 4 Scout (17B active / 109B total) still needs ~55 GB at Q4 because all experts stay resident
# Quick VRAM formula (memorize this)
VRAM (GB) = Model Size (B) x Bytes per Parameter
# Bytes per parameter by quantization:
# FP32 = 4 Q8 = 1 Q5 = 0.625 Q4 = 0.5
# Examples at Q4 (0.5 bytes/param):
7B x 0.5 = 3.5 GB raw weights -> ~8 GB comfortable GPU
70B x 0.5 = 35 GB raw weights -> ~40 GB comfortable GPUWhat Is the VRAM Formula for LLMs?
VRAM (GB) = Model Size in Billions × Bytes per Parameter
- Model size: Number of parameters (7B, 13B, 70B, etc.)
- Bytes per parameter by precision: FP32 = 4 bytes, Q8 = 1 byte, Q5 ≈ 0.625 bytes, Q4 = 0.5 bytes
Example: Llama 3 70B, FP32, no quantization:
70 billion × 4 bytes = 280 GB. Impractical on any consumer GPU.
Llama 3 70B, Q4 (4-bit) quantization:
70 billion × 0.5 bytes = 35 GB of raw weights. Add ~10-15% for KV cache and runtime overhead → ~40 GB allocated in practice, which is why 2× 24 GB GPUs (or a single 48 GB card) is the realistic buy target.
MoE Models (Sparse): Active parameters drive compute, but all experts must stay loaded in VRAM. Example: Llama 4 Scout has 109B total parameters with 17B active per token. At Q4 it still needs ~55 GB of VRAM to hold every expert — it only squeezes into a 24 GB GPU at an aggressive 1.78-bit quant (~20 tok/s). Compute is cheap; memory is the constraint.
How Much VRAM Does Each Model Size Need?
| Model Size | FP32 (No Quantization) | Q8 (8-bit) | Q5 (5-bit) | Q4 (4-bit) | Recommended GPU |
|---|---|---|---|---|---|
| 3B (Phi, StableLM) | 12 GB | 3 GB | ~2 GB | 1.5 GB | RTX 2060 6 GB or RTX 5070 12 GB |
| 7B (Llama 3.3, Mistral) | 28 GB | 7 GB | ~4.4 GB | 3.5 GB | RTX 4060 Ti 8 GB or RTX 5070 12 GB |
| 13B (Llama 3.3, Mistral) | 52 GB | 13 GB | ~8 GB | 6.5 GB | RTX 3060 12 GB or RTX 4070 12 GB |
| 22B (Qwen, Gemma) | 88 GB | 22 GB | ~14 GB | 11 GB | RTX 4060 Ti 16 GB or RTX 4080 16 GB |
| 70B (Llama 3, Qwen) | 280 GB | 70 GB | ~44 GB | 35 GB | 2× RTX 4090 (48 GB total) or RTX 6000 Ada 48 GB |
| Qwen 3.6 35B-A3B (3B active, MoE)* | 12 GB | 3 GB | 2 GB | 2 GB | RTX 2060 6 GB or RTX 5070 12 GB |
| DeepSeek V4-Flash (13B active / 284B total, MoE)* | 52 GB | 13 GB | 8 GB | 7 GB | RTX 3060 12 GB or RTX 5070 12 GB |
| Llama 4 Scout (17B active / 109B total, MoE)† | 436 GB | 109 GB | 68 GB | 55 GB | 2× RTX 4090 (48 GB) — fits 24 GB only at 1.78-bit (~20 tok/s) |
| gpt-oss:20b (3.6B active / 21B total, MoE)* | 84 GB | 21 GB | 13 GB | 12 GB | RTX 5070 12 GB or any 16 GB GPU |
| Kimi K2.6 (32B active / 1T total, MoE)* | 128 GB | 32 GB | 20 GB | 16 GB | 2× RTX 4090 or RTX 5090 32 GB (Q4 only) |
* MoE models: VRAM is calculated from active parameters only, not total model size. † Llama 4 Scout keeps all 109B parameters resident, so it needs ~55 GB at Q4 despite only 17B active per token.
Best Local LLM by VRAM Tier
Most people already know their GPU's VRAM and want the reverse lookup: "I have 12 GB — what's the best model?" Use this table instead of the formula.
📍 In One Sentence
Match your GPU's VRAM to the largest model tier it fits at Q4, then check the by-model-size table above for the exact model.
💬 In Plain Terms
If you already know how much VRAM your GPU has, skip the model-size math — find your card's VRAM in the table below and use the model tier next to it.
| VRAM Tier | Best-Fit Model (Q4) | Example GPUs | What Fits |
|---|---|---|---|
| 4 GB | Llama 3.2 3B or Phi-3.5-mini (~1.5-2 GB) | RTX 3050 4 GB, GTX 1650 | 3B-class only — 7B models will OOM |
| 6 GB | Qwen3 4B, or Llama 3.1 8B at aggressive Q4 (~3-4 GB) | RTX 2060 6 GB, RTX 3050 Ti | 4B comfortably; 8B is tight |
| 8 GB | Llama 3.3 8B or Qwen3 8B (~3.5-5 GB) with real headroom | RTX 4060 Ti 8 GB, RTX 3070 | The most common sweet spot for 7-8B models |
| 12 GB | Qwen3 14B (~6.5 GB), or 7-8B at Q5/Q8 for higher quality | RTX 3060 12 GB, RTX 4070 | 14B comfortably, room for larger context |
| 16 GB | Qwen3 14B at Q5/Q8, or 22B-class (Gemma, Qwen) at Q4 (~11 GB) | RTX 4060 Ti 16 GB, RTX 4080 | 22B is the practical ceiling at this tier |
| 24 GB | 22-32B comfortably at full Q8; 70B only at sub-4-bit with quality loss | RTX 4090, RTX 3090 | 70B genuinely needs ~35 GB — this tier is not quite enough |
| 32 GB | 22-32B at full precision, or as one card in a dual-GPU 70B setup | RTX 5090 | Still short of a full 70B Q4 alone (~35 GB needed) |
| 48 GB | 70B models comfortably at Q4 (~35 GB), tightly at Q5 (~44 GB) | 2× RTX 4090, RTX 6000 Ada | First tier that fits 70B without quality-losing quantization tricks |
These are Q4 buy-target figures with realistic overhead included — see the by-model-size table above for bare compute-minimum numbers per quantization level.
MoE Models Need Far Less VRAM Than Their Size Suggests
Mixture-of-Experts (MoE) models split their parameters across many "expert" sub-networks and activate only a fraction for each token. Active parameters cut compute and speed up inference, but for most MoE models every expert must still be loaded into VRAM — so memory usage tracks total parameters, not active ones.
Dense model rule: VRAM = total_params × bytes_per_param
MoE model rule (compute): active_params drive tokens/sec — but VRAM still scales with total resident weights.
Example: Llama 4 Scout has 109B total parameters with only 17B active per token. It runs fast for its size, but at Q4 it still needs ~55 GB of VRAM to hold all experts — out of reach for a single 24 GB GPU except at an aggressive 1.78-bit quant (~20 tok/s on an RTX 4090).
Some runtimes can stream or offload inactive experts to system RAM, trading speed for a smaller VRAM footprint. The headline takeaway: do not assume an MoE model fits in active-parameter-sized VRAM — check the actual on-disk size at your quant level.
How Does Quantization Reduce VRAM Requirements?
Quantization reduces the number of bits needed to represent each model parameter.
- FP32 (32-bit float): Full precision. 1 parameter = 4 bytes. No loss. Slowest.
- Q8 (8-bit): 1 parameter = 1 byte. ~6% accuracy loss. 75% VRAM savings.
- Q5 (5-bit): 1 parameter = 0.625 bytes. ~2% accuracy loss. 84% VRAM savings.
- Q4 (4-bit): 1 parameter = 0.5 bytes. ~1% accuracy loss. 87.5% VRAM savings.
For most users, Q4 is the sweet spot: imperceptible accuracy loss, 87% smaller VRAM footprint.
As of April 2026, Q4 is standard. Q5 and Q8 are available if you have extra VRAM and want marginal quality gains.
VRAM determines model size, but prompt design determines output quality. Techniques like chain-of-thought and few-shot prompting can close the quality gap between smaller and larger models. Explore the full prompt engineering toolkit to get more from the models your hardware supports. If you have 12–16 GB VRAM and want a concrete coding workload to put that toolkit against, Replace GitHub Copilot With a Local LLM maps the Continue.dev + Ollama + Qwen3-Coder stack onto exactly those VRAM tiers.
What About Batch Size and Multi-User Inference?
Batch size affects throughput (tokens per second), not single-inference latency.
A single user prompting "What is 2+2?" uses the same VRAM whether batch size is 1 or 32.
Batch size = 32 means processing 32 prompts in parallel. This uses ~32× more VRAM, but generates 32 responses faster.
For single-user (typical local LLM usage): Batch size = 1. VRAM is model size + 1-2GB overhead.
For multi-user server: Allocate batch size × model VRAM. A 70B model at batch=4 needs ~160GB (40GB × 4).
Do You Need More VRAM Than the Model Size?
Yes. Beyond the model weights, add:
- KV cache (key-value cache for context): ~5-10% extra VRAM.
- Optimizer state (if fine-tuning): 2-4× model size (only relevant for training, not inference).
- System overhead (OS, drivers, Ollama/LM Studio runtime): ~1-2GB.
Rule: A 70B model Q4 (35GB raw weights) + KV cache (~3GB) + system (~2GB) = ~40GB allocated in practice.
Always buy GPUs with at least 1-2GB headroom above theoretical minimums.
LM Studio Hardware Requirements
LM Studio runs GGUF models on llama.cpp under the hood, so its VRAM math is identical to the tables above — it adds only its own ~1-2GB runtime/UI overhead on top.
- Minimum: 8GB system RAM for small (3B) models running CPU-only. A dedicated GPU is optional but 10-20× faster.
- Recommended: 8GB VRAM for comfortable 7-8B inference; 12GB+ for 13-14B models.
- GPU support: NVIDIA (CUDA), AMD (ROCm on Linux, Vulkan on Windows), Apple Silicon (Metal, unified memory counts as VRAM).
- LM Studio's model browser shows a compatibility indicator (green/yellow/red) per model based on your detected VRAM — it flags a model before download if it won't fit.
Use the VRAM tier table above to pick a model size, then match it against LM Studio's indicator for your specific GPU.
Ollama Hardware Requirements
Ollama also runs on llama.cpp, so its VRAM requirements match the tables above one-to-one — there's no separate "Ollama tax" on top of the model itself.
- Minimum: 8GB system RAM for CPU-only inference of 3B-class models.
- GPU support: NVIDIA (CUDA 11+), AMD (ROCm — officially supported card list is narrower than LM Studio's), Apple Silicon (Metal via unified memory).
- `ollama ps` shows live VRAM usage per loaded model. `ollama run <model> --verbose` reports load time and token throughput.
- If a model doesn't fully fit, Ollama automatically splits layers between GPU and CPU RAM instead of failing outright — expect a 30-50% speed penalty on the offloaded portion (see overhead above).
Same rule as LM Studio: use your GPU's VRAM tier to pick the model size, not the other way around.
Common VRAM Misconceptions
- More VRAM = faster inference. False. VRAM size doesn't affect speed. Memory bandwidth (GB/sec) does, and that's fixed per GPU.
- Batch size = sequential token limit. False. Batch size = parallel requests. Single inference uses batch=1 regardless of VRAM size.
- You need 24GB for any 70B model. False. A genuine 70B fit needs ~35-40GB at Q4, and 70GB+ at Q8. 24GB only works with aggressive sub-4-bit quantization and a real quality hit.
VRAM Calculator
Select your model size and quantization to estimate VRAM requirements.
Popular Models
Base Model
6.50 GB
Context OH
1.50 GB
Batch OH
0.00 GB
System OH
1.00 GB
Total Minimum
9.00 GB
Recommended (with 25% safety margin)
11.25 GB
👉 Look for a GPU with at least 11.25 GB VRAM
Compatible GPUs
RTX 3060 (12 GB)
0.8 GB headroom
RTX 4070 (12 GB)
0.8 GB headroom
RTX 4070 Ti (12 GB)
0.8 GB headroom
RTX 4080 (16 GB)
4.8 GB headroom
RTX 4090 (24 GB)
12.8 GB headroom
Mac mini M5 (16 GB) (16 GB)
4.8 GB headroom
Mac mini M4 (16 GB) (16 GB)
4.8 GB headroom
MacBook Pro (24 GB) (24 GB)
12.8 GB headroom
M3 Max (36 GB) (36 GB)
24.8 GB headroom
💡 Pro Tips:
- Always use the "with safety margin" figure when buying a GPU
- Q4 gives 90-95% quality with 25% size reduction. Q5 is better if you have room
- Context overhead grows with conversation length. Budget 1-3 GB for typical usage
- Batch size matters for multi-user APIs. Single-user chat can ignore batch overhead
📋 Share this configuration:
Frequently Asked Questions
Can I run Mistral Small on a 6GB GPU?
Barely, at Q4 with tight overhead. Practically, no. Buy at least 8GB. You'll hit OOM errors with 6GB.
How much VRAM do I need for fine-tuning a 7B model?
For LoRA: 12-16GB. Full fine-tuning: 28GB+. Fine-tuning requires optimizer state (2-4× model VRAM), not just inference.
Is 12GB enough for Llama 3 13B?
Yes, comfortably. A 13B model needs about 6.5GB of raw Q4 weights, so 12GB leaves plenty of headroom for KV cache and context. At Q5 or Q8 it gets tighter — 16GB is safer there.
Do I need 40GB for a 70B model?
At Q4, yes — realistically around 35-40GB once KV cache and system overhead are included. At Q5+, you need 44GB or more. 24GB GPUs are not enough for a genuine 70B fit.
Does increasing batch size reduce VRAM for single inference?
No. Single inference always uses batch=1 VRAM. Batch size only helps throughput (multi-user scenarios).
What's the best quantization for accuracy?
Q8 is nearly imperceptible loss. Q5 is ~2% loss. Q4 is ~1% loss. For most, Q4 is the sweet spot.
Can I offload some VRAM to CPU RAM?
Yes, via layer-splitting (NVLink). Llama.cpp and Ollama support this. Performance drops 30-50% but it works. Under 8 GB VRAM? See **which models run fastest on your exact hardware tier** — CPU-only, 4 GB, 6 GB, and 8 GB VRAM benchmarks with real tok/sec numbers.
What are LM Studio's system requirements for local LLMs?
LM Studio runs on llama.cpp, so its VRAM needs match the model tables on this page exactly, plus ~1-2GB of its own runtime overhead. 8GB VRAM is comfortable for 7-8B models; 12GB+ for 13-14B.
What are Ollama's hardware requirements?
Ollama also runs on llama.cpp, so VRAM needs are identical to this page's tables. Minimum 8GB system RAM for CPU-only 3B models; a GPU with 8GB+ VRAM makes 7-8B models 10-20× faster.
Sources
- NVIDIA CUDA memory architecture and shared memory model documentation
- Ollama and LM Studio official documentation: model VRAM requirements and quantization specs
- llama.cpp project GitHub: quantization levels (Q4, Q5, Q8) and memory calculations