Q4_K_M vs Q8_0: Which Should You Pick?
By Hans Kuepper · Founder of PromptQuorum · Discovery engine for open-weight & open-source AI
Q4_K_M and Q8_0 are two llama.cpp quantization formats: on Llama 3 8B, Q8_0 is practically identical to f16 by perplexity (+0.02%), while Q4_K_M adds +2.8% but needs a 4.58 GiB file instead of 7.96 GiB (Llama 3.1 8B: 7.95 GiB). The choice mostly depends on your VRAM.
Key Takeaways
- ✓8 GB VRAM or less: use Q4_K_M — its 8B file is 4.58 GiB, while Q8_0 (7.95 GiB) no longer fits once a KV cache is added
- ✓12+ GB VRAM: Q8_0 is worth it for near-f16 output, but it is slower: 50.93 versus 71.93 tokens/s for Llama 3.1 8B in the llama.cpp quantize README
- ✓For most users running Ollama daily, Q4_K_M is the right choice

Quick Answer
Use Q4_K_M with 8 GB VRAM or less and Q8_0 with 12+ GB. For Llama 3 8B, Q4_K_M is 4.58 GiB and Q8_0 is 7.96 GiB; Q8_0 stays closer to full precision.
- ▸Q4_K_M: 4.58 GiB for Llama 3.1 8B, about 5.1 GiB with a 4k KV cache (calculated)
- ▸Q8_0: 7.95 GiB for Llama 3.1 8B, about 8.5 GiB with a 4k KV cache (calculated)
- ▸Perplexity vs f16 on Llama 3 8B: Q4_K_M +2.8%, Q8_0 +0.02% (llama.cpp, calculated)
The Quick Verdict
On Llama 3 8B, Q8_0 is practically identical to f16 by perplexity, and Q4_K_M adds a small increase. llama.cpp's perplexity test on wikitext gives 6.2332 for f16, 6.2343 for Q8_0 and 6.4071 for Q4_K_M without an importance matrix: +0.02% and +2.8% (calculated; llama.cpp perplexity README). Perplexity is a language-modelling score, not a task benchmark, so it does not show how much a particular chat, coding or summarization task changes.
Q4_K_M is the usual default because the Llama 3.1 8B file is 4.58 GiB instead of 7.95 GiB and stays close to f16 on that test. Choose Q8_0 when memory is not the limit. We did not measure task quality ourselves; if one task matters to you, run both quantizations on it.
If your results feel wrong on Q4_K_M, a likely first thing to check is the model size and prompt structure: llama.cpp's measured perplexity gap to f16 is small.
Side-by-Side Comparison
The table uses Llama 3.1 8B file sizes from the llama.cpp quantize README, a calculated KV cache (32 layers, 8 key-value heads, head dimension 128, f16: 0.5 GiB for 4,096 tokens) and Llama 3 8B perplexity from the llama.cpp perplexity README. Both formats work with Ollama, LM Studio, and llama.cpp without any special configuration.
For context on what Q4_K_M means and how k-quant compression works, see the Q4_K_M explained guide. For the full quantization reference, see quantization levels compared.
None of the sources above tests long-document recall, multi-step math or long code generation, so this page makes no claim about them. Q8_0 has lower perplexity than Q4_K_M, so if you have the memory it is the lower-risk choice by that metric; test both on tasks where small errors matter. For a refresher, see what Q4_K_M means.
| Metric | Q4_K_M | Q8_0 |
|---|---|---|
| File size (Llama 3.1 8B) | 4.58 GiB | 7.95 GiB |
| VRAM with 4k KV cache (calculated, without runtime overhead) | ~5.1 GiB | ~8.5 GiB |
| Perplexity, Llama 3 8B (f16: 6.23) | 6.41 (+2.8%) | 6.23 (+0.02%) |
| Tokens/s, Llama 3.1 8B (README) | 71.93 | 50.93 |
| Best for | 6–8 GB VRAM | 12+ GB VRAM |
Quick Answers About Q4_K_M vs Q8_0
Is Q8_0 noticeably better than Q4_K_M?▾
Does Q8_0 run faster than Q4_K_M?▾
Can I switch between Q4_K_M and Q8_0 for different tasks?▾
ollama pull llama3:8b-instruct-q4_K_M and ollama pull llama3:8b-instruct-q8_0 are separate downloads. You switch by specifying the tag in ollama run.What about Q4_K_S — is it worth using instead of Q4_K_M?▾
Want the full breakdown?
Read the complete guide →Related Prompt Bites