Skip to main content
PromptQuorumBuilt for humans. Structured for AI.

Q4_K_M vs Q8_0: Which Should You Pick?

By · Founder of PromptQuorum · Discovery engine for open-weight & open-source AI

Q4_K_M and Q8_0 are two llama.cpp quantization formats: on Llama 3 8B, Q8_0 is practically identical to f16 by perplexity (+0.02%), while Q4_K_M adds +2.8% but needs a 4.58 GiB file instead of 7.96 GiB (Llama 3.1 8B: 7.95 GiB). The choice mostly depends on your VRAM.

Key Takeaways

  • ✓8 GB VRAM or less: use Q4_K_M — its 8B file is 4.58 GiB, while Q8_0 (7.95 GiB) no longer fits once a KV cache is added
  • ✓12+ GB VRAM: Q8_0 is worth it for near-f16 output, but it is slower: 50.93 versus 71.93 tokens/s for Llama 3.1 8B in the llama.cpp quantize README
  • ✓For most users running Ollama daily, Q4_K_M is the right choice
Q4_K_M vs Q8_0: Which Should You Pick?

Quick Answer

Use Q4_K_M with 8 GB VRAM or less and Q8_0 with 12+ GB. For Llama 3 8B, Q4_K_M is 4.58 GiB and Q8_0 is 7.96 GiB; Q8_0 stays closer to full precision.

  • ▸Q4_K_M: 4.58 GiB for Llama 3.1 8B, about 5.1 GiB with a 4k KV cache (calculated)
  • ▸Q8_0: 7.95 GiB for Llama 3.1 8B, about 8.5 GiB with a 4k KV cache (calculated)
  • ▸Perplexity vs f16 on Llama 3 8B: Q4_K_M +2.8%, Q8_0 +0.02% (llama.cpp, calculated)
Quantization & VRAMIntermediate

The Quick Verdict

On Llama 3 8B, Q8_0 is practically identical to f16 by perplexity, and Q4_K_M adds a small increase. llama.cpp's perplexity test on wikitext gives 6.2332 for f16, 6.2343 for Q8_0 and 6.4071 for Q4_K_M without an importance matrix: +0.02% and +2.8% (calculated; llama.cpp perplexity README). Perplexity is a language-modelling score, not a task benchmark, so it does not show how much a particular chat, coding or summarization task changes.

Q4_K_M is the usual default because the Llama 3.1 8B file is 4.58 GiB instead of 7.95 GiB and stays close to f16 on that test. Choose Q8_0 when memory is not the limit. We did not measure task quality ourselves; if one task matters to you, run both quantizations on it.

If your results feel wrong on Q4_K_M, a likely first thing to check is the model size and prompt structure: llama.cpp's measured perplexity gap to f16 is small.

Side-by-Side Comparison

The table uses Llama 3.1 8B file sizes from the llama.cpp quantize README, a calculated KV cache (32 layers, 8 key-value heads, head dimension 128, f16: 0.5 GiB for 4,096 tokens) and Llama 3 8B perplexity from the llama.cpp perplexity README. Both formats work with Ollama, LM Studio, and llama.cpp without any special configuration.

For context on what Q4_K_M means and how k-quant compression works, see the Q4_K_M explained guide. For the full quantization reference, see quantization levels compared.

None of the sources above tests long-document recall, multi-step math or long code generation, so this page makes no claim about them. Q8_0 has lower perplexity than Q4_K_M, so if you have the memory it is the lower-risk choice by that metric; test both on tasks where small errors matter. For a refresher, see what Q4_K_M means.

MetricQ4_K_MQ8_0
File size (Llama 3.1 8B)4.58 GiB7.95 GiB
VRAM with 4k KV cache (calculated, without runtime overhead)~5.1 GiB~8.5 GiB
Perplexity, Llama 3 8B (f16: 6.23)6.41 (+2.8%)6.23 (+0.02%)
Tokens/s, Llama 3.1 8B (README)71.9350.93
Best for6–8 GB VRAM12+ GB VRAM

Quick Answers About Q4_K_M vs Q8_0

Is Q8_0 noticeably better than Q4_K_M?▾
Possibly, but these sources do not show where. Q8_0 has less quantization error: +0.02% perplexity vs f16 on Llama 3 8B, against +2.8% for Q4_K_M (llama.cpp, calculated). Perplexity does not measure chat, coding or summarization quality directly, so test both on your own task.
Does Q8_0 run faster than Q4_K_M?▾
No. Q8_0 is larger and needs more memory bandwidth per token. The llama.cpp quantize README lists 71.93 tokens/s for Q4_K_M and 50.93 tokens/s for Q8_0 on Llama 3.1 8B (speeds depend on hardware). See what Q4_K_M means for the underlying reason.
Can I switch between Q4_K_M and Q8_0 for different tasks?▾
Only by pulling and running different model tags. In Ollama: ollama pull llama3:8b-instruct-q4_K_M and ollama pull llama3:8b-instruct-q8_0 are separate downloads. You switch by specifying the tag in ollama run.
What about Q4_K_S — is it worth using instead of Q4_K_M?▾
Q4_K_S is 0.22 GiB smaller than Q4_K_M (4.36 vs 4.58 GiB for Llama 3.1 8B) and has higher perplexity (6.50 vs 6.41 on Llama 3 8B without an importance matrix). Use Q4_K_S only if Q4_K_M does not fit; otherwise Q4_K_M is the better choice.
MCP