What Is Q4_K_M Quantization?
By Hans Kuepper · Founder of PromptQuorum · Discovery engine for open-weight & open-source AI
Q4_K_M is a llama.cpp k-quant format: 4-bit quantization with a medium-size mix of tensor types. On Llama 3 8B it scores a perplexity of 6.41 against 6.70 for Q4_0, for a file 0.24 GiB larger.
Key Takeaways
- ✓Q4_K_M = 4-bit quantization with k-quant compression at medium quality — lower perplexity than Q4_0 for a slightly larger file
- ✓Llama 3.1 8B at Q4_K_M is 4.58 GiB on disk; with a 4k-token KV cache (about 0.5 GiB, calculated) it needs about 5.1 GiB of VRAM before runtime overhead
- ✓Use Q4_K_M as your default — on the Llama 3 8B test it adds under 3% perplexity over f16 at about 58% of the Q8_0 file size

Quick Answer
Q4_K_M means 4-bit quantization using k-quant (K) compression at medium (M) quality. It is a common default for most models: lower perplexity than Q4_0, smaller than Q8_0.
- ▸Q = quantized, 4 = 4-bit, K = k-quant, M = medium
- ▸Lower perplexity than Q4_0 (6.41 vs 6.70 on Llama 3 8B) for a file 0.24 GiB larger
- ▸Use Q4_K_M as your default quantization
What Each Letter in Q4_K_M Means
Q4_K_M exists because the older 4-bit format Q4_0 loses more quality than necessary. On Llama 3 8B, llama.cpp's perplexity test gives 6.41 for Q4_K_M and 6.70 for Q4_0 (lower is better; f16 is 6.23), for a file that is 0.24 GiB larger (4.58 vs 4.34 GiB). Source: llama.cpp perplexity README.
The "K" marks the k-quant family introduced in llama.cpp pull request #1684. Q4_K_M mixes two tensor types: it uses 6-bit Q6_K for half of the attention.wv and feed_forward.w2 tensors and for output.weight, and 4-bit Q4_K for the remaining tensors. On Llama 3.1 8B this averages 4.89 bits per weight (llama.cpp quantize README), above the nominal 4.
The "M" is the size variant within the family. Q4_K_S (small) uses Q4_K for every tensor: 4.36 GiB versus 4.58 GiB for Q4_K_M on Llama 3.1 8B, with higher perplexity (6.50 vs 6.41 on Llama 3 8B without an importance matrix). Q5_K_M is the next step up at 5.33 GiB.
K-quant works on blocks of weights. Q4_K groups 32 weights into a block and 8 blocks into a super-block, storing block scales and minimums with 6 bits, which is 4.5 bits per weight (pull request #1684). Q4_K_M lands at 4.89 because of its Q6_K tensors, which is why it sits between Q4_K_S (4.67) and Q5_K_M (5.70) in size. For when the M tier is not enough, see Q4_K_M vs Q8_0.
How Q4_K_M Compares to Other Quantizations
The table shows llama.cpp's Llama 3 8B perplexity test (wikitext, no importance matrix; lower is better). Percentages are calculated against f16 (6.2332). Q4_K_M adds under 3% perplexity at 4.58 GiB, about 58% of the Q8_0 file size (calculated: 4.58 / 7.96).
For a direct comparison of Q4_K_M vs Q8_0, see the Q4_K_M vs Q8_0 decision guide. For the full quantization reference, see the quantization levels comparison.
| Format | Size, Llama 3 8B (GiB) | Perplexity (change vs f16) |
|---|---|---|
| Q4_0 | 4.34 | 6.70 (+7.5%) |
| Q4_K_M | 4.58 | 6.41 (+2.8%) |
| Q5_K_M | 5.33 | 6.29 (+0.9%) |
| Q8_0 | 7.96 | 6.23 (+0.02%) |
Quick Answers About Quantization
Is Q4_K_M the same as Q4_0?▾
Which quantization should I use for 8 GB VRAM?▾
What does the 'M' in Q4_K_M stand for?▾
Which models on Ollama use Q4_K_M by default?▾
llama3.1:8b-instruct-q5_K_M or llama3.1:8b-instruct-q8_0.Want the full breakdown?
Read the complete guide →