Skip to main content
PromptQuorumBuilt for humans. Structured for AI.

What Is Q4_K_M Quantization?

By · Founder of PromptQuorum · Discovery engine for open-weight & open-source AI

Q4_K_M is a llama.cpp k-quant format: 4-bit quantization with a medium-size mix of tensor types. On Llama 3 8B it scores a perplexity of 6.41 against 6.70 for Q4_0, for a file 0.24 GiB larger.

Key Takeaways

  • ✓Q4_K_M = 4-bit quantization with k-quant compression at medium quality — lower perplexity than Q4_0 for a slightly larger file
  • ✓Llama 3.1 8B at Q4_K_M is 4.58 GiB on disk; with a 4k-token KV cache (about 0.5 GiB, calculated) it needs about 5.1 GiB of VRAM before runtime overhead
  • ✓Use Q4_K_M as your default — on the Llama 3 8B test it adds under 3% perplexity over f16 at about 58% of the Q8_0 file size
What Is Q4_K_M Quantization?

Quick Answer

Q4_K_M means 4-bit quantization using k-quant (K) compression at medium (M) quality. It is a common default for most models: lower perplexity than Q4_0, smaller than Q8_0.

  • ▸Q = quantized, 4 = 4-bit, K = k-quant, M = medium
  • ▸Lower perplexity than Q4_0 (6.41 vs 6.70 on Llama 3 8B) for a file 0.24 GiB larger
  • ▸Use Q4_K_M as your default quantization
Quantization & VRAMIntermediate

What Each Letter in Q4_K_M Means

Q4_K_M exists because the older 4-bit format Q4_0 loses more quality than necessary. On Llama 3 8B, llama.cpp's perplexity test gives 6.41 for Q4_K_M and 6.70 for Q4_0 (lower is better; f16 is 6.23), for a file that is 0.24 GiB larger (4.58 vs 4.34 GiB). Source: llama.cpp perplexity README.

The "K" marks the k-quant family introduced in llama.cpp pull request #1684. Q4_K_M mixes two tensor types: it uses 6-bit Q6_K for half of the attention.wv and feed_forward.w2 tensors and for output.weight, and 4-bit Q4_K for the remaining tensors. On Llama 3.1 8B this averages 4.89 bits per weight (llama.cpp quantize README), above the nominal 4.

The "M" is the size variant within the family. Q4_K_S (small) uses Q4_K for every tensor: 4.36 GiB versus 4.58 GiB for Q4_K_M on Llama 3.1 8B, with higher perplexity (6.50 vs 6.41 on Llama 3 8B without an importance matrix). Q5_K_M is the next step up at 5.33 GiB.

K-quant works on blocks of weights. Q4_K groups 32 weights into a block and 8 blocks into a super-block, storing block scales and minimums with 6 bits, which is 4.5 bits per weight (pull request #1684). Q4_K_M lands at 4.89 because of its Q6_K tensors, which is why it sits between Q4_K_S (4.67) and Q5_K_M (5.70) in size. For when the M tier is not enough, see Q4_K_M vs Q8_0.

How Q4_K_M Compares to Other Quantizations

The table shows llama.cpp's Llama 3 8B perplexity test (wikitext, no importance matrix; lower is better). Percentages are calculated against f16 (6.2332). Q4_K_M adds under 3% perplexity at 4.58 GiB, about 58% of the Q8_0 file size (calculated: 4.58 / 7.96).

For a direct comparison of Q4_K_M vs Q8_0, see the Q4_K_M vs Q8_0 decision guide. For the full quantization reference, see the quantization levels comparison.

FormatSize, Llama 3 8B (GiB)Perplexity (change vs f16)
Q4_04.346.70 (+7.5%)
Q4_K_M4.586.41 (+2.8%)
Q5_K_M5.336.29 (+0.9%)
Q8_07.966.23 (+0.02%)

Quick Answers About Quantization

Is Q4_K_M the same as Q4_0?▾
No. On Llama 3 8B, Q4_K_M reaches a perplexity of 6.41 against 6.70 for Q4_0, for a file 0.24 GiB larger (4.58 vs 4.34 GiB; llama.cpp, no importance matrix). Prefer Q4_K_M over Q4_0. See the Q4_K_M vs Q8_0 guide for when to go higher.
Which quantization should I use for 8 GB VRAM?▾
Q4_K_M for 8B models: the Llama 3.1 8B file is 4.58 GiB, or about 5.1 GiB with a 4k-token KV cache (calculated, before runtime overhead). Q5_K_M is 5.33 GiB, about 5.8 GiB with the same cache (calculated), and cuts the perplexity increase over f16 from +2.8% to +0.9% on Llama 3 8B. At a 4k context both fit in 8 GB before runtime overhead; Q8_0 (7.95 GiB file) does not.
What does the 'M' in Q4_K_M stand for?▾
Medium: it is the size variant within the k-quant family. Q4_K_S (small) is smaller (4.36 GiB vs 4.58 GiB on Llama 3.1 8B) with higher perplexity, and Q4_K_M (medium) is the usual default. llama.cpp's quantize tool has no Q4_K_L type; the next step up is Q5_K_M.
Which models on Ollama use Q4_K_M by default?▾
Several. For Llama 3.1 8B, Qwen3 8B, Mistral 7B and Phi-4 mini, the default tag in the Ollama library shares its digest with a tag that has q4_K_M in its name (for Mistral 7B: the v0.3 tag) (checked 2026-11-06); the library pages do not label the default quantization, and other models can differ, so check the tag list of the model. Other quantizations are separate tags whose names vary by model, for example llama3.1:8b-instruct-q5_K_M or llama3.1:8b-instruct-q8_0.
MCP