LLM Quantization Explained: Q4_K_M vs Q4_0 vs Q8_0 (2026)

Quantization converts 16-bit model weights (FP16) to 4-bit or 8-bit integers, reducing RAM by 50-75% with only 1-3% quality loss at Q4_K_M. A large language model stores its learned knowledge as billions of numerical weights. By default, these are stored as 16-bit floating-point numbers (FP16) -- two bytes per weight. A 7B model has 7 billion weights, so the FP16 file size is approximately 14 GB.

Quantization replaces these 16-bit floats with lower-precision integers. At 4-bit quantization, each weight uses 0.5 bytes instead of 2 -- cutting memory to ~3.5 GB for the weights alone. With metadata overhead, a quantized 7B model at Q4_K_M is approximately 4.5 GB.

This matters for local inference because consumer hardware has limited RAM. Without quantization, a 7B model requires 16 GB of RAM to run. With Q4_K_M quantization, the same model runs on 6 GB of RAM, making it accessible on most modern laptops.

Q4_K_M is a 4-bit GGUF quantization format used in llama.cpp and Ollama. The "K" means it uses K-quants (mixed precision), and "M" = medium — a balance between model size, speed, and quality loss. Q4_K_M stores most weights at 4-bit but uses 6-bit for the most sensitive layers, giving it a better quality-to-size ratio than pure 4-bit Q4_0.

Q4_K_M at 4-bit is the standard recommendation -- approximately 4.5 GB RAM for a 7B model with only 1-3% quality loss vs FP16. Quantization names follow a pattern: Q{bits}_{variant}. The bit count is the weight precision; the variant affects how the quantization is applied:

Use this calculator to compute exact VRAM requirements for any combination of model, quantization, context, and batch size. Select your configuration and see which GPUs fit.

Q8_0 is an 8-bit GGUF quantization format that is effectively lossless — under 0.5% quality degradation versus FP16 — at roughly half the file size. Each weight is stored in 8 bits plus a small per-block scale, so a 7B model is ~7.7 GB instead of ~14 GB at FP16. Unlike the K-quants (Q4_K_M, Q5_K_M), Q8_0 uses uniform 8-bit precision for every weight — there is no mixed-precision "K" variant because 8 bits already preserves almost all information.

Choose Q4_K_M over Q4_0. Both average 4 bits per weight, but Q4_K_M is a K-quant that stores the most sensitive layers at 6-bit, recovering 5–8% quality at the same ~4.5 GB footprint for a 7B model. Q4_0 is the original uniform 4-bit format from early llama.cpp and exists today only for legacy compatibility. There is no size or speed reason to pick Q4_0 when Q4_K_M is available.

Q4_K_M is llama.cpp's standard 4-bit K-quant. Q4_K_XL is not a standard type — it is an Unsloth "Dynamic" GGUF variant that keeps a 4-bit base but upcasts the most sensitive layers (embeddings, attention and output) to higher precision. File size and quality land between Q4_K_M and Q5_K_M — the same principle Q8_K_XL applies over Q8_0, just on a 4-bit base.

Exact Q4_K_XL file sizes vary by model and by how many layers Unsloth upcasts, so check the size shown in LM Studio or on Hugging Face before downloading rather than planning around a fixed number. Practical rule: if Q5_K_M fits your VRAM, take Q5_K_M — it is a standard format with wider tool support. Q4_K_XL is the in-between step for when Q4_K_M fits but Q5_K_M just misses.

Q4_K_M and Q4_K_S are both 4-bit K-quants; the difference is how many layers stay at higher precision. Q4_K_M (Medium) keeps more sensitive layers at 6-bit, while Q4_K_S (Small) pushes more weights to 4-bit to save ~0.3–0.4 GB on a 7B model. Measured on llama.cpp, Q4_K_S adds about +0.11 perplexity at 7B versus +0.05 for Q4_K_M — roughly 3–5% more quality loss. Pick Q4_K_S only when those few hundred megabytes decide whether the model fits in VRAM.

For most chat and writing tasks, Q4_K_M is the better trade — it uses ~4.5 GB for a 7B model versus ~7.7 GB for Q8_0, with only 1–3% more quality loss. Choose Q8_0 (needs 16+ GB VRAM) when you need maximum fidelity for coding, math, or agentic tool use, where small errors compound. Q8_0 loses under 0.5% versus FP16; Q4_K_M loses 1–3%. The gap is imperceptible in everyday use but can matter on precise numerical reasoning.

Q8_0 is the standard llama.cpp 8-bit quant — every weight at 8-bit, ~7.7 GB for a 7B model, under 0.5% loss versus FP16. Q8_K_XL is not a stock llama.cpp type: it is an Unsloth "Dynamic" GGUF variant that keeps an 8-bit base but upcasts the most sensitive layers (embeddings, attention, and output) to 16-bit (BF16/F16), pushing quality closer to full FP16 at a slightly larger file size. Q8_K_XL targets users who want the last fraction of a percent of accuracy and have VRAM to spare.

Exact Q8_K_XL file sizes vary by model and by how many layers Unsloth upcasts, so verify the size shown in your tool (LM Studio or Hugging Face) before downloading. For a 7B–8B model expect it to sit slightly above Q8_0; on very large models the gap is wider. Because Q8_0 is already effectively lossless for most users, Q8_K_XL is worth it only when you specifically need maximum fidelity and the extra VRAM is free.

GGUF (GPT-Generated Unified Format) is the single-file standard for quantized LLM weights, containing model weights, metadata, and tokenizer -- used by Ollama, LM Studio, and llama.cpp. It was created by the llama.cpp project and replaces the older GGML format.

A GGUF file contains: the quantized model weights, all model metadata (architecture, tokenizer, context length), and a format version number. This self-contained design means a single `.gguf` file is everything needed to run the model -- no separate tokenizer files, no configuration JSON.

GGUF is the standard format for Ollama, LM Studio, Jan AI, and GPT4All. When you run `ollama pull llama3.1:8b`, Ollama downloads a GGUF file internally. When LM Studio shows model file sizes, those are GGUF file sizes.

The quantization level is part of the filename: `Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf` is a Q4_K_M quantized GGUF of Llama 3.1 8B.

Quantization cuts RAM by 50–75%: a 7B model drops from ~14 GB at FP16 to ~4.5 GB at Q4_K_M, with under 1% quality loss on standard benchmarks.

Q4_K_M loses 1-3% on MMLU benchmarks vs FP16 -- imperceptible in most practical tasks. Q3_K_S loses 5-10% and is noticeable on math and reasoning. Quality loss from quantization is measured by comparing benchmark scores between full-precision and quantized versions. The established findings are:

Quantization reduces memory usage but can degrade output quality. Well-engineered prompts compensate: techniques like few-shot examples and explicit output constraints help quantized models maintain accuracy. See prompt engineering techniques for methods that work at any quantization level.

Choose based on your available VRAM, not on the model size alone. The table below shows which quantization to select for different hardware constraints.

LM Studio (desktop app) shows available quantization variants for each model download. When searching for a model, you\'ll see multiple GGUF options: Q2_K, Q3_K_S, Q4_K_M, Q5_K_M, Q6_K, Q8_0.

Step 1: Open LM Studio → Navigate to the "Local Models" tab. Search for a model (e.g., "Llama 3.1 8B"). Step 2: Each model shows available quantizations. Look at the file size to estimate VRAM usage. Q4_K_M for a 7B model is usually listed as ~4.5 GB. Step 3: Click the download icon next to your chosen quantization.

Recommended defaults for LM Studio:

• If your GPU has 6-8 GB VRAM (RTX 4060, RTX 3060 Ti, RTX 4060 Ti): Download the Q4_K_M variant (smallest file with acceptable quality).

• If your GPU has 12-16 GB VRAM (RTX 4070, RTX 4080): Download Q5_K_M or Q6_K (better quality, still well within VRAM).

• If your GPU has 24+ GB VRAM (RTX 4090, RTX 5090): Download Q8_0 or FP16 (maximum quality, minimal speed penalty).

LM Studio\'s "GPU offload" feature: Check the "Use GPU" toggle in the chat interface. LM Studio will automatically move as many model layers to GPU as VRAM allows, offloading the rest to CPU RAM. If your system RAM is sufficient, this allows running models slightly larger than your GPU VRAM (e.g., Llama 3.3 70B Q4_K_M on RTX 4090 with 64+ GB system RAM).

When VRAM is full, models can offload (move) layers to system RAM. Offloading trades speed for capacity.

Scenario: Running 70B Q4 model on RTX 4090 (24 GB). Model needs 35 GB. With offloading, run at ~5-10 tokens/sec (80% to RAM).

Offloading is a last resort -- it makes inference impractical. Use only for offline batch processing or experimentation.

Modern inference engines (vLLM, llama.cpp) can split a model across multiple GPUs automatically. Learn more about Multi-GPU Local LLMs for advanced setups.

Example: 70B model with 2× RTX 4090:

• Without splitting: Impossible (needs 40+ GB VRAM in one GPU).

• With splitting: Half the model weights on each GPU. Inference speed: ~100 tokens/sec (communication overhead is minimal).

Layer splitting is practical for production deployments and is transparent to the user.

KV Cache quantization reduces the memory required to store attention key-value pairs during inference, particularly important when processing long contexts (32K+ tokens). While model weight quantization (Q4_K_M) is most common, KV cache quantization targets a different memory bottleneck.

During inference, the model maintains running key-value (KV) pairs for each token in the context. For a 7B model processing a 32K-token context, KV cache alone can consume 8–16 GB of VRAM depending on the precision. Standard KV cache uses FP16 (2 bytes per value); quantizing the KV cache to FP8 or Q8 reduces this by 50%.

How to enable KV Cache quantization:

• Ollama: Automatic on compatible models; no user configuration needed.

• LM Studio: Check "KV cache quantization" toggle in Settings (if available on your version).

• llama.cpp: Start the server with `--cache-type-k q8_0 --cache-type-v q8_0` (short form: `-ctk q8_0 -ctv q8_0`). Use `q4_0` instead of `q8_0` to halve the KV cache again.

Trade-offs: KV cache quantization has minimal quality impact (<1% degradation even with aggressive quantization) because attention patterns are more robust to lower precision than model weights. Recommended for models processing 16K+ contexts on constrained hardware.

Best results come from combining all three techniques. See VRAM Requirements Guide for specific hardware planning.

Scenario 1: 70B on single RTX 4090 (24 GB)

• Quantize to Q4 (35 GB → 18 GB)

• Use offloading for remaining 6 GB (to system RAM)

• Result: ~8-10 tokens/sec (slow but works)

Scenario 2: 70B on 2× RTX 4090

• Quantize to Q5 (43.75 GB)

• Use layer splitting across 2 GPUs (22 GB each)

• Result: ~100 tokens/sec (practical)

Each technique trades VRAM reduction for speed penalties. Quantization has minimal impact; offloading causes 5–10× slowdown; layer splitting adds ~5% overhead.

Mac Studio with M5 Ultra (from $5,499, 96 GB unified memory standard, up to 512 GB) runs Llama 3.3 70B at Q4 natively — no offloading, no layer splitting required.

Apple refreshed the Mac Studio lineup on August 25, 2026 with two chips: M5 Max ($2,499, up to 128 GB unified memory, up to 614 GB/s bandwidth per Apple's published specs) and M5 Ultra ($5,499 starting at 96 GB, up to 512 GB, up to 1.2 TB/s bandwidth per Apple's published specs). Both replace the previous M2 Ultra generation (192 GB max, ~800 GB/s). DDR5 system RAM offloading is still capped at ~90 GB/s -- that bandwidth gap is why unified memory outperforms CPU-RAM offloading for 70B-class models.

Independent throughput benchmarks for M5 Max and M5 Ultra are not yet available. Standard configurations ship September 22, 2026, and the 512 GB M5 Ultra configuration ships in late October 2026. PromptQuorum has not tested this hardware -- tokens-per-second figures below are omitted for the new chips until independent measurements exist.