Key Takeaways
- Best reasoning at small scale: Phi-4 Mini 3.8B -- 68% MMLU, 70% HumanEval, runs on 4 GB RAM.
- Fastest on CPU: Gemma 2 2B -- 40-60 tok/sec on any modern laptop, 1.7 GB RAM.
- Best small coding model: Qwen2.5 3B -- 65% HumanEval at ~2 GB RAM.
- Best general-purpose 3B: Llama 3.2 3B -- most community support, 128K context, 2.5 GB RAM.
- As of April 2026, no sub-2B model produces output quality suitable for professional tasks. Use 3B+ for real work.
What Is a "Small" Local LLM and When Should You Use One?
A small local LLM is typically defined as a model with fewer than 4 billion parameters. At Q4_K_M quantization, these models require 1.5-3 GB of RAM -- well within the constraints of entry-level laptops with 4-8 GB total memory.
As of April 2026, small models are appropriate for: quick summarization, simple Q&A, code snippet explanation, translation of short texts, and classification tasks. They are not suitable for multi-step reasoning, complex code generation, or writing long-form coherent documents.
The quality gap between a 3B and 7B model is significant -- roughly equivalent to the gap between GPT-3.5 Mini and GPT-3.5 Turbo. For users with 8 GB RAM, a 7B model at Q4_K_M is almost always the better choice if the machine has headroom. See Best Beginner Local LLM Models for 7B recommendations.
Which Model Should You Use? Quick Decision Guide?
Phi-4 Mini 3.8B -- Best Reasoning Performance in the Sub-4B Class
Microsoft Phi-4 Mini achieves 68% on MMLU and 70% on HumanEval -- scores that exceed many 7B models released before 2025. This is possible because Phi-4 Mini was trained on a curated synthetic dataset focused on reasoning and problem-solving, rather than broad web text.
As of April 2026, Phi-4 Mini is the recommended choice for users who primarily need reasoning (math, logic, step-by-step explanations) or coding assistance on hardware with 4-6 GB RAM.
| Spec | Value |
|---|---|
| MMLU | 68% |
| HumanEval | 70% |
| RAM (Q4_K_M) | ~2.5 GB |
| Context | 128K tokens |
| CPU speed | 30-50 tok/sec |
| Ollama command | ollama run phi4-mini |
Gemma 2 2B -- Fastest Small Local LLM on CPU
Google Gemma 2 2B generates 40-60 tokens/sec on a modern laptop CPU -- the fastest of any model at this quality tier. Its 1.7 GB RAM footprint leaves ample memory for the OS and other applications on a 4 GB machine.
Quality is lower than Phi-4 Mini or Llama 3.2 3B on reasoning tasks. The 8K context window (vs. 128K on Phi-4 Mini and Llama 3.2) is a practical limitation for longer documents. Gemma 2 2B is the right choice when response speed matters more than output depth.
| Spec | Value |
|---|---|
| MMLU | 52% |
| RAM (Q4_K_M) | ~1.7 GB |
| Context | 8K tokens |
| CPU speed | 40-60 tok/sec |
| Ollama command | ollama run gemma2:2b |
Qwen2.5 3B -- Best Small Model for Coding Tasks
Qwen2.5 3B scores 65% on HumanEval -- 5 percentage points above Llama 3.2 3B -- making it the best choice for coding tasks at the 3B scale. It includes JSON mode and function calling support, and natively handles 29 languages.
For non-coding tasks in English, Llama 3.2 3B and Phi-4 Mini produce more natural prose. Choose Qwen2.5 3B specifically when coding or multilingual output is the primary use case.
| Spec | Value |
|---|---|
| MMLU | 62% |
| HumanEval | 65% |
| RAM (Q4_K_M) | ~2 GB |
| Context | 128K tokens |
| CPU speed | 25-40 tok/sec |
| Ollama command | ollama run qwen2.5:3b |
Llama 3.2 3B -- Best General-Purpose Small Model
Meta Llama 3.2 3B is the most widely documented and community-supported 3B model. It scores 58% on MMLU and 60% on HumanEval -- slightly below Phi-4 Mini on both -- but has the widest tool support, the most fine-tunes available, and the largest collection of community guides.
The 128K context window is the same as larger Llama 3.x models, making it suitable for summarizing medium-length documents. For a first small model, Llama 3.2 3B remains the safest choice due to predictable behavior and extensive documentation.
| Spec | Value |
|---|---|
| MMLU | 58% |
| RAM (Q4_K_M) | ~2.5 GB |
| Context | 128K tokens |
| CPU speed | 25-45 tok/sec |
| Ollama command | ollama run llama3.2:3b |
Llama 3.2 1B -- Absolute Minimum for Any Useful Output
Llama 3.2 1B requires only 1.3 GB of RAM and generates 60-90 tok/sec on CPU -- the fastest locally-runnable model. Output quality is marginal: it handles very simple classification and keyword extraction but struggles with coherent multi-sentence responses. As of April 2026, use Llama 3.2 1B only when RAM is genuinely the binding constraint (under 3 GB available) or for testing tool integrations.
Full Comparison: Best Small Local LLMs Under 4B Parameters
| Model | MMLU | HumanEval | RAM | Context | Best For |
|---|---|---|---|---|---|
| Phi-4 Mini 3.8B | 68% | 70% | 2.5 GB | 128K | Reasoning, coding |
| Qwen2.5 3B | 62% | 65% | 2 GB | 128K | Coding, multilingual |
| Llama 3.2 3B | 58% | 60% | 2.5 GB | 128K | General use, first model |
| Gemma 2 2B | 52% | 38% | 1.7 GB | 8K | Speed, very low RAM |
| Llama 3.2 1B | 32% | 28% | 1.3 GB | 128K | Absolute minimum RAM |
Small Local LLMs by Region
EU / GDPR: For EU professionals running AI on constrained hardware -- field work, air-gapped environments, older enterprise laptops -- small local models provide GDPR-compliant inference with zero data egress. A Phi-4 Mini 3.8B running on a standard-issue corporate laptop (8 GB RAM) keeps all processed text on-device under GDPR Article 5 (data minimization). For German BSI compliance documentation: Phi-4 Mini (Microsoft, MIT licence) and Llama 3.2 3B (Meta, Llama Community licence) both provide versioned model identifiers via their Ollama tags, satisfying AI tool documentation requirements. Mistral does not currently offer a sub-4B model. For EU organizations preferring an EU-origin model at this size class, options are limited until Mistral releases a sub-4B variant.
Japan (METI): For Japanese-language tasks at the small model tier, Qwen2.5 3B is the only model in this comparison with native Japanese tokenization. Llama 3.2 3B handles Japanese but with lower token efficiency. For Japanese summarization or translation on constrained hardware: `ollama run qwen2.5:3b`. The speed advantage of small models is particularly relevant for Japanese enterprise use: 25-40 tok/sec on CPU provides adequate real-time response for chat interfaces on standard-issue office hardware.
China: Qwen2.5 3B (Alibaba, Apache 2.0) is the natural choice for Chinese-language small model deployment. Native Chinese tokenization processes Mandarin text 30-40% more efficiently than Llama at equivalent parameter count. For IoT and edge deployments under China's Data Security Law (数据安全法): `ollama run qwen2.5:3b` runs on any Linux device with 4 GB RAM and processes all text on-device with no external API calls.
What Are the Common Mistakes When Running Small Local LLMs?
- Using Q8_0 quantization instead of Q4_K_M: Q8_0 requires nearly double the RAM of Q4_K_M for minimal quality improvement at small scale. A Llama 3.2 3B model at Q8_0 needs ~3.8 GB RAM vs ~2.5 GB for Q4_K_M. On a 4 GB machine, Q8_0 may trigger swap usage and make inference 3-5× slower. Always use Q4_K_M as the default for sub-4B models.
- Running a base model instead of the instruct variant: Base models (e.g., `llama3.2:3b-text`) are pre-fine-tuning checkpoints trained to predict the next token in text. They do not follow instructions. When you ask a base model "What is 2+2?", it may complete the sentence as a quiz rather than answer "4". Always use the instruct variant: `llama3.2:3b` (Ollama defaults to instruct for named models).
- Expecting 7B model quality from a 3B model: A 3B model at 68% MMLU (Phi-4 Mini) performs similarly to a 2023-era GPT-3.5 Mini on general tasks. Complex reasoning chains, long-form writing, and nuanced code generation will produce noticeably lower quality than a 7B model. If output quality is insufficient, upgrade to a 7B model -- the RAM difference is ~2 GB (2.5 GB → 4.5 GB).
Understanding Quantization: RAM vs Quality Trade-off
Common Questions About Small Local LLM Models
What is the smallest local LLM that produces useful output?
As of April 2026, the practical minimum for useful output is a 3B model at Q4_K_M quantization. Models below 2B parameters (Llama 3.2 1B, Gemma 2 2B) produce coherent single sentences but struggle with multi-step instructions, longer responses, and complex reasoning. For tasks like summarization and simple Q&A, Gemma 2 2B is usable. For anything more complex, start with a 3B model.
Can a 3B model run on a phone?
Yes -- Llama 3.2 1B and 3B are specifically designed for on-device mobile deployment. Meta provides optimized builds for iOS (via MLC LLM) and Android. Inference on a modern phone (Snapdragon 8 Gen 3 or Apple A17 Pro) produces 15-30 tok/sec for 1B models. LM Studio and Ollama do not currently run on iOS or Android -- mobile requires separate frameworks.
Are small models good for summarization?
Yes -- summarization is one of the strongest use cases for small models. Gemma 2 2B and Llama 3.2 3B reliably produce accurate summaries of texts up to ~4,000 words (their practical context limit for quality output). For longer documents, use a model with a large context window like Phi-4 Mini or Llama 3.2 3B (both 128K tokens).
How much faster is a 2B model than a 7B model on the same hardware?
Approximately 2-3× faster on CPU. Gemma 2 2B generates 40-60 tok/sec vs 10-20 tok/sec for Mistral 7B on the same laptop CPU. On a GPU, the speed advantage narrows because GPU throughput is less constrained by model size. The speed difference is most noticeable on CPU-only machines.
Do small models support function calling?
Some do. Qwen2.5 3B supports function calling and JSON mode. Llama 3.2 3B has basic tool use support. Gemma 2 2B does not support function calling. Check the model's documentation before building a pipeline that depends on structured output.
Which small model is best for languages other than English?
Qwen2.5 3B supports 29 languages natively including Chinese, Japanese, Korean, and Arabic. Gemma 2 2B and Phi-4 Mini are primarily English-optimized. For non-English tasks at the small model scale, Qwen2.5 3B is the clear choice. See Qwen vs Llama vs Mistral multilingual comparison for a full language comparison.
What is the difference between Phi-4 Mini and Llama 3.2 3B for everyday tasks?
Phi-4 Mini outperforms Llama 3.2 3B on reasoning, math, and coding (68% vs 58% MMLU, 70% vs 60% HumanEval) at nearly identical RAM (2.5 GB each). For everyday tasks -- Q&A, summarization, simple explanations -- the quality gap is noticeable but not dramatic. Llama 3.2 3B has broader community support and more fine-tunes available. Choose Phi-4 Mini for structured reasoning; Llama 3.2 3B for general chat and broader compatibility.
Can I run two small models simultaneously?
Yes, if total RAM permits. Two 3B models at Q4_K_M use ~5 GB combined -- feasible on an 8 GB machine with a lean OS. Ollama loads one model at a time per process by default. Run two Ollama instances on different ports (OLLAMA_HOST=:11434 and OLLAMA_HOST=:11435) to serve two models in parallel. This is useful for A/B testing outputs.
Do small models work for RAG (retrieval-augmented generation)?
Yes for simple RAG. Llama 3.2 3B and Phi-4 Mini can answer questions over retrieved document chunks reliably. For RAG over large knowledge bases requiring multi-hop reasoning, 7B+ models perform more consistently. GPT4All's LocalDocs feature uses a 3B model for document Q&A and works well for personal document collections.
Is Phi-4 Mini better than Llama 3.2 3B for coding?
Yes. Phi-4 Mini scores 70% on HumanEval vs 60% for Llama 3.2 3B -- a meaningful 10-point gap at this scale. For coding assistance on 4-6 GB RAM machines, Phi-4 Mini is the recommended choice. For multilingual coding (non-Python), Qwen2.5 3B at 65% HumanEval is competitive with Phi-4 Mini while also supporting function calling.
Sources
- Hugging Face Open LLM Leaderboard -- open-llm-leaderboard.hf.space (MMLU and HumanEval scores)
- Microsoft Phi-4 Technical Report -- microsoft.com/en-us/research/publication/phi-4-technical-report/
- Meta Llama 3.2 Model Card -- huggingface.co/meta-llama/Llama-3.2-3B-Instruct
- Google Gemma 2 Technical Report -- storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf