Skip to main content
PromptQuorumBuilt for humans. Structured for AI.
Home/Local LLMs
Best Models

Long Context Local LLMs 2026: 128K, 256K and 1M Models Compared

·10 min readBy · Founder of PromptQuorum · Discovery engine for open-weight & open-source AI

In 2026, 128K tokens is the entry level for local LLMs, not the ceiling: Qwen3.5, Qwen3.8 and Gemma 4 (12B and larger) support 256K, NVIDIA Nemotron 3 Nano 30B supports 1M and Llama 4 Scout lists 10M. The real limit is memory: the KV cache at 128K takes from about 0.75 GiB (Nemotron 3 Nano) to 20 GiB (Mistral Small 3.2) depending on the architecture.

Key Takeaways

  • 256K is mainstream, 1M is possible. Qwen3.5, Qwen3.8 and Gemma 4 (12B, 26B, 31B) list 256K windows in the Ollama library. Nemotron 3 Nano 30B supports 1M. Gemma 4 E2B/E4B, Mistral Small 3.2, gpt-oss 20B and Llama 3.1 8B stay at 128K.
  • The KV cache decides memory, and it depends on the architecture. At 128K the f16 cache takes 16 GiB for Llama 3.1 8B, 8 GiB for Qwen3.8 27B, about 2.7 GiB for Gemma 4 26B-A4B and about 0.75 GiB for Nemotron 3 Nano.
  • Hybrid architectures make long windows cheap. Sliding-window attention (Gemma 4, gpt-oss), Gated DeltaNet (Qwen3.5, Qwen3.8) and Mamba-2 (Nemotron 3 Nano) replace many full-attention layers, and only full-attention layers grow with the window.
  • Ollama defaults to 4K tokens below 24 GiB of VRAM, 32K from 24 to 48 GiB and 256K from 48 GiB. Set the context explicitly and check it with ollama ps.
  • Usable context is shorter than advertised. RULER and NoLiMa found large accuracy drops at 32K for many models that claim 128K or more. Put critical facts at the start or end and test at your target length.
  • Choose by task. RAG suits search across many documents. Long context suits reasoning across one coherent document, such as a contract, a codebase or a book.

📍 In One Sentence

In 2026, 128K tokens is the entry level for local LLMs; Qwen3.5, Qwen3.8 and Gemma 4 (12B and larger) support 256K and NVIDIA Nemotron 3 Nano 30B supports 1M, but memory, not the window, sets the practical limit -- the f16 KV cache at 128K ranges from 0.75 GiB to 20 GiB depending on the architecture.

💬 In Plain Terms

Context length is how much text a model can read in one request. 128K tokens is about 100,000 words, 256K about 200,000 words. Bigger windows cost memory, and models read the middle of very long inputs less reliably, so test with your own documents before you depend on a long window.

In 2026, 128K tokens is the entry level for local LLMs, not the ceiling. The Ollama library lists 256K windows for Qwen3.5, Qwen3.8 and Gemma 4 (12B and larger), 1M for NVIDIA Nemotron 3 Nano 30B and 10M for Llama 4 Scout. Memory is the real limit: the KV cache at 128K takes from under 1 GiB (Nemotron 3 Nano) to 20 GiB (Mistral Small 3.2) depending on the architecture, and Ollama defaults to only 4K tokens on GPUs with under 24 GiB of VRAM. This guide lists which models support which window, what each costs in memory and how to set the context length.

Slide Deck: Long Context Local LLMs 2026: 128K, 256K and 1M Models Compared

The slide deck below covers: which local LLMs reach 128K, 256K and 1M tokens, KV-cache memory per model at 32K, 128K and the maximum window, why architecture decides memory, the usable-versus-advertised context gap (RULER, NoLiMa) and how to set num_ctx in Ollama. Download the PDF as a long context local LLM reference card.

Browse the slides below or download as PDF for offline reference. Download Reference Card (PDF)

Long Context Local LLMs 2026: 128K, 256K and 1M Models Compared

What Is Context Length and Why Does It Matter for Local LLMs?

Context length is the maximum number of tokens a model can hold in one request: your prompt, any documents, the conversation so far and the answer it writes, added together. One token is about 0.75 English words, so 128K tokens is about 100,000 words and 256K tokens about 200,000 words.

The advertised window is the maximum the model architecture supports. The usable window is the length at which answers stay accurate for your task, and it is shorter.

Long windows matter for whole codebases, contracts, long transcripts, book chapters and agent sessions where tool output piles up. The Ollama documentation recommends at least 64,000 tokens for agents, web search and coding tools.

Three things set the limit on your own machine: the model's window, the memory its KV cache needs at that length and how reliably it uses text at that length. The next sections cover each in turn.

Which Local LLMs Support More Than 128K Tokens in 2026?

Several do: Qwen3.5, Qwen3.8 and Gemma 4 (12B and larger) list 256K windows in the Ollama library, NVIDIA Nemotron 3 Nano 30B lists 1M and Llama 4 Scout lists 10M. Models released before 2026 mostly stop at 128K.

The table shows the window listed by the model card or the Ollama library. A longer-context extension, such as YaRN, is a vendor-documented option that needs extra configuration and memory.

Llama 4 Scout lists 10M tokens, but its model card reports its long-context benchmarks at a 128K window, and the Ollama download is 67 GB. Llama 4 Maverick lists 1M but is a 245 GB download, so it needs server hardware.

Ollama tags can differ in window size: qwen3:8b and qwen3:14b list 40K, while qwen3:4b and qwen3:30b list 256K. Read the context figure on the model's Ollama page before you plan around it.

Model
Listed window
Longer context
Ollama tag
Nemotron 3 Nano 30B-A3B1M (default 256K)—ollama run nemotron-3-nano:30b
Qwen3.8 27B256KUp to 1M (YaRN)ollama run qwen3.8:27b
Qwen3.5 9B / 27B256KUp to 1.01M (YaRN)ollama run qwen3.5:9b
Qwen3 30B-A3B-2507256K1M (DCA, ~240 GB GPU)ollama run qwen3:30b
Gemma 4 12B / 26B / 31B256K—ollama run gemma4:26b
Gemma 4 E2B / E4B128K—ollama run gemma4:e4b
Llama 4 Scout 109B10M listedBenchmarked at 128Kollama run llama4:16x17b
Mistral Small 3.2 24B128K—ollama run mistral-small3.2
gpt-oss 20B128K—ollama run gpt-oss:20b
Llama 3.1 8B128K—ollama run llama3.1:8b
Qwen3 8B / 14B40K—ollama run qwen3:14b
Listed context windows of local LLMs on a log scale: 128K is the entry level, 256K is common, Nemotron 3 Nano supports 1M and Llama 4 Scout lists 10M.
Listed context windows of local LLMs on a log scale: 128K is the entry level, 256K is common, Nemotron 3 Nano supports 1M and Llama 4 Scout lists 10M.

How Much Memory Does Long Context Need?

Memory for long context is the model weights plus the KV cache, and the KV cache depends on the architecture, not only on the window. The KV cache stores the attention keys and values of every token in the context and is allocated on top of the weights.

KV bytes per token = 2 (keys and values) × full-attention layers × KV heads × head dimension × bytes per value (2 for the default f16 cache). Llama 3.1 8B: 2 × 32 × 8 × 128 × 2 = 131,072 bytes (128 KiB) per token, which is 16 GiB at 128K tokens, more than three times its 4.9 GB download.

Newer architectures cut this cost. Gemma 4 uses sliding-window attention in most layers (a 1,024-token window in the 12B, 26B and 31B models), gpt-oss in every second layer (128 tokens), Qwen3.5 and Qwen3.8 use Gated DeltaNet linear attention in three of every four layers, and Nemotron 3 Nano has only 6 attention layers out of 52. Only full-attention layers grow with the window.

The values below are calculated from each model's published configuration, not measured. Runtimes add compute buffers and cache implementations differ, so confirm real use with ollama ps.

Quantizing the KV cache shrinks it: according to the Ollama documentation, q8_0 uses about half the memory of f16 and q4_0 about a quarter, with a small-to-medium precision loss for q4_0.

Model
Download
KV at 32K
KV at 128K
KV at max window
Llama 3.1 8B4.9 GB4 GiB16 GiB16 GiB (128K)
Mistral Small 3.2 24B15 GB5 GiB20 GiB20 GiB (128K)
gpt-oss 20B14 GB0.8 GiB3 GiB3 GiB (128K)
Qwen3.5 4B3.3 GB1 GiB4 GiB8 GiB (256K)
Qwen3.5 9B6.6 GB1 GiB4 GiB8 GiB (256K)
Gemma 4 12B7.7 GB0.8 GiB2.3 GiB4.3 GiB (256K)
Gemma 4 26B-A4B16 GB0.8 GiB2.7 GiB5.2 GiB (256K)
Gemma 4 31B19 GB3.3 GiB10.8 GiB20.8 GiB (256K)
Qwen3.8 27B18 GB2 GiB8 GiB16 GiB (256K)
Qwen3.5 35B-A3B22 GB0.6 GiB2.5 GiB5 GiB (256K)
Qwen3 30B-A3B-250719 GB3 GiB12 GiB24 GiB (256K)
Nemotron 3 Nano 30B-A3B18 GB0.2 GiB0.8 GiB6 GiB (1M)
Weights plus f16 KV cache at a 128K context: Llama 3.1 8B needs about 21 GB, while Qwen3.5 9B needs about 11 GB and Nemotron 3 Nano about 19 GB.
Weights plus f16 KV cache at a 128K context: Llama 3.1 8B needs about 21 GB, while Qwen3.5 9B needs about 11 GB and Nemotron 3 Nano about 19 GB.

Which Long-Context Model Fits Your Machine?

Pick by memory first: weights plus KV cache at the context you need must fit with 2 to 3 GB to spare for the operating system and the runtime. The picks below are arithmetic from the table above, not quality rankings; we did not benchmark long-context accuracy.

  • 8 GB: Qwen3.5 4B (3.3 GB download). The KV cache is 1 GiB at 32K and 2 GiB at 64K; 128K needs 4 GiB, which is tight on 8 GB.
  • 16 GB: Qwen3.5 9B (6.6 GB) fits 128K with about 4 GiB of KV cache. Gemma 4 12B (7.7 GB) fits its full 256K window with about 4.3 GiB.
  • 24 to 32 GB: Gemma 4 26B-A4B (16 GB) needs about 5.2 GiB at 256K. Qwen3.8 27B (18 GB) needs about 8 GiB at 128K. gpt-oss 20B (14 GB) needs about 3 GiB at its 128K maximum.
  • 48 GB and more: Gemma 4 31B (19 GB) needs about 21 GiB at 256K. Mistral Small 3.2 24B (15 GB) needs 20 GiB at 128K with an f16 cache and about 10 GiB with q8_0.
  • 1M tokens: by its configuration, Nemotron 3 Nano 30B needs about 6 GiB of KV cache at 1M on top of its 18 GB download, so memory alone does not rule it out on a 32 GB machine. NVIDIA ships 256K as the default and lists H100 and A100 GPUs as supported hardware, and Unsloth warns that 1M may trigger out-of-memory crashes, so treat it as an experiment. Qwen states that 1M tokens on Qwen3-30B-A3B-Instruct-2507 needs about 240 GB of GPU memory.

Why Is Usable Context Shorter Than the Advertised Maximum?

The advertised window is an architectural upper bound, not a quality guarantee. Liu et al. (2023) found that language models answer most accurately when the relevant text is at the start or the end of the input and less accurately when it is in the middle, the "lost in the middle" effect.

RULER (Hsieh et al., 2024) tested 17 models that all claim 32K tokens or more; only about half kept satisfactory performance at 32K. NoLiMa (Modarressi et al., 2025) tested 13 models that claim at least 128K; at 32K, 11 of them fell below 50% of their short-context score, and GPT-4o dropped from 99.3% to 69.7%.

Those studies predate most models in this guide, so read them as evidence for the pattern, not as scores for any model listed here. To check a model yourself, hide one fact at 10%, 50% and 90% of a long prompt, ask about it in different words and repeat at 32K, 64K and 128K.

When accuracy matters, put critical instructions and facts at the start or the end of the prompt, retrieve only the relevant passages (see the RAG section below) or process long documents in overlapping sections.

Schematic of the "lost in the middle" effect: accuracy is highest for information at the start and the end of the prompt and lower in the middle (shape reported by Liu et al., 2023; illustrative, not measured values).
Schematic of the "lost in the middle" effect: accuracy is highest for information at the start and the end of the prompt and lower in the middle (shape reported by Liu et al., 2023; illustrative, not measured values).

How Do You Set Context Length in Ollama, LM Studio and llama.cpp?

Ollama picks its default context from your VRAM: 4K tokens below 24 GiB, 32K from 24 to 48 GiB and 256K at 48 GiB or more. Input beyond the active context is truncated, so the model never sees the start of a long document.

You can set the context with the Ollama app slider, the OLLAMA_CONTEXT_LENGTH environment variable, /set parameter num_ctx inside ollama run, a Modelfile or the num_ctx option of the API. Confirm the result with ollama ps.

In LM Studio, open My Models, click the gear icon of a model and set the context size and Flash Attention. In llama.cpp, -c sets the context size (0 loads it from the model), -fa controls Flash Attention and --cache-type-k and --cache-type-v quantize the KV cache.

Ollama quantizes the KV cache only when Flash Attention is on, and the setting is global for all models. OLLAMA_KV_CACHE_TYPE=q8_0 uses about half the memory of f16 and q4_0 about a quarter.

bash
# 1. Check what a loaded model really uses (see the CONTEXT column)
ollama ps

# 2. Server-wide default (the Ollama app has a context slider instead)
OLLAMA_CONTEXT_LENGTH=65536 ollama serve

# 3. One interactive session
ollama run qwen3.5:9b
>>> /set parameter num_ctx 65536

# 4. A persistent custom model
cat << EOF > Modelfile
FROM qwen3.5:9b
PARAMETER num_ctx 131072
EOF
ollama create qwen3.5-9b-128k -f Modelfile
ollama run qwen3.5-9b-128k

# 5. Optional: shrink the KV cache (global setting, needs Flash Attention)
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve

# llama.cpp equivalent
llama-server -m model.gguf -c 131072 -fa on --cache-type-k q8_0 --cache-type-v q8_0
Ways to set the context length in Ollama, from server default to per-request, and how to verify the active value with ollama ps.
Ways to set the context length in Ollama, from server default to per-request, and how to verify the active value with ollama ps.

Do Non-English Documents Use More of the Context Window?

They can: Petrov et al. (2023) found that the same text can need up to 15 times more tokens in one language than in another, depending on the tokenizer. Whether Japanese, Chinese, Korean or Arabic text is affected depends on the model's tokenizer.

Do not convert pages to tokens with the English rule of 0.75 words per token. Count tokens with the model you plan to use: ollama run <model> --verbose prints the prompt token count after a reply.

Budget the window per language and per model family, because tokenizers differ between model families.

Long Context or RAG for Documents?

Use RAG when the answer lives in a few passages of a large collection, and use long context when the model must reason across one coherent document.

RAG (retrieval-augmented generation) retrieves the most relevant chunks, often a few thousand tokens in total, and keeps the prompt short. Long context keeps cross-references intact, such as clauses in a contract or call chains in a codebase, that chunking can split.

Many setups combine both: retrieve whole sections instead of fragments and place them in a 32K to 128K window. See Best Local RAG Tools for local options and RAG Explained for the method.

Common Mistakes with Long Context Local LLMs

  • Trusting the default context: Ollama defaults to 4K tokens below 24 GiB of VRAM, whatever the model supports, so a 100-page document is truncated before the model reads it. Set the context explicitly and confirm it in the CONTEXT column of ollama ps.
  • Budgeting memory from the download size: the KV cache can exceed the weights. Llama 3.1 8B is a 4.9 GB download but needs about 16 GiB of KV cache at 128K. Use the formula or the table above before you raise the context.
  • Assuming every tag has the long window: in the Ollama library qwen3:14b lists 40K while qwen3:30b lists 256K. Read the context figure on the model page before you plan around it.
  • Treating the advertised window as the usable window: RULER and NoLiMa found sharp accuracy drops at 32K for many models that claim 128K or more. Test with your own documents at the length you plan to use.
  • Reaching for 1M tokens by default: prefill time and memory grow with the prompt, and Unsloth warns that 1M may trigger out-of-memory crashes for Nemotron 3 Nano. Use 1M for occasional jobs and prefer RAG or chunking for everyday work.
  • Setting q4_0 for the KV cache without testing: Ollama documents a small-to-medium precision loss for q4_0 that can be more noticeable at higher contexts, and the setting applies to every model. Start with q8_0.

Frequently Asked Questions

What is the longest context window a local LLM can use in 2026?

Llama 4 Scout lists 10M tokens, but its model card reports benchmarks at 128K and the Ollama download is 67 GB. Among models that run on one consumer machine, NVIDIA Nemotron 3 Nano 30B-A3B supports 1M tokens (256K is the default configuration), and Qwen3.8 and Qwen3.5 support 256K natively with up to roughly 1M through YaRN. Whether a long window is usable depends on memory and on how accurately the model reads long inputs.

Is a 128K context window still enough?

128K tokens is about 100,000 words, enough for most single documents. A 300-page book is typically 75,000 to 100,000 words, roughly 100K to 135K tokens, so it usually exceeds or barely fits a 128K window with no room for questions and answers. A 256K window holds the book and the answer; beyond that, use RAG or chunking.

How much memory does a 1M-token context need?

It depends on the architecture. By the published configurations, the f16 KV cache at 1M tokens is about 32 GiB for Qwen3.5 9B, about 64 GiB for Qwen3.8 27B and about 6 GiB for Nemotron 3 Nano 30B-A3B, on top of the weights. Qwen states that 1M tokens on Qwen3-30B-A3B-Instruct-2507 needs about 240 GB of GPU memory. A q8_0 cache halves the KV figures, and the time to read a 1M-token prompt grows with its length, so reserve it for occasional jobs.

Why do models with the same context window need different amounts of memory?

The KV cache grows with full-attention layers × KV heads × head dimension × tokens. Llama 3.1 8B needs 128 KiB per token, Gemma 4 26B-A4B about 20 KiB because only 5 of its 30 layers use full attention, and Nemotron 3 Nano about 6 KiB because only 6 of its 52 layers are attention layers. Sliding-window, linear-attention and Mamba layers keep a fixed-size state instead of growing with the window.

What is the KV cache and how can I make it smaller?

The KV cache holds the attention keys and values of every token in the context so the model does not recompute them for each new token. It is allocated on top of the weights and grows with context length. To shrink it, lower num_ctx, enable Flash Attention and set OLLAMA_KV_CACHE_TYPE to q8_0 (about half the memory of f16) or q4_0 (about a quarter, with more precision loss at long contexts), or choose a model with a smaller cache per token. In Ollama the cache type is global and applies to every model.

Why does my local model forget the start of my document?

Most likely the active context is smaller than your document. Ollama defaults to 4K tokens on GPUs with under 24 GiB of VRAM, 32K from 24 to 48 GiB and 256K from 48 GiB, whatever the model supports, and input beyond the context is truncated. Check the CONTEXT column of ollama ps and raise it with OLLAMA_CONTEXT_LENGTH, /set parameter num_ctx, a Modelfile or the API num_ctx option.

How do I set a 128K context in Ollama?

Create a Modelfile with FROM qwen3.5:9b and PARAMETER num_ctx 131072, run ollama create qwen3.5-9b-128k -f Modelfile, then ollama run qwen3.5-9b-128k and verify with ollama ps. Alternatively start the server with OLLAMA_CONTEXT_LENGTH=131072 ollama serve. Make sure weights plus KV cache fit in memory first; for Qwen3.5 9B that is about 6.6 GB plus 4 GiB.

Does a longer context make a local LLM slower?

Yes. The model must read the whole prompt (prefill) before it writes the first token, so time to first token grows with prompt length, and attention over a larger cache also lowers generation speed. Linear-attention and Mamba layers reduce the growth but do not remove it. Measure on your hardware with ollama run <model> --verbose, which prints prompt-evaluation and generation rates.

What is the "lost in the middle" problem and does it still apply?

Liu et al. (2023) found that language models answer most accurately when the relevant text is at the start or the end of the input and less accurately when it is in the middle. RULER (2024) and NoLiMa (2025) later showed that many models claiming 32K to 128K windows lose accuracy well before their limit. Those studies tested earlier models, so run your own check: hide a fact at 10%, 50% and 90% of a long prompt and ask about it. Put critical instructions at the start or end, or retrieve only the relevant passages.

Is long context or RAG better for document question answering?

RAG is usually cheaper in memory and time when the answer sits in a few passages of a large collection, because it keeps the prompt to a few thousand tokens. Long context is better when the model must reason across one coherent document, such as a contract, a codebase or a book chapter, where chunking would split cross-references. Many setups combine both by retrieving whole sections into a 32K to 128K window.

Can I summarize an entire book with a local LLM?

A 300-page book is roughly 100K to 135K tokens, so it fits a 256K window (Qwen3.5, Qwen3.8, Gemma 4 12B and larger) and may not fit a 128K window once the summary is counted. With a 128K model or less memory, summarize chapter by chapter and then summarize the summaries. Set num_ctx explicitly so the whole book is in the active context.

Do non-English documents use more of the context window?

They can. Petrov et al. (2023) found that the same text can need up to 15 times more tokens in one language than in another, depending on the tokenizer. Count tokens with your model instead of using the English rule of 0.75 words per token: ollama run <model> --verbose prints the prompt token count.

Which long-context models run on a 16 GB machine?

By memory arithmetic, Qwen3.5 9B (6.6 GB download plus about 4 GiB of KV cache at 128K) and Gemma 4 12B (7.7 GB plus about 4.3 GiB at its full 256K window) fit in 16 GB with a few GB to spare. These are calculations from published configurations, not measured runs; confirm with ollama ps.

Sources

Need to size memory for a long context? Start with the hardware guide.

Local LLM Hardware Guide →

A Note on Third-Party Facts

This article references third-party AI models, benchmarks, prices, and licenses. The AI landscape changes rapidly. Benchmark scores, license terms, model names, and API prices can shift between the time of writing and the time you read this. Before making deployment or compliance decisions based on this article, verify current figures on each provider’s official source: Hugging Face model cards for licenses and benchmarks, provider websites for API pricing, and EUR-Lex for current GDPR and EU AI Act text.

Run PromptQuorum with a local LLM, your own API keys, or both — you pick the backend.

Download the PromptQuorum Beta →

← Back to Local LLMs

MCP