What Is Context Length and Why Does It Matter for Local LLMs?
Context length is the maximum number of tokens a model can hold in one request: your prompt, any documents, the conversation so far and the answer it writes, added together. One token is about 0.75 English words, so 128K tokens is about 100,000 words and 256K tokens about 200,000 words.
The advertised window is the maximum the model architecture supports. The usable window is the length at which answers stay accurate for your task, and it is shorter.
Long windows matter for whole codebases, contracts, long transcripts, book chapters and agent sessions where tool output piles up. The Ollama documentation recommends at least 64,000 tokens for agents, web search and coding tools.
Three things set the limit on your own machine: the model's window, the memory its KV cache needs at that length and how reliably it uses text at that length. The next sections cover each in turn.
Which Local LLMs Support More Than 128K Tokens in 2026?
Several do: Qwen3.5, Qwen3.8 and Gemma 4 (12B and larger) list 256K windows in the Ollama library, NVIDIA Nemotron 3 Nano 30B lists 1M and Llama 4 Scout lists 10M. Models released before 2026 mostly stop at 128K.
The table shows the window listed by the model card or the Ollama library. A longer-context extension, such as YaRN, is a vendor-documented option that needs extra configuration and memory.
Llama 4 Scout lists 10M tokens, but its model card reports its long-context benchmarks at a 128K window, and the Ollama download is 67 GB. Llama 4 Maverick lists 1M but is a 245 GB download, so it needs server hardware.
Ollama tags can differ in window size: qwen3:8b and qwen3:14b list 40K, while qwen3:4b and qwen3:30b list 256K. Read the context figure on the model's Ollama page before you plan around it.
Model | Listed window | Longer context | Ollama tag |
|---|---|---|---|
| Nemotron 3 Nano 30B-A3B | 1M (default 256K) | — | ollama run nemotron-3-nano:30b |
| Qwen3.8 27B | 256K | Up to 1M (YaRN) | ollama run qwen3.8:27b |
| Qwen3.5 9B / 27B | 256K | Up to 1.01M (YaRN) | ollama run qwen3.5:9b |
| Qwen3 30B-A3B-2507 | 256K | 1M (DCA, ~240 GB GPU) | ollama run qwen3:30b |
| Gemma 4 12B / 26B / 31B | 256K | — | ollama run gemma4:26b |
| Gemma 4 E2B / E4B | 128K | — | ollama run gemma4:e4b |
| Llama 4 Scout 109B | 10M listed | Benchmarked at 128K | ollama run llama4:16x17b |
| Mistral Small 3.2 24B | 128K | — | ollama run mistral-small3.2 |
| gpt-oss 20B | 128K | — | ollama run gpt-oss:20b |
| Llama 3.1 8B | 128K | — | ollama run llama3.1:8b |
| Qwen3 8B / 14B | 40K | — | ollama run qwen3:14b |
How Much Memory Does Long Context Need?
Memory for long context is the model weights plus the KV cache, and the KV cache depends on the architecture, not only on the window. The KV cache stores the attention keys and values of every token in the context and is allocated on top of the weights.
KV bytes per token = 2 (keys and values) × full-attention layers × KV heads × head dimension × bytes per value (2 for the default f16 cache). Llama 3.1 8B: 2 × 32 × 8 × 128 × 2 = 131,072 bytes (128 KiB) per token, which is 16 GiB at 128K tokens, more than three times its 4.9 GB download.
Newer architectures cut this cost. Gemma 4 uses sliding-window attention in most layers (a 1,024-token window in the 12B, 26B and 31B models), gpt-oss in every second layer (128 tokens), Qwen3.5 and Qwen3.8 use Gated DeltaNet linear attention in three of every four layers, and Nemotron 3 Nano has only 6 attention layers out of 52. Only full-attention layers grow with the window.
The values below are calculated from each model's published configuration, not measured. Runtimes add compute buffers and cache implementations differ, so confirm real use with ollama ps.
Quantizing the KV cache shrinks it: according to the Ollama documentation, q8_0 uses about half the memory of f16 and q4_0 about a quarter, with a small-to-medium precision loss for q4_0.
Model | Download | KV at 32K | KV at 128K | KV at max window |
|---|---|---|---|---|
| Llama 3.1 8B | 4.9 GB | 4 GiB | 16 GiB | 16 GiB (128K) |
| Mistral Small 3.2 24B | 15 GB | 5 GiB | 20 GiB | 20 GiB (128K) |
| gpt-oss 20B | 14 GB | 0.8 GiB | 3 GiB | 3 GiB (128K) |
| Qwen3.5 4B | 3.3 GB | 1 GiB | 4 GiB | 8 GiB (256K) |
| Qwen3.5 9B | 6.6 GB | 1 GiB | 4 GiB | 8 GiB (256K) |
| Gemma 4 12B | 7.7 GB | 0.8 GiB | 2.3 GiB | 4.3 GiB (256K) |
| Gemma 4 26B-A4B | 16 GB | 0.8 GiB | 2.7 GiB | 5.2 GiB (256K) |
| Gemma 4 31B | 19 GB | 3.3 GiB | 10.8 GiB | 20.8 GiB (256K) |
| Qwen3.8 27B | 18 GB | 2 GiB | 8 GiB | 16 GiB (256K) |
| Qwen3.5 35B-A3B | 22 GB | 0.6 GiB | 2.5 GiB | 5 GiB (256K) |
| Qwen3 30B-A3B-2507 | 19 GB | 3 GiB | 12 GiB | 24 GiB (256K) |
| Nemotron 3 Nano 30B-A3B | 18 GB | 0.2 GiB | 0.8 GiB | 6 GiB (1M) |
Which Long-Context Model Fits Your Machine?
Pick by memory first: weights plus KV cache at the context you need must fit with 2 to 3 GB to spare for the operating system and the runtime. The picks below are arithmetic from the table above, not quality rankings; we did not benchmark long-context accuracy.
- 8 GB: Qwen3.5 4B (3.3 GB download). The KV cache is 1 GiB at 32K and 2 GiB at 64K; 128K needs 4 GiB, which is tight on 8 GB.
- 16 GB: Qwen3.5 9B (6.6 GB) fits 128K with about 4 GiB of KV cache. Gemma 4 12B (7.7 GB) fits its full 256K window with about 4.3 GiB.
- 24 to 32 GB: Gemma 4 26B-A4B (16 GB) needs about 5.2 GiB at 256K. Qwen3.8 27B (18 GB) needs about 8 GiB at 128K. gpt-oss 20B (14 GB) needs about 3 GiB at its 128K maximum.
- 48 GB and more: Gemma 4 31B (19 GB) needs about 21 GiB at 256K. Mistral Small 3.2 24B (15 GB) needs 20 GiB at 128K with an f16 cache and about 10 GiB with q8_0.
- 1M tokens: by its configuration, Nemotron 3 Nano 30B needs about 6 GiB of KV cache at 1M on top of its 18 GB download, so memory alone does not rule it out on a 32 GB machine. NVIDIA ships 256K as the default and lists H100 and A100 GPUs as supported hardware, and Unsloth warns that 1M may trigger out-of-memory crashes, so treat it as an experiment. Qwen states that 1M tokens on Qwen3-30B-A3B-Instruct-2507 needs about 240 GB of GPU memory.
Why Is Usable Context Shorter Than the Advertised Maximum?
The advertised window is an architectural upper bound, not a quality guarantee. Liu et al. (2023) found that language models answer most accurately when the relevant text is at the start or the end of the input and less accurately when it is in the middle, the "lost in the middle" effect.
RULER (Hsieh et al., 2024) tested 17 models that all claim 32K tokens or more; only about half kept satisfactory performance at 32K. NoLiMa (Modarressi et al., 2025) tested 13 models that claim at least 128K; at 32K, 11 of them fell below 50% of their short-context score, and GPT-4o dropped from 99.3% to 69.7%.
Those studies predate most models in this guide, so read them as evidence for the pattern, not as scores for any model listed here. To check a model yourself, hide one fact at 10%, 50% and 90% of a long prompt, ask about it in different words and repeat at 32K, 64K and 128K.
When accuracy matters, put critical instructions and facts at the start or the end of the prompt, retrieve only the relevant passages (see the RAG section below) or process long documents in overlapping sections.
How Do You Set Context Length in Ollama, LM Studio and llama.cpp?
Ollama picks its default context from your VRAM: 4K tokens below 24 GiB, 32K from 24 to 48 GiB and 256K at 48 GiB or more. Input beyond the active context is truncated, so the model never sees the start of a long document.
You can set the context with the Ollama app slider, the OLLAMA_CONTEXT_LENGTH environment variable, /set parameter num_ctx inside ollama run, a Modelfile or the num_ctx option of the API. Confirm the result with ollama ps.
In LM Studio, open My Models, click the gear icon of a model and set the context size and Flash Attention. In llama.cpp, -c sets the context size (0 loads it from the model), -fa controls Flash Attention and --cache-type-k and --cache-type-v quantize the KV cache.
Ollama quantizes the KV cache only when Flash Attention is on, and the setting is global for all models. OLLAMA_KV_CACHE_TYPE=q8_0 uses about half the memory of f16 and q4_0 about a quarter.
# 1. Check what a loaded model really uses (see the CONTEXT column)
ollama ps
# 2. Server-wide default (the Ollama app has a context slider instead)
OLLAMA_CONTEXT_LENGTH=65536 ollama serve
# 3. One interactive session
ollama run qwen3.5:9b
>>> /set parameter num_ctx 65536
# 4. A persistent custom model
cat << EOF > Modelfile
FROM qwen3.5:9b
PARAMETER num_ctx 131072
EOF
ollama create qwen3.5-9b-128k -f Modelfile
ollama run qwen3.5-9b-128k
# 5. Optional: shrink the KV cache (global setting, needs Flash Attention)
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve
# llama.cpp equivalent
llama-server -m model.gguf -c 131072 -fa on --cache-type-k q8_0 --cache-type-v q8_0Do Non-English Documents Use More of the Context Window?
They can: Petrov et al. (2023) found that the same text can need up to 15 times more tokens in one language than in another, depending on the tokenizer. Whether Japanese, Chinese, Korean or Arabic text is affected depends on the model's tokenizer.
Do not convert pages to tokens with the English rule of 0.75 words per token. Count tokens with the model you plan to use: ollama run <model> --verbose prints the prompt token count after a reply.
Budget the window per language and per model family, because tokenizers differ between model families.
Long Context or RAG for Documents?
Use RAG when the answer lives in a few passages of a large collection, and use long context when the model must reason across one coherent document.
RAG (retrieval-augmented generation) retrieves the most relevant chunks, often a few thousand tokens in total, and keeps the prompt short. Long context keeps cross-references intact, such as clauses in a contract or call chains in a codebase, that chunking can split.
Many setups combine both: retrieve whole sections instead of fragments and place them in a 32K to 128K window. See Best Local RAG Tools for local options and RAG Explained for the method.
Common Mistakes with Long Context Local LLMs
- Trusting the default context: Ollama defaults to 4K tokens below 24 GiB of VRAM, whatever the model supports, so a 100-page document is truncated before the model reads it. Set the context explicitly and confirm it in the CONTEXT column of ollama ps.
- Budgeting memory from the download size: the KV cache can exceed the weights. Llama 3.1 8B is a 4.9 GB download but needs about 16 GiB of KV cache at 128K. Use the formula or the table above before you raise the context.
- Assuming every tag has the long window: in the Ollama library qwen3:14b lists 40K while qwen3:30b lists 256K. Read the context figure on the model page before you plan around it.
- Treating the advertised window as the usable window: RULER and NoLiMa found sharp accuracy drops at 32K for many models that claim 128K or more. Test with your own documents at the length you plan to use.
- Reaching for 1M tokens by default: prefill time and memory grow with the prompt, and Unsloth warns that 1M may trigger out-of-memory crashes for Nemotron 3 Nano. Use 1M for occasional jobs and prefer RAG or chunking for everyday work.
- Setting q4_0 for the KV cache without testing: Ollama documents a small-to-medium precision loss for q4_0 that can be more noticeable at higher contexts, and the setting applies to every model. Start with q8_0.
Frequently Asked Questions
What is the longest context window a local LLM can use in 2026?
Llama 4 Scout lists 10M tokens, but its model card reports benchmarks at 128K and the Ollama download is 67 GB. Among models that run on one consumer machine, NVIDIA Nemotron 3 Nano 30B-A3B supports 1M tokens (256K is the default configuration), and Qwen3.8 and Qwen3.5 support 256K natively with up to roughly 1M through YaRN. Whether a long window is usable depends on memory and on how accurately the model reads long inputs.
Is a 128K context window still enough?
128K tokens is about 100,000 words, enough for most single documents. A 300-page book is typically 75,000 to 100,000 words, roughly 100K to 135K tokens, so it usually exceeds or barely fits a 128K window with no room for questions and answers. A 256K window holds the book and the answer; beyond that, use RAG or chunking.
How much memory does a 1M-token context need?
It depends on the architecture. By the published configurations, the f16 KV cache at 1M tokens is about 32 GiB for Qwen3.5 9B, about 64 GiB for Qwen3.8 27B and about 6 GiB for Nemotron 3 Nano 30B-A3B, on top of the weights. Qwen states that 1M tokens on Qwen3-30B-A3B-Instruct-2507 needs about 240 GB of GPU memory. A q8_0 cache halves the KV figures, and the time to read a 1M-token prompt grows with its length, so reserve it for occasional jobs.
Why do models with the same context window need different amounts of memory?
The KV cache grows with full-attention layers × KV heads × head dimension × tokens. Llama 3.1 8B needs 128 KiB per token, Gemma 4 26B-A4B about 20 KiB because only 5 of its 30 layers use full attention, and Nemotron 3 Nano about 6 KiB because only 6 of its 52 layers are attention layers. Sliding-window, linear-attention and Mamba layers keep a fixed-size state instead of growing with the window.
What is the KV cache and how can I make it smaller?
The KV cache holds the attention keys and values of every token in the context so the model does not recompute them for each new token. It is allocated on top of the weights and grows with context length. To shrink it, lower num_ctx, enable Flash Attention and set OLLAMA_KV_CACHE_TYPE to q8_0 (about half the memory of f16) or q4_0 (about a quarter, with more precision loss at long contexts), or choose a model with a smaller cache per token. In Ollama the cache type is global and applies to every model.
Why does my local model forget the start of my document?
Most likely the active context is smaller than your document. Ollama defaults to 4K tokens on GPUs with under 24 GiB of VRAM, 32K from 24 to 48 GiB and 256K from 48 GiB, whatever the model supports, and input beyond the context is truncated. Check the CONTEXT column of ollama ps and raise it with OLLAMA_CONTEXT_LENGTH, /set parameter num_ctx, a Modelfile or the API num_ctx option.
How do I set a 128K context in Ollama?
Create a Modelfile with FROM qwen3.5:9b and PARAMETER num_ctx 131072, run ollama create qwen3.5-9b-128k -f Modelfile, then ollama run qwen3.5-9b-128k and verify with ollama ps. Alternatively start the server with OLLAMA_CONTEXT_LENGTH=131072 ollama serve. Make sure weights plus KV cache fit in memory first; for Qwen3.5 9B that is about 6.6 GB plus 4 GiB.
Does a longer context make a local LLM slower?
Yes. The model must read the whole prompt (prefill) before it writes the first token, so time to first token grows with prompt length, and attention over a larger cache also lowers generation speed. Linear-attention and Mamba layers reduce the growth but do not remove it. Measure on your hardware with ollama run <model> --verbose, which prints prompt-evaluation and generation rates.
What is the "lost in the middle" problem and does it still apply?
Liu et al. (2023) found that language models answer most accurately when the relevant text is at the start or the end of the input and less accurately when it is in the middle. RULER (2024) and NoLiMa (2025) later showed that many models claiming 32K to 128K windows lose accuracy well before their limit. Those studies tested earlier models, so run your own check: hide a fact at 10%, 50% and 90% of a long prompt and ask about it. Put critical instructions at the start or end, or retrieve only the relevant passages.
Is long context or RAG better for document question answering?
RAG is usually cheaper in memory and time when the answer sits in a few passages of a large collection, because it keeps the prompt to a few thousand tokens. Long context is better when the model must reason across one coherent document, such as a contract, a codebase or a book chapter, where chunking would split cross-references. Many setups combine both by retrieving whole sections into a 32K to 128K window.
Can I summarize an entire book with a local LLM?
A 300-page book is roughly 100K to 135K tokens, so it fits a 256K window (Qwen3.5, Qwen3.8, Gemma 4 12B and larger) and may not fit a 128K window once the summary is counted. With a 128K model or less memory, summarize chapter by chapter and then summarize the summaries. Set num_ctx explicitly so the whole book is in the active context.
Do non-English documents use more of the context window?
They can. Petrov et al. (2023) found that the same text can need up to 15 times more tokens in one language than in another, depending on the tokenizer. Count tokens with your model instead of using the English rule of 0.75 words per token: ollama run <model> --verbose prints the prompt token count.
Which long-context models run on a 16 GB machine?
By memory arithmetic, Qwen3.5 9B (6.6 GB download plus about 4 GiB of KV cache at 128K) and Gemma 4 12B (7.7 GB plus about 4.3 GiB at its full 256K window) fit in 16 GB with a few GB to spare. These are calculations from published configurations, not measured runs; confirm with ollama ps.
Sources
- Ollama. (2026). "Context length" documentation. docs.ollama.com -- default context by VRAM tier, OLLAMA_CONTEXT_LENGTH, 64,000-token recommendation for agents.
- Ollama. (2026). "FAQ". docs.ollama.com -- /set parameter num_ctx, OLLAMA_FLASH_ATTENTION, OLLAMA_KV_CACHE_TYPE (f16, q8_0, q4_0).
- Ollama. (2026). "qwen3.8" library page. ollama.com -- 27B, 18 GB, 256K context.
- Ollama. (2026). "qwen3.5" library page. ollama.com -- 0.8B to 122B tags, 256K context.
- Ollama. (2026). "qwen3" library page. ollama.com -- 40K for 8B/14B/32B, 256K for 4B/30B.
- Ollama. (2026). "gemma4" library page. ollama.com -- E2B/E4B 128K, 12B/26B/31B 256K.
- Ollama. (2026). "nemotron-3-nano" library page. ollama.com -- 30B tag lists 1M context, 18 GB.
- Ollama. (2026). "llama4" library page. ollama.com -- Scout 67 GB and 10M, Maverick 245 GB and 1M.
- Ollama. (2026). "mistral-small3.2", "gpt-oss" and "llama3.1" library pages. ollama.com -- 128K windows, 15 GB, 14 GB and 4.9 GB downloads.
- Qwen Team. (2026). "Qwen3.8-27B" model card. huggingface.co -- 262,144 native, up to 1,000,000 with YaRN, Apache-2.0.
- Qwen Team. (2026). "Qwen3.5-9B" model card. huggingface.co -- 262,144 native, up to 1,010,000 with YaRN; Gated DeltaNet hybrid layout.
- Qwen Team. (2025). "Qwen3-30B-A3B-Instruct-2507" model card. huggingface.co -- 262,144 native, 1M with Dual Chunk Attention, about 240 GB GPU memory.
- Google. (2026). "Gemma 4 31B" model card. huggingface.co -- 128K for E2B/E4B, 256K for the larger sizes.
- NVIDIA. (2025). "NVIDIA-Nemotron-3-Nano-30B-A3B-BF16" model card. huggingface.co -- 23 Mamba-2 and MoE layers plus 6 attention layers, 1M supported, 256K default.
- Meta. (2025). "Llama-4-Scout-17B-16E-Instruct" model card. huggingface.co -- 109B total, 10M listed, long-context benchmarks at 128K.
- Unsloth. (2026). "NVIDIA Nemotron 3 Nano -- How To Run Guide". unsloth.ai -- 1M context can run out of memory, default 262,144.
- Liu et al. (2023). "Lost in the Middle: How Language Models Use Long Contexts". arxiv.org -- accuracy is highest at the start and end of the input.
- Hsieh et al. (2024). "RULER: What's the Real Context Size of Your Long-Context Language Models?". arxiv.org -- only about half of 17 models claiming 32K+ stay satisfactory at 32K.
- Modarressi et al. (2025). "NoLiMa: Long-Context Evaluation Beyond Literal Matching". arxiv.org -- 11 of 13 models fall below 50% of their short-context score at 32K.
- Petrov et al. (2023). "Language Model Tokenizers Introduce Unfairness Between Languages". arxiv.org -- up to 15 times more tokens for the same text.
- llama.cpp. "llama-server" README. github.com -- --ctx-size, --flash-attn, --cache-type-k, --cache-type-v.
- LM Studio. "Per-model default parameters". lmstudio.ai -- context size and Flash Attention per model.
- KV cache sizes in this guide are calculated from the config.json of each model repository on Hugging Face (layers, KV heads, head dimension, attention layer types) and are not measured.
