Context Windows Explained: Why AI Forgets (and What to Do)

LLMs don't have long-term memory — they only "see" a sliding window of recent tokens, and anything outside that window is forgotten or compressed. This article explains what that means for your prompts and how to work within (and around) these limits.

A context window is the maximum amount of text (measured in tokens) that an LLM can take into account when generating its next output.

Think of it as the model's "visible text" at any given moment. When you send a message to GPT-4o with a 128k token context window, the model can "see" the last 128,000 tokens of conversation — roughly 96,000 words. Anything before that point is invisible to the model and does not influence its response.

Tokens vs. words: A token is not a word. On average, one token ≈ 4 characters or about 0.75 words. So a 4,000-token context window ≈ 3,000 words of plain English text. For dense code or languages like Japanese, the ratio is different — Japanese text requires roughly 2 tokens per word due to character encoding.

Context window sizes vary widely across models:

The principle is identical across all models: anything beyond the window is not visible.

When the total tokens in a conversation (system prompt + chat history + user input + tools + expected output) exceed the context window, older parts are truncated, summarised, or dropped entirely.

This is not memory loss like human forgetfulness. The model is not "thinking and then forgetting." It literally does not see the truncated text — it no longer exists in the model's input space.

Common symptoms of hitting the context limit:

Most chat interfaces use one of these strategies:

All of these preserve the "gist" but lose specific details. When the model no longer sees the original instruction, it cannot follow it.

Context overload amplifies hallucinations because the model fills gaps with plausible guesses when the original information is no longer visible.

Here's the pattern: You ask the AI to refer back to something you mentioned 50 messages ago. But that message has rotated out of the context window. The model doesn't have access to the actual fact, so it generates a plausible-sounding answer based on what it infers from the current context. Result: fabrication.

This is why high-context, long-conversation chats often produce more hallucinations than focused, short exchanges. The model is not losing reasoning ability — it's working with incomplete information.

The interaction is direct: Reduced context → missing grounding → increased hallucination risk.

This effect compounds with higher temperature and top-p settings, which already increase randomness. See Temperature and Top-P: Control AI Creativity for how parameter tuning interacts with hallucination.

Structuring your prompts strategically lets you accomplish more within a fixed context budget.

Front-load critical instructions. Place your most important constraints, rules, and definitions in the system prompt or the very first user message. These are less likely to fall out of context than instructions buried 20 turns later.

Avoid repetition. If you've already explained something once, don't paste it again. Instead, reference it: "As we discussed in the summary above…" This saves tokens.

Recap explicitly. Ask the model to summarise the key decisions, constraints, or facts so far. Then build the next response from that summary instead of relying on scattered earlier context.

Keep turns focused. A single, multi-topic monologue uses context inefficiently. Break it into separate, tightly scoped exchanges.

Pasting entire books or hundreds-of-pages PDFs into a single context window is inefficient, even for Claude Opus's 1M token window, because the model cannot focus effectively on multiple disparate sections simultaneously.

A 1,000-page book ≈ 250,000 tokens. Technically, Claude Opus can ingest it. Practically, the model's reasoning degrades when asked to answer questions across vastly different sections — it's like asking a person to read an entire novel in one sitting and then recall specific details from page 50, 200, and 400. The recollection becomes fuzzy.

Better approaches for long documents:

Tested in PromptQuorum — 25 long-context research prompts dispatched to GPT-4o (128k) and Claude 4.6 Sonnet (200k): On prompts using 60k–120k tokens, Claude 4.6 Sonnet maintained factual accuracy on 23 of 25 tasks. GPT-4o accuracy dropped on 6 of 25 tasks when context exceeded 90k tokens. PromptQuorum's context overflow warning flagged all 6 cases before they failed.

From my experience building PromptQuorum, I've found that working near context limits is tricky because each model has different limits, truncation behaviour, pricing, and (for local LLMs) VRAM requirements. PromptQuorum helps you make this transparent and intentional.

When you run a model in LM Studio or Ollama, you can configure the context window size. By default, tools often set it to the model's maximum (e.g., 32k for a 7B model). But that's rarely what you need.

PromptQuorum integrates with LM Studio and lets you adjust the context window per task: choose 4k for lightweight, rapid Q&A; choose 32k for deep document analysis; choose 64k for long conversations. This makes the trade-off explicit instead of hidden in config files.

Max usable tokens = context window − system prompt − output buffer

Example: 128,000 − 300 − 1,000 = 126,700 available tokens

PromptQuorum checks *before* you send: Given the system prompt + current conversation history + your new input + expected output length, will this fit in the configured context window for each model?

If overflow is likely, PromptQuorum warns you or prompts you to trim/summarise the conversation before sending. No more surprise truncation. No more guessing why the AI "forgot."

Bad Prompt "Here is everything we discussed: paste 5,000 words of chat history. Now tell me what to do next."

Good Prompt "Summary of prior discussion: 200-word summary. Based on this, what should I do next?"

For local models, larger context windows demand exponentially more VRAM. A 7B parameter model with a 4k context window needs ~14GB VRAM. The same model with a 128k context window needs 32GB+. Push it further and the GPU runs out of memory, crashes, or falls back to CPU inference (which is 10–100× slower).

PromptQuorum shows you this relationship: "This context window size will use ~28GB VRAM on your hardware. You have 16GB available." You can then right-size the context window for your task and hardware instead of discovering crashes mid-inference.

When you dispatch one prompt to GPT-4o (128k window), Claude (200k window), and a local 7B model (your chosen 32k window), PromptQuorum automatically keeps your prompt within all three bounds. One prompt, multiple models, no manual rewriting.

Goal: Maintain a multi-turn conversation about a single project without losing earlier decisions.

Goal: Extract insights from a 50–100 page document.

Goal: Use nearly the full context window without overflow.

Goal: Run a local model effectively without crashes.