Context windows have grown to a million tokens. Four studies suggest the part a model can actually use is much smaller, and that what you leave out matters more than what you put in.

Suppose you had to brief a sharp new colleague on a problem. You could hand them one page with the three facts that matter. Or you could hand them four hundred pages of meeting minutes with those same three facts somewhere inside, and wish them luck.

Nobody would choose the second option for a person. We choose it for AI models every day, because the window is large and filling it feels thorough.

This story was written with the assistance of an AI writing program. Every figure in it comes from the papers listed at the end.

Over the past two years, researchers have measured what that habit costs. The results are consistent enough that the effect now has a name: context rot.

The experiment that named it

In July 2025, Kelly Hong, Anton Troynikov and Jeff Huber at Chroma published a technical report covering 18 models from Anthropic, OpenAI, Google and Alibaba, including Claude Opus 4, GPT-4.1, Gemini 2.5 Pro and Qwen3.

Their approach was simple. Keep the task the same, make the input longer, and watch what happens.

The cleanest test used LongMemEval, a benchmark of questions about a long chat history. Each model answered every question twice. Once with the full history, about 113,000 tokens. Once with only the relevant excerpt, about 300 tokens.

The answer was sitting in both versions. Every model did significantly better with the 300-token one.

Think about what that means. The long prompt contained everything the short one did. The extra 112,700 tokens added no missing fact. They only added things to wade through, and that alone was enough to lower the score.

Simple copying breaks too

You might object that finding one fact in a long conversation is hard. So the Chroma team also tried something close to trivial.

They gave models a list of one word repeated many times, with a single different word placed somewhere in it, and asked for the exact same text back. No reasoning, no retrieval. Just copy.

Across 1,090 variations, from 25 words up to 10,000, accuracy fell as the list got longer for every model. Some of the failures were strange. Gemini models began producing random words at around 500 to 750 words. Others kept writing the repeated word until they ran out of output. Claude Opus 4 declined about 2.9% of the attempts and GPT-4.1 about 2.6%.

A task a child could do with a pencil became unreliable through length alone.

Near misses do the most damage

The third finding is the one I would want every developer to know.

The researchers planted "distractors" in the text: passages on the same topic as the answer that did not quite answer the question. One distractor lowered accuracy. Four lowered it further.

How the models failed differed by family. Claude models tended to say they could not determine the answer. GPT models had the highest rate of confident wrong answers.

Now picture a long working session with a coding agent. It holds the first version of your plan and the third. A file as it was before the refactor and after. An error message from a bug you fixed an hour ago. Each of those is a near miss by the report's definition: on topic, plausible, and wrong.

There was one more odd result. When the surrounding text was shuffled into nonsense order, all 18 models found the answer more easily than when it read as a coherent document. The report gives no settled explanation. But it does suggest that well-organised, closely related material is not harmless padding. It competes for the model's attention.

The gap between advertised and usable

A second team reached the same place by another route. NoLiMa, a benchmark from researchers at Adobe and LMU Munich presented at ICML 2025, removed a crutch that older tests allowed. In classic needle-in-a-haystack tests the question shares words with the answer, so the model can match strings. NoLiMa's questions share almost none. The model has to make the connection itself.

They tested 13 models that all claim to support at least 128,000 tokens. Under 1,000 tokens they did well. At 32,000 tokens, a quarter of the advertised window, 11 of the 13 had dropped below half of their short-context score. GPT-4o, one of the best performers, went from 99.3% to 69.7%.

Then in February 2026, Ravi Raju and colleagues looked at this in real bug fixing on SWE-bench Verified. They noticed that successful agent runs usually stayed under 20,000 to 30,000 tokens, and that runs which accumulated more context succeeded less often.

To test long context directly, they did what many of us do by hand. They put all the relevant files into one 64,000-token prompt and asked for a patch. Qwen3-Coder resolved 7% of the tasks. GPT-5-nano resolved none. The failures included invented diffs and patches aimed at the wrong file.

Their conclusion is worth repeating: agents do well because they break work into short steps, not because they reason well over long inputs.

What actually works

A paper from March 2026 gives the hopeful half of the story. Weili Cao and colleagues took tasks with huge inputs, from 188,000 tokens to three trillion, and did not load the text into the prompt at all. They left it in files and let a coding agent search it with ordinary tools. The agents beat the published state of the art by 17.3% on average.

So the model reads long material badly and searches it well. That points to a few habits.

Start a new session for a new task. A long conversation is a pile of near misses. Clearing it costs you a minute of re-explaining and removes hours of stale detail.

Point, don't paste. Give the agent a file path and let it open what it needs. Anthropic's engineering team calls this just-in-time retrieval, and it keeps the window small by default.

Send exploration elsewhere. A subagent can read fifty files and report back in a page. Anthropic describes those summaries as often 1,000 to 2,000 tokens. Your main session gets the conclusion without the fifty files.

Keep notes outside the window. Have the agent write decisions and progress to a file. A fresh session can read ten lines of notes far more reliably than it can recall turn 40 of 200.

Delete what is wrong, not only what is irrelevant. Outdated docs, abandoned plans and old error logs are distractors. They are more dangerous than unrelated text because they look like answers.

Use the document's own words. NoLiMa shows models lean on literal overlap. If the spec says "tenant", ask about tenants, not customers.

Caveats

Chroma sells a retrieval database, so a finding that says "retrieve less, and choose carefully" suits its business. The report publishes its method and code, which helps.

Most of these tests are artificial. Hiding a sentence in essays is not the same as real work, and the models in the 2025 studies are no longer the newest. The NoLiMa authors note in a later version that newer models such as GPT-4.1 hold up better, though they still decline well short of their claimed length.

The bug-fixing study used small, inexpensive models and 100 samples. Larger models would score higher. The direction is what carries over, not the exact numbers.

The takeaway

The size of a context window tells you how much a model will accept. It tells you very little about how much it will use well.

In the cleanest test we have, 300 well-chosen tokens beat 113,000 that contained the same answer. The work that pays is the editing: deciding what the model does not need to see.

That is the same thing a good briefing does for a person. It turns out the one-page note was the right choice all along.

Sources: Kelly Hong, Anton Troynikov and Jeff Huber, "Context Rot: How Increasing Input Tokens Impacts LLM Performance," Chroma Technical Report (July 2025). Ali Modarressi et al., "NoLiMa: Long-Context Evaluation Beyond Literal Matching," ICML 2025, arXiv 2502.05167. Ravi Raju et al., "The Limits of Long-Context Reasoning in Automated Bug Fixing," arXiv 2602.16069 (February 2026). Weili Cao et al., "Coding Agents are Effective Long-Context Processors," arXiv 2603.20432 (March 2026). Anthropic, "Effective context engineering for AI agents" (September 2025).