Your agent demo went beautifully for ten minutes. An hour into the same session it was re-reading files it had already read, re-proposing a fix it had already rejected, and ignoring a constraint you gave it in turn three. Nothing crashed. No rate limit, no stack trace, no exception in the logs, just a slow sag in quality you can feel but cannot point at. That failure has a name: context rot.
Every provider quotes a window in six or seven figures now: 128K, 200K, a million tokens. It is easy to read that as a promise that the model will use all of it equally well. Benchmarks say otherwise. A bigger window changes what fits, not how reliably what fits gets used, and that gap is where long-running agents go to die.
What is context rot?
Context rot is the measured tendency of a large language model to answer less reliably as its input grows, even when the relevant information is present and the input still fits the context window. Quality degrades gradually rather than failing outright, so long agent sessions decay quietly instead of erroring.
The term was popularized by a July 2025 technical report from Chroma titled "Context Rot", which named what engineers had spent a year calling "the agent got dumb halfway through". Before the name, each instance looked like a one-off prompt problem; after it, one class of failure with shared fixes.
It hides because there is no failure signal. A rate limit throws. A malformed tool call throws. Context rot returns a fluent, confident, subtly worse answer with a 200 status code, and your monitoring stays green while users stop trusting the system.
What the benchmarks actually show
The first widely cited result predates the term. In 2023, Liu and colleagues published "Lost in the Middle", which moved a relevant passage through different positions in a long input. Accuracy traced a U-shape: strong near the beginning, strong again near the end, weakest in the middle. Only the position changed.
The Chroma report evaluated 18 leading models and found performance degrading as input grew, even on tasks as simple as repeating back a list of words. Two secondary findings matter more to builders than that headline. Degradation accelerated when the question and the relevant text shared less literal wording, forcing a match on meaning rather than overlapping strings. And plausible-but-wrong distractors, content that looks like an answer but is not, hurt more as context grew.
The NoLiMa benchmark, published in 2025, pushed on exactly that case. It plants needles, facts the model must locate, worded to share almost no vocabulary with the query, so keyword overlap cannot rescue retrieval. Most models tested fell below 50% of their own short-context performance by around 32K tokens. That is measured against their own short-input scores, not some absolute bar.
Two caveats: benchmarks are stylized, so your workload's curve is not theirs and it shifts with every model version. And "degrades" describes a slope, not a wall at token N, which is why it is so easy to miss.
| Study or benchmark | What it measured | What it found |
|---|---|---|
| Lost in the Middle (Liu et al., 2023) | Recall of a relevant passage as its position moves through a long input | A U-shape: strong at the start and the end, weakest when the passage sits in the middle |
| Chroma "Context Rot" report (July 2025) | 18 leading models from the GPT, Claude, Gemini and Qwen families, including word-repetition tasks | Performance drops as input grows; worse on non-literal matches and with plausible wrong distractors |
| NoLiMa benchmark (2025) | Finding a planted fact when the query shares almost no literal wording with it | Most models tested fell below 50% of their own short-context performance by around 32K tokens |
Why more context makes models worse
There is no closed-form explanation of context rot, only leading explanations that fit the observed behaviour. Three stand out, and they point at the same conclusion.
Attention is a budget, not a guarantee
Transformer attention computes, for every token, a weighted view over every other token it can see, and those weights are normalized to sum to one. Add tokens and you are not adding attention, you are dividing a fixed allocation across more candidates: more places to look, more ways to be pulled toward the wrong one. That is a simplification rather than a proof, but it matches what the benchmarks measure.
Positions deep in the context are undertrained territory
Long-context ability is usually extended after the fact. A model is pretrained mostly on sequences far shorter than its advertised limit, then stretched to accept more by rescaling the values that tell it where each token sits and by continued training on longer samples. A model that accepts 200K tokens has not necessarily seen many examples of reasoning across 200K tokens. The capability is real but thinner at depth than the headline implies.
Agent loops manufacture their own distractors
This one bites production systems hardest, and it is structural rather than architectural. An agent loop appends. Every tool result, every abandoned plan, every file it read and deemed irrelevant stays in the transcript, competing for attention with what matters now. The reported finding that plausible-but-wrong distractors hurt more as context grows is concrete here: a plan abandoned in turn nine reads exactly like a plan the agent is following, because the same model wrote both. Irrelevant tokens are not free, and in a long session they are most of your context.
The 10,000th token is not processed as reliably as the 100th. Plan for that instead of hoping the window saves you.
The context engineering playbook
Context engineering, deciding what occupies the model's input on every turn and what does not, matters far more than prompt wording once your system runs in a loop. Everything above was diagnosis; what follows is treatment. Seven practices, ordered roughly by leverage. The first two do most of the work.
1. Set a context budget and make overruns attributable
Pick a working ceiling well below the advertised window and treat it as a hard operating limit. Somewhere in the 40 to 60% range is a defensible start: practice, not vendor guidance, so move it on your own measurements. The value of a number is that it forces a decision when you cross it, not when the provider rejects the request.
A budget you cannot attribute is a number in a design doc. Instrument the loop so every turn logs a token breakdown by component: system prompt, tool schemas, history, tool results, retrieved documents. The first look usually turns up something structural. Tool definitions are a fixed tax paid on every turn: a registry of a few dozen verbose descriptions can consume several thousand tokens before the user says a word.
2. Compact at checkpoints, do not truncate at the cliff
The reflexive fix near the limit is to drop the oldest messages. Resist it. Truncation deletes exactly the material that explains why the agent is doing what it is doing: the original instruction, the constraint from turn three, the approach already tried and rejected. What survives is the tail the model could mostly reconstruct anyway.
Compaction replaces that span with a structured note instead: decisions made, tasks still open, constraints that still bind, and the identifiers the agent will need again. Keep the last several turns verbatim so immediate working state stays intact, and run the swap at a natural boundary rather than when you run out of room.
01const CONTEXT_BUDGET = 60_000; // tokens - well under the advertised window02const KEEP_RECENT = 8; // last N turns stay verbatim0304// Call at a checkpoint (a step just finished), never mid tool call.05async function maybeCompact(history: Turn[]): Promise<Turn[]> {06 if (countTokens(history) < CONTEXT_BUDGET) return history;0708 const old = history.slice(0, -KEEP_RECENT);09 const summary = await summarize(old, {10 keep: ["decisions", "open tasks", "constraints", "file paths", "ids"],11 });1213 // One structured note replaces thousands of stale tokens14 return [asSystemNote(summary), ...history.slice(-KEEP_RECENT)];15}Three details separate a compaction routine that works from one that quietly corrupts state.
- Compact at a checkpoint, never mid-tool-call. A tool call and its result are a matched pair; splitting them leaves an orphaned call that most APIs reject.
- Keep the summary schema fixed. Compaction is lossy and lossy compounds, because the second pass summarizes a summary. Fixed fields resist drift far better than prose and stay diffable.
- Carry identifiers verbatim. "The config file" will not survive three generations of summary; a literal path will.
3. Externalize memory into something the agent can re-read
A transcript is the worst possible database: unindexed, append-only, and re-attended over in full on every turn. Durable facts do not belong there. Give the agent a workspace, a directory or a table or a key-value store, plus tools to read and write it, and keep the transcript for right now.
The economics flip. A fact in a file costs nothing until it is fetched; the same fact in the transcript costs attention on every turn until the session ends. A memory the agent can grep beats a transcript it must attend over.
The failure mode is discipline, not technology. Memory never written stays empty, memory never pruned goes contradictory, and an agent holding two conflicting notes will confidently follow the wrong one. Make writing a step in the loop and give stale notes a way to be superseded.
4. Isolate wide work behind sub-agents
Some work is token-hungry and its output is tiny. Settling one question about a codebase might mean reading dozens of files to produce two sentences. Inline, all those files sit in the orchestrator's context permanently. In a sub-agent, a fresh worker with its own clean context, budget and narrow contract, the two sentences come back and the rest is discarded. The orchestrator then scales with conclusions rather than with volume examined.
It is not free. A sub-agent cannot see the parent's context, so ambiguity in your task description becomes divergence you only notice when the answer comes back addressing a different question. Spend some of the saved tokens on a precise contract: what to investigate, what to return, in what shape. Sub-agents also cut context length while raising total tokens, which belongs in your agent cost model.
5. Prune tool results before they reach the context
Raw tool output is the biggest rot accelerant in most agent systems. An API response a human would skim in five seconds can arrive as thousands of tokens of nested JSON, most of it metadata the model never reads.
Do the trimming at the tool boundary, not in the prompt. Have the tool return a projection, only the fields the agent uses, and write the full payload out-of-band, returning a handle so the agent can fetch the rest deliberately. Cap output at a fixed token ceiling and make truncation visible.
Then go further: old tool results can be replaced with a stub once acted on. A listing from thirty turns ago has done its job; keeping the conclusion and dropping the listing is the difference between context that grows with the task and context that grows with everything you ever looked at.
6. Put instructions first and the live question last
This is the cheapest intervention here and it follows from the positional results. Instructions and constraints go at the start, where recall is strongest; the immediate question goes at the end, where recall is also strong. The large middle is where you put material the model can afford to sample rather than absorb.
The pattern to avoid is burying the ask. A prompt that opens with instructions, follows with sixty thousand tokens of retrieved context and never restates what it wants is asking the model to hold the request where holding is weakest. Restating it in one sentence at the end costs a few dozen tokens and is often the largest single-line quality gain available.
A happy accident: the stable prefix good placement wants at the front is also what prompt caching wants, where the provider stores the encoded front of your prompt and skips re-processing it.
7. Measure it instead of vibing it
Every practice above is a hypothesis until you measure it. Build a small evaluation that replays representative long sessions and scores quality against context length, then watch that curve the way you watch latency regressions.
Long context vs RAG vs memory: pick per job
These four get argued about as competitors. They are not. They are different answers to one question: where does information live between the moment you have it and the moment the model needs it? The window keeps it in the prompt. RAG, retrieval-augmented generation, keeps it in a searchable corpus and pastes in only the matching passages. Memory files keep it on disk. Sub-agents keep it in a worker you discard.
| Approach | Best at | Failure mode | Cost profile |
|---|---|---|---|
| Stuff the window | Small, self-contained tasks where every input fits and is current | Rot: quality sags as input grows and middle material gets skimmed | Highest tokens per call, lowest build effort |
| RAG (retrieve, then answer) | Large corpora where only a few passages matter for any one question | Retriever misses, and loosely related chunks land as distractors | Cheap per call, plus an index and embeddings to operate |
| External memory files | Durable cross-session facts: decisions, constraints, IDs, file paths | Stale or conflicting notes, or an agent that never re-reads them | Near-zero tokens, real write and pruning discipline |
| Sub-agents | Wide, token-hungry investigation where only the conclusion travels up | Divergence from a vague task contract; nuance lost in the handoff | Small orchestrator context, higher total token spend |
A production agent uses all four at once, and the design work is deciding which category each piece of information belongs to: ephemeral working state in the window, reference corpora behind retrieval, durable facts in memory, wide investigations in sub-agents.
Two adjacent decisions determine how well it holds. Tools that return narrow results keep context lean by construction, which is why response shape matters as much as the tool's logic when building MCP servers for production, MCP being the standard interface agents use to reach external tools. And an agent that can checkpoint and resume need not carry a session in its window: moving that state into a real datastore, the pattern behind a Postgres-backed checkpointer for LangGraph, turns a context problem into a storage problem.
How to measure context rot in your own system
You cannot tune what you cannot see, and context rot is invisible to every monitor you already run. Error rates stay flat. Latency stays flat. Token counts rise, which reads as usage growth. The only thing moving is quality as a function of input length. Four pieces fix that.
- A replay harness. Record real sessions, full message lists with tool calls and results, and replay them deterministically so the model and your context strategy are the only variables.
- A fixed probe set. Questions with known answers planted at known depths: near the start, deep in the middle, near the end. Include one worded to share almost no vocabulary with its target, since that semantic case degrades fastest.
- A quality-versus-length curve. Score the probes at several lengths and plot, do not average: one aggregate hides the shape you want, which is where your curve bends.
- A drift alarm. Re-run the curve when the model version changes, when you add tools, and when you change the compaction policy. All three move it.
The whole thing can be a few hundred lines and a nightly job, and it buys you the ability to say "quality holds to roughly 45K tokens on this workload and bends after that", where the number is yours. That turns an argument about vibes into a budget.
Traces make degradation legible. When you can open one long session and see context length, the per-component token breakdown, every compaction event and output quality on one timeline, rot stops being a mystery. That instrumentation is central to how we approach LLMOps, the practice of running LLM systems with evaluation, tracing and monitoring attached. The systems still reliable at hour four are the ones where somebody watches that curve.
FAQ
What is context rot in LLMs?
Context rot is the degradation in a language model's output quality as its input grows longer, even when every fact it needs is inside the window. Benchmarks show accuracy falling with input length on tasks the same model handles perfectly at short length, so the failure is length-driven rather than task-driven.
Does a bigger context window fix context rot?
No. A bigger window changes what fits, not how reliably the model uses what is in it. Reported results show degradation well before advertised limits, so moving from a 200K to a 1M window mostly buys room to accumulate more distractors. Manage the context you actually send on each turn instead.
Is context rot the same as lost in the middle?
Not quite. Lost in the middle describes a positional effect: information placed in the middle of a long input is recalled worse than the same information at the start or the end. Context rot is the broader phenomenon of quality falling as input length grows, of which the positional effect is one component.
Does RAG eliminate context rot?
No, it relocates the problem. Retrieval keeps the input short, which genuinely helps, but a retriever returning loosely related chunks injects plausible-but-wrong material, exactly the distractors that reported benchmarks show hurting more as context grows. Retrieval quality and chunk count become the new levers to tune and measure.
How do I prevent context rot in AI agents?
Set a token budget well under the window and track usage per component. Compact old turns into structured notes at checkpoints instead of truncating at the limit. Keep durable facts in files the agent re-reads, push wide exploration to sub-agents, trim raw tool output, and put instructions first with the live question last.
Build for context rot from the first sprint
Context rot is not a bug you fix once. It is a property of the tools, so it is a constraint you design around, like network latency or rate limits. The teams shipping agents people keep using decided early what goes in the window, what goes in a file, and what never enters context.
That is the work we do. Zenthos builds agentic AI systems with context budgets, compaction and memory architecture in from the first sprint rather than bolted on after a long session goes sideways. We set up the LLMOps side too, so degradation surfaces on a dashboard instead of in a support ticket. And when a smaller model with a tight context beats a bigger window, LLM fine-tuning is how you get there.
If your agent is brilliant in demos and unreliable an hour in, that is solvable and usually quick to diagnose. Get in touch for a free consultation, bring a transcript that went wrong, and we will walk through where the context went. The context engineering checklist is a one-page version of this playbook for your next design review.