Why Long Contexts Degrade: Context Rot Explained

Context rot is the tendency for a model to become less reliable at using information as its context gets longer. The details are still in the window, but the model may overlook them, mix them up or give weight to the wrong parts. Large windows reduce hard failures such as running out of space, but they do not remove this soft failure.

What causes it

Language models use an attention mechanism to decide which earlier tokens matter for the next token. Attention is spread across the whole context. When the context holds a few thousand relevant tokens, each gets a meaningful share. When it holds a few hundred thousand mixed tokens, the relevant ones compete with a lot of noise. Models are also trained mostly on shorter sequences than the largest windows allow, so behaviour at the extreme end is often less polished. The exact degree varies a lot by model and task, so measure on your own workload instead of trusting a general claim.

Symptoms you will see in agents

• The agent ignores a rule from the system prompt that it followed earlier in the run.
• It repeats a search it already did, because the earlier result is buried.
• It confuses two similar items, such as two files with similar names.
• It answers from an outdated tool result instead of a newer one.
• Quality drops gradually as a session gets longer, then recovers in a fresh session.

The last symptom is a strong clue. If restarting with a short, clean context fixes the problem, the context has rotted.

Distractors matter

Research on long-context tests, including the well-known needle-in-a-haystack style, shows that finding a single fact is easier than reasoning over many facts, and that similar-looking but irrelevant passages make errors more likely. In practice this means that stuffing in "related" documents can hurt more than leaving them out. Quality of context beats quantity.

A small experiment you can run

The script below hides a fact inside filler text of different lengths and checks whether the model can retrieve it. It uses only the Claude SDK. The filler is repeated sentences, so this is a crude test, but it shows how to measure instead of guess.

import anthropic

client = anthropic.Anthropic()
MODEL = "claude-sonnet-5-5"

FACT = "The deployment password for project Falcon is 'blue-harbor-42'."
FILLER = "The quarterly report notes steady progress across all teams. " * 1

def run(filler_repeats: int) -> bool:
    filler = FILLER * filler_repeats
    half = len(filler) // 2
    doc = filler[:half] + " " + FACT + " " + filler[half:]
    response = client.messages.create(
        model=MODEL,
        max_tokens=2000,
        messages=[{
            "role": "user",
            "content": doc + "\n\nWhat is the deployment password for project Falcon?",
        }],
    )
    text = "".join(b.text for b in response.content if b.type == "text")
    return "blue-harbor-42" in text

for repeats in (10, 500, 5000):
    print(repeats, "repeats ->", run(repeats))

On many models this simple test passes at every length, which is itself informative: easy retrieval survives well. The harder case is when the fact is similar to other content, or when the model must combine several facts. Post 63 shows how to build tougher tests.

How to limit context rot

1. Keep it short: include only what the current step needs.
2. Clear stale results: remove old tool output once it has been used.
3. Summarise history: replace long past sections with a short summary (posts 33 and 53).
4. Place key items well: put instructions and the question where the model attends to them (posts 29 and 64).
5. Use sub-tasks: let a separate call read a big document and return only the answer (post 67).
6. Measure: compare quality at several context lengths on your own tasks.

Why compaction and clearing exist

Several techniques later in this course are direct answers to context rot. Clearing old tool results removes distractors that are no longer relevant. Summarising history keeps the useful facts while dropping the long tail of detail. Sub-agents let a big reading job happen in a separate window, so the main agent only receives a short result. Just-in-time retrieval avoids loading material until the moment it is needed. If you remember one thing, remember this: each of these is a way of keeping the window small and focused, because small and focused contexts tend to behave better than large, crowded ones.

Do not over-correct

Context rot does not mean you should starve the model. Missing facts cause confident wrong answers, which are worse than slow answers. The balance is to include everything required for the current step, and nothing that is merely "possibly related". When in doubt, run a test with and without the extra material and let the results decide.

Takeaway

Treat context as a finite attention budget, not just a storage limit. The aim is not to use the whole window, but to make every token in it earn its place.