Context Rot: Why Long AI Chats Get Worse and When to Start Fresh

TL;DR: Context rot is the drop in an AI model's accuracy as the conversation or document you give it gets longer, even when everything it needs is still in there. Research has measured it across many models. The practical fix is simple: keep what the model sees short and relevant, put the important material where the model reads it best, and when a chat starts going wrong, ask for a summary and start a fresh one.
You've probably seen it happen. A chat with an AI assistant starts well. Forty messages later it forgets a rule you gave it at the start, repeats a mistake you already corrected, or mixes up the second draft with the fourth. That slow decline is called context rot. This guide explains what it is, what the research measured, and how to decide when to start a fresh chat.
What context rot means
Every AI model reads from a context window: all the text it can see when it writes a reply. In a chat, that's the whole conversation so far plus any files you attached, and each new message adds to it.
Anthropic's documentation describes the context window as the model's "working memory" and states the problem plainly: "As token count grows, accuracy and recall degrade, a phenomenon known as context rot." It adds that this "makes curating what's in context just as important as how much space is available" (Anthropic, Context windows).
So a large window tells you how much the model will accept. It doesn't tell you how well the model will use all of it. Our glossary entry on context rot has the one-paragraph version.
What the research found
Four studies point the same way.
Position matters: Lost in the Middle
In 2023, a team of researchers moved one important passage around inside a long input and measured how well models found and used it. They report that "performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models" (Liu et al., arXiv 2307.03172). Our paper digest of Lost in the Middle explains the setup.
For a chat, that suggests a detail buried in message twenty of forty may be harder for the model to use than your first instructions or your latest message, though the next study shows that position isn't the whole story.
Length alone hurts: Chroma's context rot report
In July 2025, Chroma published a technical report called Context Rot that tested 18 models. Its conclusion is that models "do not maintain consistent performance across input lengths." It also found that "Even a single distractor reduces performance relative to the baseline", where a distractor is text that looks relevant but isn't the answer. And it found that models did better on focused prompts of about 300 tokens than on the full 113,000-token version of the same conversation (Chroma, Context Rot).
A long chat is full of distractors: earlier drafts, abandoned ideas and corrected facts.
Even perfect recall doesn't save it
A 2025 study asked a sharper question: what if the model can find every relevant fact? Across five models on math, question answering and coding tasks, performance still fell by 13.9% to 85% as input length grew, and it still fell when the irrelevant text was replaced with blank whitespace (Du et al., arXiv 2510.05381). The authors conclude that "the sheer length of the input alone can hurt LLM performance."
Long conversations add their own problem
The last study moves from length to the shape of a chat. Researchers compared giving a model a whole task in one message with revealing the same task a piece at a time over several turns, the way people often talk. Across six generation tasks, the top models they tested did worse in the multi-turn version, with an average drop of 39%. Their summary: "when LLMs take a wrong turn in a conversation, they get lost and do not recover." The models often made assumptions early and then relied on them (Laban et al., arXiv 2505.06120).
More messages rarely fix a chat that went wrong early, because the wrong turn is still sitting in the context.
Signs your chat has rotted
You don't need a benchmark to spot it. Watch for these:
- It drops a rule you set early, such as a word limit, a tone or a format.
- It brings back something you already corrected.
- It quotes or edits an older version of your document instead of the current one.
- Answers get longer and vaguer while the question stays the same.
- It keeps building on an assumption you never made.
One sign alone can be a normal mistake. Two or three in a row usually mean the context is working against you.
When to start a fresh chat
A good rule: start fresh when the task changes, when you've corrected the same thing twice, or when the chat has collected several versions of the thing you're working on. Keep going when the chat is short and on track, or when the next step depends on detail that would be hard to restate.
The researchers behind the multi-turn study give the same advice to users. They suggest that "starting a new conversation that repeats the same information" can work better than pushing on, and that "consolidating instruction requirements into a single instruction" helps (Laban et al., arXiv 2505.06120). Anthropic's guidance for long-running agents that keep their state in files also treats a fresh start as a real option: when a context window is cleared, "consider starting with a brand new context window rather than using compaction" (Anthropic, Prompting best practices).
Starting fresh doesn't mean starting from zero. Carry over a short, clean summary and leave the clutter behind.
The fresh-start handoff prompt
Paste this at the end of the old chat, check the result, then paste the result into a new chat.
We're going to continue this work in a new chat. Write a handoff
note I can paste there. Use these headings:
1. Goal: what we're trying to produce, in two sentences.
2. Rules: every instruction I gave about format, tone, length and
audience that still applies.
3. Decisions: what we agreed, as a short bulleted list.
4. Current version: the latest draft or result in full, and nothing
older.
5. Open questions: what's still unresolved.
Leave out anything I corrected or rejected. If you're unsure whether
something still applies, list it under Open questions instead of
guessing.
Why this works: It does what the multi-turn study recommends, collecting requirements spread over many messages into one instruction. It keeps only the current version, which removes the old drafts that act as distractors. And it puts the goal and rules at the start of the new chat, a strong position according to Lost in the Middle. Read the note before you paste it, because a rotted chat can also write a rotted summary.
How to slow context rot down
You can't switch it off, but you can give the model less to get lost in.
Put long material first and your question last. For inputs of 20,000 tokens or more, Anthropic's docs advise placing long documents "near the top of your prompt, above your query, instructions, and examples", and note that "Queries at the end can improve response quality by up to 30 percent in tests, especially with complex, multidocument inputs" (Anthropic, Prompting best practices). If you're not sure how long your document is, paste it into the token counter.
Ask for quotes before the answer. For long documents, the same docs suggest asking the model to "quote relevant parts of the documents first before carrying out its task", which "helps Claude focus on the relevant content and ignore the rest of the document." The 2025 length study reached a similar idea: having the model recite the evidence first turns a long task into a short one (Du et al., arXiv 2510.05381).
Send less. Attach the three pages that matter instead of the whole 80-page report. Anthropic's guide to context engineering describes context as "a finite resource with diminishing marginal returns" and sets the goal as finding "the smallest set of high-signal tokens that maximize the likelihood of your desired outcome" (Anthropic, Effective context engineering for AI agents).
One task per chat. A chat that drafted an email and then debugged a spreadsheet carries both into every reply.
Know what your app does when the chat gets long. Some apps quietly trim or summarize old messages. Anthropic notes that chat interfaces "can also manage the context window on a rolling 'first in, first out' basis" (Anthropic, Context windows), which means your earliest instructions may be the first to go. Restating key rules, or starting fresh with a handoff note, keeps them in view.
For people building agents
Agents fill their context window faster than chats because every tool result lands in it. The fixes are compaction, which summarizes older turns automatically, clearing old tool results, and handing side tasks to a subagent with its own clean window. Our guide to context engineering for agents across 50 turns walks through a full setup, context engineering vs prompt engineering covers token budgets, and when to use subagents covers the delegation side.
Frequently asked questions
What is context rot in AI?
Context rot is the drop in a language model's accuracy and recall as its input gets longer. It happens even when the information the model needs is still inside the context window. Anthropic's documentation uses the term, and Chroma's 2025 report measured it across 18 models.
Why does ChatGPT or Claude get worse in long conversations?
Every message stays in the context the model reads, so a long chat holds old drafts, corrections and side topics that compete for its attention. Research shows models use information at the start and end of a long input better than the middle, and that length alone can lower accuracy. A wrong assumption made early in a chat also tends to stick.
When should I start a new chat with AI?
Start a new chat when the task changes, when you've corrected the same mistake twice, or when the chat holds several versions of your work. Ask the old chat for a short handoff note with the goal, rules, decisions and the latest version, check it, and paste it into the new one.
Does a bigger context window fix context rot?
No. A bigger window lets the model accept more text, but studies found accuracy still falls as input grows, including on models built for long contexts. What you put in the window matters more than how big it is.
How do I stop an AI from forgetting my instructions?
Keep the chat focused on one task, put long material first and your question last, and restate key rules when the chat gets long. If the model keeps drifting, start a fresh chat with your rules at the top of the first message.
Terms in this article
From the library
- Token Budget & Cost Estimator for Production LLM AppsPlaybook
- AI Agent Security Checklist: The Lethal TrifectaPlaybook
- Chain-of-Verification Reduces Hallucination in Large Language ModelsPaper
- Tree of Thoughts: Deliberate Problem Solving with Large Language ModelsPaper
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsPaper
Rather have someone set it up for you?
That's a fair call. LTFI, the agentic business operating system from Kief Studio, can set up the work these guides describe and run it with you. No website yet? Kief Studio will build you one for free.

