DevelopmentGlossary
Context Window
Context windows have grown from a few thousand tokens in early chat models to hundreds of thousands, and a million or more on some current models. Limits change with each release, so check the provider's documentation for the model you use. Bigger windows change what's possible: full codebases, entire books, hour-long meeting transcripts can fit in one call.
Caveats: (1) cost scales with window usage, so a 1M-token request is not free; (2) attention quality degrades with distance (the lost-in-the-middle effect is real, and so is context rot, (3) some models charge different rates for long vs short context. Design for the window size you actually need, not the one your model allows.
Example
# Managing context window in a long conversation
MAX_CONTEXT = 200_000 # look up your model's limit
SYSTEM_TOKENS = count(system_prompt)
RESERVED_FOR_OUTPUT = 4_000
budget = MAX_CONTEXT - SYSTEM_TOKENS - RESERVED_FOR_OUTPUT
# Trim old turns until history fits in `budget`
while count(history) > budget:
history = summarize_oldest(history)When to use it
- Planning context composition for agents and RAG
- Deciding whether to chunk a large document or feed it whole
- Sizing the conversation history buffer
When not to use it
- Filling the window just because you can, since attention quality degrades
- Relying on 1M+ windows for precise recall; use retrieval instead
Articles that use this
- The System Prompt Architecture for Coding Agents: What Claude Code and Cursor Get RightDevelopment
- Context Rot: Why Long AI Chats Get Worse and When to Start FreshTechniques
- AI for Administrative Tasks: Inbox, Meeting Notes and Documents
- Your Agent's 'Done' Is a Lie: Gate Completion on Disk Evidence, Not Its Final MessageAgentic
- Stop Trusting Free-Form Chain-of-Thought. Make Your Model Sign a Logical Certificate InsteadTechniques

