You've rewritten your system prompt six times. You've tweaked the wording, added emphasis, bolted on "make sure to include X" instructions. The output still misses key facts, hallucinates details, or just feels off.

The problem isn't your words. It's what the model sees before it starts generating.

Prompt Engineering Hits a Wall

Prompt engineering treats the instruction text as the primary lever. Reword, restructure, add constraints. It works -- up to a point. But when you're building real systems that chain multiple steps, pull from external data, and maintain conversation state, prompt wording accounts for maybe 20% of output quality.

The other 80% is context engineering: deciding what information fills the context window, where it sits, and what gets cut when space runs out.

Andrej Karpathy framed it cleanly in mid-2025: the LLM is the CPU, the context window is RAM, and you're the operating system. Your job isn't writing better instructions. Your job is memory management.

Gartner made it official shortly after, telling enterprise AI leaders to prioritize context over prompts. And LangChain's 2025 State of Agent Engineering report found that among the 57% of organizations running AI agents in production, 32% cited quality as the top barrier -- with most failures traced to poor context management, not model capability.

The model isn't dumb. It just never saw the right information.

Why More Tokens Make Things Worse

Here's the counterintuitive part. Bigger context windows should help, right? More room for information means better outputs.

Not even close.

Stanford's "Lost in the Middle" research (published in TACL, 2024) measured a 30%+ accuracy drop on multi-document QA when the answer document moved from position 1 to position 10 in a 20-document context. Same words, same documents, same prompt. Only the position changed.

And Amplitude's AI team discovered that a focused 300-token context outperforms an unfocused 113,000-token one. They'd hit a ceiling where prompt tweaks stopped helping entirely. The fix was restructuring what filled the context window -- memory, retrieval results, tool outputs, conversation history -- not rewording instructions.

Morph LLM's analysis of SWE-rebench data shows a clear performance ceiling around 1M tokens. Past a threshold, more context actively degrades agent performance. The industry sold bigger windows as progress. Without context engineering, bigger windows just mean more noise.

The Token Budget Framework

Think of your context window as a fixed budget. Every token you spend on one thing is a token you can't spend on something else. Here's how to allocate a 128k token window for a production system:

System instructions: 2-5k tokens (fixed). Your persona, rules, and constraints. This is the smallest slice, and you never cut it. If your system prompt is 20k tokens, you're over-prompting.

Current state and tool outputs: 20-40k tokens (dynamic). The highest-priority dynamic content. What the model needs to act on right now. This gets the biggest allocation because it changes every turn.

Retrieved documents: 10-30k tokens (dynamic). RAG results, reference material, source docs. Filter aggressively for relevance. Five highly relevant paragraphs beat fifty tangentially related ones.

Conversation history: 5-15k tokens (sliding window). Summarize older turns. One team cut conversation history from 1,500 to 500 tokens per message and saved $3,000/month at 100k messages -- $6,600 down to $3,600, zero change to prompt instructions.

Output reserve: 25-50% of total. This is what the model needs to actually generate a response. Starve this and you get truncated, degraded output.

Buffer: 10-15%. Headroom for unexpected tool results or retrieval spikes.

When you hit your ceiling, the system has to decide what to drop. That decision architecture IS context engineering.

Before and After: The Real Fixes

Here's what this looks like in practice. Each row is a problem you've probably hit, the prompt engineering reflex, and the context engineering fix that actually works.

Problem: Output ignores key facts. Prompt fix: "Make sure to include X, Y, and Z." Context fix: Put X, Y, and Z in the first 500 tokens of context. Position matters more than instruction.

Problem: Inconsistent tone. Prompt fix: "Write in a professional, authoritative tone." Context fix: Inject three few-shot examples of the desired tone into the context. Show, don't tell.

Problem: Hallucinated details. Prompt fix: "Only use verified information. Do not make things up." Context fix: Supply verified source documents via retrieval. Cut irrelevant documents that dilute the signal. The model can't hallucinate as easily when the real answer is right there.

Problem: Agent forgets prior steps. Prompt fix: "Remember what you did in step 2." Context fix: Write state to a structured progress file. Reload relevant state on each turn. Don't ask the model to remember -- give it the information.

Problem: Costs too high. Prompt fix: Make the prompt shorter. Context fix: Trim conversation history, cache static content with prompt caching, route simple queries to a smaller model. One customer service operation cut costs 75% by routing 80% of queries to a budget model and saving the premium model for the 20% that actually needed it.

A Token Budget in Action

Here's a concrete prompt that applies context engineering principles. Instead of telling the model what to do with more words, you structure what it sees.

The Prompt:

<system>
You are a technical documentation writer. Match the tone of the examples provided.
</system>

<examples>
[3 short examples of desired output style -- 200 tokens total]
</examples>

<current_document>
[The specific section being documented -- 500 tokens, trimmed to relevant portion]
</current_document>

<api_reference>
[Only the endpoints referenced in current_document -- 800 tokens, filtered from 15k full spec]
</api_reference>

Write the documentation section for the current_document, following the style shown in the examples.
Use only the API details provided in api_reference.

Why This Works: The instruction is 2 sentences. The context does the heavy lifting: few-shot examples set the tone (no need to describe it), the document scope is trimmed to what's relevant (not the whole codebase), and the API reference is pre-filtered (not the full 15k-token spec). Total context: ~1,500 tokens. Compare that to a 3,000-token system prompt that tries to describe the desired output in words.

Expected Output:

A documentation section that matches the provided examples in tone and structure, references only the specific API endpoints supplied, and stays tightly scoped to the current section. No hallucinated endpoints, no tone drift, no "I don't have enough context" hedging -- because the context was engineered to contain exactly what's needed.

The 80/20 Rule for Context

In high-performing agentic systems, roughly 20% of context is static (instructions, persona, rules) and 80% is dynamic (retrieved data, memory, tool outputs, current state). If your ratio is inverted -- huge system prompt, minimal dynamic context -- you're over-prompting and under-contextualizing.

Claude Code's approach to tool loading is a good example of this in production. Instead of loading all MCP tool definitions upfront (a single Playwright MCP server consumes 11.7k tokens), it discovers and loads tools on demand -- reducing context overhead by 95%. That's not a prompt optimization. That's a structural redesign of what enters the window.

The ACE framework demonstrated the same principle from the research side: using context engineering, an open-source model (DeepSeek-V3.1) scored 59.4% on the AppWorld benchmark, nearly matching IBM's GPT-4.1-based agent at 60.3%. Better context management let a smaller, cheaper model compete with a premium one.

Start Here

Pick one system you're running today. Measure how your context window is actually allocated -- what percentage is static instructions vs. dynamic content? Where does critical information sit in the token order? What gets dropped when the window fills up?

Most teams find their system prompt is 60%+ of context usage. Flip that ratio. Trim the instructions, expand the dynamic context, and put the most important information first.

You'll get more from restructuring what the model sees than from rewriting what you tell it to do.

If your team wants to build context engineering into your AI workflows, connect with Kief Studio on Discord or schedule a session. We run hands-on training that goes beyond prompt templates into the structural patterns that actually move output quality.

context window optimization

Context Engineering Beats Prompt Engineering: How to Design Token Budgets That Actually Work