Context Engineering Beats Prompt Engineering: How to Design Token Budgets That Actually Work

You've rewritten your system prompt six times. You've tweaked the wording, added emphasis, bolted on "make sure to include X" instructions. The output still misses key facts, hallucinates details, or just feels off.

The problem isn't your words. It's what the model sees before it starts generating.

Prompt Engineering Hits a Wall

Prompt engineering treats the instruction text as the primary lever. Reword, restructure, add constraints. It works -- up to a point. But when you're building real systems that chain multiple steps, pull from external data, and maintain conversation state, prompt wording is only a small part of output quality.

Most of the rest is context engineering: deciding what information fills the context window, where it sits, and what gets cut when space runs out.

Andrej Karpathy framed it cleanly in mid-2025: the LLM is the CPU, the context window is RAM, and you're the operating system. Your job isn't writing better instructions. Your job is memory management.

Gartner made it official shortly after, telling enterprise AI leaders to prioritize context over prompts. And LangChain's 2025 State of Agent Engineering report found that among the 57% of organizations running AI agents in production, 32% cited quality as the top barrier.

The model isn't dumb. It just never saw the right information.

Why More Tokens Make Things Worse

Here's the counterintuitive part. Bigger context windows should help, right? More room for information means better outputs.

Not even close.

Stanford's "Lost in the Middle" research (published in TACL, 2024) measured a 30%+ accuracy drop on multi-document QA when the answer document moved from position 1 to position 10 in a 20-document context. Same words, same documents, same prompt. Only the position changed.

Chroma's Context Rot study (July 2025) found that models answered better from a focused 300-token context than from the full 113,000-token version of the same chat history. Amplitude's AI team reported the same lesson from production. They'd hit a ceiling where prompt tweaks stopped helping entirely. The fix was restructuring what filled the context window -- memory, retrieval results, tool outputs, conversation history -- not rewording instructions.

Morph's write-up, citing a SWE-rebench maintainer, reports a performance ceiling around 1M tokens. Past a threshold, more context actively degrades agent performance. The industry sold bigger windows as progress. Without context engineering, bigger windows just mean more noise.

The Token Budget Framework

Think of your context window as a fixed budget. Every token you spend on one thing is a token you can't spend on something else. Here's how to allocate a 128k token window for a production system:

System instructions: 2-5k tokens (fixed). Your persona, rules, and constraints. This is the smallest slice, and you never cut it. If your system prompt is 20k tokens, you're over-prompting.

Current state and tool outputs: 20-40k tokens (dynamic). The highest-priority dynamic content. What the model needs to act on right now. This gets the biggest allocation because it changes every turn.

Retrieved documents: 10-30k tokens (dynamic). RAG results, reference material, source docs. Filter aggressively for relevance. Five highly relevant paragraphs beat fifty tangentially related ones.

Conversation history: 5-15k tokens (sliding window). Summarize older turns.

Output reserve: 25-50% of total. This is what the model needs to actually generate a response. Starve this and you get truncated, degraded output.

Buffer: 10-15%. Headroom for unexpected tool results or retrieval spikes.

When you hit your ceiling, the system has to decide what to drop. That decision architecture IS context engineering.

Before and After: The Real Fixes

Here's what this looks like in practice. Each row is a problem you've probably hit, the prompt engineering reflex, and the context engineering fix that actually works.

Problem: Output ignores key facts. Prompt fix: "Make sure to include X, Y, and Z." Context fix: Put X, Y, and Z in the first 500 tokens of context. Position matters more than instruction.

Problem: Inconsistent tone. Prompt fix: "Write in a professional, authoritative tone." Context fix: Inject three few-shot examples of the desired tone into the context. Show, don't tell.

Problem: Hallucinated details. Prompt fix: "Only use verified information. Do not make things up." Context fix: Supply verified source documents via retrieval. Cut irrelevant documents that dilute the signal. The model can't hallucinate as easily when the real answer is right there.

Problem: Agent forgets prior steps. Prompt fix: "Remember what you did in step 2." Context fix: Write state to a structured progress file. Reload relevant state on each turn. Don't ask the model to remember -- give it the information.

Problem: Costs too high. Prompt fix: Make the prompt shorter. Context fix: Trim conversation history, cache static content with prompt caching, route simple queries to a smaller model. Most queries in a typical support workload can go to a budget model, with the premium model saved for the ones that actually need it.

A Token Budget in Action

Here's a concrete prompt that applies context engineering principles. Instead of telling the model what to do with more words, you structure what it sees.

The Prompt:

<system>
You are a technical documentation writer. Match the tone of the examples provided.
</system>

<examples>
[3 short examples of desired output style -- 200 tokens total]
</examples>

<current_document>
[The specific section being documented -- 500 tokens, trimmed to relevant portion]
</current_document>

<api_reference>
[Only the endpoints referenced in current_document -- 800 tokens, filtered from 15k full spec]
</api_reference>

Write the documentation section for the current_document, following the style shown in the examples.
Use only the API details provided in api_reference.

Why This Works: The instruction is 2 sentences. The context does the heavy lifting: few-shot examples set the tone (no need to describe it), the document scope is trimmed to what's relevant (not the whole codebase), and the API reference is pre-filtered (not the full 15k-token spec). Total context: ~1,500 tokens. Compare that to a 3,000-token system prompt that tries to describe the desired output in words.

Expected Output:

A documentation section that matches the provided examples in tone and structure, references only the specific API endpoints supplied, and stays tightly scoped to the current section. No hallucinated endpoints, no tone drift, no "I don't have enough context" hedging -- because the context was engineered to contain exactly what's needed.

Static vs. Dynamic Context

In high-performing agentic systems, a small share of context is static (instructions, persona, rules) and most of it is dynamic (retrieved data, memory, tool outputs, current state). If your ratio is inverted -- huge system prompt, minimal dynamic context -- you're over-prompting and under-contextualizing.

Claude Code's approach to tool loading is a good example of this in production. Instead of loading all MCP tool definitions upfront (a single large MCP server can eat thousands of tokens), it discovers and loads tools on demand. Anthropic reported that its tool search approach cut tool-definition tokens by about 85% in its own tests. That's not a prompt optimization. That's a structural redesign of what enters the window.

The ACE framework demonstrated the same principle from the research side: using context engineering, an open-source model (DeepSeek-V3.1) scored 59.4% on the AppWorld benchmark, nearly matching IBM's GPT-4.1-based agent at 60.3%. Better context management let a smaller, cheaper model compete with a premium one.

Start Here

Pick one system you're running today. Measure how your context window is actually allocated -- what percentage is static instructions vs. dynamic content? Where does critical information sit in the token order? What gets dropped when the window fills up?

If your system prompt takes up most of your context, flip that ratio. Trim the instructions, expand the dynamic context, and put the most important information first.

You'll get more from restructuring what the model sees than from rewriting what you tell it to do.

If your team wants to build context engineering into your AI workflows, connect with Kief Studio on Discord or schedule a session. We run hands-on training that goes beyond prompt templates into the structural patterns that actually move output quality.