Skip to content
Search
Techniques7 min read

Context Engineering vs Prompt Engineering: How to Design Token Budgets That Actually Work

Brian Gagne
Context Engineering vs Prompt Engineering: How to Design Token Budgets That Actually Work

TL;DR: Prompt engineering is about how you word the instruction. Context engineering is about everything the model sees when it answers: instructions, examples, retrieved documents, tool definitions, history and state. Past a certain point, rewording stops helping and the fix is what you load, where it sits, and what you leave out. Give each kind of content a token budget, put the important material first, and keep the window lean, because models get worse as input grows even when the answer is in there.

Context engineering vs prompt engineering is a question of scope. You've rewritten the system prompt six times, added emphasis and bolted on "make sure to include X". The output still misses key facts or invents details, and usually the cause is what the model saw before it started writing rather than the wording.

In June 2025, Shopify's CEO Tobi Lütke wrote that he preferred "context engineering" because it describes the core skill better: "the art of providing all the context for the task to be plausibly solvable by the LLM." About a week later Andrej Karpathy agreed, calling it "the delicate art and science of filling the context window with just the right information for the next step." Anthropic's engineering team calls context engineering "the natural progression of prompt engineering", and describes context as "a finite resource with diminishing marginal returns" (Anthropic, effective context engineering for AI agents). The glossary entry on context engineering has the short version.

Why more tokens can make answers worse

Bigger windows sound like more room for useful information. The research says more input carries its own cost.

  • Position matters. In "Lost in the Middle", performance was often highest when the relevant information sat at the start or end of the input, and dropped significantly when the model had to use information in the middle, even for long-context models (Liu et al., arXiv 2307.03172).
  • Length alone hurts. Across five models on math, QA and coding, performance fell by 13.9% to 85% as input grew, even when the models could retrieve every relevant fact. It still fell when the irrelevant text was swapped for whitespace (Du et al., arXiv 2510.05381).
  • Focused beats full. Chroma's Context Rot study found models did significantly better on focused prompts of about 300 tokens than on the full 113,000-token version of the same conversation, and concluded that models "do not maintain consistent performance across input lengths" (Chroma, Context Rot).

The practical rule: treat an advertised window size as a ceiling and budget well under it.

Budget the context window

Treat the window as a fixed budget. A token spent on one thing is a token you can't spend on another. These ranges are a starting point for a 128k window; measure your own system and adjust.

Slice Starting budget Notes
System instructions 2k to 5k, fixed Role, rules, constraints. If yours is 20k, you're describing in words what examples or retrieved text should show.
Current state and tool outputs 20k to 40k, changes every turn What the model must act on now. Highest-priority dynamic content.
Retrieved documents 10k to 30k Filter hard. Five relevant paragraphs beat fifty loosely related ones.
Conversation history 5k to 15k, sliding Summarize older turns.
Headroom 10% to 15% For a large tool result or retrieval spike.

Also leave room for the response. On many APIs the output limit is a separate setting, but long inputs still crowd it on models where input and output share the window. When you hit the ceiling, something has to go, and deciding what goes is the heart of context engineering.

Four operations: write, select, compress, isolate

Most context work falls into four moves. Anthropic's guide describes the long-run versions of three of them as compaction, structured note-taking and sub-agent architectures (Anthropic).

  • Write: save information outside the window and load it back when needed. An agent that keeps research in a notes file and holds only a summary in context is writing.
  • Select: load only what this step needs. Retrieval (RAG) is selection, and so is tool filtering. RAG-MCP retrieved only the relevant tool descriptions instead of loading all of them, and tool selection accuracy rose from 13.62% to 43.13% (Gan and Sun, arXiv 2505.03275). Anthropic reported that its tool search approach cut tool-definition tokens by 85% (Anthropic, advanced tool use).
  • Compress: cut tokens and keep the signal. Claude Code, for example, summarizes conversation history automatically as it approaches the context limit, and lets you say what to keep (Claude Code docs). Do the same on purpose in your own pipelines.
  • Isolate: give sub-agents their own clean windows and have them return short summaries. See agent handoff protocols for what to pass between them.

Research points the same way. Agentic Context Engineering (ACE) treats the context as a playbook that is edited and refined over time, and matched the top-ranked production agent on the AppWorld leaderboard average while using a smaller open-source model (Zhang et al., arXiv 2510.04618).

The usual fixes, side by side

Problem The prompt-engineering reflex The context-engineering fix
Output ignores key facts "Make sure to include X, Y and Z." Put X, Y and Z near the start of the context.
Tone drifts "Write in a professional, authoritative tone." Add three short examples of the tone you want.
Invented details "Only use verified information." Supply the source documents and cut the irrelevant ones.
Agent forgets earlier steps "Remember what you did in step 2." Write state to a progress file and reload the relevant part each turn.
Costs too high Shorten the prompt. Trim history, cache the stable prefix, route simple requests to a smaller model.

On cost, prompt caching bills repeated prefix tokens at a reduced rate on Anthropic's API and others, which rewards keeping instructions and examples stable at the top of the prompt (Anthropic prompt caching docs). Our prompt caching guide covers how to structure for it.

Example 1: a two-sentence instruction, a well-built context

The Prompt:

<system>
You are a technical documentation writer. Match the tone of the examples.
</system>

<examples>
[3 short examples of the target style, about 200 tokens in total]
</examples>

<current_document>
[Only the section being documented, about 500 tokens]
</current_document>

<api_reference>
[Only the endpoints this section uses, about 800 tokens, filtered from a 15k-token spec]
</api_reference>

Write the documentation for current_document in the style of the
examples. Use only the API details in api_reference.

Why This Works: The instruction is two sentences and the context does the rest. The examples set the tone without describing it, the document is trimmed to the section in hand, and the API reference is filtered before it reaches the model. About 1,500 tokens of context replace a long system prompt that tries to describe the output in words.

Expected Output: A section that matches the examples in tone and structure, mentions only the endpoints supplied, and stays inside the current section.

Example 2: say what to use and what to ignore

The Prompt (before):

You are a helpful assistant. Answer the user's question about our product.
[the entire 200-page product manual]
User question: How do I reset my API key?

The Prompt (after):

You are a technical support agent for Acme API Platform.

## What you're looking at
The user is asking about API key management. Below is the matching
section of our docs (last updated 2026-05-01). Use only this section.
If the answer isn't here, say so.

## Documentation
[API Key Management section only, about 2 pages]

## What to ignore
Don't bring up billing or account deletion. The user didn't ask.

## Format
Numbered steps, at most 5. Include the dashboard URL for the action.

User question: How do I reset my API key?

Why This Works: It scopes the context to one section, says what to leave out, and fixes the answer's shape. The model no longer has to work out which of 200 pages matters.

Expected Output: Five or fewer numbered steps taken from the key management section, with the dashboard URL from that section, and a plain "that isn't covered here" if the section lacks the answer.

Start here

Pick one system you run today and measure how its window is spent. What share is fixed instructions and what share is live content? Where does the critical information sit? What gets dropped when the window fills? If instructions take most of the space, flip the ratio: trim them, expand the live content and put the important material first. For agents that run for dozens of turns, continue with context engineering for agents. More techniques are in the Techniques pillar.

Frequently asked questions

What is the difference between context engineering and prompt engineering?

Prompt engineering is writing and organizing the instructions. Context engineering covers everything in the model's window when it answers, including retrieved documents, examples, tool definitions, history and saved state, and decides what gets in, where it goes and what's left out.

Is prompt engineering dead?

No. Clear instructions still matter, and they're part of context engineering. They just stop being the main lever once a system pulls in documents, tools and history, because what the model sees then shapes the answer more than how the request is worded.

Why does a bigger context window not always help?

Models get less reliable as input grows. In one study, performance fell 13.9% to 85% with longer inputs even when every relevant fact was retrievable (arXiv 2510.05381). Information buried in the middle of a long input is also used less well (arXiv 2307.03172).

What is a token budget?

It's a planned split of the context window between instructions, current state, retrieved documents, history and headroom, so the most important content always fits and the least important is dropped first.

Rather have someone set it up for you?

That's a fair call. LTFI, the agentic business operating system from Kief Studio, can set up the work these guides describe and run it with you. No website yet? Kief Studio will build you one for free.

Go further on kief.studio