Designing Tool-Use Prompts That Don't Hallucinate Tool Calls

Your agent just called a tool that doesn't exist. Not a typo in the function name. Not a missing import. The model fabricated an entire tool, passed it plausible-looking parameters, and kept going like nothing happened.

This is tool-call hallucination, and it's the single biggest reliability problem in agentic systems right now. The MCPVerse benchmark found that when tool schemas get injected into system prompts instead of using native function calling, Claude Sonnet 4 fabricated tool responses more than 70% of the time. Not edge cases. Seventy percent.

The fix isn't "write better prompts." The fix is understanding the specific failure modes and applying the right structural pattern to each one.

The Seven Ways Tool Calls Break

ToolScan, a systematic study of tool-use errors across LLMs (Kokane et al., 2025), identified seven distinct failure patterns:

1. Hallucinated function names -- the model invents a tool that isn't in the schema
2. Hallucinated argument names -- real tool, fake parameters
3. Incorrect argument values -- right parameter name, wrong content
4. Wrong argument types -- string where it expects an integer
5. Insufficient API calls -- skipping necessary steps in a chain
6. Redundant API calls -- calling the same tool repeatedly for no reason
7. Invalid output format -- mangling the structured response

Each of these has a different root cause. Treating them all as "the model needs better instructions" is why most prompt-engineering advice for tool use doesn't work.

The Format Problem Nobody Talks About

Here's the counterintuitive finding from the MCPVerse research: the 70% hallucination rate wasn't caused by bad tool descriptions. It was caused by a mismatch between how tools were presented in the prompt and how the model was trained to call them.

A perfectly written tool description, injected as text in a system prompt, will hallucinate more than a mediocre description delivered through the model's native function-calling API. The model learned tool use in a specific structural format during training. When you deviate from that format, you're asking it to generalize in a way it wasn't optimized for.

This means the first rule of tool-use prompting has nothing to do with words.

Rule 1: Use your model's native function-calling format. If your framework supports structured tool definitions (OpenAI's function calling, Anthropic's tool use, etc.), use them. Don't roll your own by stuffing JSON schemas into the system prompt.

Prompt Patterns That Fix Each Failure Mode

Phantom Tool Names: Strict Schema Enforcement

When the model invents tools, it's usually because the tool list is ambiguous or the model is interpolating between similar-sounding tools it's seen in training data.

The Prompt Pattern:

You have access to EXACTLY these tools and NO others:

- search_database: Query the product catalog by SKU, name, or category
- get_order_status: Look up order status by order_id (integer)
- create_support_ticket: File a new ticket with subject (string) and body (string)

If the user's request cannot be fulfilled with these three tools,
say so directly. Do not attempt to call any tool not listed above.
Do not invent tool names or guess at tools that might exist.

Why This Works: The explicit constraint ("EXACTLY these tools and NO others") combined with a fallback instruction ("say so directly") gives the model a safe exit when it can't find a matching tool. Without that exit, the model will fabricate one rather than admit it can't help.

Expected Output:

I don't have a tool to process refunds directly. I can create a support ticket for the billing team to handle your refund request. Want me to do that?

Wrong Parameters: JSON Schema With Types and Boundaries

Hallucinated argument names drop significantly when you enforce strict JSON schemas. Most frameworks now support strict: true mode on function definitions, which constrains the model to only emit parameters defined in the schema.

The Prompt Pattern:

{
  "name": "get_order_status",
  "description": "Look up current status of a customer order",
  "strict": true,
  "parameters": {
    "type": "object",
    "properties": {
      "order_id": {
        "type": "integer",
        "description": "Numeric order ID (e.g., 48291). NOT the tracking number."
      }
    },
    "required": ["order_id"],
    "additionalProperties": false
  }
}

Why This Works: additionalProperties: false prevents the model from inventing extra parameters. The description includes an example value AND a clarification of what the parameter is NOT. That negative example ("NOT the tracking number") blocks the most common confusion.

Unnecessary Calls: "When NOT to Use" Guidance

This is the pattern most people miss. Your tool descriptions tell the model what the tool does. They almost never say when to leave it alone.

The Prompt Pattern:

- search_database: Query the product catalog by SKU, name, or category.
  USE when the user asks about product availability, pricing, or specifications.
  DO NOT USE when the user is asking about their own order -- use get_order_status instead.
  DO NOT USE for general questions you can answer from context.

Why This Works: Short tool instructions work best when each one carries both a verb and a boundary. The "DO NOT USE when..." lines are as important as the "USE when..." lines. They give the model explicit permission to skip the tool.

Expected Output:

Your order #48291 shipped yesterday and should arrive Thursday. (I looked that up using order status -- no need to search the product catalog for this.)

Silent Failure Masking: Force Error Reporting

This one causes the most damage in production. The model calls a tool, gets a 400 error, and instead of reporting the failure, it fabricates a success response. The user thinks the action completed. It didn't.

The Prompt Pattern:

CRITICAL RULE FOR ALL TOOL CALLS:
If any tool returns an error, a timeout, or an unexpected response:
1. Report the exact error message to the user
2. Do not retry unless the user asks you to
3. Do not guess what the successful response would have been
4. Do not claim the action succeeded

Wrong: "I've updated your email address."
Right: "The update failed with error: 'invalid_email_format'. The email you
provided may have a typo. Can you double-check it?"

Why This Works: The wrong/right examples are critical. Models learn behavior from examples faster than from rules. Showing the exact failure mode you want to prevent, paired with the correct alternative, is more effective than a paragraph of instructions.

The Allow/Deny List Pattern

Unbounded tool access is one of the most common ways agents get into trouble. Explicit allow/deny lists scoped to the task context shrink what a confused or manipulated model can reach.

In practice, this means your agent shouldn't have the same tool access for every conversation turn.

The Prompt Pattern:

You are helping the user manage their account settings.

ALLOWED tools for this task:
- get_user_profile
- update_email
- update_notification_preferences

DENIED tools for this task (even though they exist in your tool list):
- delete_account
- reset_password
- modify_billing

If the user asks for something that requires a denied tool, explain that
this action requires elevated permissions and direct them to support.

Why This Works: The model has a clear scope. It knows certain tools exist but are off-limits for this context. Without the deny list, a capable model with access to delete_account might call it if the user says "just get rid of everything" during a frustrating support interaction.

The Surprising Finding: More Tools Can Help

The conventional wisdom says "fewer tools means fewer errors." The MCPVerse data says otherwise -- for capable models.

Claude performed better in Standard mode (full tool set available) than in Oracle mode (only the "correct" tools provided). With more options, the model found alternative paths when one tool didn't fit. With fewer options, it forced the wrong tool to work.

This doesn't mean you should dump 500 tools into every prompt. It means that for capable models, a well-described tool set of 15-20 tools often outperforms an artificially pruned set of 3-4. The key is quality of tool descriptions, not quantity of tools.

For truly large tool sets (50+), retrieval-based tool selection works better than listing everything. The model first describes what it needs, a retrieval layer finds matching tools, and the model picks from the shortlist.

The Chain Validation Pattern

The most dangerous failure mode is cascading hallucination. One fabricated tool output feeds into the next tool call, which feeds into the next. An inventory agent hallucinated a nonexistent SKU, then called four downstream APIs to price, stock, and ship a phantom item.

The fix isn't a better prompt. It's a deterministic validation gate between tool calls.

After EVERY tool call, before making the next one:
1. State what the tool returned (quote the key fields)
2. State what you will do next and why this tool output justifies it
3. If the tool returned no results, empty data, or an unexpected format -- STOP
   and report to the user rather than proceeding with assumptions

This forces the model to "show its work" between steps. It catches fabricated outputs because the model has to quote specific return values, which is harder to fake than summarizing.

For safety-critical flows, move this gate outside the LLM entirely. A deterministic process evaluates the tool output against expected schemas before the model sees the result. No LLM call, sub-100ms latency, zero hallucination risk on the validation itself.

The Hierarchy That Matters

If you take one thing from this post, take the fix hierarchy in order of impact:

1. Use native function-calling format (in MCPVerse, schemas pasted into the prompt pushed Claude Sonnet 4 past a 70% hallucination rate)
2. Enforce strict JSON schemas with additionalProperties: false
3. Add "when NOT to call" guidance to every tool description
4. Keep per-tool instructions under 60 words with clear verbs and boundaries
5. Add deterministic validation gates outside the LLM for anything that touches production data

Most teams jump to step 3 or 4 and wonder why their agents still hallucinate. The structural decisions in steps 1 and 2 have an outsized impact that no amount of clever wording can replace.

Tool-use prompting isn't really about prompting. It's about giving the model the right structure, the right constraints, and an explicit exit when it doesn't know what to do.

Want hands-on training on building reliable tool-use agents for your team? Connect with Kief Studio on Discord or schedule a session.