Tool Call Hallucination: Designing Tool-Use Prompts That Don't Invent Tools or Arguments

TL;DR: Tool call hallucination is when an agent calls a tool that doesn't exist, makes up an argument value, or reports a result the tool never returned. Most of it is fixed by structure: use the provider's native function calling, turn on strict schemas, write tool descriptions that say when not to use the tool, and put a validation gate between the model's call and your system.
Your agent just called a function that doesn't exist. The JSON parsed, the types were right, and the function name was made up. Or the name was real and the email address in the arguments was invented. This is tool call hallucination, and it gets past ordinary tests because the output looks correct.
Below are the failure modes researchers have catalogued, why the format you present tools in matters, and the patterns that stop each one. If tool use or function calling are new to you, read those glossary entries first.
The seven ways tool calls break
ToolScan, a benchmark from Salesforce AI Research for characterizing tool-use errors, named seven error patterns that even strong models show (Kokane et al., arXiv 2411.13547):
- Insufficient API calls: the model skips a call the task needs.
- Incorrect argument value: the right parameter with the wrong content.
- Incorrect argument name: a real tool with a parameter that doesn't exist.
- Incorrect argument type: a string where an integer belongs.
- Repeated API calls: the same call again with no reason.
- Incorrect function name: a tool that isn't in the list.
- Invalid format error: a malformed call.
Each has a different cause. A schema fixes the type and name errors and does nothing for a made-up value, so one generic instruction such as "only use real tools" rarely works alone.
The format problem
The MCPVerse benchmark tested agents on more than 550 real tools (Lei et al., arXiv 2508.16260). To test how much the calling format matters, the authors also ran models with the tool schemas pasted into the prompt as text instead of passed through native function calling. For Claude 4 Sonnet, that produced a hallucination rate above 70%, with the model often fabricating tool responses despite explicit instructions not to. In the paper's Standard mode its score fell from 62.36 with native function calling to 15.10 with prompt-based calling. The authors put it down to a template mismatch with how the model was trained to call tools.
Rule 1: use the model's native function-calling format. Define tools through the API's tools parameter. Don't build your own by pasting JSON schemas into the system prompt. When you use the tools parameter, the API builds its own tool-use system prompt from your definitions (Define tools).
Rule 2: turn on strict mode. OpenAI's docs say that setting strict to true makes function calls "reliably adhere to the function schema, instead of being best effort", and require additionalProperties: false and every field listed as required (OpenAI function calling). Anthropic's strict tool use uses grammar-constrained sampling, guarantees that tool input follows the schema, and guarantees that the tool name is one you provided (Strict tool use). That closes off invented names, argument names and types. It leaves invented values wide open. {"account": "1234567890", "amount": 50000} passes every structural check even when nobody in the conversation ever gave that account number.
Prompt patterns for each failure mode
Phantom tools: give the model an exit
A model reaches for a tool that doesn't exist when nothing on its list fits the request. The survey of agent hallucinations by Lin et al. describes this directly: if no suitable tool is available, whatever tool the agent picks "is likely to be irrelevant or even fabricated" (arXiv 2509.18970). The fix is to make "I can't do that" an allowed answer.
The Prompt:
You have access to EXACTLY these tools and no others:
- search_catalog: Query the product catalog by SKU, name, or category.
- get_order_status: Look up order status by order_id (integer).
- create_support_ticket: File a ticket with subject (string) and body (string).
If the request cannot be done with these three tools, say so plainly
and offer the closest thing you can do. Never call a tool that is not
listed above, and never guess at tools that might exist.
Why This Works: The closed list plus the fallback instruction gives the model a safe answer when no tool matches. Without that exit it tends to pick the nearest tool or invent one.
Expected Output: "I don't have a tool that processes refunds. I can open a support ticket so the billing team handles it. Want me to do that?"
Wrong tool chosen: say when not to use it
Anthropic's guidance calls detailed descriptions "by far the most important factor in tool performance" and asks for what the tool does, when to use it and when not to, what each parameter means, and its limits, in at least three to four sentences per tool (Define tools).
The Prompt:
search_catalog: Searches the product catalog by SKU, product name or
category and returns price, stock and specifications. Use it when the
customer asks about a product they might buy. Do not use it for a
customer's own order; use get_order_status for that. Do not use it for
general questions you can answer from the conversation.
Why This Works: The "do not use" lines carry as much weight as the "use" line. They give the model explicit permission to skip the tool and name the right one instead.
Made-up values: preconditions on every argument
This is the failure strict mode can't touch. The model needs an email address it doesn't have, so it writes a plausible one.
The Prompt:
send_email: Sends an email to a contact.
- recipient_email: MUST be an address the user typed in this
conversation. Never generate, complete or guess an address.
- subject: string, at most 200 characters.
- body: string, at most 5000 characters.
PRECONDITION: Do not call send_email until the user has given the
recipient's address. If they say "email John" with no address, ask for it.
Why This Works: The precondition turns an unstated assumption into a rule with a clear boundary.
Expected Output (user says "email John about the meeting"): "Sure. What's John's email address?" Without the precondition you'd see something like send_email(recipient_email="[email protected]", ...), and the message either bounces or reaches a stranger.
Failures reported as successes: make errors visible
A tool returns an error and the model says it worked. Nobody goes back to check, so this one does real damage.
The Prompt:
If any tool returns an error, a timeout or an unexpected response:
1. Report the exact error message to the user.
2. Do not retry unless the user asks.
3. Do not guess what a successful result would have been.
4. Never say the action succeeded.
Wrong: "I've updated your email address."
Right: "The update failed with 'invalid_email_format'. The address may
have a typo. Can you check it?"
Why This Works: The paired wrong and right examples show the exact behaviour to avoid and its replacement, which gives the model a concrete pattern to copy instead of a rule alone. On the protocol side, MCP tools report execution failures in the result with isError: true, so your client can catch them before the model narrates them (MCP tools specification).
Scope the tool list per task
An agent that can reach every tool on every turn does the most harm when it's confused, so scope access to the job in front of it.
The Prompt:
You are helping the user change their account settings.
ALLOWED for this task: get_user_profile, update_email,
update_notification_preferences.
NOT ALLOWED for this task, even though they exist: delete_account,
reset_password, modify_billing.
If the user asks for something that needs a tool that is not allowed,
explain that it needs extra permissions and point them to support.
Why This Works: Naming the forbidden tools gives the model a concrete rule and lowers the chance of confusing them with similar allowed ones. Enforce the same list in code too, by sending only the allowed tool definitions for that task.
MCPVerse found most models did worse as the tool set grew, although agentic models held their accuracy or improved slightly with a larger set; Claude 4 Sonnet scored 62.4 with the 220-plus tools of the Standard set against 62.3 with only the tools each task needed (arXiv 2508.16260). For long lists on other models, let the model search for tools instead of loading them all, as Anthropic's tool search tool does.
Put a validation gate between the call and your system
The worst case is a chain, such as an inventory agent that invents a SKU and then prices, reserves and ships an item that doesn't exist. Prompts reduce this, and only a check outside the model stops it.
The Prompt (validator):
You are a tool-call validator. Given a proposed call and the
conversation so far, check three things:
1. TOOL EXISTS: is the function in the registered list? If not,
reject with UNKNOWN_TOOL.
2. VALUES GROUNDED: is every argument either stated in the
conversation, a documented default, or computed from those? If not,
reject with UNGROUNDED_PARAMETER and name the field.
3. PRECONDITIONS MET: are the tool's documented preconditions true
right now? If not, reject with PRECONDITION_FAILED and name it.
Reply with APPROVED, or REJECTED: <reason>.
Why This Works: Named checks give the validator a concrete job instead of an open "does this look fine?" question.
Expected Output (for an invented account number): REJECTED: UNGROUNDED_PARAMETER: account
Write checks 1 and 3 as plain code where you can, and keep the model for the grounding check. For anything that touches production data, the MCP specification also says clients should show tool inputs to the user and ask for confirmation on sensitive operations (MCP tools specification).
Treat tool descriptions you didn't write as untrusted
If your agent loads tools from MCP servers you don't run, their descriptions land in its context. The MCP specification says descriptions of tool behaviour such as annotations "should be considered untrusted, unless obtained from a trusted server", and that hosts must get the user's consent before invoking any tool (MCP specification). Keep your own descriptions factual and free of instructions aimed at the agent. Our MCP server design guide covers how to write tool interfaces agents use correctly.
The order that matters
- Native function calling, never schemas pasted into the prompt.
- Strict mode on every tool.
- Descriptions that say when to use the tool and when not to.
- Preconditions on every argument that must come from the user.
- A validation gate outside the model for anything that writes data.
Steps 1 and 2 cost one setting each and remove whole classes of error. For more on making agents prove their work, see gate completion on disk evidence, and for the wider topic see the Agentic pillar.
Frequently asked questions
What is tool call hallucination?
It's when a model calls a tool that doesn't exist, passes an argument name or value that isn't real, or describes a tool result it never received. ToolScan lists seven concrete patterns, from incorrect function names to repeated calls (arXiv 2411.13547).
Does strict mode stop hallucinated function calls?
Partly. Strict mode makes the call match your schema, and on Anthropic's API it also guarantees the tool name is one you supplied (Strict tool use). It can't tell whether a value is true, so you still need preconditions and a validation step for made-up arguments.
Should I put tool definitions in the system prompt?
No. Pass them through the API's native tools parameter. In MCPVerse, prompt-based tool calling pushed one model's hallucination rate above 70% and cut its Standard-mode score from 62.36 to 15.10 (arXiv 2508.16260).
Terms in this article
From the library
Rather have someone set it up for you?
That's a fair call. LTFI, the agentic business operating system from Kief Studio, can set up the work these guides describe and run it with you. No website yet? Kief Studio will build you one for free.

