Skip to content
Search
Applied8 min read

Why Does AI Make Things Up? Three Checks That Catch It

Meelie Gagne
Why Does AI Make Things Up? Three Checks That Catch It

TL;DR: AI makes things up because it writes the most likely next words, and when it doesn't know something it tends to guess instead of saying so. You can't switch that off with a prompt, but three checks help you catch it before it reaches anyone else: trace every fact to a source you open yourself, make the assistant quote the exact sentence it relied on, and ask the key facts again in a separate chat.

You asked an AI assistant for a summary, a statistic or a list of sources, and it gave you a confident answer. Then a colleague pointed out that one of the numbers, names or references didn't exist. If you're wondering why AI makes things up, you're asking about what researchers call a hallucination. This guide explains in plain language why it happens, where invented details tend to hide, and three checks that catch them before your work reaches anyone else.

Why AI makes things up

It predicts words, it doesn't look up facts

The assistant in your browser runs on a large language model. The US National Institute of Standards and Technology describes the problem this way: confabulation, its formal term for hallucination, is "The production of confidently stated but erroneous or false content (known colloquially as 'hallucinations' or 'fabrications') by which users may be misled or deceived" (NIST AI 600-1, Generative AI Profile).

NIST also explains where it comes from. Confabulations "are a natural result of the way generative models are designed: they generate outputs that approximate the statistical distribution of their training data; for example, LLMs predict the next token or word in a sentence or phrase." Most of the time the likeliest words are also true. When they aren't, the sentence still reads smoothly, so nothing about it looks wrong.

It's rewarded for guessing

A 2025 research paper puts the second cause simply: "Like students facing hard exam questions, large language models sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty." The authors argue that the way models are trained and scored rewards a confident guess over "I don't know" (Kalai et al., "Why Language Models Hallucinate", arXiv 2509.04664). On a test that only marks answers right or wrong, a guess sometimes earns a point and "I don't know" never does.

It goes along with your question

A study of legal questions, later published in the Journal of Legal Analysis, found that models "often fail to correct a user's incorrect legal assumptions" (Dahl et al., "Large Legal Fictions", arXiv 2401.01301). If your question contains a wrong assumption, such as a policy that doesn't exist or a date that's off by a year, the answer may build on it instead of pointing it out.

It may not know what's recent

Each model learns from text collected up to a date called its knowledge cutoff. Unless the tool searches the web or reads documents you give it, it can't know about anything after that date, and it may fill the gap with something that sounds current.

Where invented details hide

Hallucinations usually look like ordinary, confident text. They hide in the small, specific details that make text look trustworthy:

  • Numbers: statistics, totals, percentages and prices
  • Names: people, companies, products and job titles
  • Dates and deadlines
  • Quotes, especially ones attributed to a real person
  • Sources: book titles, report names, court cases and web links. An invented reference has its own name, a hallucinated citation.

NIST notes that models "may also include confabulated logic or citations that purport to justify or explain the system's answer". A neat explanation or a reference list doesn't prove the answer is right.

How often this happens depends heavily on the task. In that legal study, which tested four models available in 2023, asking specific, verifiable questions about random federal court cases produced legal hallucinations between 58% of the time (GPT-4) and 88% of the time (Llama 2) (Dahl et al.). Newer models may do better, but the pattern still holds: the risk climbs when the assistant has to supply facts from memory. Everyday tasks such as rewording your own email carry less risk, because you supplied the facts.

You also can't just ask the assistant whether it's sure. The same study found evidence that models "cannot always predict, or do not always know, when they are producing legal hallucinations." That's why the checks below rely on sources and separate questions instead of the model's own confidence.

Check 1: Trace every fact to a source you open yourself

This is the check that matters most. Before anything goes to your manager, a client or the public, list every number, name, date, quote and reference in it, and find each one in a source you open yourself.

  • If the assistant gave a link, open it and find the exact sentence. A real page that doesn't say what the assistant claims is as bad as a fake page.
  • If it named a report or study, search for the title and check the author and year.
  • If a figure has no source at all, either find one or take the figure out.

It's slow the first time. It gets quicker once you notice that most of a draft is wording, and only a handful of facts in it need checking.

Check 2: Make it quote the exact sentence

A safer way to use AI with facts is to give it the facts yourself. Paste the document, report or email thread, and ask it to answer only from that text and quote the sentence behind each answer. This is called grounding. Then you check each quote against the original, which takes seconds with your browser or document's Find command.

The Prompt:

Answer the questions below using ONLY the document I've pasted.

For each answer:
1. Give a short answer.
2. Quote the exact sentence from the document that supports it,
   word for word, in quotation marks.
3. If the document doesn't answer the question, write
   "Not in the document" and nothing else.

Do not use anything you know from outside this document.

Questions:
1. [your question]
2. [your question]

Document:
[paste the document, with personal details removed]

Why This Works: Asking for a word-for-word quote gives you something you can check with a single search. Allowing "Not in the document" gives the assistant an honest answer to fall back on, so it has less reason to guess. If a quote doesn't appear in your document, you've caught an invented answer.

Before you paste a work document, strip out personal data and secrets. Qurtoo's free PII redactor swaps them for placeholders in your browser and puts them back in the answer afterwards.

Check 3: Ask the key facts again, separately

For the two or three facts that matter most, open a new chat (a temporary one, if your assistant remembers past chats) and ask about each one on its own, without showing the first answer. Then compare.

This copies the idea behind a research method called Chain-of-Verification. The model drafts an answer, plans verification questions, "answers those questions independently so the answers are not biased by other responses", and then writes a final answer. The authors report that this reduced hallucinations across several kinds of tasks (Dhuliawala et al., 2023, arXiv 2309.11495; our summary of the paper).

The Prompt:

I'm checking a few facts. Answer each question on its own.
If you don't know or aren't sure, say "Not sure" instead of guessing.

1. [the first key fact, phrased as a neutral question]
2. [the second key fact, phrased as a neutral question]

Why This Works: A fresh chat that can't see the first answer can't simply agree with it. If your assistant remembers past chats, use a temporary or private chat with memory off for this check. Neutral wording ("When did the policy change?" instead of "The policy changed in March, right?") avoids leading the model toward your assumption.

A match raises your confidence, but it doesn't prove the fact, because two answers can agree and both be wrong. If they disagree, you've found a fact that needs Check 1.

Which check to use when

  • Rewording your own text: read it once against your original. The facts came from you, so look for anything that changed.
  • Summarizing a document you have: Check 2, then spot-check a few quotes.
  • Anything with facts the assistant supplied from memory: Check 1 on every fact, with Check 3 on the ones that matter most.
  • Anything legal, medical, financial or public-facing: Check 1 on everything, and a person who knows the subject reads it before it goes out.

Telling the assistant "don't make anything up" doesn't do the job of these checks. Our article on why "never hallucinate" doesn't work explains why, and the verification prompt pattern shows how technical teams build the same checks into their systems.

Frequently asked questions

Why does AI make things up?

AI assistants write the most likely next words based on their training, and they don't look facts up unless the tool is connected to search or documents. Research also suggests that the way models are trained and scored rewards a confident guess over admitting uncertainty, so when they don't know, they sometimes guess.

Can you stop AI from hallucinating?

Not completely, but you can make it less likely and easier to catch. Give the assistant the source material, ask it to quote the sentence behind each answer, let it say "not in the document", and check every name, number and reference yourself before you use it.

Why does AI make up sources and citations?

A reference list is a pattern the model has seen many times, so it can produce titles, authors and links that look right without checking whether they exist. Open every link and search every title before you rely on a source an assistant gave you.

How do I know if AI is making things up?

Usually you can't tell from the writing, because invented details read just as smoothly as true ones. Look hardest at numbers, names, dates, quotes and sources, and confirm each one in a source you open yourself.

Does asking AI "are you sure?" help?

Not reliably. Research on legal questions found that models don't always know when they're producing hallucinations, so a confident "yes" proves little. Asking the question again in a fresh chat, or asking for a quote from a document you supplied, gives you something you can actually check.

Rather have someone set it up for you?

That's a fair call. LTFI, the agentic business operating system from Kief Studio, can set up the work these guides describe and run it with you. No website yet? Kief Studio will build you one for free.

Go further on kief.studio