You shipped RAG. The vector store returns the right PDFs. The bot still invents a refund policy that does not exist.

That is not a retrieval mystery. The generation prompt never forced the model to use what it was given.

Most teams treat the RAG prompt as glue: dump chunks, ask a question, hope. The model already "knows" things from training. When the prompt does not put a hard contract on the context, parametric knowledge wins. Or the model fuses two policies into a third rule nobody wrote. Or it cites a chunk that does not support the claim.

This post is about the generation contract. The part that decides whether retrieved text is used, cited, or quietly ignored.

The honest numbers (not the headline fantasy)

There is no clean study that says "better RAG prompts cut hallucinations by 80%." Keep that title as a target for how wrong naive systems get, not as a measured result you can put in a client proposal.

What the research does show:

Even with relevant context, strong models often fail to clear about 80% answer correctness on common long-context RAG benchmarks. Roughly one answer in five is still wrong when retrieval is already in the ballpark. Wood and Forbes (arXiv:2412.05223) cite Databricks' long-context RAG work on that ceiling.

Legal tools did not escape this. Stanford-linked evaluation of commercial legal research assistants found hallucinations on 17–33% of queries, with the best system around ~65% accuracy. "Grounded" in marketing copy is not the same as grounded in production.

Prompt patterns do move the needle with measured gains, just not a magic 80% reduction. Meta's Chain-of-Verification (CoVe) cut fabricated list items hard on Wikidata-style tasks and lifted FActScore from 55.9 to 71.4 (~28% relative). Self-RAG's reflection loop improves citation quality and long-form factuality versus naive RAG. Strict "answer only from context / refuse otherwise" prompts cut forced wrong answers in controlled settings.

So the real thesis is narrower and more useful: when retrieval is right, prompt structure decides whether the model grounds, invents, or merges.

Why good chunks still get ignored

Four failure modes show up again and again.

Parametric fill-in. You never gave an exact refuse string. The model treats "helpful" as "finish the answer from memory."

Lost in the middle. Liu et al. (arXiv:2307.03172) showed multi-document QA peaks when the gold passage sits at the start or end of the context and drops when it sits in the middle. Dumping top-k in raw rank order is a silent accuracy tax.

False synthesis. Two policies conflict. The model invents a compromise. You never said "list both; do not reconcile."

Entity collision. Passages about calcium. Question about magnesium. The model binds the wrong properties to the wrong noun. Acurai's RAGTruth walkthrough is the clean example: retrieval was fine; packaging was not.

Retrieval bugs still exist. Parsing splits table cells. Query language does not match document vocabulary. The wrong page ranks first. When that happens, a perfect grounding prompt can still "faithfully" answer from bad evidence (Shi’s 2026 PDF RAG writeups make that point hard). You need logs of retrieved spans next to answers so you can tell which brick failed.

When the spans are right and the answer is wrong, fix the prompt contract.

Naive prompt vs grounded contract

Before (typical production default):

You are a helpful support assistant. Use the context below to answer the user.

Context:
{{retrieved_chunks}}

User question: {{question}}

That invites fill-in, silent merges, and citation theater ("Source: policy.pdf" with no supporting span).

After (hard ground + refuse + conflict rule + citation map):

The Prompt:

You answer ONLY from the numbered Context below.

Rules:
1. Use only facts that appear in Context. Do not use prior knowledge.
2. After every factual claim, cite the chunk id like [2].
3. If sources conflict, list each position with its citation. Do not merge them into one rule.
4. If the Context does not contain enough information to answer, reply with exactly:
   Not in the provided sources.
   Do not guess. Do not partially invent.

Context:
[1] {{chunk_1}}
[2] {{chunk_2}}
[3] {{chunk_3}}

Question: {{question}}

Answer format:
- Direct answer (or the exact refuse line)
- Supporting quotes: short spans from the cited chunks

Why This Works: Exact refuse text blocks "helpful" fill-in. Numbered citations make audits mechanical. The conflict rule stops fake policy synthesis. Quotes force the model to touch the chunks before it summarizes.

Expected Output:

Refunds are available within 14 days of purchase for unused licenses [1]. Extended hardware warranties are non-refundable after activation [2].

Supporting quotes: [1] "Software license refunds: 14 days from purchase if unused." [2] "Activated hardware warranties cannot be refunded."

Note: Sources disagree on restocking fees for returned accessories ([1] 15%, [2] no restocking fee). Both are listed; no single fee is applied.

If the chunks never mention refunds:

Not in the provided sources.

Evidence-first and typed refusal

"Cite your sources" is not enough. Models invent plausible citations. Force span-level support, or force a structured refusal.

The Prompt:

Task: Answer the question using ONLY Context.

Procedure:
1. Find the shortest quote from Context that supports the answer. Copy it verbatim.
2. If no supporting quote exists, set complete_answer_found = false and answer = null.
3. If a quote exists, set complete_answer_found = true and write answer using only that evidence.
4. confidence: high | medium | low based on whether the quote fully covers the question.

Return JSON only:
{
  "complete_answer_found": boolean,
  "answer": string | null,
  "evidence": [{"chunk_id": string, "quote": string}],
  "confidence": "high" | "medium" | "low"
}

Context:
[1] {{chunk_1}}
[2] {{chunk_2}}

Question: {{question}}

Why This Works: Missing data becomes a boolean, not a smooth paragraph. Structured outputs (schema-forced generation) make this reliable in production without fine-tuning Self-RAG.

Expected Output:

{ "complete_answer_found": false, "answer": null, "evidence": [], "confidence": "low" }

That pattern matches the generation fix in enterprise PDF demos where models invent a 2026 oil price from a 2025 row: without complete_answer_found, the model pads. With it, silence is valid.

Reorder before you generate

After retrieval, do not paste rank order blindly.

Put the top one or two passages first and last. Park lower-ranked chunks in the middle. That lines up with the U-curve in Lost in the Middle.

Cheap rule in code:

Score/rank as usual.
Build context as: [best, second_best, ...rest..., second_best, best] (or best first, rest, best last if length is tight).
Cap total tokens; prefer fewer strong chunks over a long middle soup.

This is not a wording trick. It is prompt layout. Same chunks, different attention.

Light CoVe for multi-fact answers

When the user wants a list of features, vendors, or dates, single-shot RAG invents extras. CoVe structure:

Draft answer from context.
List verification questions for each claim.
Answer each verification question from context only, in a separate pass if you can.
Final answer keeps only claims that survived.

On multi-fact tasks, CoVe’s measured win is fewer fabricated items. For SMBs, a two-pass version is enough: draft, then "for each bullet, quote or delete."

Entity isolation when names look alike

If the question mixes similar entities (two products, two minerals, two clients), split the call. One entity per generation. Rewrite messy PDF chunks into short subject–predicate lines before the model sees them when binding errors keep recurring.

That is packaging, not a bigger embedding model.

How to tell prompt failure from retrieval failure

Log three things next to every answer:

The exact question
The chunk ids and short quotes the model claimed
Whether a human (or a second model pass) finds those quotes in the returned text

If the quotes are real and the answer still invents, fix the contract. If the quotes are wrong or empty while the "right" PDF never ranked, fix chunking, parsing, or ranking. Do not spend a week rewriting system prose for a parser that split a table cell.

Five-minute audit for a client or internal bot

Open the production prompt and check:

Is there an exact refuse string (not "if unsure, say so")?
Are chunks numbered, and are citations required per claim?
Is there a conflict rule (list both; do not merge)?
Do you reorder for edge placement of top evidence?
For hard domains, do you force evidence spans or structured refusal before free text?
Do you keep a 10–20 question eval set with known-good answers and known-missing answers?

If you sell "grounded AI" to regulated clients, residual error rates in legal research tools (still ~1 in 6 on hard queries in published work) are the reality check. Prompt contracts plus verification beats a shinier model alone.

What to ship today

Start with hard ground + refuse + conflict + numbered citations. Add typed JSON refusal for support bots that must not invent. Reorder chunks. Split entity-colliding questions. Log evidence next to answers.

RAG without a generation contract is retrieval bolted onto hope. The patterns above are small, copy-pasteable, and model-agnostic across Claude, GPT, Gemini, and Llama. Differences show up mainly in how strictly each model follows schema and refuse strings -- test your exact stack on the missing-answer cases first.

If your team is building client knowledge bases or support bots and wants live practice on grounding contracts, eval sets, and failure triage, connect with Kief Studio on Discord or schedule a session.

RAG prompt engineering

Grounding Prompts With Retrieved Context: The RAG Prompt Patterns That Cut Hallucinations by 80%