Every team shipping an LLM feature hits the same wall: the model confidently states something that is simply false. A wrong price, a made-up API method, a citation to a paper that does not exist. This is hallucination, and the first thing to accept is that you reduce it, you do not eliminate it. An LLM generates the most probable next token given its training distribution and your prompt. It has no built-in concept of "true" versus "plausible." So the goal is a system that makes plausible-but-wrong output rare, cheap to catch, and hard to ship to a user.

The techniques below stack. Each one closes a different gap, and the more of them you apply, the lower your hallucination rate goes.

Start with grounding: give the model the facts

The single biggest lever is grounding. If the answer depends on facts the model was never reliably trained on, or on facts that change (your product docs, a customer's account state, today's prices), do not ask the model to recall them. Retrieve them and put them in the prompt.

This is the whole point of retrieval-augmented generation. You embed the user's query, pull the most relevant chunks from a vector store or a keyword index, and inject them into context with an instruction to answer only from the provided material. If you are building this from scratch, our guide to building a RAG pipeline walks through chunking, indexing, and retrieval.

Grounding alone is not enough, though. Two failure modes remain:

  1. The retrieved context does not actually contain the answer, and the model fills the gap from its parametric memory anyway.
  2. The context contains the answer, but the model paraphrases it into something subtly wrong.

You handle the first with an explicit escape hatch, and the second with verification. Both are below.

Let the model say "I don't know"

Models hallucinate partly because we implicitly reward them for always producing an answer. Flip that. Tell the model, in the system prompt, that abstaining is a correct and valued outcome when the context is insufficient.

Answer the user's question using ONLY the information in <context>.
If the context does not contain enough information to answer,
respond exactly with: "I don't have enough information to answer that."
Do not use outside knowledge. Do not guess.

This one instruction removes a surprising amount of confident nonsense. It works best when paired with grounding, because now the model has a clear signal (empty or irrelevant context) that maps to a clear action (abstain). Measure the abstention rate: if it is near zero on a query set that includes unanswerable questions, your instruction is being ignored and you need a stronger prompt or a stricter model.

Constrain the output shape

Free-form prose gives the model maximum room to improvise. Structured output narrows it. When you force the model to emit JSON matching a schema, every field becomes a specific question rather than an open invitation to write.

Most modern APIs support schema-constrained or "structured" generation, where the decoder is restricted to tokens that keep the output valid against your schema. Combine that with fields that carry evidence:

from pydantic import BaseModel

class Answer(BaseModel):
    answer: str
    supported_by_context: bool
    source_quote: str  # verbatim span from the retrieved context

# Pass Answer as the response schema to your provider's
# structured-output API, then reject any result where
# supported_by_context is True but source_quote is not
# actually a substring of the retrieved context.

The source_quote field is the trick. Asking the model to quote the exact span that supports its answer makes fabrication harder and gives you something to validate mechanically. If the quote is not a literal substring of what you retrieved, you caught a hallucination without a second model call.

Add guardrails and validation

Guardrails are the deterministic checks that sit between the model and the user. Structured output is one; there are more. Validate that referenced entities exist (does that product ID resolve? does that URL return 200?). Check numbers against your source of truth rather than trusting the model's arithmetic. Reject answers that cite sources not present in the retrieved set. For code generation, run the output through a linter or type checker before showing it.

The principle: anything you can verify with code, verify with code. A regex or a database lookup is faster, cheaper, and more reliable than a second LLM, and it never hallucinates. Reserve model-based checks for the fuzzy claims that resist mechanical validation.

Verification passes

For the claims code cannot check, add a verification pass. Three patterns, roughly in order of cost:

Self-check. Ask the same model to critique its own answer against the source. This is cheap and catches obvious contradictions, though a model that hallucinated a fact will sometimes defend it.

Second-model check. Route the answer and its context to a different model (or the same model in a fresh, unbiased context) with a narrow verification prompt. This is the LLM-as-a-judge pattern applied to factuality.

You are a fact-checker. Below is a CONTEXT and a CLAIM.
Reply "SUPPORTED" only if the CLAIM is fully entailed by the CONTEXT.
Reply "UNSUPPORTED" if any part of the CLAIM is not in the CONTEXT,
even if it seems true. Then give the sentence from CONTEXT that
supports it, or "none".

CONTEXT: {context}
CLAIM: {model_answer}

Claim-checking against sources. Decompose the answer into atomic claims and verify each one against the retrieved documents. This is the most thorough and the most expensive, so reserve it for high-stakes outputs (legal, medical, financial) where a single wrong claim is costly.

Verification passes trade latency and tokens for reliability. Apply them selectively: full claim-checking on the answers that matter, a lightweight self-check on the rest, and nothing on outputs a guardrail already validated.

Measure it, or you are guessing

None of this is worth much if you cannot tell whether it helped. Build a hallucination eval set: real queries paired with the correct answer and, ideally, the supporting source. Include unanswerable questions so you can measure whether the model correctly abstains instead of inventing.

Score each response for factual accuracy against the source. You can automate scoring with an LLM judge for scale, but calibrate it against a human-labeled sample first, because the judge hallucinates too. Track the numbers that matter:

  • Hallucination rate: answers containing at least one unsupported claim.
  • Abstention accuracy: correct "I don't know" on unanswerable questions, minus wrongful abstention on answerable ones.
  • Groundedness: fraction of claims traceable to the provided context.

Run this suite on every prompt change, model swap, and retrieval tweak. Our guide to evaluating LLMs covers how to build and maintain these sets so they stay honest as your system evolves.

The honest tradeoffs

Every layer here costs something. Retrieval adds infrastructure and latency. Abstention prompts occasionally make the model too cautious and refuse answerable questions. Structured output can fail on genuinely open-ended tasks. Verification passes multiply your token bill and slow responses. Evals take real engineering time to build and keep current.

So do not bolt on all of it blindly. Start with grounding and an abstention instruction, because together they are cheap and catch the most. Add schema constraints where your output is structured anyway. Add verification passes only on the outputs where a wrong answer actually hurts. And build the eval set early, because it is the only thing that tells you which of these layers is earning its keep. Reducing hallucination is not one fix; it is a budget you spend where the failures cost the most.