Most LLM features ship without any real evaluation. Someone tries a handful of prompts, the output looks good, and it goes to production. Then a model upgrade, a prompt tweak, or a retrieval change silently degrades quality, and nobody notices until a user complains. Evaluation is the discipline that turns "looks good to me" into a number you can track over time and gate merges on.
This guide covers how to build that system: what to measure offline versus online, how to assemble a golden dataset, when to use deterministic metrics versus an LLM-as-a-judge, how to design rubrics that don't lie to you, and how to wire it all into CI. If you want the conceptual grounding first, read the LLM evals glossary entry.
Why eval matters more than it looks
The core problem with LLMs is that their behavior is non-deterministic and their failure surface is enormous. A traditional unit test checks one input against one expected output. An LLM feature has effectively infinite valid inputs and a fuzzy notion of "correct" output. You cannot enumerate correctness with assertions, so you need a statistical view: over a representative sample, how often does the system do the right thing, and did that rate move when you changed something?
Without eval you are flying blind on every change. Prompt edits, temperature changes, model version bumps, retrieval index rebuilds, and even provider-side model updates all shift behavior. If you're choosing between providers, that decision needs eval data too — see our OpenAI vs Anthropic vs Gemini API comparison for where the differences actually show up. Eval is the only way to make those changes with evidence instead of vibes.
Offline vs online eval
These are two different jobs and you need both.
Offline eval runs against a fixed dataset before you ship. It's reproducible, cheap to run repeatedly, and it's what you gate CI on. The dataset is a frozen set of inputs with known-good expectations. Because the inputs are fixed, a score change between two runs is attributable to your code change, not to traffic drift. Offline eval answers: "Did this change make things worse on cases I already care about?"
Online eval runs against live production traffic. It's the only way to catch problems your dataset never anticipated — new user phrasings, edge cases, distribution shift. Online eval usually can't use ground-truth labels (you don't know the right answer in real time), so it leans on proxy signals: implicit feedback (did the user retry, thumbs-down, abandon?), guardrail violation rates, and sampled human or LLM-judge review of production outputs. Online eval answers: "Is quality holding up in the real world, and what should go into the next version of my golden dataset?"
The relationship is a loop. Online eval surfaces failure modes; those failures become new golden-dataset cases; offline eval then prevents regression on them forever.
Building a golden dataset
The golden dataset is the foundation. Get this wrong and every metric downstream is noise.
Start small and real. Fifty to a hundred hand-curated cases drawn from actual usage beat a thousand synthetic ones. Each case is an input plus whatever "correct" means for your task: an exact answer, a set of acceptable answers, a reference document, or a rubric to grade against.
Cover the distribution deliberately. Include the common happy path, but weight toward the cases that hurt: ambiguous queries, adversarial inputs, out-of-scope requests you want refused, and the specific failure modes you've already seen in production. A dataset that's all easy cases will report 98% and tell you nothing.
Version it like code. Store it in the repo (or a versioned store), review changes in PRs, and never edit a case silently. When you add a case because of a production incident, that's a permanent regression test. For retrieval-heavy systems, your dataset also needs to pin the retrieved context — otherwise you're evaluating the retriever and the generator at once and can't tell which one broke. See how to build a RAG pipeline for keeping those layers separable.
Deterministic metrics vs LLM-as-a-judge
Two families of scoring. Reach for the cheaper, more reliable one first.
Deterministic metrics are code, not models. Exact match, regex/schema validation, JSON parse success, presence of required fields, string containment, numeric tolerance, or a task-specific checker. They're free, instant, perfectly reproducible, and unbiased. Use them whenever the task has structure: classification, extraction, function-calling arguments, format compliance, refusal detection. A surprising amount of "LLM quality" reduces to "did it return valid, correctly-typed output," which is a deterministic check.
def score_extraction(output: dict, expected: dict) -> float:
required = {"invoice_number", "total", "due_date"}
if not required.issubset(output):
return 0.0
hits = sum(1 for k in required if output.get(k) == expected.get(k))
return hits / len(required)
LLM-as-a-judge uses a model to grade output, and you need it for open-ended tasks where no deterministic check applies: summarization quality, tone, helpfulness, factual grounding against a source. The judge reads the input, the output, and a rubric, then returns a score and a justification.
JUDGE_PROMPT = """You are grading an assistant's answer.
Question: {question}
Reference facts: {reference}
Answer: {answer}
Score 1-5 on faithfulness to the reference facts ONLY.
5 = every claim supported. 1 = contradicts the reference.
Return JSON: {{"score": <int>, "reasoning": "<one sentence>"}}"""
Judge outputs are themselves LLM outputs, so treat them with suspicion. Ask for a justification (it improves scoring and gives you an audit trail), use a coarse scale (1–5 beats 1–100, where models can't discriminate meaningfully), and pin the judge model version so your grades don't drift when the provider updates the model.
Rubric design
A good rubric is specific, single-axis, and anchored. Don't ask a judge for "quality" — decompose it into named dimensions like faithfulness, relevance, and completeness, and score each separately. Give concrete anchors for what each score means ("5 = every claim traceable to the source; 3 = mostly grounded, one unsupported claim; 1 = fabricated content"). Vague rubrics produce judges that regress to the mean and rate everything a 3 or 4. This is also where eval connects to hallucination work: a faithfulness rubric graded against a reference is your primary detector for fabricated claims, and reducing them is its own discipline — see how to reduce LLM hallucinations.
Validate the judge against humans before you trust it. Have a person grade 20–30 cases, run the judge on the same cases, and check agreement. If the judge disagrees with humans more than humans disagree with each other, fix the rubric before you rely on the number.
Regression gating in CI
The payoff is automation. Run offline eval on every PR that touches prompts, model config, or retrieval, and fail the build if scores drop.
- name: Run eval suite
run: python -m eval.run --dataset golden/v3.jsonl --out results.json
- name: Gate on regression
run: python -m eval.gate --results results.json --baseline main --threshold 0.02
Gate on a delta from the baseline branch, not an absolute number. Because judge scores have variance, allow a small tolerance so noise doesn't flap the build, and run enough cases that the mean is stable. Post per-case diffs on the PR so a reviewer can see which cases regressed, not just that the aggregate moved. The failing cases are usually more informative than the score.
Honest tradeoffs and pitfalls
Judge bias is real. LLM judges favor longer answers, prefer outputs in their own style, and can be swayed by confident phrasing regardless of correctness. Using the same model family as both generator and judge amplifies this. Mitigate with a different judge model, blind position-swapped comparisons for pairwise grading, and periodic human recalibration.
Datasets go stale. A golden dataset frozen a year ago tests last year's product. Usage shifts, features change, new failure modes appear. Budget recurring work to fold online-eval failures back into the dataset, and treat a dataset that hasn't changed in months as a warning sign, not a success.
Cost compounds. LLM-as-a-judge means every eval run is more inference. A large dataset times several judge dimensions times every PR times a frequently-changing team adds up fast. Keep judge use for what genuinely needs it, put deterministic checks first as a cheap gate, and consider a smaller cheaper judge model for high-volume online sampling with the stronger judge reserved for release gates.
Metrics can drift from reality. A number that goes up while users get unhappier is worse than no number. Periodically spot-check that your eval score still correlates with actual user satisfaction. Eval is a proxy — keep checking that the proxy still points at the thing you care about.
Start with ten real cases and one deterministic check wired into CI this week. That beats a perfect eval framework you never ship.