Every serious codebase gates merges on tests and types. Then we bolt a probabilistic component into the middle of the system, one that changes behavior when a provider ships a new snapshot or someone rewrites a prompt, and we gate that with nothing but vibes and a demo query.
The result is a class of regression that no unit test can see. Retrieval recall drops from 0.93 to 0.87 because someone "improved" the chunking. Answers get subtly worse after a model version bump. The test suite is green the whole time, because the tests verify the plumbing, and the plumbing is fine.
The live demo on this site is guarded against that class of bug the same way it is guarded against type errors: a gate in CI that runs on every pull request and can say no. This is the machinery, piece by piece.
The golden set: small, real, versioned
The foundation is a golden dataset: questions paired with the chunks and answers a correct system should produce, drawn from real documents. Ours lives in the repo next to the code, versioned with it, seeded with a fixtures file so the suite can score offline.
Size matters less than honesty. Twenty to fifty question-answer pairs that reflect what users actually ask will catch more regressions than five hundred synthetic ones generated by the same model family you are trying to evaluate. Start embarrassingly small and grow it from production queries that went wrong.
Judge cassettes: deterministic and free
The standard objection to evals in CI is cost and flakiness: an LLM judge means API calls on every PR, bills that scale with commit frequency, and a gate that fails randomly when the provider hiccups. A flaky, metered merge gate gets deleted within a month, usually by the person it blocked unfairly.
The fix is the same one HTTP testing solved years ago: cassettes. Record the judge's scoring responses once, commit the recordings, and replay them in CI. Every run after that is offline, deterministic, and costs nothing. The routing layer has a mock override so CI can run cassette-only, with no network path to a model at all.
The trade-off is staleness: when the golden set or scoring rubric changes, cassettes must be re-recorded, and that is a deliberate, reviewed act. That is a feature. Your quality bar should change by commit, not by whatever mood the judge model is in this Tuesday.
The delta comment: the number lives next to the diff
Output matters as much as measurement. Our eval workflow posts a comment on the pull request with the score delta against main. A reviewer looking at a chunking refactor sees, in the same scroll, the code change and what it did to recall@k.
This changes review conversations. "Looks fine to me" becomes "this loses 4 points of recall for 80ms of latency, is that the trade we want?" The number turns a taste debate into an engineering decision.
One boring, hard-won detail: sanitize what goes into that comment. Eval output flows from dataset files into rendered markdown on your PR, which is an injection path. Ours also needed an allowlist on which dataset paths the eval endpoint may read, because "evaluate an arbitrary file" is an arbitrary-file-read primitive wearing a lab coat. Eval infrastructure is production infrastructure, with the attack surface to match.
What the gate has actually caught
The point of the machinery is the day it says no, or the day it lets you say yes quickly.
When the demo migrated embeddings to gemini-embedding-001 at 768 dimensions, the migration shipped only after an eval spike showed retrieval held up. When the routing policy moved the whole demo to Gemini, the suite's assertions moved with it in the same commit, because the gate forces routing changes and quality checks to travel together. And when the model serving the demo was retired upstream, the recovery was a config change plus a green eval run, not a week of "does it still work?" spot-checking.
That last one is the underrated payoff. The gate does not just block bad changes. It makes good changes cheap, because "prove it still works" is one CI run instead of an afternoon of manual queries.
Score the stages, not the vibe
A single end-to-end score tells you something broke, never what. The suite scores retrieval on its own terms: did the right chunks come back, at what rank. The RAG article walks the pipeline this gate protects; the short version is that recall is the ceiling on everything downstream, so it gets its own number, its own threshold, and its own delta in the PR comment.
Generation quality rides on top and is judged separately, which is exactly where the cassettes earn their keep: judging free-form answers is the expensive, nondeterministic part, and recording it is what makes the whole gate fast enough that nobody is tempted to skip it.
Where to start
If you run any LLM system without this, the on-ramp is a weekend, not a quarter. Twenty golden questions from real usage. A script that scores retrieval against them. A CI job that runs it on PRs and posts the delta. A threshold that blocks the merge. Grow each piece when it earns it.
The first article in this series put "an eval in front and a human behind" at the center of the honest method. This is the front half, mechanized. The runbooks and eval reports behind the client systems are in the operator brief, and the pipeline this particular gate protects is live right now.