---
title: "What are LLM evals?"
excerpt: "Evals are automated tests for non-deterministic model output. Without them you're shipping vibes. With them, prompt and model changes stop being scary."
definition: "LLM evals are automated tests that score a model or pipeline's outputs against expected behavior on a fixed dataset, so you can measure whether a change to prompts, models, or retrieval makes the system better or worse."
date: "2026-07-09"
lastModified: "2026-07-09"
author: "Teo Deleanu"
authorAvatar: "/team/teo.jpg"
tags: ["Evals", "AI Engineering", "Production AI"]
keywords:
  - "what are LLM evals"
  - "LLM evaluation explained"
  - "how to test LLM applications"
---

LLM evals are how you turn "this feels better" into "this scored 8 points higher on our test set." They are the unit tests of AI systems: a fixed dataset of inputs, an expected-behavior spec, and a scorer that tells you whether a change helped or hurt.

## Why it matters in production

LLM output is non-deterministic, so you cannot assert exact string equality the way normal tests do. Without evals, every prompt tweak is a gamble and every model upgrade is terrifying — you have no way to know what you broke. Evals convert that fear into a number, and a number you can gate a deploy on. This is exactly why we run [evals as CI](/blog/evals-as-ci): a change that regresses the score never merges.

## The common mistake

Waiting until "later" to build evals, then never building them. The second mistake is scoring only the final answer while ignoring retrieval — a RAG system can produce a fluent wrong answer because the [retrieval](/glossary/retrieval-augmented-generation) step failed, and an answer-only eval never catches it. Measure each stage.

## How we actually use it

We build a small, honest eval set early — real inputs, real failure cases — and grow it every time production surprises us. Where an answer is open-ended, we score it with an [LLM as a judge](/glossary/llm-as-a-judge) rather than brittle string matching. Evals are also the safety net that makes a [model migration](/blog/surviving-a-model-migration) a measured decision instead of a leap of faith.