---
title: "LLM evaluation methods"
excerpt: "A taxonomy of LLM evaluation: benchmarks, LLM-as-judge, human eval, and task-specific metrics — what each measures and when it applies."
date: "2026-07-09"
lastModified: "2026-07-09"
summary: "There is no single way to evaluate an LLM — there's a taxonomy of methods, each measuring something different. This explainer maps the landscape: public benchmarks, deterministic task metrics, LLM-as-a-judge, and human evaluation, with the strengths, blind spots, and correct use of each."
keyTakeaways:
  - "Evaluation methods form a taxonomy, not a hierarchy — benchmarks, deterministic metrics, LLM-as-judge, and human eval each answer a different question."
  - "Public benchmarks rank general capability but tell you little about your specific application; task-specific evaluation is what predicts production quality."
  - "Match the method to the task's structure: deterministic metrics for structured output, LLM-as-a-judge for open-ended quality, human eval as ground truth."
author: "Teo Deleanu"
authorAvatar: "/team/teo.jpg"
tags: ["LLM Evals", "AI Engineering", "Testing", "Production AI"]
keywords:
  - "llm evaluation methods"
  - "llm benchmarks"
  - "llm as a judge"
  - "human evaluation llm"
  - "task specific llm metrics"
  - "how to evaluate language models"
---

There is no single number that tells you whether an LLM is good. "Good" depends entirely on what you're asking it to do, and the field has developed a whole taxonomy of evaluation methods, each measuring a different thing under different assumptions. Confusing them is the source of most bad evaluation: teams cite a benchmark score to justify a production decision the benchmark says nothing about, or reach for an expensive human panel to check something a regex would settle in a millisecond. This explainer maps the landscape so you can pick the right instrument for the question you're actually asking.

If you want the conceptual grounding first, the [LLM evals glossary entry](/glossary/llm-evals) is the primer; for the hands-on version of wiring this into a pipeline, [how to evaluate LLMs](/guides/how-to-evaluate-llms) is the procedural companion to this taxonomy.

## The first split: capability versus application

The most important distinction, and the one most often blurred, is *what you're evaluating*.

**Capability evaluation** asks how good a model is in general — its reasoning, knowledge, coding, and language ability in the abstract. This is what public benchmarks and leaderboards measure, and it's what you consult when *choosing* a base model.

**Application evaluation** asks how good *your system* is at *your task* — your prompts, your retrieval, your model, on your data. This is what predicts whether users are well-served, and it's what you consult when *shipping and maintaining* a feature.

A model can top every leaderboard and still be wrong for your application because your task, data, and constraints differ from the benchmark's. The methods below split along this line. Use capability evals to choose; use application evals to ship. Never substitute one for the other.

## Public benchmarks

Benchmarks are standardized test sets — MMLU for broad knowledge, GSM8K and MATH for math reasoning, HumanEval for code, and many others — that produce a comparable score across models. Their value is real: they let you compare models on a level field, they track the field's progress over time, and they're a reasonable *first* filter when picking a base model or deciding whether a new release is worth evaluating further.

Their blind spots are just as real, and you must hold them in mind. **Contamination** is endemic: benchmark questions leak into training data, so a high score can reflect memorization rather than capability, and scores inflate over time for reasons unrelated to real improvement. **Benchmarks measure the benchmark, not your task** — MMLU accuracy tells you nothing about how the model handles your customer's phrasing, your document format, or your output schema. And **a single aggregate hides the distribution** — a model can post a great average while failing systematically on the exact slice you care about. Treat benchmarks as a coarse screen for model selection, never as evidence about your application. If you're weighing providers, benchmark scores are one input among several — [OpenAI vs Anthropic vs Gemini API](/compare/openai-vs-anthropic-vs-gemini-api) covers the ones benchmarks miss.

## Deterministic task-specific metrics

When your task has structure, the best evaluation is code, not judgment. Deterministic metrics compare output against a known-correct answer with a fixed rule, and they should be your default whenever they apply.

The family is broad: exact match and accuracy for classification; precision, recall, and F1 for extraction and retrieval; schema/JSON validation and format compliance for structured output; numeric tolerance for calculations; execution-based checks for code (does the generated function pass the tests?); string-overlap metrics like BLEU or ROUGE for translation and summarization where a reference exists. For [RAG](/glossary/retrieval-augmented-generation) specifically, retrieval quality has its own deterministic metrics — hit rate, recall@k, and mean reciprocal rank measure whether the right passages were fetched, independently of what the model then did with them.

Their virtues are decisive where they fit: free, instant, perfectly reproducible, and unbiased. The catch is that they only work when "correct" is well-defined, and they can be brittle — a semantically perfect answer that differs from the reference by a synonym scores zero under exact match. So use them for what has structure, and don't force them onto open-ended tasks where surface overlap doesn't equal quality. A surprising fraction of "LLM quality" reduces to "did it return valid, correctly-typed output," which is a deterministic check hiding in plain sight.

## LLM-as-a-judge

For open-ended output — summaries, explanations, conversational tone, factual grounding — there's often no reference answer and no deterministic rule. [LLM-as-a-judge](/glossary/llm-as-a-judge) uses a capable model to grade output against a rubric, and it's the workhorse method for quality that resists code.

It comes in two shapes. **Pointwise** scoring grades one output against a rubric (score this answer 1–5 on faithfulness). **Pairwise** comparison asks which of two outputs is better — often more reliable, since relative judgments are easier than absolute ones, and it's the basis of most model-vs-model quality comparisons. Its strength is scale: it approximates human judgment on subjective quality at a fraction of the cost and time, which makes it the only practical way to grade thousands of open-ended outputs per day.

But a judge is an LLM, so it inherits LLM failure modes, and using it well means respecting them. **Judge bias is documented**: judges favor longer answers, prefer outputs in their own style, and can be swayed by confident phrasing over correct content. Using the same model family as generator and judge amplifies the effect. Mitigations that matter: require a written justification (it improves the score and gives an audit trail), use a coarse scale (1–5, not 1–100, where models can't discriminate), swap positions in pairwise grading to cancel order bias, pin the judge's model version so grades don't drift, and — non-negotiable — **validate the judge against human labels** before trusting it. If the judge disagrees with humans more than humans disagree with each other, the rubric is broken.

## Human evaluation

Human judgment is the ground truth every other method approximates. When a person rates outputs, runs a blind A/B preference test, or does expert review in a domain like law or medicine, you're measuring the thing itself rather than a proxy. Nothing else captures nuance, safety edge cases, and genuine helpfulness as faithfully.

The cost is why it can't be your everyday gate: humans are slow, expensive, and — without careful rubrics and inter-rater agreement checks — inconsistent. So human eval plays specific, high-leverage roles rather than running on every commit. It **calibrates** your automated methods (the human-labeled set that validates your LLM judge). It **arbitrates** the decisions too important to trust to automation — a major release, a safety-critical behavior. And it **discovers** failure modes your automated evals can't see because they weren't designed to look for them; those discoveries become new deterministic checks and new judge rubrics. Treat human eval as the calibration standard and the court of last resort, not the assembly line.

## Choosing a method: match the instrument to the question

The taxonomy resolves into a decision, and the decision is driven by two things: the task's structure and the question you're asking.

Start with structure. If the task has a well-defined correct answer — classification, extraction, structured output, code, retrieval — use **deterministic metrics**. They're the cheapest, most reliable option, and you should exhaust them before reaching for anything model-based. If the output is open-ended and quality is subjective, use **LLM-as-a-judge**, validated against humans. Reserve **human evaluation** for calibrating the judge, arbitrating high-stakes calls, and hunting unknown failures.

Then layer in the question. Choosing a base model? **Benchmarks** for a first screen, then a small application-specific eval to confirm on your task. Shipping and maintaining a feature? An **application eval** on a golden dataset — mostly deterministic checks, LLM-judge where needed — gating your CI, with human review as the periodic calibration.

In practice a mature system runs several of these at once: deterministic checks as a fast, cheap first gate; LLM-as-a-judge for the open-ended dimensions; human review on a sampled slice to keep the automated methods honest. The methods aren't rivals competing for the title of "best eval." They're complementary instruments, and the skill is knowing which one answers the question in front of you — and never letting a benchmark score stand in for the application evidence that actually decides whether you ship.

## A word on cost and the online/offline split

One axis cuts across the whole taxonomy: *when* the evaluation runs. **Offline evaluation** runs against a fixed dataset before you ship — reproducible, repeatable, and the thing you gate CI on. **Online evaluation** runs against live production traffic and catches what your dataset never anticipated: new phrasings, distribution shift, real-world edge cases. Online eval rarely has ground-truth labels, so it leans on proxy signals — implicit user feedback (retries, thumbs-down, abandonment), guardrail-violation rates, and sampled judge or human review of production outputs. The relationship is a loop: online eval surfaces failures, those failures become new offline test cases, and offline eval then prevents them forever.

Cost is the constraint that shapes all of this. Deterministic metrics are effectively free, so run them everywhere. LLM-as-a-judge means every eval run is more inference — a large dataset times several judge dimensions times every pull request adds up fast, so reserve the strong judge for release gates and use a cheaper judge (or pure deterministic checks) for high-volume online sampling. Human eval is the most expensive by far, so spend it where it has the most leverage: calibrating the cheaper methods and arbitrating the decisions that matter most. A good evaluation strategy isn't the one with the most rigor everywhere; it's the one that spends each method's cost where that method's signal is worth it.