---
title: "What is LLM-as-a-judge?"
excerpt: "Using one LLM to score another's output. It scales evaluation of open-ended answers — if you remember the judge is fallible and needs its own calibration."
definition: "LLM-as-a-judge is an evaluation technique that uses a language model to score or compare the outputs of another model against a rubric, enabling automated grading of open-ended responses that exact-match tests cannot handle."
date: "2026-07-09"
lastModified: "2026-07-09"
author: "Teo Deleanu"
authorAvatar: "/team/teo.jpg"
tags: ["Evals", "AI Engineering", "Production AI"]
keywords:
  - "what is llm as a judge"
  - "llm as a judge evaluation"
  - "using llm to grade llm output"
---

LLM-as-a-judge means asking a model to grade output — its own or another model's — against a rubric. It exists because most useful AI output is open-ended: there is no single correct string for "summarize this call" or "answer this support question," so you cannot assert equality. A judge model can, at scale, say which of two answers is better and why.

## Why it matters in production

It is what makes [LLM evals](/glossary/llm-evals) practical for subjective tasks. Human grading does not scale to every commit; a well-prompted judge does. That is what lets us keep [evals in CI](/blog/evals-as-ci) even for tasks with no exact answer key.

## The common mistake

Trusting the judge blindly. A judge model has its own biases — it favors longer answers, rewards confident tone, and can be swayed by position order in a pairwise comparison. If you never calibrate the judge against human labels on a sample, you are optimizing toward the judge's quirks, not real quality. A judge is a measurement instrument, and instruments drift.

## How we actually use it

We calibrate the judge against a human-labeled slice first, use pairwise comparison rather than absolute scores where we can (models are more reliable at "A vs B" than "rate this 1-10"), and pin the judge model so its behavior does not shift underneath us during a [model migration](/blog/surviving-a-model-migration). It scores the answer; separate [retrieval](/glossary/retrieval-augmented-generation) checks score whether the right context was even available.