LLM-as-a-judge means asking a model to grade output — its own or another model's — against a rubric. It exists because most useful AI output is open-ended: there is no single correct string for "summarize this call" or "answer this support question," so you cannot assert equality. A judge model can, at scale, say which of two answers is better and why.
Why it matters in production
It is what makes LLM evals practical for subjective tasks. Human grading does not scale to every commit; a well-prompted judge does. That is what lets us keep evals in CI even for tasks with no exact answer key.
The common mistake
Trusting the judge blindly. A judge model has its own biases — it favors longer answers, rewards confident tone, and can be swayed by position order in a pairwise comparison. If you never calibrate the judge against human labels on a sample, you are optimizing toward the judge's quirks, not real quality. A judge is a measurement instrument, and instruments drift.
How we actually use it
We calibrate the judge against a human-labeled slice first, use pairwise comparison rather than absolute scores where we can (models are more reliable at "A vs B" than "rate this 1-10"), and pin the judge model so its behavior does not shift underneath us during a model migration. It scores the answer; separate retrieval checks score whether the right context was even available.