Evaluation Methods: How to Actually Score AI Output

The scoring problem

The case for evaluation is easy to make and easy to agree with. Nobody senior argues against measuring AI quality. The evals deep-dive covers that strategy layer: golden sets, gates, maturity. This article sits one level down, at the question that strategy quietly assumes is solved. Given a model output and some notion of what good looks like, how do you actually produce a number?

That question is harder than it sounds because the output is open-ended. A classifier emits one of five labels and you compare it to a known answer; scoring is a lookup. A support assistant emits a paragraph, and there are hundreds of correct paragraphs, phrased a thousand ways, some correct in substance but wrong in tone, some fluent and confidently false. There is no single reference string to compare against, so "did it match" is the wrong question. The right question is "how good is this, along the dimensions I care about," and that requires choosing a scoring method, each of which measures something slightly different from what you actually want.

The rest of this piece is a tour of those methods, from the cheap and rigid to the expensive and nuanced: deterministic reference checks, semantic similarity, rubric scoring and LLM-as-judge, and human review. For each, the useful knowledge is not how it works but where it fails, because every method is a proxy, and a proxy trusted past its blind spot is worse than no measurement at all.

Reference-based methods

Reference-based scoring compares an output against a known-correct answer. It splits into two families with very different characters. The first is deterministic checks: exact match, regular expressions, JSON schema validation, numeric tolerance. These are precise, free to run, and perfectly reproducible. If the task is "return valid JSON with these four fields" or "the answer must contain the order ID," a code-graded check is not just adequate, it is correct, and reaching for a model to grade it is waste dressed up as sophistication.

The second family is text-overlap metrics such as BLEU and ROUGE, which score how many words or n-grams an output shares with a reference. They were built for machine translation and summarization, where a reference translation is a reasonable target. Applied to open-ended generation they mislead in both directions: a paraphrase that is entirely correct scores low because it reuses few of the reference's exact words, and a fluent answer that overlaps heavily can still be wrong on the one fact that mattered. Overlap measures surface form, not correctness.

The rule of thumb: use deterministic checks aggressively as a first line, because a large fraction of what teams call "quality" is really format and constraint compliance that code can verify for free. Treat overlap metrics with suspicion for anything genuinely open-ended, and never let a rising ROUGE number stand in for correctness. When wording can vary but meaning must hold, you have outgrown reference-based scoring and need the next method.

Semantic similarity

Semantic methods answer the objection that ends the previous section. Instead of comparing surface words, they embed both the output and a reference answer into vectors with an embedding model, then measure the distance between them, usually cosine similarity. Two sentences that share almost no vocabulary but mean the same thing land close together; the paraphrase that ROUGE punished now scores as it should. For open-ended generation where the correct answer can be phrased many ways, this is a real improvement, and it is cheap enough to run on every case.

The blind spots are specific and worth memorizing, because they are the mirror image of the overlap problem. Embedding similarity captures topical and semantic closeness, not factual correctness. Two answers about the same subject, in the same register, that disagree on the decisive number will still sit near each other in vector space, because they are talking about the same thing. The metric rewards being on-topic and fluent, which is not the same as being right.

• Similar wording, wrong fact: "the refund window is 30 days" and "the refund window is 14 days" are near-identical embeddings and opposite answers.
• Reference dependence: the score is only as good as the reference answer, and for genuinely open tasks a single reference under-represents the space of correct responses.
• Model-bound blind spots: the similarity inherits whatever the embedding model does and does not distinguish, including domain terms it was never trained to separate.

Semantic similarity earns a place as a cheap, meaning-aware signal, a strict upgrade over n-gram overlap for open-ended text. It does not earn trust as a correctness check. It is best used to flag outputs that drift far from an expected answer, not to certify that a nearby one is true. When the thing you actually need to score is faithfulness, tone, or reasoning quality, similarity cannot see it, and you move to criteria-based scoring.

Rubrics and LLM-as-judge

Most quality dimensions that matter, such as faithfulness, helpfulness, tone, and completeness, are graded rather than binary, and no distance metric expresses them. The tool for these is a rubric: an explicit set of criteria with anchored levels that turn a vague judgment into a repeatable one. The economical way to apply a rubric at scale is LLM-as-judge, where a strong model reads the output and the rubric and returns a score. It comes in two shapes: pointwise, scoring one output against the rubric, and pairwise, deciding which of two outputs is better, which is the natural fit for model and prompt bake-offs.

Judges scale beautifully and they are biased in reproducible ways. The three with the strongest evidence: position bias, where in pairwise comparisons judges favor one slot, often the first; verbosity bias, where longer answers score higher at equal correctness; and a general tendency to reward confident fluency over accuracy. None are exotic, and any one can flip a bake-off's conclusion. The defenses are mechanical: randomize or swap ordering and discard cases whose verdict flips, anchor every rubric level in concrete language, and, above all, calibrate the judge against human labels before you trust a single number it produces.

  criterion: faithfulness to retrieved context
  ------------------------------------------------
  0  makes claims the context does not support
  1  mostly supported; one unsupported detail
  2  every claim traceable to the context
  ------------------------------------------------
  output order: randomized   |  judge != model
  calibrated vs human labels: kappa = 0.81
            

Calibration is the dividing line between a measurement and a guess. Have experts label 100 to 200 cases, measure agreement between judge and human (Cohen's kappa or plain percent agreement), and publish that figure beside every judge-scored metric. A judge that agrees with your experts most of the time is a useful instrument. One nobody checked is an opinion generator, and a judge model upgrade resets the baseline because it is an instrument change.

Human evaluation

Human expert review is the gold standard, and it is worth being precise about why. For nuanced quality, whether a legal summary is defensible, whether a clinical note is safe, whether an answer is genuinely helpful rather than merely plausible, a qualified human is the only instrument that measures the thing directly instead of a proxy for it. Everything cheaper in this article is ultimately calibrated against human judgment, which makes human eval both the most expensive rung and the reference standard the others are measured by.

The cost is real and it is not only money. Expert time is slow and scarce, so human eval does not fit inside a per-commit gate. And human judgment is not automatically reliable: two reviewers handed the same output and a vague instruction will disagree, and that disagreement is noise you have paid for. Turning human review into a measurement rather than a collection of opinions requires protocol discipline.

Three practices carry most of that discipline. First, grade against a written rubric with the same anchored levels a judge would use, so reviewers apply a shared definition rather than private taste. Second, use independent double-labeling on a sample, treating disagreements as rubric bugs to fix rather than noise to average away. Third, measure and report inter-rater agreement (Cohen's kappa or plain percent agreement); when agreement is low, the rubric, not the model, is what you are actually testing, and no downstream score built on it means anything.

The pattern that works is a division of labor. Humans set the standard on a modest, carefully labeled sample and periodically re-anchor it; automated methods run that standard at scale in between. Human eval is the survey benchmark you measure against a few times a year, not the tape you carry on every commit.

Being statistically honest

Every method above returns a number, and a number invites a conclusion. This is where most eval work quietly falls apart, because the number is a sample statistic and gets treated as a fact. A pass rate measured on 40 cases is an estimate with a wide confidence interval, and language models add their own run-to-run variance on top: the same prompt scored twice can differ. Report a bare "82 percent" without a sample size and a spread and you have published a point estimate as if it were the population truth.

The discipline is the ordinary statistics of comparison, applied honestly. When you claim system B beats system A, you are testing whether an observed gap is larger than the noise. On small sets, a two or three point difference is almost always noise, and teams that treat it as signal spend quarters chasing ghosts and shipping changes that did nothing. A few useful habits: size the set to the effect you need to detect, run enough repetitions to see the variance rather than assuming it away, use a paired comparison so each case faces both systems, and apply a significance test before you write "improved" in a deck.

Statistical honesty is unglamorous and it is what separates an eval program from an eval performance. It rarely changes the method you use; it changes what you are willing to claim from the method's output, and how many cases you insist on before claiming it. A modest true gain reported with its uncertainty is worth more than a dramatic one reported without, because only the first survives contact with production.

The architect view

No method here is the answer, and any team selling one as universal is selling a blind spot. The competence is in matching the instrument to the task and layering them by cost. Deterministic checks catch format and constraint violations for free on every commit. Semantic similarity flags meaning drift cheaply. Calibrated judges grade nuanced dimensions at scale. Human review sets and periodically re-anchors the standard the judges are measured against. Each rung covers a failure the cheaper rung below it cannot see, which is why a mature stack runs several at once rather than betting on one.

Two principles hold the layering together. Push every check to the cheapest rung that can honestly express it, because expensive methods spent on cheap questions are budget you no longer have for the questions that need them. And keep the expensive rungs as calibration for the cheap ones rather than running them everywhere: human eval is the standard, not the gate. The connecting tissue is statistical honesty, which decides what any of these numbers licenses you to claim.

These mechanics are what the strategy layer assumes. The evals deep-dive shows how the methods here become golden sets and release gates; RAG evaluation applies them to retrieval and faithfulness specifically; and versioning and regression is where statistical honesty stops being a virtue and becomes the thing that tells you, run over run, whether a change actually moved quality or just moved the noise. A working harness lives in the Eval Harness in the Lab.