Evals Are the New Unit Tests: Building a Quality System for AI

Why “it looks good” doesn’t scale

A deterministic function either passes its unit test or it does not. A language model gives you a distribution: run the same prompt twice and you can get two different answers, both plausible, one subtly wrong. That single property breaks the quality assumptions most engineering organizations are built on. Code review cannot catch a regression that appears in 4 percent of outputs. Manual QA cannot re-check ten thousand behaviors every time a prompt changes.

The failure mode that actually hurts is the silent regression. You swap in a newer model version to cut latency, or reword a system prompt to fix one complaint, and a behavior three features away quietly degrades. Nothing crashes and no alert fires. The first signal is a customer escalation weeks later, and by then you cannot tell which of the last eleven changes caused it. Teams that ship on vibe checks ("we tried a few prompts, it looks fine") are not lazy; they simply have no other instrument. But vibes do not scale. They live in the heads of two or three senior people, which caps team size, and every release needs those people's attention, which caps cadence.

Evals fix this by converting taste into a repeatable gate. An eval is a defined input set, a scoring method, and a threshold. It runs the same way for the newest hire and the CTO, at 2 a.m. and before the board demo. Over time something more interesting happens: the suite stops being a test artifact and becomes the spec. When a stakeholder asks what the assistant actually guarantees, the honest answer is whatever the suite measures, at the pass rate it currently holds.

The eval taxonomy

Evaluation methods form a ladder from cheap-and-rigid to expensive-and-nuanced. The engineering skill is matching each behavior you care about to the cheapest rung that can actually measure it. Format compliance never needs a human; clinical tone in a healthcare assistant never trusts a regex.

Two disciplines keep the ladder honest. First, push every check down to the cheapest rung that can express it: a surprising fraction of "quality" is schema validity, length bounds, and citation presence, all code-gradable. Second, use the expensive rungs to calibrate the cheap ones, not to run in every pipeline. Human review is a measurement standard, not a release gate. The broader catalog of methods lives in Eval Methods.

Building your first golden set

A golden set is a versioned collection of inputs paired with expected outcomes or labeled reference answers. It is the single highest-return artifact an AI team can build, and most teams overthink it. The right first move is to sample real traffic: pull a few hundred production queries (redacted as your privacy review requires), cluster them by intent, and select cases proportionally so the set mirrors what users actually ask. Then add synthetic edge cases deliberately, because real traffic under-represents exactly the inputs that break systems.

• Representative cases sampled from production, stratified by intent and difficulty.
• Edge cases: ambiguous requests, very long inputs, adversarial phrasing, mixed languages.
• Out-of-scope probes: questions the system should refuse or route elsewhere.
• Known failures: every production incident becomes a permanent case, the AI analog of a regression test written from a bug report.

On size: 30 to 50 well-chosen cases will catch gross regressions, the kind where a prompt edit drops task success by 20 points. That is enough to start gating releases this quarter. A few hundred cases is where statistical confidence begins; below that, a two or three point difference in pass rate is typically noise, and teams that treat it as signal end up chasing ghosts. Grow the set before you grow your faith in small deltas.

Label each case against a written rubric, not against one reviewer's intuition, and have two people label a sample independently; their disagreements are rubric bugs, and fixing them is where the real specification work happens. Version the set like code, because a score is meaningless without knowing which revision produced it. And schedule refreshes: a golden set frozen for a year measures last year's product.

LLM-as-judge, done right

Using a strong model to grade another model's outputs is the only economical way to apply rubric scoring at scale, and it works well enough that most mature eval stacks depend on it. It is also a measurement instrument with known, reproducible biases, and teams that ignore them ship dashboards that are precisely wrong.

The three biases with the strongest evidence behind them: position bias (in pairwise comparisons, judges systematically favor one position, often the first, with the direction varying by judge model), verbosity bias (longer answers score higher at equal correctness), and self-preference (models rate outputs from their own family more favorably). None of these are exotic; all of them will distort an uncontrolled bake-off enough to flip its conclusion.

• Randomize order in pairwise comparisons, or run both orderings and average; discard cases where the verdict flips with position.
• Score against an explicit rubric with anchored levels ("a 2 means the answer is correct but omits the required disclaimer"), never a bare "rate this 1 to 10".
• Use a different model family as judge than the one being evaluated, where procurement and data policy allow it.
• Calibrate against humans: have experts label 100 to 200 cases, measure judge-human agreement (Cohen's kappa or plain percent agreement), and publish that number next to every judge-scored metric.
• Recalibrate on change: a judge model upgrade is an instrument change and resets your baselines.

The calibration number is the part most teams skip and the part skeptical stakeholders should ask for first. A judge that agrees with your experts 90 percent of the time is a useful instrument; one you never checked is an opinion generator with a per-token price.

Evaluating agents and RAG

For agents, end-to-end task success is the headline number: given a realistic goal, did the agent finish the job? But a single pass/fail per task tells you the failure rate, not the failure location. Add step-level checks: tool-call correctness (did it pick the right tool with valid arguments), trajectory efficiency (steps taken versus a reference path, since an agent that succeeds in 40 calls instead of 6 is a cost and latency bug that task success hides), and recovery from injected failures (return a timeout or a malformed payload mid-run and verify the agent retries, re-plans, or degrades gracefully instead of hallucinating a result).

RAG systems demand the same decomposition. Measure retrieval quality (did the right passages reach the context window, expressed as recall and precision at k against labeled relevant chunks) separately from generation quality (is the answer faithful to the retrieved context, and does it actually address the question). Conflating them is the most common RAG evaluation mistake, because the fixes live in different subsystems.

 user query
     |
     v
+-----------+   top-k chunks   +-----------+   answer
| RETRIEVER | ---------------> | GENERATOR | --------> user
+-----------+                  +-----------+
     |                              |
 measure here:                 measure here:
 recall@k                      faithfulness
 precision@k                   answer relevance
            

This is the payoff of component metrics: they localize failures. Low recall sends you to the index and chunking strategy; low faithfulness with high recall sends you to the prompt or the model. Detailed patterns are in Evaluating RAG and Agent & RAG Evals.

Wiring evals into CI/CD

The mechanical rule: anything that changes model behavior is a code change and gets gated like one. That includes prompt edits, model version bumps, retrieval index rebuilds, embedding model swaps, temperature changes, and tool schema updates. A pull request that touches a prompt file should trigger the eval suite automatically, and a score below threshold should block the merge with the same finality as a failing unit test. If the gate can be skipped with a shrug, you do not have a quality system, you have a dashboard.

Model swaps deserve a heavier path because they change everything at once. The sequence that works: full offline eval suite first, then shadow traffic (the candidate model runs against a mirror of production requests, scored offline, with no user exposure), then a canary (a few percent of live traffic, watched against automated quality and latency metrics, with instant rollback). Shadow mode is where you discover the behaviors your golden set never covered; every surprise it surfaces should become a new case before the canary starts.

Production is the last gate and the first source of new tests. Sample live outputs continuously, judge-score them, and alert on drift in score distributions or in the input mix itself. Every confirmed production failure flows back into the golden set, which is how the suite compounds. Pipeline patterns are cataloged in CI/CD for AI, and a working reference implementation lives in the Eval Harness in the Lab.

An eval maturity path

Eval capability grows in recognizable stages, and it is worth being honest about where your organization sits before buying tools for a stage you have not reached.

1. Level 1, manual spot checks. Engineers paste prompts into a playground before release. Quality knowledge lives in individuals. Regressions are found by customers.
2. Level 2, golden set in CI. A versioned set of 30 to a few hundred cases runs automatically on every behavior-changing merge, with a blocking threshold. Releases stop depending on who is in the room.
3. Level 3, judge-scored rubrics plus component metrics. Calibrated LLM judges grade nuanced dimensions at scale; agent and RAG pipelines report step-level and retrieval metrics that localize failures. Quality becomes debuggable, not just detectable.
4. Level 4, production feedback loop. Live traffic is continuously sampled and scored, drift triggers alerts, and confirmed failures flow back into the suite automatically. The system measures itself.

As of mid-2026, most enterprises running LLM features in production sit at level 1, with pockets of level 2 in their strongest teams. That is not primarily a tooling gap; the harness is a weekend of work. The gap is organizational. Level 2 requires someone to own the golden set as a product. Level 3 requires a labeling budget and the discipline to publish judge-agreement numbers. Level 4 requires data pipelines, privacy sign-off to learn from production traffic, and an on-call culture that treats a quality drift alert as seriously as a latency page.

The compounding effect is the strategic argument. A three-year-old eval suite refreshed from thousands of real production failures encodes what your customers actually need in a form no competitor can scrape, prompt, or license. Models will keep changing underneath you. The suite is the asset that makes every one of those changes safe to take, and it is the closest thing to a moat that applied AI offers.