Evals Are the New Unit Tests
An eval is a repeatable test that scores an LLM system's outputs against a defined notion of correct, filling the role unit tests play in deterministic software. Because the same prompt can yield different completions, a single pass/fail assertion is meaningless; instead you run a suite of hundreds of cases and gate on aggregate statistics: pass rate, mean rubric score, and score deltas against the last release. A practical suite mixes exact checks where they exist (JSON schema validity, SQL that actually executes, required citations present) with graded scoring where they do not. The architect's point is blunt: the eval suite is the executable definition of what working means for the system. Teams that skip it end up doing vibe checks in staging, so every prompt tweak, model upgrade, or retrieval change ships on faith. Budget eval construction as a first-class deliverable, typically 20 to 30 percent of initial build effort.
Eval Methods: Golden Sets, LLM-as-Judge, Rubrics
Three method families dominate practical evaluation. Golden sets pair curated inputs with reference outputs or acceptance criteria; 100 to 300 well-chosen cases sampled from real traffic beat ten thousand synthetic ones, because coverage of actual failure modes is what matters. LLM-as-judge uses a strong model to grade outputs against a rubric, which scales cheaply but inherits known biases: judges prefer longer answers, favor their own model family, and drift with answer position, so you randomize order, pin the judge model version, and write rubrics with explicit score anchors instead of a bare 1 to 10 scale. Pairwise comparison is more reliable than absolute scoring for subjective quality. The non-negotiable step is human calibration: periodically double-score a sample and measure judge-human agreement with Cohen's kappa or plain accuracy. A judge that agrees with your experts about 90 percent of the time is a usable instrument; one you never calibrated is a random number generator with confidence.
Evaluating Agents & Retrieval Systems
Agents and RAG pipelines fail in ways a single input-output score cannot see, so you evaluate the trajectory, not just the final answer. For agents that means scoring each step: did it pick the right tool, were the arguments valid, did it recover from an error, did it stop instead of looping. Tool-call correctness is often checkable programmatically, comparing emitted calls against a schema and an expected call sequence. For retrieval, measure the components separately: recall and precision at k on a labeled query set for the retriever, then faithfulness and answer relevance for generation, because a wrong answer built on correct context is a prompt problem while a wrong answer built on wrong context is an index problem. End-to-end task success rate is the right headline metric and the one executives ask for, but it is diagnostic only when paired with stepwise evals; a 70 percent success rate says nothing about where the other 30 percent died.
Tracing & Observability in Production
Tracing instruments each LLM request as a tree of spans: the user turn at the root, with child spans for every prompt render, model call, tool invocation, and retrieval step, each carrying inputs, outputs, token counts, latency, and cost. This is ordinary distributed tracing applied to a new payload, and the OpenTelemetry GenAI semantic conventions now standardize the attribute names (gen_ai.request.model, token usage, tool-call spans), so instrumentation is not locked to one vendor; platforms such as Langfuse, Arize Phoenix, and LangSmith consume the same shape. Traces are also the raw material for everything else in LLMOps: production failures become eval cases, token telemetry feeds the cost dashboard, and latency percentiles drive timeout and fallback design. The enterprise concern is data handling: traces contain prompts and user content, so they need the same PII controls, access restrictions, and retention policy as application logs, decided before launch rather than after the first audit.
Versioning & Regression: Prompts, Models, Pipelines
In an LLM system the behavior-defining artifact is not just code: it is the tuple of prompt template, model version, decoding parameters, retrieval index, and tool definitions, and a change to any one of them can shift output quality. Version them together as a single release unit: prompts in source control rather than in a dashboard someone edits on a Friday afternoon, models pinned to explicit snapshots instead of floating aliases, and index builds stamped with the embedding model and corpus version that produced them. Every change to any element triggers the regression suite: run the eval set, diff scores against the baseline, and require sign-off when regressions exceed a threshold. Model upgrades deserve particular suspicion, because a model that improves on public benchmarks can still regress your specific tasks; the upgrade path is eval first, then a canary on a few percent of traffic with automated comparison, then ramp. Silent provider-side model updates are a real supply-chain risk worth addressing contractually.
Hallucination & Failure Analysis
Hallucination is not a single failure mode, and treating it as one makes it unfixable; useful analysis starts with a failure taxonomy. Separate at minimum: fabrication (claims with no source), faithfulness errors (contradicting the retrieved context), instruction violations, format breaks, unwarranted refusals, and tool misuse, because each demands a different fix. The working method is error clustering: pull failed or low-scored traces from production, embed and cluster them, or have a model label them against the taxonomy, then rank clusters by frequency times business impact. Most teams find a Pareto shape: three or four clusters account for the large majority of failures, and the top one is frequently a retrieval or prompt defect rather than a model limitation. The flywheel that separates mature teams from the rest is feeding every confirmed production failure back into the eval set as a permanent regression case, so the suite grows to mirror reality instead of the launch-day imagination of it.
CI/CD & Release Management for AI Systems
CI/CD for AI keeps the familiar pipeline shape and adds a statistical quality gate: alongside unit tests, the pipeline runs the eval suite against the candidate (new prompt, model, or index) and blocks the merge if aggregate scores regress beyond tolerance. Because evals cost real money and minutes, teams tier them: a fast smoke set on every commit, the full suite on release candidates, and nightly runs against the live provider to catch upstream drift. Deployment then proceeds in stages: shadow traffic first, where the candidate processes real requests without serving users and its outputs are scored offline; then a canary at 1 to 5 percent with online signals (thumbs-down rate, escalations, guardrail triggers) watched against thresholds; then ramp. Define rollback triggers numerically in advance, because mid-incident nobody wants to debate what worse means. Since prompts, models, and indexes are versioned artifacts, rollback is a pointer flip, and protecting that property is the point of the whole discipline.