- The release unit is no longer just code. A prompt edit, a model version bump, or a rebuilt retrieval index can change behavior with no code diff, so the pipeline has to version and gate all of them.
- Deterministic unit tests do not fit non-deterministic output. Eval suites become the merge gate, with threshold-based pass/fail and regression sets that guard against silent quality drops.
- Ship AI changes the way you ship code: offline eval, then canary with shadow traffic, then online eval and tracing, with automatic rollback wired to metric regression.
Why AI breaks classic CI/CD
Classic CI/CD rests on an assumption so foundational that most engineers never state it: behavior changes only when code changes. The commit is the release unit, the diff is the review surface, and a green test suite means the build behaves as specified. Every tool in the pipeline, from code review to bisecting a regression, inherits that assumption. AI systems break it.
In an LLM application, the behavior you ship is a product of the code plus a prompt, a model version, a set of generation parameters, and often a retrieval index. Any of those can change while the code stays byte-for-byte identical. A colleague rewords a system prompt to fix one complaint. A provider silently rotates the weights behind a model alias. An overnight job rebuilds the vector index with a new chunking strategy. In each case the application behaves differently in production and git diff shows nothing. The instrument your organization trusts to answer "what changed" goes blind exactly when you need it.
The second break is non-determinism. A deterministic function passes its unit test or it does not. A language model returns a distribution: run the same prompt twice and you can get two plausible answers, one subtly wrong. Assertion-style tests, which demand an exact expected output, either flake constantly or get loosened until they assert nothing. You cannot gate a stochastic system with a check designed for a deterministic one.
The new release artifacts
If a change to any input can change behavior, then every such input has to become a first-class release artifact: named, versioned, pinned, and reviewed, rather than a runtime setting someone can edit in a console. The discipline is not exotic. It is the same instinct that moved configuration out of hand-edited servers and into version control, applied to the new surfaces that now steer an AI system.
Four artifacts carry the behavior and belong under the same rigor as source code. Pin each to an explicit version, record which versions shipped together, and require a reviewed change to move any of them.
| Artifact | What it pins | Failure if left as a runtime setting |
|---|---|---|
| Prompts and templates | System prompts, few-shot examples, tool descriptions, output schemas | Console edits change behavior with no review, no history, no rollback path |
| Model version and config | Exact model build, temperature, top-p, max tokens, stop sequences | A floating alias reroutes to new weights overnight; params drift per environment |
| Retrieval index | Embedding model, chunking strategy, index snapshot, source corpus version | An index rebuild silently changes answers; you cannot reproduce yesterday's output |
| Eval datasets | Golden sets, regression cases, rubrics and thresholds | A score means nothing without knowing which dataset revision produced it |
Two rules make the discipline pay off. First, pin, never float: a model alias like latest is a promise that your behavior can change without your involvement. Bind to an explicit build and upgrade deliberately. Second, version the bundle, not just the parts. A release is a specific prompt revision, against a specific model build, over a specific index snapshot, judged by a specific eval set. Record that tuple so any production behavior is reproducible and any regression is bisectable across the surfaces that actually moved.
Eval suites as the merge gate
Once you accept that behavior-changing inputs are release artifacts, you need a gate that can judge them. Assertion-style unit tests cannot: they demand an exact output, and the output legitimately varies. The replacement is an eval suite wired into the merge, treated with the same finality as a failing build. A pull request that touches a prompt, bumps a model, or rebuilds an index runs the suite automatically, and a score below threshold blocks the merge. If the gate can be waved through with a shrug, it is a dashboard, not a gate.
Because output varies run to run, the gate is threshold-based rather than binary. You are not asserting a single correct string; you are measuring a pass rate over a set of cases and comparing it to a baseline. That reframes the whole exercise as statistics. Small deltas are usually noise, so the gate should fire on a meaningful drop against the current baseline, not on a one or two point wobble that a rerun would erase. The evals deep-dive covers scoring methods, golden sets, and how to size a suite for real statistical confidence.
The regression set is the part that earns its keep over time. Every confirmed production failure becomes a permanent case, so the suite accumulates hard-won knowledge of exactly how your system has broken before. That is what catches the silent regression: the reworded prompt that fixes one complaint and quietly degrades a behavior three features away. Nothing crashes, no alert fires, and only a case that already encodes the old failure will notice.
The pipeline, end to end
Put the pieces in sequence and a shape emerges that mirrors classic delivery while adding the controls non-determinism demands. Each stage catches a different class of problem, and the value comes from their order: cheap checks first, exposure to real traffic last, and a rollback path that stays open the whole way through.
commit
|
v
+---------------+ below threshold
| OFFLINE EVAL |-----------------------+
+---------------+ |
| pass |
v v
+---------------+ regression +----------+
| GATE |----------------->| BLOCK / |
+---------------+ | ROLLBACK |
| merge +----------+
v ^
+---------------+ metric regress |
| CANARY |-----------------------+
+---------------+ |
| healthy |
v |
+---------------+ drift / alert |
| ONLINE EVAL |-----------------------+
+---------------+
| stable
v
PROMOTE
Read the stages by what each one catches. Offline eval runs the suite against golden and regression sets before anything ships, catching gross quality drops on known cases at inference cost only. The gate converts that score into a merge decision, so a bad change never reaches an environment. The canary exposes the change to a small, watched slice of live traffic, catching what the offline set never covered: real input distributions, latency under load, and interactions with downstream systems. Online eval keeps scoring once traffic is flowing, catching drift that only appears at scale or over time.
The property that makes the pipeline trustworthy is that promotion is the only path forward and rollback is always reachable. A change earns its way to full traffic by clearing each gate; it does not arrive there by default and get pulled back on complaint. That inversion, promote on evidence rather than roll back on incident, is the whole point.
Progressive delivery for models
A model or prompt change is riskier than a typical code change because it alters behavior across every input at once, and the offline suite only ever samples that space. Progressive delivery manages the gap by limiting exposure while real traffic tells you what the eval set could not. The techniques are borrowed from mature software delivery and adapted to the fact that the thing you are watching is output quality, not just error rate.
| Strategy | How it works | When to reach for it |
|---|---|---|
| Shadow / mirrored traffic | Candidate runs on a copy of live requests; output is scored, never served | Highest-risk changes, especially model swaps; surfaces uncovered behaviors with zero user exposure |
| Canary | A small percentage of live traffic hits the candidate, watched against quality and latency metrics | Default first exposure once shadow is clean; catches real-traffic issues cheaply |
| Percentage rollout | Traffic share ramps in steps, each held long enough to read metrics | Promoting a healthy canary while keeping blast radius bounded at every step |
| Automatic rollback | A metric regression past threshold reverts to the last good version without a human in the loop | Always on, underneath every strategy above; the safety net that makes the rest affordable |
Shadow mode is the underused technique and the most valuable for model swaps. Because the candidate never serves a user, you can run a full mirror of production against it and score the results offline, discovering exactly the behaviors your golden set missed. Every surprise it surfaces should become a new eval case before the canary starts, so the suite is stronger by the time real users are involved.
Automatic rollback is what makes the whole approach economical rather than nerve-wracking. If reverting a bad candidate requires a human to notice, diagnose, and act, the cost of a failed rollout is an incident. If a regression past a defined threshold reverts on its own in seconds, a failed rollout is a non-event. Wire the trigger to the same metrics the canary watches, and the pipeline defends itself faster than an on-call engineer can open a dashboard.
Online evaluation and rollback
Offline eval is necessary and insufficient. It can only measure what its cases cover, and production traffic is a wider, stranger distribution than any curated set: inputs you never imagined, phrasings your rubric never anticipated, and interactions with downstream systems that only exist in the live environment. Online evaluation closes that gap by continuously sampling real outputs, scoring them the way the offline suite does, and watching the score distribution and the input mix for drift over time.
Scoring alone tells you that quality moved but not why. Tracing supplies the why. When an online score dips, a trace lets you follow a single request through retrieval, prompt assembly, the model call, and any tool invocations, so you can localize the regression to the subsystem that actually broke rather than guessing across the whole stack. Online eval is the smoke detector; tracing is the map that tells you which room is on fire. The tracing and observability deep-dive covers how to instrument that path end to end.
The loop only closes if rollback triggers are defined in advance, not negotiated during an incident. Decide, before you ship, which signals revert a release and at what level.
- Quality regression: online score drops past a set margin against the pre-release baseline on a stable input mix.
- Safety breach: any regression in the refusal or policy-violation rate, treated as a hard trigger with no threshold to debate.
- Operational signal: latency, error rate, or cost per request crosses its budget, since a correct answer that arrives too slowly or too expensively is still a regression.
- Feedback signal: a sharp move in thumbs-down rate, escalations, or task-abandonment on the canary cohort.
Confirmed production failures do not just trigger a rollback; they flow back into the regression set. That is how the pipeline compounds: each escape becomes a permanent guard, and the offline gate that missed it once will catch its recurrence forever.
The architect view
The failure mode to design against is a team that treats prompts and models as settings and the eval gate as a favor. It works until a silent regression reaches a customer, and then it works for no one. The architect's job is to make the safe path the easy path, so that shipping an AI change without an eval gate is not a matter of discipline but something the platform simply does not offer.
That means building the gates into the paved road rather than asking each team to assemble them. The delivery platform should version prompts, models, indexes, and eval sets as a bundle by default; run the offline suite on every behavior-changing merge; carry a change through shadow, canary, and staged rollout with automatic rollback wired underneath; and score live traffic with tracing attached. When eval-driven promotion is the default template a team inherits, quality stops depending on who happened to remember it.
None of this pays off without versioning discipline underneath it. Pin models to explicit builds, keep prompts in review-gated source control, snapshot indexes, and record which versions shipped together, so every production behavior is reproducible and every regression is bisectable across the surfaces that actually move. The organizational shift, from console edits to reviewed artifacts, is harder than the tooling and matters more.