- Five stacked layers (experience, orchestration, model access, knowledge, data foundation) plus a cross-cutting governance plane give every GenAI team one paved road instead of a private stack.
- The model gateway is the highest-leverage control point: one interface for auth, routing, caching, quotas and telemetry makes swapping models a configuration change, not a code change.
- Adoption, not compliance, is the success metric: ship golden-path templates, run the platform like a product, and measure the share of inference traffic that flows through the road.
Why you need a reference architecture
Walk into any large enterprise 18 months into its GenAI program and you will find the same landscape: a dozen pilots, each wired by hand. One team calls a provider API with a key pasted into a config file. Another picked a different provider and built its own logging. A third implemented guardrails as a stack of regular expressions someone wrote on a Friday. Each decision was locally reasonable. The aggregate is a system nobody can see: no unified cost view, no consistent audit trail, and an incident response process that starts with archaeology.
The costs multiply quietly. Five teams each spend weeks rebuilding retry logic, token accounting and prompt versioning, which is waste you can at least estimate. The risks are worse because they are invisible until they are not: an unlogged prompt that carried customer data to a third party, an API key with no spend ceiling, a model deprecation that forces a code change in nine applications simultaneously. None of these show up in a status report until one of them becomes the status report.
A reference architecture is the countermeasure, and it works by economics rather than enforcement. It is a shared blueprint that names the layers of a GenAI system, fixes the interfaces between them, and assigns each cross-cutting concern (identity, logging, evals, cost control) to exactly one place. It deliberately does not mandate a single vendor or framework; it mandates where decisions live, so they are made once instead of a dozen times.
The rest of this article walks the layers concretely, names the control point in each, and closes with how to drive adoption without turning the platform team into the department of no.
The six layers at a glance
The architecture stacks five layers and cuts one plane across all of them. Each boundary between layers is an interface you standardize; each layer is a place where one team can own a concern on behalf of every other team.
+---------------------------------------------+ +----------+ | L5 EXPERIENCE | | | | chat, copilots, embedded UX, plugins | | GOVERN- | +---------------------------------------------+ | ANCE | | L4 ORCHESTRATION | | PLANE | | workflows, agents, state, tool registry | | | +---------------------------------------------+ | identity | | L3 MODEL ACCESS | | guard- | | gateway: authN/Z, routing, cache, quota | | rails | +---------------------------------------------+ | evals | | L2 KNOWLEDGE & RETRIEVAL | | tracing | | ingestion, hybrid search, entitlements | | cost | +---------------------------------------------+ | model & | | L1 DATA FOUNDATION | | prompt | | sources, pipelines, quality, lineage | | registry | +---------------------------------------------+ +----------+
Experience to orchestration. The experience layer owns the human contract: interfaces, latency budgets, streaming behavior, accessibility. Everything below it should be invisible to users. The boundary is a task or session API, so a chat surface, an IDE plugin and a batch job can all invoke the same orchestration without reimplementing it.
Orchestration to model access. Orchestration composes steps; it never holds provider credentials. Every inference call, from every workflow and agent, goes through the model gateway. This is the boundary that turns model choice into configuration.
Model access to knowledge. Retrieval is a service with an API, not a library each application embeds. Callers pass their identity with every query so the knowledge layer can enforce entitlements, covered below.
Knowledge to data foundation. The data foundation is your existing estate: warehouses, document stores, SaaS systems. The knowledge layer consumes it through ingestion pipelines but does not own the sources. The governance plane is deliberately drawn as a plane, not a layer: you do not pass through it, it touches everything.
The orchestration layer
Orchestration is where a request becomes a plan: which steps run, in what order, with which tools and which model calls. The first design decision is the split between workflows and agents. A workflow is a predefined path where model calls are steps in code you wrote; an agent lets the model decide the sequence at runtime. As of mid-2026, most enterprise value still ships as workflows, because they are testable, debuggable and predictable in cost. Agents earn their place on genuinely open-ended tasks. The reference architecture supports both on the same substrate, so a workflow can graduate to an agent without a rewrite.
Whatever executes on top, this layer owns a specific set of responsibilities that individual applications must not reimplement:
- State and memory: conversation state, long-running task checkpoints, and resumability after failure.
- Tool access: a registry of approved tools and MCP connectors, each with an owner, declared scopes and review status.
- Reliability mechanics: retries with backoff, timeouts, and fallback paths when a step fails.
- Human approvals: a standard gate for consequential actions, so "a person confirms before the write" is a platform feature, not per-app code.
- Task decomposition: fan-out and join for multi-step work, with budgets enforced per task, not per call.
The framework question matters less than teams think, provided you contain it. The framework landscape churns on a roughly annual cycle, so wrap orchestration behind your own service interface and treat the framework as an implementation detail. Teams that let framework types leak into their application contracts are signing up to rewrite those applications when the ecosystem shifts, and it will shift.
The model layer: gateway as control point
If you build only one piece of this architecture, build the model gateway: a single service between every caller and every model provider. It is the highest-leverage control point in the stack because every token of inference passes through it, which means every cross-cutting concern can be enforced there exactly once.
| Gateway function | What it does | Cost of skipping it |
|---|---|---|
| AuthN/Z | Callers present workload identity, not provider keys | Keys sprawl across repos and config files |
| Routing & fallback | Maps use cases to models; fails over on provider errors | Every outage is an application incident |
| Caching | Serves repeated calls without re-inference | Paying repeatedly for identical work |
| Quotas & budgets | Per-team and per-use-case spend ceilings | Cost surprises found in the monthly invoice |
| Telemetry | Logs every call: model, tokens, latency, caller | No audit trail, no cost attribution |
The strategic payoff is the model portfolio. With one interface in front of all providers, a model swap is a routing-table change: you can run a cheap, fast model as the default and escalate hard cases to a stronger or reasoning-class model per use case, A/B two models on live traffic, and retire a deprecated model without touching application code. Budget logic needs one wrinkle for reasoning models: their thinking tokens are billed as output tokens, so a quota defined on input size alone will underestimate spend, sometimes badly.
Keep the gateway thin. It enforces policy; it does not hold business logic, prompts or retrieval code, which belong in the layers above. And be conservative with caching: exact-match caching is safe and cheap, while semantic caching (serving a cached answer for a similar question) trades correctness for savings and should be opt-in per use case, never a silent default.
The knowledge and retrieval layer
This layer turns the enterprise's documents and records into something a model can use, and it is where most quality problems in RAG systems actually originate. The work starts with ingestion pipelines: parsing, chunking and enrichment as a managed, versioned process rather than a notebook someone ran once. Typical chunks run a few hundred tokens with roughly 10 to 20 percent overlap, but the higher-leverage moves are in enrichment. Contextual retrieval, described by Anthropic in September 2024, prepends a short chunk-specific snippet of document context to each chunk before embedding (situating that chunk within its source document) and substantially reduces retrieval failure rates; parent-child retrieval matches on small chunks but returns the larger parent section, giving the model coherent context instead of fragments.
Retrieval itself should be hybrid by default: dense vector search for semantic similarity fused with keyword search for exact terms, part numbers and names that embeddings blur, usually with a reranking stage on top. Pure vector search demos well and then fails on precisely the queries enterprise users care about most, the ones containing identifiers.
Finally, freshness is a discipline, not a property. Pin the embedding model version and re-embed the corpus when it changes, because vectors from different models do not mix. Propagate deletions from source systems on a defined SLA; a document deleted for legal reasons must not live on in the index. Define staleness budgets per corpus: policy documents may tolerate a day, pricing data may not.
The governance plane
Governance fails when it is a stage at the end of the pipeline: a review board that meets after the system is built. In this architecture it is a plane that touches every layer, and its controls live at the boundaries already described, where they execute on every request rather than once per quarter.
- Identity for humans and agents. Every request carries who asked (the human) and what is acting (the workload or agent, with its own identity, distinct from the user's). Both propagate through all layers, which is what makes entitlement-aware retrieval and per-team cost attribution possible at all.
- Guardrails at the input and output boundaries. Input screening for injection and data exfiltration attempts, output checks for policy and PII, enforced in the platform path so an application cannot forget them.
- Evals as release gates. A prompt or model change ships only after passing its evaluation suite in CI, exactly like tests gate code. This is established practice, and it is the single control that most reduces regressions in production behavior.
- Tracing and cost telemetry. Every step of every request is traceable end to end, with token spend attributed to team and use case.
- Model and prompt inventory. A registry of every model, prompt version and dataset in production, with an owner for each.
The inventory deserves emphasis because it looks bureaucratic and is not. When a regulator, customer or auditor asks "where does AI make decisions in your business, and on what data," the inventory is the difference between an answer and a scramble. It is also the mechanism that makes deprecation tractable: you cannot retire a model you cannot enumerate the consumers of.
None of these controls is novel. What the plane adds is placement: each one enforced once, in the platform path, instead of reimplemented with varying rigor by every team.
Making it real: paved roads, not mandates
The architecture on the page is the easy half. The failure mode of the hard half is predictable: publish the diagram, mandate compliance, and watch teams route around you, because the mandate added work without adding value. The alternative is to run the platform as a product whose customers are your own engineers, and to make the paved road genuinely faster than the dirt path beside it.
The core artifact is the golden-path template: a starter kit that gives a new team a working service on day one, with gateway credentials, tracing, guardrails, an eval harness and CI wiring already in place. The metric that matters for the template is time to first production call: if the road gets a team from idea to a governed, observable endpoint in days, adoption follows; if onboarding takes a quarter, no mandate will save you. A platform team operating this way holds office hours, publishes a roadmap, and treats a confusing onboarding doc as a defect.
- Prove it on one flagship. Build the road underneath your first serious use case, not beside it. The platform earns credibility by shipping something visible, and the use case hardens the platform against reality.
- Make migration the path of least resistance. Onboard the next teams with white-glove support, then add incentives: faster security review on the road, chargeback visibility off it, deprecation dates for direct provider keys.
- Measure adoption, not compliance. Track the share of inference traffic flowing through the gateway, time-to-production for new use cases, and reuse of connectors and templates. Checklist compliance can be gamed; traffic share cannot.
When those numbers move, the reference architecture has stopped being a document and become infrastructure: the boring, load-bearing kind that lets every new AI idea start at layer five instead of layer one. To sketch a stack like this interactively, the Architecture Designer in the Lab walks the same layers against your own constraints.