- A model gateway (also called an LLM or AI gateway) is a single API endpoint between your applications and every model provider, centralizing auth, routing, fallback, rate limiting, caching, telemetry and a uniform interface.
- Because every inference call passes through it, the gateway is the one place to enforce budgets, guardrails, PII filtering, model allowlists and audit logging once, for everyone, and to make a model swap configuration rather than code.
- Commercial and open-source gateways both exist; the value is the pattern, a thin control layer, not a product. Introduce it as the paved road that beats raw keys, never as a mandated chokepoint.
The problem: every team wires its own
Look inside most enterprises a year into their AI program and you find the same picture repeated in miniature across every team. Each application holds its own provider API key, pasted into a config file or a secrets manager nobody audits. Each has hand-rolled its own retry and timeout logic, usually subtly wrong. Each logs to a different place, in a different shape, if it logs the model calls at all. Every decision was locally sensible under deadline. The sum is an estate no single person can see.
The consequences are not evenly distributed between the visible and the dangerous. On the visible side is plain duplication: five teams each rebuilding backoff, token counting and streaming plumbing that a platform could have provided once. That waste you can at least estimate. On the dangerous side, cost is effectively invisible, because spend lands as one undifferentiated provider invoice with no attribution to team or use case, and it surfaces only when finance asks who spent the money and nobody can answer. Governance is not merely hard but structurally impossible: you cannot filter for PII, enforce a model allowlist or produce an audit trail on traffic that never flows through a shared point.
The quietest cost is rigidity. When a provider deprecates a model, raises a price or suffers an outage, the response is a code change in every application that embedded that provider directly. What should be a routing decision becomes a coordinated deployment across nine repositories, each owned by a different team on a different schedule.
What a model gateway is
A model gateway is a single API endpoint that sits between your applications and your model providers. Applications stop calling providers directly; they call the gateway, and the gateway calls the models on their behalf. It is one front door for the entire AI estate. The same pattern travels under several names, model gateway, LLM gateway, AI gateway, but the shape is identical: a thin service that every inference request passes through on its way out and every response passes through on its way back.
The mental model worth holding is the API gateway that most platform teams already run for their microservices, specialized for inference traffic. An API gateway gives you one place to handle authentication, rate limiting and logging for a fleet of services. A model gateway does the same job for a fleet of models, and adds the concerns that only make sense for language models: token and cost accounting, prompt-level telemetry, response caching keyed on request content, and a uniform request shape that hides the differences between providers.
That last point is what makes it more than a proxy. Each provider exposes its own request format, its own parameter names, its own streaming protocol and its own error semantics. A gateway presents callers one consistent interface and translates to whatever provider actually serves the request. The application no longer knows or cares which model answered. It knows the gateway.
The distinction to hold onto is that a gateway is a control layer, not a modeling layer. It does not own prompts, retrieval logic or orchestration; those live in the layers above it, as the reference architecture lays out. The gateway owns the traffic, and owning the traffic turns out to be the highest-leverage thing you can own.
What it centralizes
The gateway earns its place by absorbing a set of responsibilities that would otherwise be reimplemented, inconsistently, in every application. None of these is individually novel. The leverage comes from placement: each one is solved once, in a component every call already traverses, instead of a dozen times with varying rigor.
| Responsibility | What the gateway does | Left to each app instead |
|---|---|---|
| AuthN / AuthZ | Callers present a workload identity; the gateway holds provider credentials | Provider keys copied into every repo and config |
| Routing & fallback | Maps requests to models by policy; fails over to a backup provider on error | Every provider outage becomes an application incident |
| Rate limiting & quotas | Per-team and per-use-case ceilings on requests and spend | Runaway loops and cost surprises found in the invoice |
| Caching | Serves repeated requests without paying for re-inference | Paying repeatedly for identical work |
| Telemetry | Logs every call: model, tokens, latency, cost, caller | No audit trail and no cost attribution |
| Uniform interface | One request shape across all providers | Provider-specific code welded into each app |
A few of these deserve a caution rather than applause. Caching is safe and cheap in its exact-match form, where an identical request returns a stored response; its semantic form, serving a cached answer to a merely similar question, trades correctness for savings and should be opt-in per use case, never a silent default. Quotas need one wrinkle for reasoning-class models, whose internal thinking tokens are billed as output: a budget defined on input size alone will understate real spend, as the economics of inference spell out. The point of centralizing these is not that the gateway makes each decision perfectly. It is that the decision exists in exactly one place you can reason about, tune and audit.
The natural governance and cost control point
The argument for the gateway is almost topological. Governance and cost control both require that every relevant action pass through a place where policy can be applied. In a direct-to-provider world no such place exists; controls have to be replicated into every caller and trusted to stay consistent, which they never do. The gateway creates the chokepoint that makes enforce-once possible, because by construction there is no path to a model that avoids it.
WITHOUT a gateway WITH a gateway
------------------ ----------------
app A --> provider 1 app A --.
app B --> provider 1 \
app C --> provider 2 app B --> [ GATEWAY ] --> provider 1
app D --> provider 2 app C --/ | provider 2
app E --> provider 3 app D --/ | provider 3
app E --/ |
keys, logging, guardrails one place for: authZ,
duplicated per app; budgets, guardrails, PII
no single view of spend filter, allowlist, audit
Once traffic converges, the controls that were impossible become routine. Budgets stop spend before it happens rather than reporting it after. Guardrails screen inputs for injection and outputs for policy violations on the platform path, so an application cannot forget them under deadline. A PII filter runs on the boundary instead of being trusted to every team. A model allowlist means only vetted models can be reached, and adding one is a governance decision with a record, not a line someone shipped on a Friday. And every call produces a uniform audit log, which is the difference between an answer and a scramble when a regulator or customer asks where AI touches a decision and on what data.
Routing, fallback and portability
With all traffic behind one interface, model selection becomes a policy the gateway resolves rather than a choice hard-coded into each caller. The most useful routing is by task tier: a cheap, fast default model handles the bulk of requests, and the gateway escalates the genuinely hard cases to a stronger or reasoning-class model. Cost policy layers on top, so a team can cap a low-value workflow to its cheapest option while a customer-facing path is allowed to reach for quality. The application asks for a capability; the gateway decides which model actually delivers it.
Fallback is the same mechanism pointed at reliability. When a provider returns errors or goes dark, the gateway fails the request over to a backup provider or a comparable model, and the outage that would have been a page for five application teams becomes a routing event the platform absorbs. This is only possible because the caller was never bound to a specific provider in the first place.
That decoupling is the strategic payoff, and it is worth naming plainly as an answer to lock-in. When every application speaks only to the gateway, the model portfolio lives behind one interface, and swapping a model, retiring a deprecated one, or A/B testing two on live traffic is a change to a routing table, not a code change rippling through the estate.
- Route by tier: a small default model for volume, escalation to a stronger model for hard or high-stakes requests.
- Route by cost policy: per-use-case ceilings that pick the cheapest model meeting the quality bar.
- Fail over on outage: a backup provider or equivalent model when the primary errors or times out.
- Swap by configuration: a new model reaches production by editing the routing table, with applications untouched.
Build, buy or open-source
The market has caught up to the pattern. A range of commercial AI gateways now ship these responsibilities as a product, and several credible open-source projects offer the same shape for teams that prefer to run it themselves. Which to pick is a genuinely open question that depends on your constraints, and it is worth resisting anyone, including a vendor, who tells you there is one right answer. There is not, and the landscape shifts too quickly to crown one.
What does generalize is where each option fits. The decision is less about features than about who you want operating the control point in the middle of your production traffic.
- Buy a commercial gateway when you want the responsibilities on day one, value vendor support and a managed control plane, and your requirements sit inside what a general product covers. Weigh the cost of another vendor holding your provider credentials and sitting in the hot path.
- Adopt open-source when you want the pattern without the license and are prepared to own operation, upgrades and reliability. The code is free; running it in the critical path is not.
- Build your own only when your requirements are genuinely unusual, unusual integrations, unusual policy, unusual scale, that no product serves. A gateway is deceptively simple to prototype and demanding to run at production reliability, so build deliberately, not by default.
The framing that keeps this decision honest is that the value is the pattern, not the product. A gateway is a thin control layer, and a thin layer is something you can replace. Design the interface your applications depend on to be your own, so that whichever engine sits behind it, commercial, open-source or homegrown, can change without the estate noticing. The teams that get burned are the ones that let a specific product's request format become their application contract, which trades one flavor of lock-in for another.
Introducing a gateway without a bottleneck
Every objection to a gateway is really the same fear: a single point through which all traffic flows sounds like a single point of failure and a single queue at which every team waits. That fear is legitimate, and the way you introduce the gateway either confirms it or dissolves it. The failure mode is to announce the gateway, mandate it, and turn the platform team into the office that slows everyone down. The alternative is to make the gateway the path teams choose because it is genuinely easier than raw provider keys.
Paved road, not chokepoint, is the whole discipline. A team reaching for the gateway should get, for free, the credentials, retries, logging, caching and cost visibility they would otherwise build by hand, so the compliant path is also the fastest path to production. If a team can ship quicker by going around you, the gateway is a document, not infrastructure, and no mandate will fix that.
Concretely, that means migrating by incentive rather than decree. Ship a great developer experience: a client that is a one-line change from calling a provider directly, clear docs, and templates that start a new service already wired to the gateway. Then let incentives pull teams over, faster security review on the road, cost visibility they cannot get off it, and eventual deprecation dates for direct provider keys, announced early and enforced gently. Measure the share of inference traffic flowing through the gateway, because that number, unlike checklist compliance, cannot be gamed. When it climbs on its own, the gateway has stopped being a mandate and become the obvious way to reach a model. You can sketch where it sits in a full stack with the Architecture Designer in the Lab.