- Serving every request from one large model overpays on the easy majority and still risks the hard minority; a router sends each request to the model that fits it on complexity, cost, latency and required capability.
- Strategies run from heuristic rules to learned classifiers to LLM-as-judge, plus the cascade pattern that tries a cheap model first and escalates on low confidence; cheap models absorb the head of traffic while expensive models handle the thin, hard tail.
- Routing belongs in the model gateway as a policy, not scattered in application code, for central control, swappability and observability; the costs people forget are router accuracy, the added latency of the routing step, and the need to evaluate quality per route, since misrouting a hard request to a weak model is the real risk.
One model for everything is wasteful
Most enterprise LLM workloads run on a single frontier model wired behind every request, and on the surface that looks like a sound default: pick the strongest model available and let it handle whatever comes. The problem is that the traffic is not uniform. A large share of real requests are easy: a short classification, a formatting fix, an extractive lookup, a routine rewrite. A small share are genuinely hard: multi-step reasoning, ambiguous instructions, long-context synthesis. Sending both classes to the same large model means you overpay on the easy majority and, oddly, can still fall short on the hard minority when the model you standardized on was chosen for its price rather than its ceiling.
The overpayment is the obvious cost. A frontier model can be an order of magnitude more expensive per token than a small one, and it is slower, so every trivial request that a cheaper model could have answered carries a premium it did not need. At enterprise volume that premium compounds into a line item nobody planned for. But the quieter failure is on the other end. A single model is a single point on the cost-capability curve, and no single point is optimal for the whole distribution of requests. Fix the model and you have fixed a compromise.
Model routing rejects the premise that one model should serve everything. Instead a router inspects each request and sends it to the model that actually fits it, judged on complexity, cost, latency and the capability the request demands. Easy requests go to small, fast, cheap models; hard requests go to the strong, expensive ones; and the money saved on the easy majority pays for the capability reserved for the hard tail. It is portfolio thinking applied to inference.
What routing is
A router is a component that sits in front of a pool of models and decides, per request, which one should answer. It is a dispatcher, not a model in the usual sense: its job is a classification, not a completion. The inputs to that decision are properties of the request such as its estimated complexity, the capability it needs (does it require tool use, long context, code, vision?), the latency budget of the calling path, and the cost ceiling the workload is willing to pay. The output is a single choice: this request goes to model X.
Routers come in three broad forms, in rising order of sophistication and cost. A heuristic router uses hand-written rules: prompt length, keyword or task-type tags, the endpoint that called it. A learned classifier is a small, cheap model trained to predict, from the request, which tier will answer it acceptably. An LLM-as-judge router asks a small language model to read the request and nominate the right target. Each buys more nuance at more cost and latency, a trade-off the next section makes concrete.
Whatever the mechanism, the decision flow is the same shape: classify the request, apply the routing policy, dispatch to the chosen model, and, in the cascade variant, check the result and escalate if it is not good enough.
request --> [ classify ] --> complexity / capability / budget
|
v
[ route policy ]
/ | \
easy medium hard
| | |
small mid frontier
model model model
| | |
v v v
answer answer answer
Routing is therefore best understood as policy plus a model pool, not a single clever trick. The pool defines the points available on the cost-capability curve; the policy decides which point each request lands on. Change either independently and the behavior of the system changes, which is exactly the flexibility a single fixed model denies you.
Routing strategies
The strategies differ in how the routing decision is made and, crucially, in what that decision itself costs. The three predictive approaches, heuristic rules, learned classifiers and LLM-as-judge, all decide up front, before any answer model runs. The fourth, the cascade, is different in kind: it decides after the fact, by trying a cheap model first and escalating only when the cheap answer looks weak.
Heuristic rules are the cheapest to run and the easiest to reason about: a length threshold, a task tag, the calling endpoint. They cost almost nothing and are fully transparent, but they are blunt and drift out of date as traffic changes. A learned classifier captures patterns a human would not hand-code and adds only a small, fixed inference cost, but it needs labeled training data and periodic retraining. An LLM-as-judge router is the most nuanced, since it actually reads the request, but it adds a real model call to the front of every request, taxing both latency and cost.
The cascade deserves its own line because its economics are inverted. Instead of predicting difficulty, it measures it: run the small model, inspect its confidence or a quick quality check, and escalate to a stronger model only when needed. On easy traffic that resolves at the first tier, it is very cheap. On hard traffic that has to climb, it pays twice, once for the failed cheap attempt and again for the escalation.
| Strategy | How it decides | Strengths | Costs and caveats |
|---|---|---|---|
| Heuristic rules | Hand-written thresholds and tags | Near-zero overhead, transparent, easy to audit | Blunt; drifts as traffic shifts; manual upkeep |
| Learned classifier | Small trained model predicts the tier | Captures subtle patterns; low fixed cost | Needs labeled data and retraining |
| LLM-as-judge | A small LLM reads and nominates a target | Most nuanced; adapts to phrasing | Adds a model call to every request |
| Cascade | Try cheap first, escalate on low confidence | No prediction needed; cheap on easy traffic | Pays twice on the hard tail; needs a quality signal |
In practice these combine. A cheap heuristic can pre-filter the obvious cases and hand only the ambiguous middle to a classifier or a cascade, so you spend routing effort where it changes the decision and nowhere else.
The economics of routing
The economic case rests on the shape of the traffic. Request difficulty is not evenly spread; it is heavily skewed, with a large head of easy requests and a long thin tail of hard ones. Routing exploits that skew directly. The cheap models absorb the head, where they are perfectly adequate, and the expensive models are reserved for the tail, where their extra capability is actually needed. Because the head is where most of the volume lives, moving it to a model that costs a fraction as much bends the blended cost per request down sharply, without touching the quality of the answers on the hard tail.
volume
| ####
| #### easy: small model (~1x cost)
| ####
| #### ##
| #### ## ## hard: frontier (~10x cost)
+------------------------------- difficulty
head ---------------------> tail
blended cost ~= (head share x cheap)
+ (tail share x expensive)
A rough illustration makes the lever visible. Suppose the frontier model costs roughly ten times the small one per request, and suppose eighty percent of traffic is easy enough for the small model. Serving everything on the frontier model is a baseline of ten units per request. Routing eighty percent to the cheap model drops the blended cost to about (0.8 × 1) + (0.2 × 10) = 2.8 units, a large reduction, while the hard twenty percent still gets the strong model it needs. The exact ratios depend on your models and your traffic, so treat the numbers as illustrative, but the direction holds wherever difficulty is skewed.
Latency moves the same way, because small models are generally faster as well as cheaper, so the easy majority also returns quicker. Routing compounds with the rest of the cost cluster: it sits alongside prompt caching, which drives down the cost of the shared preamble, and it feeds the wider LLM economics picture, where blended cost per request is the number that actually lands on the invoice.
Routing belongs in the gateway
Once you accept routing, the question is where it runs. The tempting answer is to code it into the application: an if-statement that picks a model based on some local signal. That works for exactly one application and becomes a liability the moment you have several. Routing logic scattered across application code means every team reinvents it, every team hard-codes its own model choices, and no one has a single place to see or change how requests are being dispatched. When a model is deprecated or a cheaper one appears, you are editing many codebases instead of one policy.
Routing belongs in the model gateway, the shared layer every request already passes through on its way to a provider. Placing the router there turns model selection from application logic into platform policy, with several concrete benefits:
- Central control. One routing policy governs all traffic, so a change to tiers or thresholds is made once and applies everywhere.
- Swappability. Because applications call a logical endpoint rather than a named model, you can add, retire or re-point models behind the gateway without touching application code.
- Observability. The gateway sees every routing decision, so route distribution, cost per route and quality per route are measurable in one place instead of guessed at per team.
- Consistency. Governance, rate limits, logging and fallbacks apply uniformly, rather than being re-implemented, and re-forgotten, application by application.
The principle is the same one that justifies the gateway in the first place: cross-cutting concerns belong in a shared control plane, not duplicated in every consumer. Routing is a cross-cutting concern par excellence, since it touches cost, latency and quality on every request the enterprise makes.
The costs people forget
Routing is not free, and the costs it adds are easy to overlook when the headline savings are so visible. The first is router accuracy. A router is a classifier, and classifiers are wrong sometimes. Every misclassification sends a request to the wrong model, and the two directions are not symmetric in consequence. Routing an easy request to the expensive model merely wastes a little money. Routing a hard request to the weak model produces a bad answer, and that is the failure that matters.
The second cost is the latency of the routing step itself. Any router that is more than a trivial rule adds time before the answer model even starts, and an LLM-as-judge router adds a whole model call. On a fast path that budget can rival the saving you were chasing. The third is complexity: more models to evaluate, version and monitor, plus the router as a new moving part with its own failure modes. The fourth, and the most neglected, is evaluation. You cannot manage routing quality by watching a single aggregate score, because a healthy blended average can hide a route that is quietly failing. Quality has to be measured per route.
None of this argues against routing; it argues for treating the router as a real component with a real error budget. The savings are worth the added machinery, but only if you account for that machinery honestly and instrument it rather than assuming it works.
The architect view
My advice to a CTO standing up routing is to start deliberately small. Two tiers, a cheap model and a strong one, with a simple heuristic or a light classifier between them, capture most of the available saving at a fraction of the complexity of a many-model, LLM-judged scheme. You can always add tiers and sophistication once you have data showing where the simple split leaves money or quality on the table. Starting complex, by contrast, buys you a system you cannot yet evaluate.
Measure quality per route from day one. The number that matters is not the blended average but the answer quality on each individual route, especially the cheap ones, because that is where a silent misrouting failure hides. Sample real traffic on each route, compare against a stronger reference where you can, and alert when a route's quality drifts. The cascade pattern is attractive here precisely because it measures difficulty instead of guessing it, at the price of paying twice on the hard tail.
Put the router in the gateway, not in application code, so that the policy is central, the models are swappable and every decision is observable. And treat the router itself as a first-class component: something you evaluate, version and monitor like any model in the pool, with its own accuracy and latency budget. A router you deploy and forget will drift as traffic changes and quietly degrade, and because its failures are asymmetric, the degradation you will not notice is the expensive kind.