- Output tokens typically cost 3 to 5 times input tokens, and cached input is often roughly 10x cheaper than fresh input: prompt structure, not vendor negotiation, sets your unit price.
- A 40,000-document daily pipeline runs on the order of $31,000 a month at illustrative list prices; caching plus batch tiers cut that roughly in half with no model change.
- The API invoice is frequently the minority of true cost: evals, observability, retrieval infrastructure, and engineering time usually dominate once a system is in production.
Why token math decides AI budgets
Every LLM invoice is a multiplication problem. A single document-extraction request that costs two and a half cents looks like a rounding error in a demo. Multiply it by 40,000 documents a day and 30 days a month, and the rounding error becomes a $31,000 monthly line item. Multiply again by the three other teams who copied your prompt, and the CFO starts attending your architecture reviews. Token math is not an engineering detail; it is the budget.
The running example for this article is a document-processing workload, because it is the most common first production use case in the enterprise: an insurer ingesting 40,000 claims documents a day, extracting structured fields and a short summary from each. Each request carries about 6,000 input tokens (instructions, examples, and the document itself) and returns about 500 output tokens of JSON. We will price this pipeline several ways and watch the bill move.
Budgets fail for a predictable reason: teams price the demo, not the workload. The demo runs 50 requests against clean samples with a short prompt. The production workload runs tens of thousands of requests against messy scans, with a prompt that has quietly quadrupled through incident-driven patching, plus retries, plus fallbacks. In practice the production cost per task commonly lands several multiples above the pilot estimate unless someone models it deliberately.
Agentic patterns amplify this further. A chat completion is one model call; an agent that plans, calls tools, inspects results, and retries can consume 10 to 100 times the tokens per task, because every loop iteration re-sends the growing conversation as input. If your roadmap includes agents, your cost model needs a multiplier column before your architecture does.
Anatomy of a model price
A model's price is not one number; it is a small rate card, and the structure of that card matters more than the headline figure. Four mechanics dominate the bill.
Input and output are priced asymmetrically. Output tokens typically cost 3 to 5 times more than input tokens, because generation is sequential and compute-hungry while input processing parallelizes well. This asymmetry means a verbose model, or a prompt that invites verbosity, is billed at the premium rate. For extraction workloads like the claims pipeline, tight JSON output is a pricing decision, not a style preference.
Cached input is dramatically cheaper. Most providers discount input tokens that repeat a previously processed prefix, often by roughly 10x. A prompt structured stable-prefix-first (system instructions, schema, few-shot examples) followed by the variable document turns the majority of input spend into discounted reads.
Batch tiers trade latency for price. Asynchronous batch endpoints commonly run at about half the interactive price in exchange for completion windows measured in hours. Document pipelines are often ideal batch citizens: nobody is waiting on screen for an overnight ingestion job.
Context length quietly multiplies input spend. Long-context models make it easy to send 100,000 tokens; nothing about pricing makes it wise. Every token in context is billed on every call, and in multi-turn or agentic flows the same tokens are re-billed each iteration. Context is a recurring cost, not a one-time upload.
| Tier (illustrative, mid-2026) | Input $/M | Cached input $/M | Output $/M | Batch |
|---|---|---|---|---|
| Frontier | ~$3.00 | ~$0.30 | ~$15.00 | ~50% off |
| Mid-tier | ~$0.80 | ~$0.08 | ~$4.00 | ~50% off |
| Small | ~$0.15 | ~$0.02 | ~$0.60 | ~50% off |
The absolute numbers drift quarter to quarter; the ratios (output over input, cached over fresh, batch over interactive) have been far more stable, and the ratios are what your architecture should exploit. The pricing mechanics primer tracks these structures in more detail.
Building a workload cost model
The cost model for any workload fits on an index card, and refusing to write it down is how projects end up surprised. Estimate four quantities per use case: requests per day, cacheable input tokens per request, variable input tokens per request, and output tokens per request. Then multiply by the rate card.
daily cost = R x (Pu x Cin + Pc x Ccache + O x Cout)
R requests per day 40,000
Pu fresh input tokens per request 3,500
Pc cached input tokens per request 2,500
O output tokens per request 500
Cin price per M fresh input tokens $3.00
Ccache price per M cached input tokens $0.30
Cout price per M output tokens $15.00
40,000 x (3,500 x $3.00/M) ............... $420/day
40,000 x (2,500 x $0.30/M) ................ $30/day
40,000 x ( 500 x $15.00/M) .............. $300/day
total $750/day
Walk the pipeline through three scenarios. Baseline, no optimization: 40,000 requests at 6,000 fresh input tokens and 500 output tokens is 240 million input and 20 million output tokens a day. At the illustrative $3 and $15 rates, that is $720 plus $300: about $1,020 a day, or roughly $31,000 a month. With caching: restructure the prompt so the 2,500 tokens of instructions, schema, and examples form a stable prefix. At the near-complete hit rates that steady traffic makes realistic, input drops to $450 a day and the total to $750, about $22,500 a month: a 26 percent cut from a prompt refactor. Adding batch: route the 70 percent of documents with overnight tolerance through a batch tier at half price, and the total falls to roughly $490 a day, or $14,600 a month. The bill is now about half the baseline, with no model change and no quality risk.
Then stress the model, because the inputs will drift. Every 1,000 tokens of prompt bloat adds 40 million input tokens a day: $120 daily, about $3,600 a month at fresh-input rates. Every 200 extra output tokens costs the same $120 a day, because output is priced 5x higher here. Verbosity and bloat are silent line items; the model makes them visible before the invoice does.
Two hedges. Cache hit rates below 90 percent erode the caching scenario quickly, so measure rather than assume. And volume grows: pipelines that work attract documents. To run your own numbers interactively, the cost estimator in the Lab implements exactly this model.
The seven cost levers, ranked
Every GenAI cost program uses some combination of seven levers. The failure mode is reaching for the dramatic ones first: teams debate self-hosting for months while their prompts carry 3,000 tokens of dead instructions. Ranked by return on effort for a typical enterprise workload:
| Rank | Lever | Effort | Typical impact (illustrative) |
|---|---|---|---|
| 1 | Prompt caching | Low | 20-40% of input spend on repeated-prefix workloads |
| 2 | Output discipline (max_tokens, terse schemas) | Low | 10-30% of output spend |
| 3 | Batch processing | Low to medium | ~50% off latency-tolerant volume |
| 4 | Model routing | Medium | 40-70% where task difficulty varies |
| 5 | Context pruning | Medium | 10-40% on RAG and agent flows |
| 6 | Distillation to smaller models | High | On the order of 5-20x unit cost reduction on narrow tasks |
| 7 | Quantized self-hosting | Very high | Break-even only at high sustained utilization |
The first three are configuration and prompt structure: days of work, no model change, no quality risk worth naming. For the claims pipeline they took the bill from $31,000 to roughly $14,600 a month. Prompt caching deserves its reputation as the highest-leverage single change on any workload with a repeated prefix.
Model routing is the pivot point: real savings, but it requires an evaluation harness to prove the cheap model is good enough per task tier, and that harness is an investment. Skipping the harness converts a cost lever into a quality incident.
Distillation and self-hosting are platform bets, not tuning. They can be right at large sustained volumes or under hard data-residency constraints, but they carry engineering headcount and, for self-hosting, the GPU utilization risk treated in the TCO section below. Earn the easy 50 percent first; it usually postpones the hard conversation by a year.
Latency is a cost too
Latency has two distinct components with different economics. Time to first token (TTFT) is dominated by prompt processing and queueing, and it grows with input length: one more argument against bloated contexts. Tokens per second determines how fast the answer completes once it starts. A model can be excellent at one and mediocre at the other, and the metric that matters depends on whether a human is watching.
Streaming is perceived-latency arbitrage. Users judge responsiveness by TTFT, not by completion time; a stream that begins in 400 milliseconds feels fast even if the full answer takes eight seconds. For interactive surfaces, streaming buys you room to use a slower, cheaper, or more careful model without the experience degrading. For machine-to-machine calls, streaming buys nothing; only total completion time matters, and it gates pipeline throughput.
Reasoning models complicate the picture. They spend additional thinking tokens before answering, which you typically pay for as output and wait for as latency. On genuinely hard tasks the quality gain is real; on routine extraction it means paying premium rates to overthink. The claims pipeline sends a reasoning tier only the roughly 5 percent of documents flagged as ambiguous, and clears the rest on a fast standard model.
The practical consequence: latency budgets shape model choice as much as price does. The pipeline's overnight ingestion has an eight-hour window, which is what unlocks batch pricing. The adjuster-facing summary view has a two-second budget, which excludes both batch tiers and heavyweight reasoning modes regardless of what they cost. Write the latency budget next to the token budget; together they usually narrow the field to two or three candidate models before quality evals even begin.
From API bill to TCO
The number on the provider invoice is the most measured GenAI cost and frequently the minority of the real one. In full TCO views of document pipelines like our running example, the recurring surprise is how much sits around the model rather than in it.
Count the surrounding line items. Evaluation infrastructure: golden datasets, regression suites, and human review time, exercised on every prompt or model change. Observability: logging every request and response with token counts, plus tracing for agentic flows. Guardrails: input and output filtering, PII handling, injection defenses. Retrieval infrastructure where RAG is involved: vector storage, embedding refresh jobs, and chunking pipelines that get rebuilt at least once. And prompt maintenance, which behaves like software maintenance because it is: prompts drift, models version, and someone owns the regression backlog. Behind all of it sit people; an ML engineer plus fractions of platform, security, and product time typically dwarfs a mid-sized workload's API spend.
Self-hosting does not remove these costs; it moves and adds. The invoice line becomes GPU reservations plus an inference-serving stack plus the engineers to run both, and it introduces utilization risk: a reserved GPU cluster running at 20 percent utilization can cost more per token than premium API rates. Self-hosting typically earns its keep only with sustained high utilization, strict data-residency requirements, or token volumes large enough to amortize the headcount. Model the crossover honestly, including the hardware refresh cycle, before the platform team falls in love with the racks. The GenAI FinOps primer covers how to keep these categories visible month over month.
The architect's checklist
Everything above compresses into a short operating discipline. The teams whose GenAI budgets behave are not the ones with the cheapest model; they are the ones who treat tokens like any other metered resource, with owners, budgets, and reviews.
- Measure cost per resolved task, not per call or per token. A cheaper model that needs three retries is not cheaper. This is the unit the business understands and the only one that survives model swaps.
- Set token budgets per use case: a ceiling on input and output tokens per request, enforced with
max_tokensand monitored for drift. Prompt bloat is the leak nobody notices until the quarterly review. - Structure every prompt cache-first: stable instructions and examples up front, variable content last. Make cache hit rate a dashboard metric with an owner.
- Route by task tier: small models by default, frontier by documented exception, with an eval harness proving the routing thresholds hold.
- Batch everything without a human waiting. Latency tolerance is a standing 50 percent coupon; use it.
- Review monthly: rate cards drift, models get replaced, and last quarter's routing table is stale. Re-run the workload model against current prices.
None of this is glamorous, and that is the point. LLM economics rewards the same discipline that tamed cloud spend a decade ago: measure the unit, own the number, revisit the assumptions. The models will keep changing; the arithmetic will not. The architects who write the token math down before the build starts are the ones whose projects are still funded a year later.