Tokenization: How Text Becomes Tokens
A token is the atomic unit an LLM reads and writes: a subword fragment produced by an algorithm such as byte-pair encoding (BPE), not a word or a character. The tokenizer is trained to merge frequent character sequences into a fixed vocabulary, typically 32k to 200k entries, so common English words become one token while rare strings shatter into several; in English, one token averages roughly 3 to 4 characters, or about 750 words per 1,000 tokens. This is why models fail at letter-counting puzzles: the model never sees letters, only token IDs, so a word like strawberry may arrive as two opaque fragments. The enterprise consequences are concrete: non-English text often costs 1.5 to 3 times more tokens for the same content, a real budget line for multilingual deployments; structured formats like JSON carry billable overhead; and every context-window and pricing calculation starts with the tokenizer, not the character count.
Pricing Mechanics: Input, Output & Cached Tokens
Commercial LLM APIs price per million tokens, with separate rates for input (prompt) and output (generated) tokens, and output typically costs 3 to 5 times more because decode is the serially expensive phase of inference. Most providers add two levers: cached-input pricing, where a repeated prompt prefix is billed at a fraction of the standard input rate (often around 10 percent), and batch tiers that trade latency for roughly 50 percent discounts on asynchronous workloads. The mix matters more than the headline rate. A summarization workload is input-heavy and cache-friendly; an agentic workload inverts the profile, because every tool call and reasoning step generates output tokens and then re-feeds them as input on the next turn, so output and re-read context dominate spend. Architects should model the input-output ratio per use case before comparing vendors; a cheaper per-token price on the wrong side of the ratio saves nothing.
Latency Anatomy: TTFT, Throughput & Streaming
LLM latency is two different problems wearing one name: time to first token (TTFT) and tokens per second once generation starts. Inference runs in two phases. Prefill processes the entire prompt in parallel and is compute-bound, so TTFT scales with input length: a 100k-token context can add seconds before anything appears. Decode then generates one token at a time and is bound by memory bandwidth, typically tens to low hundreds of tokens per second per request. Streaming does not make the model faster; it changes what users perceive: showing the first token in under a second makes a 20-second generation feel responsive, while a blocking call of the same duration feels broken. The architectural consequences: budget TTFT and throughput separately in SLOs, keep prompts short where interactivity matters, and remember that agent chains compound latency per step, so a five-call chain inherits five prefills.
Inference Optimization: Quantization, Batching, KV Cache
Inference optimization is the discipline of extracting more tokens per GPU-hour without visibly degrading quality. Quantization compresses weights from 16-bit formats down to FP8 or INT4, roughly halving or quartering the memory footprint and lifting throughput, usually at a cost of a point or two on benchmarks and occasionally more on edge cases, so evaluate on your own tasks. Continuous batching interleaves many requests at the token level instead of waiting for the slowest sequence, often multiplying GPU utilization several times over. The KV cache stores attention keys and values so each new token does not recompute the whole context; reusing it across requests that share a prefix is the mechanism behind prompt caching. Speculative decoding drafts tokens with a small model and verifies them with the large one, commonly yielding 2 to 3 times faster decode. For self-hosted deployments these levers, not model choice alone, decide whether the GPU bill is defensible; for API consumers, they are why prices keep falling.
Prompt Caching Economics
Prompt caching bills a repeated prompt prefix at a steep discount because the provider reuses the KV cache from a previous prefill instead of recomputing it. Mechanically it is prefix matching: the request must repeat earlier content exactly from the first byte, cache entries typically live for minutes (around 5 is common, with paid extensions to an hour), and cached reads are usually priced at roughly one tenth of standard input. That is an order-of-magnitude saving on the dominant token class in agentic systems, which re-send a growing transcript on every turn. The catch is that caches only hit if you design for them: put stable content first (system prompt, tool definitions, reference documents), append volatile content last (user message, retrieved chunks), and never inject a timestamp or session ID near the top, because one changed byte invalidates everything after it. For agent workloads, cache-aware prompt layout is often the single largest cost lever available, worth 50 to 90 percent of input spend.
Model Routing & Fit-for-Purpose Selection
Model routing sends each request to the cheapest model that can do the job instead of defaulting all traffic to a flagship. Frontier and small models often differ by one to two orders of magnitude in per-token price, yet a large share of enterprise traffic (classification, extraction, short summaries, routing itself) sits well within a small model's competence. Practical patterns include static routing by task type, learned routers that score prompt difficulty, and cascade designs where a cheap model answers first and a confidence check escalates hard cases to a stronger one. The clean implementation point is an LLM gateway: a single egress where routing policy, fallbacks, and model allowlists are enforced centrally rather than re-decided in every codebase. The trade-off to manage is evaluation debt: every routing rule is an implicit quality claim, and without per-task evals a cheap-first policy quietly becomes a wrong-first policy.
Unit Economics: Cost per Task, not Cost per Token
The number that matters is cost per resolved task, not cost per million tokens. A task (a support ticket answered, a contract clause extracted, a code change reviewed) consumes tokens across many calls: retrieval, generation, tool use, retries, guardrail checks, and verification passes. Agentic systems amplify this: a loop averaging 10 to 30 model calls per task, each re-reading a growing context, can turn a model that looks cheap per token into an expensive system, and a pricier model that finishes in three well-aimed calls into the economical choice. Success rate belongs in the denominator too: a cheap model that resolves 70 percent of tasks while humans redo the rest can cost more than a premium model at 95 percent. Instrument tokens per task end to end, publish cost per outcome by use case, and compare models at the workflow level, where the comparison can actually change a decision.
GenAI FinOps: Budgeting, Showback & TCO
GenAI FinOps applies cost governance to a spend line that scales with usage rather than provisioned capacity and can move 10x in a quarter when one team ships an agent. The foundation is token telemetry: every model call tagged with use case, team, model, and environment, ideally captured at the LLM gateway so instrumentation is not optional per team. On top of that sit showback or chargeback per use case, budget guardrails (rate limits, per-application spend caps, alerts on cost-per-task regressions), and forecasts built on tasks and adoption curves rather than extrapolated token counts. TCO extends well beyond API fees: vector stores and retrieval infrastructure, evaluation and observability tooling, fine-tuning and data preparation, security review, and the engineering time to maintain prompts and pipelines, which routinely exceeds the model bill itself. The goal is a defensible answer to the CFO's question: what does this capability cost per unit of business outcome, and which lever changes it.