- Quantization (FP16 down to FP8, INT8, INT4/NF4) shrinks weights and raises throughput with gradual, often acceptable quality loss: measure the drop on your own task rather than trusting a benchmark.
- The KV cache, not the weights, is usually the memory that runs out first; it grows with sequence length and batch size, and PagedAttention manages it like virtual memory so the GPU stays full.
- For hosted APIs the provider owns quantization, batching and caching, and you buy against latency and throughput SLAs; self-hosting on vLLM, TGI or TensorRT-LLM exposes every knob plus the operational risk of GPU capacity.
Why inference is where the money goes
Training a frontier model is a capital event: a large, one-time expenditure that produces a fixed asset. Inference is the opposite. Every request a user makes spends GPU seconds, and those seconds recur for the entire life of the product. A model you train once you serve billions of times, and the arithmetic of serving, not the arithmetic of training, is what shows up on next quarter's cloud invoice. For any enterprise past the pilot stage, inference is the dominant AI line item and the one that grows with success.
Because the bill is a product of volume and unit cost, small improvements compound violently. Shave ten percent off latency per token and you have not just made users happier; you have freed ten percent of your fleet to serve other traffic. Cut memory per request and you fit more concurrent sequences on the same card, raising throughput without buying hardware. At a few thousand requests per second, a single-digit-percent change in cost per token is a seven-figure annual swing. This is why the optimization surface gets its own discipline: it is where efficiency turns directly into margin.
That surface is broad. Numeric precision, memory management, how requests are grouped, how tokens are generated, and where the workload runs are all independent levers, and they interact. The rest of this article walks the four techniques that move the needle most (quantization, the KV cache, batching and speculative decoding) and then draws the line between what a hosted-API customer controls and what a self-hoster owns. The tie back to the wider bill lives in the LLM economics deep-dive.
Quantization: trading precision for speed
A model's weights are just numbers, and by default they are stored at 16 bits each (FP16 or BF16). Quantization stores them at lower precision: 8 bits (FP8 or INT8), or 4 bits (INT4, including the NF4 format popular for compressed fine-tuning). Fewer bits per weight means a smaller model in memory, less data to move from memory to the compute units, and, since inference is frequently memory-bandwidth bound, higher throughput. A model that no longer fits on one GPU at FP16 may fit comfortably at INT8, collapsing a multi-GPU deployment into a single card.
The cost is precision, and the useful surprise is how gracefully quality degrades. Dropping from FP16 to FP8 or INT8 is often nearly free in output quality for many tasks, because the weights that matter survive the rounding. INT4 pushes harder and the loss becomes visible sooner, especially on reasoning-heavy or long-form tasks, but for classification, extraction and retrieval-grounded answering it is frequently acceptable. Activation quantization (compressing the intermediate values, not just the stored weights) is harder to do without visible loss than weight-only quantization, which is why weight-only INT4 is the common starting point.
| Format | Bits | Memory vs FP16 | Typical use (illustrative) |
|---|---|---|---|
| FP16 / BF16 | 16 | baseline | Training and quality-critical serving |
| FP8 | 8 | ~half | Serving default on newer GPUs |
| INT8 | 8 | ~half | Mature, broadly supported |
| INT4 / NF4 | 4 | ~quarter | Memory-bound, latency-tolerant work |
The one rule that matters: do not assume, measure. Quantization loss is task-dependent, and a format that is invisible on a summarization benchmark can quietly break a numeric-extraction workflow. Before committing a quantized deployment, run your own evaluation set at each candidate precision and read the quality-cost trade-off off real numbers. The transformers deep-dive covers why some layers tolerate compression better than others.
The KV cache and why memory is the bottleneck
Transformers generate one token at a time, and each new token attends to every token before it. Naively, that means recomputing the keys and values for the entire history at every step, which is quadratically wasteful. The KV cache is the fix: it stores the keys and values already computed for past tokens so each new step only computes its own, then reads the rest from memory. This turns generation from repeated full recomputation into a cheap incremental append, and it is why autoregressive decoding is viable at all.
The cache is not free, and at scale it, not the model weights, is what exhausts GPU memory first. Its size grows linearly with sequence length and multiplies by the number of concurrent requests. A long-context chat with many users in flight can spend more memory on cached keys and values than on the model itself.
KV cache size (bytes), per request: 2 x L x n_kv x d_head x S x precision 2 keys and values L transformer layers n_kv key/value heads d_head dimension per head S sequence length (prompt + generated so far) precision bytes per element (2 = FP16, 1 = FP8) grows linearly with S, then multiplies by batch size B
The historical waste was fragmentation. Classic serving reserved a contiguous block sized for the longest possible sequence, so a request that finished early stranded the unused remainder. PagedAttention, introduced in vLLM, treats KV memory like an operating system treats RAM: the cache is split into fixed-size pages allocated on demand and freed when a sequence ends, with no requirement that a request's pages be contiguous. The effect is far less wasted memory, which means far more concurrent sequences on the same hardware. That density gain is the foundation the batching techniques in the next section build on.
Batching: filling the GPU
A GPU is a wide parallel machine that is most efficient when it processes many sequences at once. Batching is how you feed it. The naive approach, static batching, collects a fixed group of requests, runs them together, and returns the whole group when it is done. The flaw is that requests finish at different lengths: a batch of four where one answer is 500 tokens and the rest are 50 leaves three slots idle for the long tail while the batch waits to release together. On mixed real traffic, static batching can leave much of the GPU idle much of the time.
Static batch (4 slots), requests finish at different lengths: R1 ####.......... idle waiting...... | R2 ###############################... | releases only R3 ####.......... idle waiting...... | when the slowest R4 ########...... idle waiting...... | request finishes Continuous (in-flight) batch, finished slots refill at once: R1 ####|R5#######|R9##...... R2 ###############################... R3 ####|R6####|R10###...... R4 ########|R8#######......
Continuous batching (also called in-flight batching) fixes this at the granularity of a single decode step. When any sequence in the batch completes, its slot is freed immediately and a waiting request is admitted in its place, so the GPU never idles on a finished sequence. Combined with PagedAttention for the memory side, this keeps utilization high and is the default behavior of modern serving stacks.
The trade-off is the one worth stating plainly to stakeholders: batching optimizes throughput, and throughput can pull against per-request latency. A larger batch amortizes fixed overhead across more requests (more tokens per second, per dollar) but any individual request may wait fractionally longer to be scheduled and shares compute with its neighbors. Interactive chat weights latency; overnight document pipelines weight throughput. The serving configuration should reflect which one your workload actually cares about, a decision the latency anatomy primer unpacks further.
Speculative decoding
Autoregressive generation is latency-bound because it is sequential: token N+1 cannot start until token N exists, and each step pays the full cost of a forward pass through a large model. Speculative decoding attacks that serialization. A small, cheap draft model runs ahead and proposes several tokens at once. The large target model then verifies all of them in a single forward pass, accepting the longest prefix that matches what it would have produced itself and correcting at the first divergence.
The elegant property is that the target model still decides every token. Because verification checks the draft against the target's own distribution, the output is identical in quality to what the target would have generated alone; the draft only supplies a guess that is cheap to confirm. When the draft guesses well, several tokens emerge from one target pass and the accepted ones are effectively free latency. When it guesses badly, the target falls back to producing the next token itself, so the worst case degrades toward ordinary decoding rather than breaking anything.
The economics depend on acceptance rate, which depends on how well the small draft model predicts the large one on your traffic. A well-matched draft yields a meaningful latency reduction; a poorly matched one adds overhead for little gain, since you pay for draft passes whose tokens get rejected. Speedups are real but workload-specific, so treat any headline multiplier as illustrative until you have measured it on your own prompts.
Hosted endpoints vs self-hosting
Every technique in this article is a knob, and the first architectural question is whether you are allowed to turn it. With a hosted API, you are not, and mostly that is a feature. The provider chooses the quantization, runs continuous batching, manages the KV cache, and may or may not apply speculative decoding, all below the line you call. You do not tune these; you buy an outcome. Your levers are which endpoint to select, which price and latency tier to commit to, and how to structure prompts (caching, output length) to spend less against a rate card you cannot change. The upside is that GPU capacity, driver upgrades and serving reliability are someone else's pager.
Self-hosting inverts the deal. Running an open-weights model on vLLM, TGI (Text Generation Inference) or TensorRT-LLM hands you every knob: you pick the quantization format, set batch and KV parameters, enable speculative decoding, and can specialize the whole stack to one workload. That control is genuinely valuable for data-residency requirements, steady high-volume traffic where owned hardware beats per-token pricing, or models a vendor will not host. It also hands you the operational risk: procuring and saturating scarce GPUs, absorbing traffic spikes without a provider's elastic pool, and owning uptime.
| Dimension | Hosted API | Self-hosted (vLLM / TGI / TensorRT-LLM) |
|---|---|---|
| Quantization, batching, KV | Provider's choice, below the API | Yours to configure and tune |
| Your primary lever | Endpoint, tier, prompt structure | The full serving configuration |
| Capacity risk | Provider absorbs it | You own GPU supply and spikes |
| Best fit | Variable or early-stage traffic | Steady high volume, residency, custom models |
The honest framing for a steering committee is that self-hosting trades a predictable per-token price for a fixed infrastructure cost plus an operations team. It pays off past a volume threshold and under specific constraints, not by default. Model the crossover before committing, and route the decision through the same FinOps discipline that governs the rest of the AI spend.
What the architect actually controls
For the large majority of enterprise workloads, which run on hosted APIs, the architect's real control is selection, not tuning. You choose endpoints against explicit latency and throughput SLAs, and you hold the provider to them. A workload that needs sub-second first-token response and one that batches overnight are different purchases even from the same vendor, and treating them identically is how both the bill and the user experience drift. Write the SLA down first, then pick the endpoint that meets it at the lowest price.
Where quantized or smaller-precision options are offered, understand the quality-cost frontier before choosing a point on it. A cheaper quantized endpoint is only cheaper if it still passes your task's evaluation set; the discount is meaningless if it fails the extraction accuracy or reasoning check that matters to the business. Make quantization a measured decision with a number attached, not a default reached for to save money.
The discipline that ties it together is benchmarking on real traffic. Vendor throughput figures and published speedups are generated on idealized conditions; your prompts, your context lengths, and your concurrency profile will produce different numbers. Measure latency and cost per token on a representative sample of your own workload, re-measure when models or tiers change, and let those numbers, not the marketing, drive the choice.