The Anatomy of LLM Latency

Latency has structure

Ask a team how fast their LLM feature is and you will usually get a single number back: a p95, a median, a round figure someone remembers from a demo. That number is real, but it is an average over two very different physical processes, and averaging them together throws away exactly the information you need to make the thing faster. LLM latency has internal structure. Learn the structure and the optimization work stops being guesswork.

An inference request runs in two phases. First the model reads the entire input prompt and builds the internal state it needs to answer: the prefill phase. Then it produces the answer one token at a time, each new token conditioned on everything before it: the decode phase. These phases run on the same hardware and the same weights, but they stress completely different resources. Prefill is bound by raw compute and can process the whole prompt in parallel. Decode is bound by memory bandwidth and cannot be parallelized across the tokens it generates, because token N+1 does not exist until token N has been emitted.

The consequence is that a single latency figure hides which phase is hurting you. A retrieval-augmented request stuffing 30,000 tokens of context into a two-sentence answer is prefill-heavy. A short prompt that asks for a 2,000-token report is decode-heavy. Both might show the same total, and the fix for one does nothing for the other. Measure them apart and the problem names itself.

TTFT versus per-token latency

The two phases surface as two metrics, and every serious latency conversation should name which one it means. Time-to-first-token (TTFT) is how long the user waits before anything appears. It is dominated by prefill: the model has to read the whole prompt before it can emit even the first output token, so TTFT scales with input length. Per-output-token latency, sometimes called TPOT or inter-token latency, is the steady cadence at which tokens arrive once generation is underway. It is set by decode, and it multiplies against every token in the answer.

These compose into a simple, useful approximation for total latency:

Read that bottom row carefully, because it is where most latency intuition goes wrong. Total latency is approximately TTFT plus the per-token time multiplied by the number of output tokens. TTFT is paid once. The per-token term is paid on every single token the model writes. So an answer that is twice as long is close to twice as slow in its decode portion, while a prompt that is twice as long mostly enlarges the one-time TTFT. The two knobs are not symmetric, and treating a global latency budget as one undifferentiated pool obscures that.

Numbers here are workload-specific and move with hardware, model, and load, so treat any figure as illustrative. What is durable is the shape: one fixed cost at the front, one recurring cost per output token. Once you hold that shape in mind, the next question is why the recurring cost is the one that usually hurts.

Prefill versus decode

The asymmetry comes down to parallelism. During prefill the model already has every input token in hand, so it can push them through the network together and compute their internal representations in one heavily parallel pass. Modern accelerators are built for exactly this kind of wide matrix work, so prefill is compute-bound and remarkably fast on a per-token basis: thousands of prompt tokens can be absorbed in the time it takes to generate a handful of output tokens.

Decode has no such luxury. Each output token depends on the one before it, so generation is inherently sequential, one forward pass per token. Worse, each of those passes is dominated by reading the model's weights and the growing attention cache out of memory rather than by arithmetic, which makes decode memory-bandwidth-bound. The compute units sit underused while data moves. This is the structural reason decode is the usual bottleneck: you cannot parallelize away a dependency chain, and you are gated by how fast memory feeds the chip, not by how fast it can multiply.

PREFILL  (one parallel pass over the whole prompt)

  [t1 t2 t3 ... t500]  --->  first output token
   \___ all at once ___/       (compute-bound, fast/token)

DECODE   (one pass per token, strictly sequential)

  o1 --> o2 --> o3 --> o4 --> ... --> oN
   |      |      |      |               |
   +-- each waits for the previous -----+
        (memory-bandwidth-bound, slow/token)

This is why the same total latency can have opposite cures. If prefill dominates, you are paying for input you may be able to shorten or cache. If decode dominates, which is the common case for anything that writes more than a sentence or two, no amount of prompt trimming helps much; the fix has to attack the per-token cost or the token count. The inference optimization deep-dive covers the serving-side machinery, the KV cache and batching, that decides how well the decode chain actually runs.

What drives latency

Five things move LLM latency in practice, and they act on different phases. Input length drives prefill and therefore TTFT: more prompt tokens is more to read before the first output. Output length drives decode and therefore the bulk of total latency, because every generated token pays the full per-token cost. Model size raises both, but bites hardest on decode, since a larger model means more weight data to stream from memory on each of the many sequential passes. Batching on the serving side trades individual latency for aggregate throughput. And network overhead adds a fixed tax on top of everything, particularly for hosted APIs.

The one people underweight is output length. Input length is visible; you can see the giant prompt you assembled. Output length feels free because you did not write it, but decode is sequential, so an answer twice as long takes close to twice as long to produce in its decode portion. A prompt that quietly invites a verbose response, no length limit, no format constraint, an open-ended please explain, is often the single largest latency cost in the whole request, and it is invisible until you measure token counts.

None of this means input length is free; a 100,000-token context has a real prefill cost and a real price. But for the interactive workloads most enterprises ship, chat, drafting, summarize-and-respond, the decode term dominates the clock, and output length is the term you most directly control from the application side without touching infrastructure at all.

Streaming and perceived latency

There is a persistent myth that streaming makes LLM responses faster. It does not. Streaming changes nothing about total latency: the model still runs the same prefill and generates the same number of tokens at the same per-token rate, and the final token arrives at the same wall-clock moment whether you streamed or waited. What streaming changes is when the user first sees progress. Instead of holding every token until the answer is complete and delivering it in one block, a streaming response emits each token as it is generated.

The effect is entirely perceptual, and perception is worth a great deal. A user who watches text appear a fraction of a second after they hit send experiences a responsive system, even if the full answer takes the same eight seconds it always did. A user staring at a spinner for those same eight seconds experiences a slow one. Streaming converts a long TTFT-plus-decode wait into a short TTFT followed by visible, steady progress, which reads as fast because humans forgive a system that is obviously working. This is why nearly every chat interface streams.

The distinction to hold onto is real speed versus perceived speed. Streaming is a perceived-speed technique; it buys user experience, not throughput or cost. If your latency problem is that a batch pipeline finishes too late, or that per-request cost is too high, streaming does nothing for you, because there is no human watching tokens arrive. Reserve it for interactive surfaces, and do not let a smooth-looking stream lull you into thinking the underlying latency is solved. How the stream is delivered over the wire, the protocol and the token framing, is the province of the LLM API anatomy deep-dive.

The latency levers

Because latency is two phases, the levers sort cleanly by which phase they attack, and that is the most useful way to hold them. Match the lever to the phase that dominates your workload and you get a result; reach for a decode lever on a prefill-bound problem and you have spent effort for nothing.

Shorter outputs and a smaller or faster model are the application-side moves: cap the tokens you ask for, and pick the lightest model that passes your evaluation. Prompt caching attacks prefill directly by storing the computed state for a stable prefix, a long system prompt, a fixed document, so repeated requests skip re-reading it and TTFT drops sharply; the mechanics and the pricing are in the prompt caching deep-dive. Speculative decoding attacks decode by letting a small draft model guess several tokens ahead that the target model then verifies in a single pass, cutting per-token latency without changing output quality when the guesses land.

Two cautions keep this honest. First, several of these levers trade against something: a smaller model may cost accuracy, aggressive output caps may truncate a genuinely needed answer, and speculative decoding's payoff depends on how well the draft predicts the target on your traffic. Second, the speedups are workload-specific, so treat any headline multiplier as illustrative until you have measured it on your own prompts. The inference optimization deep-dive goes deeper on the serving-side levers a self-hoster controls.

The architect view

The discipline that separates teams who improve latency from teams who merely complain about it is measuring the two phases separately. Instrument TTFT and per-token latency as distinct metrics, on real traffic, and the dominant phase becomes obvious. A dashboard that reports only total latency will tell you the feature is slow; a dashboard that splits prefill from decode tells you whether to shorten the answer, cache the prompt, or change the model. The first is a complaint, the second is a work item.

From there the rule is to optimize the phase that dominates your workload, not the phase that is easiest to reason about. Interactive chat that writes paragraphs is decode-bound, so its wins live in output length, model choice, and speculative decoding. A classifier fed enormous documents that returns a single label is prefill-bound, so its wins live in prompt caching and input trimming. Applying the wrong family of fixes is the most common way latency work produces graphs that do not move.

Finally, budget latency the way you budget cost, because the two are the same physics seen twice. Both are driven overwhelmingly by tokens, and output tokens most of all. Set an explicit latency target per surface, decompose it into a TTFT allowance and a per-token allowance, and hold designs to it the way you hold them to a cost-per-request ceiling. Latency that is measured, split, and budgeted is a manageable engineering quantity; latency treated as one mysterious number is a recurring surprise.