Transformers & Attention, Explained for Architects

Why architects need this mental model

Most production incidents with LLMs are not model failures. They are mismatches between what the team assumed the model was and what it actually is. Someone treated the context window as durable memory, or expected a retrieved fact to be recalled verbatim, or was shocked that the same prompt cost ten times more once the input grew. Each of these is predictable the moment you understand the machine underneath, and mysterious otherwise. The architecture is not academic trivia. It is the specification that governs cost, latency, and failure modes.

Three properties do most of the explanatory work, and all three come straight from the design. First, context is expensive, and expensive in a specific, quantifiable way that scales with how much you put in. Second, the output is a probability distribution over the next token, sampled one token at a time, not a retrieved answer. Third, there is no database inside the model. Facts are not stored as rows to be looked up; they are patterns compressed into billions of weights during training, reconstructed on demand and sometimes reconstructed wrongly.

Hold those three and most LLM behavior stops being surprising. Prompt sensitivity, context cost, hallucination, the value of retrieval, the ceiling on reliability: each is a direct consequence, not a bug the next release will quietly fix.

From tokens to vectors

A transformer never sees text. Before anything else happens, the input string is cut into tokens: sub-word chunks drawn from a fixed vocabulary, typically tens of thousands of entries. Common words are single tokens; rarer words, code, and unusual formatting fragment into several. This is why token counts rarely match word counts, and why your bill is denominated in tokens rather than characters. The mechanics and their cost implications are covered in Tokenization.

Each token id is then mapped to an embedding: a vector of numbers, often on the order of a few thousand dimensions, that positions the token in a high-dimensional space where geometric closeness encodes learned similarity. These vectors are not hand-designed; they are learned during training, so that tokens used in similar ways end up near each other. From this point on, the model operates entirely on vectors.

One problem remains. Attention, the core operation coming next, is order-agnostic: on its own it treats the input as a set, so "the cat sat on the mat" and a shuffled version look identical. Order has to be injected explicitly. The original 2017 architecture added fixed sinusoidal positional encodings to the embeddings; many modern LLMs instead use rotary position embeddings (RoPE), which encode relative position inside the attention computation. The specific scheme varies, but the requirement does not: without it, sequence would carry no meaning.

  "attention scales"
          |
          v   tokenize
  [ atten | tion | _sca | les ]     4 token ids
          |
          v   embed
  [ v1 ][ v2 ][ v3 ][ v4 ]          vectors, ~thousands of dims
          |
          v   + position
  [ v1'][ v2'][ v3'][ v4']          order now encoded
          |
          v
     into the stack
            

Attention: how tokens read each other

Attention is the mechanism that lets a token's representation depend on the other tokens around it, and it is the whole reason the architecture is named for it. The intuition is routing: at each position, the model decides which other positions are relevant and pulls information from them, weighted by relevance. The word "bank" gathers different context in "river bank" than in "bank account," and attention is how that context reaches it.

The mechanism is built from three learned projections of each token vector. The query represents what this token is looking for. The key represents what each token offers as a match. The value is the information a token contributes once matched. For every token, the model compares its query against every token's key using a dot product: a large dot product means a strong match. Those scores are scaled and passed through a softmax, which turns them into weights that sum to one, and the weights blend the value vectors into a new, context-aware representation of the token.

That is the entire operation: score with queries and keys, normalize with softmax, average the values. The phrase you will see in papers, scaled dot-product attention, describes exactly these steps. No lookup table, no rules, just similarity-weighted averaging where the notion of similarity was learned during training.

Decoder-only LLMs add one constraint: a token may attend to earlier tokens and itself, but not to future ones. This causal masking is what makes left-to-right next-token prediction well-defined, and it is why the model generates one token at a time rather than filling in a sentence all at once.

Multiple heads, stacked layers

A single attention operation can only express one notion of relevance at a time. Language needs many at once: a verb relating to its subject, a pronoun to its antecedent, a bracket to its match, a word to the topic three sentences back. Multi-head attention solves this by running several attention operations in parallel, each with its own query, key, and value projections, so each head can specialize in a different kind of relationship. Their outputs are concatenated and combined, giving the layer a richer, multi-relational read of the sequence.

Attention is only half of each layer. After it comes a feed-forward network, a per-token transformation that does much of the model's raw pattern storage and non-linear computation. Two details make deep stacks trainable. Residual connections add each sub-layer's input back to its output, giving gradients a clean path and letting layers refine rather than replace. Layer normalization keeps the numbers in a stable range. This block, attention then feed-forward, each wrapped in a residual, repeats dozens to over a hundred times.

Depth is where abstraction comes from. Empirically, earlier layers track surface features (tokens, syntax, local phrases) while later layers carry more abstract, task-relevant structure. No layer is programmed to do this; the division of labor emerges from training. For an architect, the takeaway is that capability scales with depth and width, which is precisely why frontier models are large and why running them costs what it does.

Why it scaled when RNNs did not

The architecture that transformers displaced was the recurrent neural network. An RNN reads a sequence one step at a time, carrying a hidden state forward: to process token 500 it must first process tokens 1 through 499 in order. That sequential dependency is fatal on modern hardware. GPUs are massively parallel, thousands of operations at once, and an RNN's step-by-step recurrence leaves most of that silicon idle during training.

The 2017 paper "Attention Is All You Need" (Vaswani et al.) removed the recurrence entirely. Because attention compares all positions with all positions in one operation, a transformer can process every token in a training sequence in parallel. The all-pairs design that costs so much at long context is exactly what makes training saturate a GPU. This is the pivotal trade the architecture makes: it spends more total computation, but arranges it so the hardware can do it all at once.

That single change unlocked the modern era. Once training parallelized, teams could train far larger models on far more data economically, and the empirical scaling laws emerged: loss falls predictably and smoothly as parameters, data, and compute grow together. Capability became something you could plan and budget for rather than stumble upon, and the industry poured capital into ever larger runs. Two properties of the transformer made all of it possible.

• Training parallelism: a whole sequence is processed at once, so GPUs run near full utilization instead of waiting on a recurrence.
• Predictable returns: scaling laws made more compute reliably buy more capability, which justified the investment that produced today's frontier models.

One asymmetry is worth noting: training parallelizes across the sequence, but generation does not. Producing text is still autoregressive, one token at a time, because each new token depends on the ones before it. Training is embarrassingly parallel; inference is stubbornly sequential, and that gap shapes serving cost.

The cost hidden in the design

The all-pairs attention that makes transformers powerful carries a bill. Comparing every token with every token means work that grows with the square of the sequence length: double the input and attention does roughly four times the work, not twice. In notation, self-attention is O(n²) in sequence length n. This is not an implementation detail to be optimized away; it is the shape of the computation, and it is the reason long context is expensive rather than merely large.

Two engineering advances make long context practical without changing that shape. FlashAttention and similar kernels reorganize the computation to use memory far more efficiently and run faster on real hardware. They are genuinely important, but they optimize the constant factors and the memory traffic; the asymptotic O(n²) scaling is unchanged. A 100,000-token prompt still costs on the order of a hundred times the attention work of a 10,000-token prompt.

The second advance targets generation. Because decoding is one token at a time and each new token attends to all prior tokens, a naive implementation would recompute every past token's keys and values at every step. The KV cache stores those keys and values from prior steps and reuses them, so each new token only computes its own. This is what makes token-by-token generation affordable, and it is also why serving long contexts consumes so much GPU memory: the cache grows with every token in the window.

Figures are illustrative orders of magnitude to show the shape, not vendor quotes. The practical consequence is direct: context length is a primary cost and latency lever, so a retrieval step that puts only the relevant few thousand tokens in front of the model usually beats stuffing the entire corpus into a giant window. The full economics are worked in LLM Economics.

What it means for how you build

Read the architecture forward and the design rules write themselves. Context is not memory. The window is working state that the model re-reads in full on every call, at quadratic cost, and it vanishes when the request ends. Persistence is your job: a database, a retrieval layer, a session store. Treating the context window as durable memory is the single most common and most expensive misconception in production systems.

Prompts steer a predictor. A prompt does not query a knowledge base; it sets the conditions under which the model computes the most probable next token. That is why phrasing, examples, and ordering move outputs so much, and why prompting is real engineering rather than incantation. You are shaping a distribution, not issuing a command.

Hallucination is fluent extrapolation, not a lookup miss. Since facts live as patterns in weights rather than rows in a store, the model has no internal boundary between recalling something and plausibly reconstructing it. When the pattern is thin, it produces confident, well-formed text that happens to be wrong, and it cannot reliably tell the two apart. Grounding through retrieval and verifying consequential outputs is not defensive polish; it is compensating for a property of the machine. Retrieval patterns are covered in Context Engineering.

One last orientation point. Nearly all general-purpose LLMs today are decoder-only: a single stack that reads left to right and predicts the next token, the design this article has described throughout. The encoder-decoder split from the original paper still suits translation-style tasks, but the decoder-only transformer is the dominant shape of the models you will actually deploy, and understanding its one core operation, attention, explains most of what they can and cannot do.