Transformers & Attention, for Architects
The transformer is the architecture behind essentially every model an enterprise will evaluate, and its core mechanism, self-attention, is simpler than its reputation. Each token is projected into query, key, and value vectors; a token's query is scored against every other token's key, and those scores weight how much of each value flows into its updated representation. Because this is all matrix multiplication, an entire sequence is processed in parallel, which is why transformers displaced recurrent networks: they turned sequence modeling into work GPUs are good at. Stacked into dozens of layers with feed-forward blocks, the result is a function with billions of parameters whose quality improves predictably with data and compute, the scaling laws that justified frontier training budgets. What an architect needs to retain: attention cost grows quadratically with sequence length, and everything the model knows is learned statistics, not a queryable knowledge base. Both facts drive cost and failure modes downstream.
The Training Pipeline: Pretraining, SFT, RLHF
Every production model goes through roughly the same three-stage pipeline, and each stage explains a different behavior you will observe. Pretraining runs next-token prediction over trillions of tokens of web, code, and licensed text, producing a base model that is a powerful pattern completer with no notion of being helpful. Supervised fine-tuning (SFT) then teaches the instruction-following format using curated prompt-response pairs, on the order of tens to hundreds of thousands of examples. Finally, preference optimization, classically RLHF with a learned reward model and increasingly simpler methods like DPO, tunes the model toward responses humans rate highly. This last stage is where helpfulness, refusal behavior, sycophancy, and hedging come from: they are trained-in preferences, not reasoning. For an architect, the practical implication is that model behavior is a product of vendor alignment choices you cannot see, so behavioral regression testing across model versions is not optional, it is a core operational control.
The Model Landscape: Frontier, Open-Weight, MoE, SLMs
The model market has settled into a few distinct families, and matching workload to family is now a first-order architecture decision. Frontier models from the large labs lead on raw capability and are consumed as managed APIs. Open-weight models (the Llama, Mistral, Qwen, and DeepSeek lines) trail the frontier by months rather than years and can be run wherever you control the hardware. Cutting across both is mixture-of-experts (MoE): instead of activating all parameters per token, a router selects a few expert subnetworks, so a model can carry hundreds of billions of total parameters while paying inference cost on perhaps a tenth of them. At the other end, small language models in the 1B to 12B range, often distilled from larger teachers, handle classification, extraction, and routing at a fraction of the cost. The architect's job is portfolio construction: reserve frontier capacity for hard reasoning, and push volume workloads down-market.
Reasoning Models & Test-Time Compute
Reasoning models are trained, typically with reinforcement learning on verifiable problems, to produce a long internal chain of thought before answering, effectively spending inference compute to buy answer quality. This is test-time compute as a dial: the same weights can produce a fast, cheap answer or think for tens of thousands of tokens on a hard problem, and accuracy on math, code, and multi-step planning scales with how long the model deliberates. The costs are real: latency moves from seconds to minutes, token bills multiply because you pay for the thinking tokens, and the deliberation is largely invisible or summarized in most APIs. The architecture pattern that follows is routing, not defaulting: send the 5 percent of traffic that genuinely involves multi-step analysis to a reasoner, keep conversational and extraction workloads on standard models, and cap thinking budgets so a pathological query cannot burn unbounded spend.
Multimodal Models: Vision, Audio, Realtime
Multimodal models accept and increasingly generate images, audio, and video alongside text, and the dominant design maps each modality into the same token space: a vision encoder turns image patches into embeddings the language model attends over exactly as it does text tokens. In practice the enterprise killer application is document understanding: invoices, engineering drawings, scanned contracts, and dashboard screenshots go straight to the model, replacing brittle OCR-plus-template pipelines that broke on every new layout. Native audio models and realtime APIs go further, streaming speech in and out over a persistent connection (typically WebSocket or WebRTC) with sub-second turn latency, which makes voice agents viable without a separate speech-to-text stack. The architect's cautions: image tokens are expensive, often hundreds to thousands per image; vision accuracy on dense tables and small fonts still needs task-level evaluation; and realtime voice changes your infrastructure profile from stateless request-response to stateful sessions that must be load-balanced and secured accordingly.
Context Windows & Long-Context Trade-offs
The context window is the number of tokens a model can attend over in a single request, and it has grown from 4k a few years ago to 128k as a standard and 1M or more at the high end. Growth came from engineering, not magic: rotary position embeddings extended and interpolated, attention variants like grouped-query and sliding-window attention, and heavy KV-cache optimization, because naive attention cost scales quadratically and the KV cache alone can consume tens of gigabytes of GPU memory at long lengths. Two facts temper the headline number. Effective recall degrades well before the advertised limit, the documented lost-in-the-middle effect, where facts buried mid-context are retrieved less reliably than those at the edges. And context is not memory: it is rebuilt and repaid on every call, so stuffing a corpus into the window is usually the most expensive and least reliable alternative to retrieval. Long context complements RAG; it rarely replaces it.
Reading Benchmarks Without Being Fooled
Public benchmarks are the industry's scoreboard, and they mislead in well-understood ways. Classic academic suites like MMLU are effectively saturated, with frontier models compressed into a few points near the ceiling, so differences there tell you little. Worse, benchmark questions leak into pretraining corpora, and this contamination can inflate scores on material the model has memorized rather than learned to solve. The corrective instruments each have their own biases: arena-style Elo from human head-to-head votes rewards confident, well-formatted answers as much as correct ones, and agentic suites like SWE-bench Verified measure a specific harness and scaffold as much as the underlying model. The architect's discipline is to treat leaderboards as a shortlisting filter only, then build a private evaluation set from your own tickets, documents, and edge cases, roughly 100 to 500 graded examples, and let that decide. A model that wins your eval and loses the leaderboard is the right model.
Open-Weight vs Closed: the Enterprise Decision
The open-versus-closed question is less an ideology than a placement decision for each workload. Closed frontier APIs buy you peak capability, fast model refresh, and zero infrastructure burden, at the price of per-token pricing you do not control, data flowing to a third party under contractual rather than physical guarantees, and exposure to deprecation schedules. Open-weight models invert the trade: full control over residency, versioning, and fine-tuning, and at sustained high volume a self-hosted 70B-class model on reserved GPUs can undercut API pricing meaningfully, but you now own serving infrastructure, capacity planning, security patching, and an MLOps team. Regulated industries add a forcing function: some workloads simply cannot leave the boundary, whatever the capability delta. In practice, mature enterprises converge on a hybrid portfolio: frontier APIs for the hardest reasoning, open-weight models inside the perimeter for sensitive or high-volume paths, and a routing and abstraction layer so switching costs stay low as the market moves.