AI Compute: The Hardware Under the Tokens

The layer under the tokens

The token is a convenient abstraction. You pay per million of them, you budget latency around them, and most of the Atlas reasons about them without asking where they come from. But every token is produced by a physical device doing matrix multiplication, and the properties of that device explain most of what you experience at the API boundary: why prices sit where they sit, why decode latency has a floor no prompt engineering can break, why capacity gets rationed during demand spikes, and why the industry's news cycle is dominated by chip supply and data-center construction.

You do not need to be a hardware engineer to run enterprise AI well. You do need a working mental model of the compute layer, for three practical reasons. First, sanity-checking: vendors quote throughput numbers that are technically true and operationally misleading, and the difference is visible only if you know what the numbers omit. Second, capacity planning: whether you consume tokens through a managed API or serve models yourself, someone in the chain is doing tokens-per-second-per-device arithmetic, and if it is not you, you are trusting theirs. Third, reading the news: accelerator announcements, memory supply constraints and interconnect roadmaps move the cost curve that your multi-year plan quietly depends on.

This article is the tour of that layer. We start with why GPUs won the workload, then get to the less obvious and more important point: memory, not raw compute, is usually the wall. From there we survey the alternative accelerator classes and what they trade, work through the capacity arithmetic that converts specs into real throughput, and finish with the procurement decision, buy, rent or reserve, in a market where scarcity is a recurring feature rather than a bug.

Why GPUs won

A transformer forward pass is, overwhelmingly, matrix multiplication. Attention layers, feed-forward layers, the projections between them: nearly all of the arithmetic is multiplying large matrices of numbers, and almost none of it is branching, pointer-chasing or complex control flow. That workload shape is the whole story. The question was never which chip is fastest in general; it was which chip is fastest at enormous volumes of very regular linear algebra.

CPUs are built for the opposite problem. A modern CPU spends most of its silicon making a handful of cores individually clever: deep caches, branch prediction, speculative execution, all in service of running unpredictable general-purpose code with low latency. A GPU inverts the bet. It spends its silicon on thousands of simple cores that execute the same operation across large blocks of data in lockstep. For branchy code this is a terrible trade. For matrix multiplication, where every element follows the identical recipe, it is close to optimal: throughput per chip, per watt and per dollar all favor the many-simple-cores design by a wide margin.

Modern accelerators go a step further with dedicated tensor units: hardware blocks that execute small matrix-multiply-accumulate operations as a single instruction, often at reduced numeric precision, because neural networks tolerate lower precision remarkably well. These units accelerate exactly, and almost only, the operation transformers spend their lives doing.

One honest caveat: hardware alone did not decide the outcome. A decade of accumulated kernels, libraries, tooling and developer habit forms a moat around the dominant software ecosystem that rival silicon has found genuinely hard to cross. When you evaluate alternatives, the software maturity question usually matters more than the hardware question.

Memory is the wall

Here is the part vendor slides underplay: for serving large language models, the binding constraint is usually memory, not compute. It binds twice, in two different ways, and the distinction matters.

Capacity decides what fits. A model's weights must live in the accelerator's high-bandwidth memory (HBM) to be served at speed. Billions of parameters times bytes per parameter is a hard budget: it determines which models fit on one device, which need quantization to squeeze in, and which must be sharded across multiple devices connected by high-bandwidth interconnect, with all the cost and failure-mode complexity sharding brings.

Bandwidth decides how fast you decode. Generating output is sequential: one token per step per sequence, and each step must stream essentially all of the active weights from HBM through the compute units. The arithmetic per byte moved is small, so the chip spends most of its time waiting on memory rather than multiplying. This is why the decode phase is memory-bandwidth-bound while prefill, which processes the whole prompt in parallel, is more compute-bound, exactly the split described in Latency Anatomy. It is also why per-token decode speed tends to improve with new memory generations more than with new compute generations.

And the same memory is contested. The KV cache, the stored attention state for every active sequence, grows with context length and batch size and lives in the same HBM as the weights. Long contexts and big batches eat the memory that batching needs, which caps concurrency, which caps throughput.

+------------------------------------------------------+
|  On-chip caches / SRAM     tiny, fastest             |
+------------------------------------------------------+
|  HBM: weights + KV cache   capacity = what fits      |
|                            bandwidth = decode speed  |
+------------------------------------------------------+
|  Peer devices over         sharding, when one        |
|  high-bandwidth links      device is not enough      |
+------------------------------------------------------+
|  Host DRAM and storage     large, far too slow       |
|                            for the decode loop       |
+------------------------------------------------------+

Beyond GPUs

GPUs won by being very good at everything neural. That generality has a price, and a growing family of accelerators bets that giving some of it up buys efficiency. The classes matter more than the brand names, so this section stays deliberately generic.

Training-scale accelerators in the TPU style are built around very large matrix-multiply engines and deploy in tightly coupled pods with fast collective communication, because training is a synchronized, all-devices-talking workload. They can be extremely efficient at exactly that job, but they are typically bound to one cloud provider's stack, which makes adopting them as much a strategic commitment as a technical choice. Inference-specialized silicon takes the opposite cut: narrower model shapes, aggressive use of on-chip memory, dataflow designs tuned for the decode loop, all chasing low latency and high tokens per watt. The recurring risk is software and model coverage: model architectures evolve faster than silicon, and a chip tuned for last year's shapes can age quickly. At the small end, NPUs in phones and laptops run quantized small models on-device, trading capability ceiling for privacy, offline operation and per-unit economics, the pattern covered in Edge AI and Small Language Models.

The evaluation rule is constant across classes: benchmark your workload, on your models, through the vendor's actual software stack. Specs describe the silicon; the stack decides what you get.

Capacity arithmetic

Every serving capacity plan reduces to one line: real throughput equals tokens per second per device, times utilization, times batching efficiency. Each factor is a place where optimistic plans go to die, so take them one at a time.

Tokens per second per device is what benchmarks report, and the number is meaningful only with its conditions attached: which model, which quantization, what context lengths, what batch size. A figure quoted at ideal batch on short prompts can be several multiples of what your traffic will see. Batching efficiency captures the gap between that ideal and your reality: real request streams have uneven arrivals, mixed prompt lengths and long-context outliers that fragment batches, and the KV cache limits how many sequences fit in flight, as covered in Inference Optimization. Utilization captures time: diurnal load curves, provisioning for peak, failover headroom and maintenance all mean the fleet spends much of its life below capacity.

This is why utilization, not peak specs, decides economics. Two operators with identical hardware can see cost-per-token profiles that differ by multiples, purely on how well they batch and how full they run. It is also why managed API providers, who pool demand across thousands of tenants, can often beat the unit economics of a self-hosted cluster serving one spiky internal workload. Before comparing per-token prices, run this arithmetic on your own assumptions; the Cost Estimator in the Lab exists for exactly that exercise.

Buy, rent or reserve

Once you know your effective tokens per second, the procurement question is classic capacity planning: match a demand curve to a supply commitment. What makes AI compute unusual is the supply side, which in recent years has cycled between scarcity and glut, with allocation queues, long lead times for current-generation hardware, and data-center power and grid interconnection emerging as real siting constraints, a theme the sustainability deep-dive covers from the energy angle.

The three postures trade flexibility against unit cost and certainty. On-demand cloud capacity is the most flexible and typically the most expensive per unit, and it carries a risk that surprises teams: during scarcity phases, the capacity you assumed was elastic may simply not be available in your region when you need it. Reserved or committed capacity trades a one-to-multi-year commitment for better rates and, often more valuable, an actual availability guarantee. Owned hardware can win on unit economics at sustained high utilization, but you inherit procurement lead times, depreciation risk against a fast-moving hardware curve, and the operational burden of running accelerator fleets, a skill set most enterprises do not have and should think hard before building.

For enterprises consuming models through managed APIs, the provider has made this decision for you, but it still leaks through as rate limits, provisioned-throughput tiers and regional availability. The same logic applies one level up: committed API throughput is reserved capacity by another name.

The architect view

You will probably never buy a rack of accelerators, and you do not need to. But every consequential decision in this domain, model choice, latency budget, self-host versus API, vendor contract, sits on top of hardware assumptions, and the architects who surface those assumptions make visibly better calls than those who treat the API as the bottom of the stack.

• Know where the wall is. Memory capacity decides what fits per device; memory bandwidth decides decode speed; the KV cache competes for both. When a latency or cost conversation gets stuck, it is usually stuck here.
• Plan in effective tokens per second. Device throughput times utilization times batching efficiency, measured on your traffic, not the datasheet's.
• Put hardware assumptions in the TCO model. Utilization targets, batching assumptions, capacity commitments and hardware-generation risk belong in the spreadsheet as named line items, because they move the answer more than the per-token price does.
• Distrust claims that ignore batching and utilization. Any throughput or cost-per-token figure quoted without those conditions is a peak spec wearing a business case.
• Watch the supply market like a dependency. Scarcity cycles, lead times and power constraints are planning inputs, not background noise.

The tokens are the product. The silicon is the cost structure. An architect who can read both layers can tell the difference between a price that reflects physics and a price that reflects negotiation, and that distinction is worth real money on every contract you sign.