Reasoning Models & Test-Time Compute: Buying Quality with Tokens

A new scaling axis: compute at inference time

For a decade the recipe for a better model was a bigger pretraining run: more parameters, more data, more GPUs for more months. That axis still works, but each increment of quality now costs disproportionately more. Frontier training runs are reported on the order of hundreds of millions of dollars, and the marginal gain per dollar has visibly flattened. Enterprises felt this indirectly: model generations improved, but the improvements arrived on the vendor's schedule and at the vendor's chosen price point, and there was nothing an architect could do about quality except pick a bigger model.

Starting in late 2024 the field opened a second axis. OpenAI's o1 preview (September 2024) popularized the idea of a model that thinks before answering, generating a long internal chain of reasoning tokens and only then producing its reply. DeepSeek-R1 (January 2025) showed the same recipe working in an open-weight model with the full chain of thought exposed. The consistent finding across this lineage: on hard, checkable tasks such as competition math and difficult code, accuracy climbs as the model is allowed more thinking tokens, roughly smoothly over a useful range, before eventually flattening.

The architecturally interesting part is not the benchmark scores. It is that this new axis is controlled at request time. The same deployed model can answer a routine question in two seconds of light thinking or grind through a gnarly planning problem for a minute, and the caller decides which, per call, through effort or budget parameters.

How reasoning models actually work

A reasoning model is not a standard model with "think step by step" baked into its system prompt. The training recipe, as published for DeepSeek-R1 and broadly consistent with what other vendors have described, is reinforcement learning on tasks with verifiable answers: math problems whose results can be checked mechanically, code that either passes a test suite or does not. The model generates a long chain of thought, the final answer is verified, and reward flows back. Whatever internal behaviors made verified answers more likely get reinforced, at scale, over enormous numbers of attempts.

What emerges from that loop is a recognizable set of search behaviors visible in exposed reasoning traces:

• Decomposition: breaking a problem into subgoals and solving them in sequence.
• Self-checking: re-deriving an intermediate result to confirm it before building on it.
• Backtracking: noticing an approach is failing, abandoning it, and restarting from an earlier point.
• Alternative generation: producing several candidate approaches and comparing them before committing.

This is the substantive difference from prompted chain-of-thought. A "let's think step by step" prompt rearranges behaviors the base model already has; it produces the appearance of deliberation, and it does help on some tasks. But prompting cannot teach a model a new search policy, because nothing in prompting rewards knowing when to abandon a dead end. A trained reasoner backtracks because backtracking was reinforced thousands of times during training. The gap shows up exactly on the problems where the first plausible approach is wrong.

One operational caveat: visibility varies by vendor. The o-series hides or summarizes its thinking (you are billed for tokens you never see), while R1-lineage open models expose the full chain. And even a fully visible chain is best treated as a debugging aid, not an audit-grade explanation: there is no guarantee the prose faithfully describes the computation that produced the answer.

The cost and latency math

The billing rule is simple and its consequences are not: reasoning tokens are billed as output tokens, and output tokens are typically the expensive direction on any price sheet. A task a standard model answered in 300 output tokens might consume 3,000 to 20,000 thinking tokens plus the answer when routed to a reasoner at high effort. As a planning figure, a 5x to 100x per-task cost multiplier versus a direct answer is a realistic envelope, with the wide range driven by task difficulty and the effort setting.

All figures are illustrative orders of magnitude, not quotes; check your vendor's price sheet and measure your own tasks. The latency profile is the part that surprises teams. Prefill is unchanged, so where the vendor streams reasoning (or summaries of it) time to first token barely moves; where thinking is fully hidden, the first visible answer token waits behind the entire chain. Either way, what grows is total completion time, roughly linearly with thinking length. A request that thinks for 20,000 tokens is a request your user, your timeout configuration, and your connection pool all wait on for the better part of a minute.

The mitigations are the effort and thinking-budget controls that several vendors now expose. Treat a maximum thinking budget the way you treat a timeout: explicit, enforced, and set per route rather than globally. The broader latency mechanics are covered in Latency Anatomy.

When to route to a reasoner

Reasoners earn their multiplier on tasks with two properties: a solution space that rewards search, and enough structure that intermediate steps can be checked. In practice that means multi-step math and quantitative analysis, hard code (multi-file debugging, algorithm design, tricky migrations), planning and constraint satisfaction (scheduling, configuration, resource allocation), and acting as a judge or verifier over another model's work. On these, the accuracy gap over a strong standard model is often the difference between usable and not.

The multiplier is wasted on extraction, classification, templated generation, and routine summarization. Frontier non-reasoning models already sit near the quality ceiling on these tasks, so thinking tokens buy nothing, and on trivially easy inputs extended thinking has been observed to hurt, with the model talking itself out of an obviously correct answer. Latency-sensitive interactive paths are equally poor candidates regardless of difficulty.

Because the cost gap is an order of magnitude or two, routing is where the economics are decided. Three policies cover most enterprise deployments:

1. Complexity heuristics: route on observable request features: task category, input length, presence of code or math, workflow stage. Crude, cheap, and a strong first version.
2. Try cheap, then escalate: the cheap model answers first; failed validations, low-quality judge scores, or an explicit user retry escalate the same request to a reasoner. Most traffic never escalates.
3. Confidence signals: escalate when self-consistency sampling disagrees, a verifier rejects the draft, or schema and unit checks fail. Highest precision, most engineering.

All three belong in the model gateway, next to fallback and quota logic, not scattered through application code. The gateway pattern is detailed in Platforms.

Hybrid patterns: reasoners inside the architecture

The most cost-effective deployments as of mid-2026 do not choose between cheap models and reasoners; they compose them so the expensive thinking happens exactly once, where it matters. Four patterns recur.

Planner-executor split. The reasoner produces a plan: a step list, a query strategy, a refactoring spec. A cheap model then executes each step. You pay reasoning prices for the one artifact that benefits from search and commodity prices for the volume work. Reasoner as verifier. Invert it: a cheap model drafts, the reasoner judges, selects among candidates, or reviews only the high-stakes outputs. Verification is often an easier task than generation, so even a modest thinking budget goes far here.

                 user request
                       |
                       v
              +----------------+
              |     ROUTER     |
              +----------------+
              simple |   | hard
                     v   v
        +---------+     +----------+
        |  CHEAP  |     | REASONER |
        |  MODEL  |     | (plans)  |
        +---------+     +----------+
             |               | step plan
             |               v
             |          +----------+
             |          |  CHEAP   |
             |          | EXECUTOR |
             |          +----------+
             v               |
           response <--------+
            

Escalation ladders chain these: small model, then frontier standard model, then reasoner at low effort, then high effort, each rung gated by a validation signal, so each request stops at the cheapest rung that passes. Batch offline reasoning removes latency from the equation entirely: run the reasoner overnight to precompute artifacts (canonical answers to your top thousand questions, extraction schemas, remediation playbooks, plans awaiting human approval) that cheap models consume at runtime. Offline work also tends to qualify for discounted batch pricing. A working escalation loop is on display in the Self-Correcting Agent in the Lab.

Evaluating reasoning quality

Public benchmarks are the wrong instrument for reasoner selection. The famous math and code suites are saturated or near-saturated at the frontier, the deltas between top models sit inside contamination and prompt-sensitivity noise, and none of them price in the tokens spent. A model that scores two points higher while thinking four times longer is not obviously better; for most enterprise routes it is worse. How to read leaderboards skeptically is covered in Reading Benchmarks.

Overthinking deserves particular respect because it is counterintuitive. It is a documented failure mode, not an anecdote: past a task-dependent point, more thinking degrades accuracy. The model second-guesses a correct early answer, wanders into irrelevant branches, or manufactures complexity that is not in the problem. The practical response is a budget sweep: run your eval set at several thinking budgets, plot quality against tokens, and find the knee. Then set that budget in the gateway, per route.

Evaluate on your tasks with outcome checks first (verifiable endpoints: correct extraction, passing tests, valid plans), plus periodic human spot review of reasoning traces where the vendor exposes them, to catch process failures that happen to reach right answers. And measure variance: run every case three to five times, because a reasoner that is right on 60 percent of runs is a different product from one right on 95 percent, and a single-run eval cannot tell them apart. The full quality-system playbook is in Evals Are the New Unit Tests.

What this changes for architects

The lasting change is that inference budget joins latency and availability as a first-class design parameter. Every route through your system should carry an explicit thinking budget the way it carries an SLA: written down, enforced at the gateway, and reviewed when the numbers drift. "How much should this endpoint think" is now a design review question with a wrong answer in both directions.

Capacity planning changes shape. A request that generates 20,000 tokens occupies KV-cache memory and a concurrency slot for minutes, not seconds, so throughput math, timeout policy, retry behavior, and autoscaling signals all need to assume minute-scale requests on reasoning routes. Undersizing here produces the worst kind of incident: queues that build silently behind a handful of long thinkers.

Product design inherits a genuinely new question: when is quality worth a wait? The emerging answers are architectural: synchronous fast paths for everything routine, explicit thinking affordances (progress indicators, streamed reasoning summaries) where users will wait for a hard answer, and asynchronous delivery for deep work, where the reasoner runs for minutes and the result arrives as a notification or a document. Forcing every interaction through one latency profile wastes the new axis.

The discipline, as ever, is measurement before enthusiasm. Test-time compute is a dial, and dials invite maximalism. The teams that win with reasoners will be the ones that know, route by route and in numbers, where thinking pays for itself, and quietly spend nothing everywhere else. Where the research frontier goes next is tracked in Frontier: Test-Time Compute.