The Context Engineering Playbook: Budgets, Memory and Rot

From prompt engineering to context engineering

For a couple of years the industry treated output quality as a wording problem. Teams traded incantations, benchmarked adjectives, and hired for a skill that amounted to persuasive phrasing. That era ended quietly. Frontier models as of mid-2026 are largely robust to how you ask; rewording a competent instruction typically moves quality by small margins, while changing what information reaches the window moves it by multiples. The model is rarely the bottleneck anymore. The window is.

A production AI system does not receive a prompt. It receives an assembled payload: a system prompt, tool definitions, retrieved documents, memory, and a growing conversation transcript, all competing for a finite token budget. Every token in that payload costs money, adds latency, and dilutes the model's attention across everything else. The question that decides quality is no longer "how do I phrase this" but "what earns a place in the window, at what fidelity, in what order, and when does it leave."

Context engineering is the discipline that answers those questions. The defining move is to treat context the way you treat any other constrained production resource, like a connection pool or a cache tier: it has a budget, per-component allocations, instrumentation that shows composition per request, and a change process. The system prompt is one component among six, not the whole game.

This reframing matters to architects because it converts a dark art into reviewable engineering. A prompt is a string someone tweaks; a context pipeline is a design you can diagram, budget, test, and hold to an SLO. The rest of this article works through that design: the anatomy of a production window, how it degrades, how to compress it, what to persist beyond it, and how to lay it out so the cache works for you.

Anatomy of a production context window

Most production windows contain six components, each with its own change frequency and failure mode. The system prompt and tool definitions are static per release. Memory and retrieved knowledge change per session or per turn. Conversation history only grows. The current turn plus output headroom is whatever is left, and it is the part users actually feel when it runs out.

+------------------------------------------------------+
| System prompt          ~2-5%    static per release   |
+------------------------------------------------------+
| Tool definitions       ~5-15%   static per release   |
+------------------------------------------------------+
| Memory / state         ~3-8%    changes per session  |
+------------------------------------------------------+
| Retrieved knowledge    ~10-30%  changes per turn     |
+------------------------------------------------------+
| Conversation history   ~30-50%  append-only, grows   |
+------------------------------------------------------+
| Current turn + output headroom (never let this hit 0)|
+------------------------------------------------------+
            

The percentages deserve the same treatment as a cloud cost allocation: set them deliberately, then measure drift. Tool definitions are the band teams most often underestimate; an agent with forty verbose tool schemas can burn a five-figure token count before the user says a word. Retrieved knowledge is the band with the highest variance, which is why retrieval depth (how many chunks, at what size) belongs in configuration, not hardcoded in the RAG layer.

Two rules fall out of the anatomy. First, budget top-down: decide the split before you build, because every component's owner will otherwise expand to fill the window. Second, reserve output headroom explicitly. A window that is 98 percent full does not fail loudly; it produces truncated, degraded answers that look like a model problem and are actually a capacity problem.

Context rot and long-session failure

Windows do not fail at the boundary; they degrade well before it. "Context rot" is the operational name for quality decaying as a session grows, and it has several distinct mechanisms that are worth separating because they have different fixes.

The first is attention dilution: as the window fills, the model's effective attention per token drops, and instructions that were reliably followed at turn 3 get missed at turn 40. The second is positional. Research since 2023 has documented "lost in the middle" effects, where models recall material at the start and end of a long context noticeably better than material buried in the middle. The practical consequence: critical constraints placed mid-window, where retrieved chunks and old turns tend to land, are the most likely to be silently ignored.

The third mechanism is accumulation of stale and contradictory instructions. The user changed the requirement at turn 12, but the old requirement still sits in the transcript with equal standing; the model now has two authorities and picks unpredictably. The fourth is poisoning: a bulky failed tool call, an error dump, or a retrieved document with misleading content stays in the window and keeps steering every subsequent turn.

Symptoms are consistent across systems: instruction drift, forgotten constraints, answers that get vaguer and more repetitive, an agent that re-attempts things it already ruled out. Teams typically misread these as model regressions and reach for a bigger model, which treats none of the four mechanisms.

Compression: summarization, compaction, pruning

Once you accept that history cannot grow forever, the design question becomes which compression strategy to apply where. All of them trade fidelity for headroom; the craft is choosing what fidelity you can afford to lose. Production agent harnesses, including the current generation of coding agents, ship compaction as a standard feature for exactly this reason.

Two implementation notes matter more than the taxonomy. First, compress asymmetrically: constraints, decisions, and identifiers must survive compression verbatim, while pleasantries and dead-end explorations can be discarded entirely. A summary that loses a customer ID is worse than no summary. Second, compaction is itself a model call, so it can hallucinate. Checkpoint briefs should be generated against a fixed template and, for high-stakes workflows, validated (do all task IDs in the brief exist, are all hard constraints carried over) before the old transcript is released.

The strongest teams stop thinking of compression as an emergency valve and treat it as scheduled maintenance: triggers defined in config, fired at thresholds, logged like any other state transition.

Memory that survives the session

Compression buys headroom inside a session. Memory is the complementary problem: what should outlive it. The useful mental model is three tiers with different lifetimes and write policies.

• Session memory is the window itself plus scratch state: everything here is disposable by design and should be treated that way.
• Episodic memory is a record of past sessions (what was tried, what was decided, what failed) stored outside the model and retrieved when relevant, usually through the same retrieval stack that serves documents.
• Long-term memory is a small, curated set of durable facts: user preferences, environment specifics, standing constraints. It is loaded eagerly at session start rather than retrieved on demand.

The architectural failure mode is writing too much. A system that persists every turn builds a landfill; retrieval then surfaces stale or contradictory memories and reintroduces the rot you engineered out of the window. The fix is an explicit write policy: persistence must be earned, typically by a rule ("user stated a durable preference", "a decision closed an open question") or by a model judgment call that is itself logged and auditable. Deletion policy matters just as much; memories should carry provenance and a review date.

A pattern that has proven durable in practice is the memory file: the agent maintains its own compact notes (a Markdown file per project or per user is common in coding agents) that it reads at startup and edits deliberately during work. It is transparent, diffable, and reviewable by humans, which makes it far easier to govern than an opaque vector store. Most mature systems end up hybrid: memory files for curated long-term facts, retrieval-backed stores for episodic bulk. The broader pattern catalog lives in Memory Architectures.

Designing for the cache

Everything so far is about quality. Window layout also decides cost and latency, because prompt caching only rewards you if your payload is structured for it. The mechanics are strict: caching discounts apply to stable prefixes. The provider caches the processed form of the first N tokens, and you get the discount only when the bytes preceding the cache point are identical to the previous request. One changed byte early in the payload invalidates everything after it.

That single rule dictates the ordering discipline. Static content goes first: system prompt, then tool definitions, unchanged between requests and between users where possible. Slowly changing content (memory, session state) comes next. Conversation history follows and must be append-only; if compaction rewrites it, you eat one cold request and then cache the new prefix. Volatile content, retrieved chunks and the current user turn, goes last where it can change freely without disturbing the prefix. Conveniently, this is the same ordering that serves attention well, with standing instructions at the positions models attend to most reliably.

The economics are material. Cached input tokens are priced at a significant discount to fresh ones (the exact ratio varies widely by provider and tier), and cache hits also cut time to first token because the prefix is not reprocessed. For a chatty agent whose payload is 80 percent stable prefix, cache-aware layout is often the single largest cost lever available, ahead of model downgrades. The detailed math is in Prompt Caching Economics.

The playbook

Context engineering rewards teams that treat it as configuration management rather than craft. The checklist below is the minimum operating standard I hold production systems to; none of it requires new tooling, only the decision to manage the window deliberately.

1. Set budgets per component. Allocate the window across system prompt, tools, memory, retrieval, history, and output headroom before you build. An unbudgeted component will grow until it crowds out the others.
2. Instrument token composition. Log the per-component token count on every request. You cannot detect budget drift, cache regressions, or a bloating tool schema without this one histogram.
3. Define compaction triggers in config. Fill thresholds, summarization templates, and pruning rules are policy, not code. They should be changeable without a deploy and observable when they fire.
4. Eval after any context change. Reordering components, tightening a summary template, or adding a tool is a behavioral change and can regress quality as surely as a model swap. Gate it the same way; the harness is described in the evals deep-dive.
5. Version the layout. The full context assembly (ordering, budgets, templates) is a versioned artifact with an owner, a changelog, and a rollback path, exactly like a schema or an API contract.

The through-line is simple. Prompt engineering asked what to say to the model. Context engineering asks what the model should be looking at, and accepts that the answer is a system: budgeted, compressed, persisted, cache-ordered, and versioned. Teams that make that shift stop chasing model upgrades to fix problems the window created, and start shipping assistants whose fortieth turn is as sharp as their first.