Agent Memory: Session, Episodic and Long-Term

The context window is not memory

The single most common category error in agent design is treating the context window as memory. A larger window feels like a better memory, so teams reach for the biggest one on offer and assume the remembering problem is handled. It is not. The window is working state: the tokens the model is attending to right now, for this request. When the session ends, or when compaction rewrites the transcript, that state is gone. Nothing about a long context makes it persistent.

Memory is the property of surviving the session. Yesterday's decision, a preference the user stated three weeks ago, the outcome of a task the agent ran last month: none of that lives in the window unless something outside the model puts it there. That something is an external system, typically a database or a vector store, whose relevant entries are retrieved into the window at the start of a turn and dropped again at the end. The window is the desk; memory is the filing cabinet. Confusing the two produces agents that are articulate within a session and amnesiac across them.

This distinction is not pedantry, because the two have opposite economics and opposite failure modes. Window space is scarce, expensive per token, and degrades with length; you want it small and fresh. Memory is cheap to store, effectively unbounded, and its risk is the reverse: it accumulates stale and contradictory material that quietly poisons future turns. You cannot solve a memory requirement by buying window, and you cannot solve a context-rot problem by storing more.

The tiers of memory

Human-memory analogies are imperfect, but the taxonomy they suggest is genuinely useful for design, because each tier has a different lifetime, a different write trigger, and a different read path. Four tiers cover almost every agent I have built or reviewed. Treating them as one undifferentiated store is how memory systems rot.

  WORKING     the current window: this turn's context,
  (volatile)  scratch reasoning. Dies with the session.
     |
     v  what happened?
  EPISODIC    past interactions: sessions, events,
  (log)       outcomes. Retrieved when relevant.
     |
     v  what is true?
  SEMANTIC    durable facts: user preferences, domain
  (facts)     knowledge, standing constraints.
     |
     v  how to do it?
  PROCEDURAL  learned methods: workflows and skills
  (skills)    the agent reuses across tasks.
            

Working memory is the window itself. It holds the current task and any intermediate reasoning, and it is disposable by design. Episodic memory is the record of what happened: past conversations, tool runs, decisions and their outcomes, stored as timestamped events and retrieved when a new query resembles an old one. Semantic memory is the durable, mostly timeless facts, such as the user's role, their preferred format, a policy that always applies. Procedural memory is the least common and the most interesting: methods the agent has learned to reuse, closer to a skill library than a fact table.

The value of naming the tiers is that it forces a write policy per tier rather than one global rule. Episodic writes are cheap and frequent; semantic writes are rare and should be deliberate, because a wrong durable fact costs far more than a wrong log entry. Different lifetimes, different eviction rules, different retrieval strategies. Collapse them and you get the landfill.

Retrieval-backed long-term memory

Once memory lives outside the model, reading it back is a retrieval problem, and it is worth seeing that clearly: long-term memory is retrieval-augmented generation where the corpus is the user's own history plus a curated set of facts, rather than a document library. The same stack applies. You embed and index memories, and at query time you pull the relevant few into the window instead of loading everything the agent has ever known.

That "relevant few" framing is the whole point. The naive alternative, replaying the entire history into the window, fails on both axes at once: it overruns the token budget and it dilutes attention across mostly irrelevant material, so the agent is both more expensive and less accurate. A retrieval step that surfaces the three or four memories that actually bear on this turn is what makes long-term memory usable rather than a liability. Recency, relevance and importance all belong in the ranking, not similarity alone; a memory can be a close semantic match and still be stale.

Because it is retrieval, it inherits retrieval's failure modes and its tooling. Poor recall means the agent forgets things it stored; poor precision means it drags in irrelevant memories that mislead the current turn. Hybrid lexical-plus-vector ranking, the same technique that helps document search, helps here too when memories contain identifiers or exact names, as covered in the hybrid search deep-dive. The practical consequence for architects: you do not need a novel "memory database." You need your retrieval stack pointed at a well-curated memory corpus, and most of the difficulty moves upstream to how that corpus gets written.

Write policies: what deserves to persist

Retrieval quality is capped by what you chose to store, which is why the write path, not the read path, is where memory systems are won. The instinct to persist every turn is the original sin. It builds a store full of pleasantries, dead ends and duplicates, and retrieval then faithfully surfaces that noise. A disciplined write path does four things before anything reaches long-term storage.

Conflict resolution deserves the most thought, because it is where memory stops being an append-only log and becomes a maintained state. When a user who "prefers email" now says "call me instead," the system must decide: does the new fact supersede the old one, or are both valid in different contexts? A memory that carries provenance and a timestamp can be superseded cleanly; a memory that does not becomes a landmine. The safe default is that newer explicit statements override older inferred ones, and that overridden memories are marked, not silently deleted, so the change is auditable.

None of these steps is free. Extraction and summarization are themselves model calls, so they can hallucinate, which argues for constrained templates and, in high-stakes domains, validation before a write commits. The recurring theme across mature systems: spend your engineering budget on the write path. A modest store of clean, deduplicated, conflict-resolved facts outperforms a vast store of raw turns on every axis that matters.

Agent-maintained memory files

Not all memory needs a vector store. One of the most durable patterns in practice is the simplest: the agent keeps its own structured notes in a file, reads them at the start of work, and edits them deliberately as it goes. Coding agents popularized this with a per-project or per-user Markdown file, but the pattern generalizes to any long-horizon task where the agent needs a stable, curated view of what it knows.

The appeal is inspectability. A memory file is human-readable, diffable, and version-controllable, which means a person can open it, see exactly what the agent believes, correct a wrong fact, and review the change like any other commit. Contrast that with an opaque vector store, where "what does the agent remember about this user" is answerable only by running queries and hoping the ranking surfaces the relevant records. When memory drives real decisions, the ability to read and edit it directly is a governance feature, not a convenience.

The limits are real and worth stating. A file does not scale to thousands of records or to semantic search across a large history; past a certain size, reading the whole file back into the window defeats the purpose and you need retrieval. And an agent editing its own notes can corrupt them, so the write operations deserve the same discipline as any other write path: append deliberately, restructure rarely, and keep the file small enough that its entire contents earn their place in the window. Within those bounds, a memory file is the highest-transparency memory you can ship.

Forgetting, decay and privacy

Every article on memory sells remembering; the harder engineering is forgetting. An agent that never forgets is not a better agent, it is a growing liability. Three problems compound as an unbounded store ages. Staleness: facts that were true and now mislead. Contradiction: old and new versions of a fact coexisting with equal standing. And exposure: the more you retain, the larger the surface of sensitive data you are now responsible for. Memory that stores user information is regulated data, and it inherits every obligation that comes with that.

The countermeasures are policies, not features you can bolt on later. They belong in the design from the start.

• Decay. Weight memories by recency and access, so rarely used, aging entries rank lower and eventually fall out of retrieval rather than lingering forever.
• Eviction. Set explicit lifetimes and size caps per tier. Episodic logs can expire on a schedule; semantic facts persist until superseded, not indefinitely by default.
• User control and deletion. Users must be able to see what is stored about them and delete it, and deletion has to reach every copy, including embeddings and any cached summaries, not just the primary row.
• PII handling and access control. Treat the memory store as a system of record for personal data: minimize what you extract, control who and what can read it, and log access.

Deletion is the requirement teams underestimate most. A "delete my data" request has to propagate to the vector index, any derived summaries, and any memory files, or you have deleted the appearance of the data and kept the substance. Design the store so that a memory has a single authoritative identity and every derivative can be traced back to it; otherwise honoring deletion becomes archaeology. The reframing that keeps you out of trouble: memory is not a growing asset to maximize, it is a managed liability to keep as small as the use case allows.

Designing a memory system

Pulled together, a memory strategy is a set of explicit decisions, not a product you switch on. The failure mode this whole article argues against is the tempting shortcut: buy a bigger context window, call it memory, and move on. A window is working state. A memory system is an architecture with tiers, policies and an evaluation loop, and the work is in specifying each one.

1. Choose tiers by use case. Do not build all four by reflex. A support agent may need episodic and semantic memory and nothing else; a coding agent leans on procedural and a memory file. Add a tier when a concrete requirement demands it.
2. Define the write policy explicitly. Decide what gets extracted, how it is summarized, how duplicates are caught, and how conflicts resolve. This is where quality is won, so it deserves the most design time.
3. Define the read policy. Specify how many memories enter the window, and how recency, relevance and importance combine in the ranking. Treat it as the retrieval problem it is.
4. Define the forget policy. Set decay, eviction, deletion and access control up front. Retrofitting privacy onto a store that assumed permanence is expensive and error-prone.
5. Evaluate memory like retrieval. Measure recall (did it remember what it should), precision (did it avoid dragging in noise), and correctness after conflicts. The evals deep-dive covers the harness; point it at memory the same way you point it at RAG.

Frameworks now offer memory abstractions, and they can save real work, but treat them as implementations of these decisions rather than substitutes for making them. None is a settled standard, and adopting one before you have written down your own write, read and forget policies just moves the ambiguity into a dependency. Related agent patterns, including how memory interacts with multiple cooperating agents, are covered in the multi-agent systems deep-dive. The through-line is the one this article opened with: the window is not memory, and a bigger window is not a memory strategy. Memory is an architecture, and the agents users trust across sessions are the ones whose architects built it deliberately.