How Agents Remember: Memory and State Across Multi-Step Reasoning
A context window is not memory. It is a shrinking budget that every additional step spends down — and the three real strategies agent builders use to make it last.
An agent mid-loop is not thinking with a blank slate. Every thought, tool call, and observation from every prior step is still sitting in its context window — and that window is a hard token budget, not memory in any durable sense. Once the loop ends, or once the window fills, whatever wasn't captured somewhere else is gone. Understanding how production agents actually manage that budget — what they keep, what they compress, and what they externalize — is the difference between a loop that degrades gracefully over fifty steps and one that falls apart by step ten.
The Budget You're Actually Working With
Context windows have grown fast, and the growth is real: Claude's API now offers a 1M-token context window on its larger models (200K on others), Google's Gemini line was the first model class built around a 1-million-token window, and OpenAI's GPT-4o ships with a 128,000-token window. That is orders of magnitude more room than models had three years ago.
But a bigger window is a bigger budget, not a solved problem. Anthropic's own documentation names the failure mode directly: as token count grows, "context rot" sets in — accuracy and recall degrade even though the tokens are still technically "in context." Anthropic's engineering team explains the mechanism in more detail: attention has to spread across a quadratically growing number of token-pair relationships as the window fills, so the model's effective ability to find and use any one piece of information falls even while its nominal capacity keeps rising. A 1M-token window that has degraded halfway through is not meaningfully bigger than a 200K window used carefully.
This is why "just use a bigger window" is not a memory strategy. It buys you more steps before the problem hits, but it doesn't change what happens when it does.
Strategy One: The Rolling Buffer
The simplest approach — and the default for short loops — is to keep the raw transcript. Every thought, action, and observation stays in the context window verbatim, in order, and the model reads the whole thing back on every subsequent call. This is exact: nothing is lost, nothing is paraphrased, and the model sees precisely what happened.
The cost is that it grows linearly with every step. A five-step loop with a raw buffer is carrying five steps' worth of tokens on the fifth call; a fifty-step loop is carrying fifty. Past some point, the buffer alone consumes enough of the window that context rot starts working against the agent regardless of how good the underlying model is. Buffers are the right choice for short, bounded loops — the ones this cluster's cost analysis already treats as the common case — and the wrong choice for anything that needs to run long.
Strategy Two: Summarization
The next strategy — documented in LangChain's own memory framework, LangMem, as a "background" or "subconscious" process — periodically reflects on the accumulated transcript and compresses it down to what actually matters going forward. Rather than carrying every raw observation, the agent (or a separate call) rewrites the older parts of its history into a short summary: what was tried, what was learned, what's still open.
This trades exactness for durability. A summarized transcript can carry the gist of a hundred-step process in a fraction of the tokens a raw buffer would need, which is what makes long-running agents possible at all. The risk is the same risk any compression carries: if the summarization step drops something that turns out to matter three steps later, the agent has no way to recover it — it isn't in the window anymore, and it was never written anywhere else.
Strategy Three: External Retrieval
The third strategy stops trying to keep everything in the context window at all. Anything that needs to survive past the current session — a user's stated preference, a fact learned in a previous conversation, a decision made yesterday — gets written to an external store, typically a vector database, and pulled back in only when it's relevant to the current step. Pinecone's own description of this pattern is direct: embed and save the information once, then "query the vector store, retrieve the top relevant pieces, and feed them back into the LLM" — memory becomes something the agent looks up on demand rather than something it permanently carries.
This is the only one of the three strategies that scales past a single session, and it's why production coding agents use it. Cognition's Devin, for instance, documents its own "Memories" system — auto-generated, workspace-scoped records that persist across sessions alongside user-defined rules, so the agent doesn't start from zero every time it's invoked on the same project. The tradeoff is retrieval quality: the agent only remembers what it successfully retrieves, and a bad query returns nothing even if the right memory exists in the store.
Matching the Strategy to the Lifetime
| Strategy | What it keeps | Survives past this loop? | Failure mode |
|---|---|---|---|
| Rolling buffer | Everything, verbatim | No | Grows until context rot degrades every step |
| Summarization | A compressed gist | Within a long session | Silently drops details that later turn out to matter |
| External retrieval | Anything explicitly saved | Yes, across sessions | Retrieval misses the right memory even when it exists |
The practical decision is rarely "which one is best" — it's matching the strategy to how long the information actually needs to live. A single five-step lookup only needs a buffer. A long research loop needs summarization to avoid drowning in its own history. Anything a user or system genuinely needs the agent to remember tomorrow needs external retrieval, because nothing else in this list survives the session ending. Using a buffer where retrieval was needed loses state silently; using retrieval where a buffer would do adds latency and failure surface for no benefit.
The Bottom Line
Memory in an agent loop is never free, and it is never automatic past the current context window. The three real strategies — buffer, summarize, retrieve — trade exactness, durability, and cost against each other, and the right choice depends entirely on how long the information needs to survive. Providers building bigger context windows are buying agent builders more room before this decision has to be made, not making the decision unnecessary. The agents that hold up over long, complex loops are the ones that picked deliberately instead of defaulting to whatever the window could hold.
Related reading: The Artificial Intelligence hub, Tree-search and hierarchical agents in production, The real cost of agentic loops at scale
Is a larger context window the same thing as better memory?+
No. A larger window raises the ceiling on how much an agent can hold at once, but providers including Anthropic document that recall accuracy degrades as token count grows — a larger window delays the problem, it does not solve it.
What is the simplest way an agent remembers what happened in earlier steps?+
A rolling buffer — the raw transcript of thoughts, actions, and observations kept in the context window as-is. It is exact but grows linearly with every step, so it is only practical for short loops.
How do long-running agents avoid running out of context?+
By periodically summarizing older parts of the transcript down to what the next step actually needs, and by moving anything that needs to persist beyond the current session into an external store (often a vector database) that gets queried back in only when relevant.
What is "context rot"?+
A documented effect, described by Anthropic in its own engineering writing, where a model's ability to accurately recall information degrades as the number of tokens in its context window increases — attributed to attention diluting across a growing number of token-pair relationships.