AI agents don't have one single kind of memory. They can read messages, inspect files, call tools, remember what happened a few steps ago, and use all of that information to decide what to do next.

But there is an important distinction between having information in the context window and actually having useful memory. Context helps an agent with the task in front of it. Memory helps it across tasks. The engineering challenge is deciding what crosses that boundary, and when.

The context window is more like the agent's working desk. It contains the information the agent currently needs to work on a task. Once that task is over, most of that information may no longer be useful.

Long-term memory is different. It stores information that may still matter later, such as a user's preferences, an important project decision, or a fact that should not have to be explained again in every conversation. That gives us two basic layers to design for: short-term memory for the current task, and long-term memory for what needs to survive across tasks.

The difficult part of designing an agent's memory is not simply choosing where to store information. You need to decide what to keep, where to keep it, when to retrieve it, and when to let it go.

A quick way to keep the terms straight:

  • Context is whatever is currently available to the model.
  • Short-term memory is working information kept for the current task, usually represented through that active context.
  • Long-term memory is information stored outside the active context and retrieved when it's needed again.

To make this concrete, imagine you're building an AI coding agent for a small dev team. During a single session, it needs the current error message, the file it's editing, and the latest test output. Across sessions, it might need to remember that the project runs Python 3.12, that the team prefers pytest over unittest, or that a particular architectural decision was already made and shouldn't be revisited every sprint. The first kind of information is short-term. The second is long-term. We'll come back to this same agent throughout.

Short-Term Memory: The Active Context

The active context is the information an agent needs while it is working on something right now.

This can include recent conversation turns, the current goal, tool outputs, files it has inspected, previous actions, and temporary information produced during the task.

Back to the coding agent: during a debugging session, it might have the

current file open, the latest code changes, recent test results, an error message, and a few instructions from the user in its context.

That information is extremely useful while debugging. But most of it does not need to become permanent memory.

This is similar to how coding agents such as Claude Code can work with a context window during a debugging session. The current code, recent test output, and conversation history help the agent understand what it is doing. They are working information for that session rather than facts that necessarily need to follow the user forever.

It's worth being precise here: the context window is not itself a memory system. It's the mechanism through which the model receives working information for the current task. Memory is the separate question of what should still be around once that context window is gone.

The problem is that a context window is not free storage.

As more information gets added, the agent has more material to process. Even before reaching the maximum token limit, a very crowded context can make it harder to focus on the information that actually matters. This is often described as context rot. If an agent is carrying fifty loosely relevant facts alongside the three it actually needs for the task, those three have to compete for attention with the other forty-seven.

So a good short-term memory system should not simply keep adding information. It should also remove, summarize, or deprioritize information that is no longer useful.

Long-Term Memory: Persistent Storage Outside the Prompt

Long-term memory is information that should survive beyond the current task or conversation. For our coding agent, that's the Python version, the testing framework, the architectural decisions the team already made — the kind of thing nobody wants to repeat every time they open a new session.

This could be a durable preference, an important project decision, a relationship between two pieces of information, or another fact that will probably still be useful later.

Instead of putting all of this information into every context window, an agent can keep it in a persistent storage system and retrieve it when needed.

Anthropic's memory tool for Claude provides a useful example. It uses a

filesystem-like structure where information can be organized into files such as /profile.md or project-specific files. The agent can read relevant information when needed and write new information back to the memory store.

This approach separates storage from context. The information can exist permanently without being loaded into every prompt.

ChatGPT's memory feature is another example of the same general idea. Durable information can be extracted from conversations and made available in future conversations instead of requiring the user to repeat it every time. (Memory features like these change quickly, so it's worth checking the current behavior before treating any specific detail as settled.)

A simple test for deciding whether something belongs in long-term memory is:

Would this still be true or useful three months from now?

If the answer is probably yes, it may be a good candidate for long-term memory. If it only matters because of what the agent is doing right now, it probably belongs in short-term memory instead.

Moving Information from Short-Term to Long-Term

The interesting part is what happens between these two layers.

A conversation produces a huge amount of information, but only a small portion of it should become permanent memory.

A basic memory pipeline looks like this:

Conversation → identify a candidate fact → decide if it is durable → write it to memory → retrieve it later when relevant.

The agent needs some rules for making that decision.

For example, is the information stable? Is it specific only to the current task? Would forgetting it cause a problem later? Has the information already changed before?

One important rule is that one mention is not necessarily a pattern.

If someone says, "I'm using dark mode today," that does not necessarily mean they permanently prefer dark mode. The agent should be careful about turning a temporary statement into a long-term preference.

The opposite problem is under-writing.

An agent that refuses to save anything may repeatedly ask the user for the same information. Imagine having to tell an agent your project name every time you start a new session.

Over-writing creates the opposite problem. If the agent permanently records every casual comment, the memory store becomes noisy, potentially invasive, and difficult to retrieve from. Good memory design needs to sit between these two extremes.

Retrieval: The Real Bottleneck

Saving information is only half of the memory problem.

Imagine an agent has stored hundreds of facts about a user and their projects. If it cannot identify which three facts are relevant to the current question, having all those memories does not help much.

This makes retrieval one of the most important parts of an agent's memory system. The system needs to answer four basic questions:

  • What information should be retrieved?
  • When should it be retrieved?
  • How much information should be loaded?
  • What should happen when two memories conflict?

Claude's memory system provides an interesting example of selective retrieval. Rather than forcing the entire memory store into the context, the agent can work with a listing that provides short descriptions of memory files. It can then decide which file is relevant before loading its full contents.

This is a simple but important idea: the agent should not have to read everything just because everything is available.

For example, if the coding agent is asked to explain a test failure, pulling in unrelated memories about the user's favorite writing style, an old unrelated project, or last week's conversation may add noise rather than improve the answer. Good retrieval is selective by design, not just by accident of a small memory store.

Handling Updates, Conflicts, and Staleness

Long-term memory has another problem: facts change.

A memory system that only adds new information can slowly fill up with outdated facts.

Imagine the agent has stored a fact: the user prefers tabs over spaces.

Six months later, the user joins a new team that follows a style guide requiring spaces, and mentions this in passing while asking for help with something else.

If the system simply appends the new statement, it now has two conflicting memories. Which one should the agent trust?

This is why a useful memory system needs an edit path, not just an append path.

Depending on the situation, the system might overwrite the old fact, keep both facts with timestamps or context, or ask the user when it cannot determine which information is current.

There is also a practical reason to write information as it becomes clear instead of waiting until the end of a session. Users do not always review or confirm everything that happened during a conversation. If an important decision was made halfway through a task, waiting until the end creates an opportunity for that information to be missed.

Privacy and Trust Boundaries

Not everything that could be remembered should be remembered.

Some information can be useful to an agent but still require special handling, especially sensitive information involving health, finances, identity, or other private matters. A memory system should have explicit rules for what it is allowed to store rather than assuming that useful information is automatically safe to keep.

Users should also have visibility into what has been stored about them.

More importantly, there needs to be a real way to remove information. If a user says, "forget that," the system should have an actual deletion or editing mechanism. Simply hiding the information from future responses is not the same thing as removing it from persistent storage.

This is part of the trust boundary of an agent. Users need to know that persistent memory is something they can inspect and control, rather than a black box that keeps accumulating information forever.

A Simple Reference Architecture

A basic agent memory architecture does not have to be complicated.

The flow can look like this:

User message → short-term context assembly → retrieve relevant long-term memory → generate response → selectively write new durable information.

The short-term context contains the current conversation and working information. The retrieval layer adds only the long-term facts that are relevant to the task. After the response or during the task, the system can identify information that is worth keeping and write it back to persistent storage.

Spelled out a bit further, one pass through that flow looks like this:

User asks a question
      ↓
Assemble current conversation and task context
      ↓
Search relevant stored memories
      ↓
Rank and filter what actually applies
      ↓
Add the selected memories to context
      ↓
Generate a response
      ↓
Identify any new durable information
      ↓
Write it back to the memory store

The storage layer can be as simple as a set of files or a database. For larger collections where meaning-based search is useful, a vector store can also be used — the choice depends on how much you're storing and how it needs to be searched, not on the pattern above.

Under the single label of "long-term memory," there's usually more than one kind of information living side by side: preferences about the user, facts about a specific project, decisions made along the way, and general knowledge the agent has picked up. They don't all need separate systems, but knowing what kind of information you're storing makes it easier to decide how that information should be handled later.

This general pattern appears in several agent systems, although the implementation differs. Claude's memory tool uses persistent files. ChatGPT has a memory system that can carry selected information across conversations. Projects such as MemGPT and Letta use a tiered approach where some information remains in the active context while other information is stored outside the immediate context and retrieved when needed.

The underlying idea is the same: not all memory needs to be in the agent's active context at the same time.

Common Memory Design Pitfalls

There are a few mistakes that show up repeatedly when building agent memory.

is useful for the current task, but it is not a substitute for persistent memory.

information eventually conflicts with new information.

noise and can create unnecessary privacy risks.

agent becomes repetitive and forgetful.

automatically mean better reasoning. Retrieve what is relevant instead.

  • Treating the context window as permanent storage. Context
  • Having no update or delete path. Without maintenance, old
  • Writing too aggressively. Saving every statement creates
  • Writing too little. If nothing useful is remembered, the
  • Loading the entire memory store. More information does not

These problems are often more important than the database or storage technology being used.

Conclusion

Designing memory for an AI agent is less about finding a place to store information and more about deciding what deserves to be remembered.

An agent that gets this right does not feel like it has a database attached to it. It feels like it simply remembers you: your name without being asked twice, the decision you made three weeks ago, the fact that changed last month and should no longer be treated as current.

Active context handles the current task. Long-term memory carries information about the person or project over time. The hard part is the boundary between them: what gets promoted, how it gets retrieved, and how outdated information gets replaced.

Get that boundary right and the storage mechanism stops mattering much. A good memory system isn't the one that remembers the most. It's the one that remembers the right things and brings them back at the right time.