LLMs are stateless — every call starts fresh. "Memory" is the machinery you build around the model to make it feel continuous.
Three layers
- Short-term (context window) — the messages in the current conversation. Managed by summarization and windowing when it overflows.
- Long-term (retrieval) — facts, past conversations, and documents stored in a vector or hybrid index and pulled in on demand. See RAG survey.
- Working memory / scratchpad — the agent's own notes across steps, often stored as structured JSON.
Libraries worth knowing
- mem0 — extract, store, and recall user-level facts.
- Zep — long-term memory + temporal knowledge graph.
- LangGraph memory — short and long-term memory primitives inside the graph runtime.
Design tips
- Store facts, not raw transcripts. Summarize before you save.
- Attach a source and timestamp to every memory so you can expire or override it.
- Retrieve with a filter for the current user — memory leaks across users are the number one bug.