Many agent frameworks now ship some kind of long-term memory: a store of summaries, extracted facts, graphs or learned notes that the agent writes as it works and reads back later. A new paper from researchers at the University of Illinois Urbana-Champaign, Tsinghua University, Renmin University of China, Verdent.ai and the University of Waterloo asks a blunt question about all of that machinery: does it actually beat just searching the raw chat history?
In many of their tests, it does not. The authors call the gap memory tax: the queries a memory system gets wrong that a naive retrieval baseline gets right.
What they tested
The team compared 11 open-source memory systems, grouped by how they store information:
- Flat stores: A-Mem, SimpleMem, MemOS
- Hierarchical or lifecycle designs: MemoryOS, LightMem, OpenViking
- Graph-based: HippoRAG2, PlugMem, MAGMA
- Trained memory policies: Mem-T, Memory-R2
The reference point is naive chunk-based RAG over the raw history, in a one-round and a two-round variant. The paper also reports no-context, full-context and gold-context runs.
If RAG is new to you, our plain-English RAG explainer covers how chunking, embedding and retrieval work. For how production memory layers such as Mem0, Letta and Zep store and fetch facts, see our agent memory frameworks comparison.
Every system ran on three backbones (Qwen3.5-4B, Qwen3.6-27B and GPT-5.6 Luna at low effort) with the same embedding model, Qwen3-Embedding-0.6B. An LLM judge backed by Qwen3.6-27B scored the answers. The benchmarks were LongMemEval-s, PersonaMem-v2 and MINT-Eval. Because of cost, the team used stratified subsets: 20% of PersonaMem-v2 and 10% of MINT-Eval.
The headline result
On LongMemEval-s, no memory system beat two-round RAG on any of the three backbones. With Qwen3.6-27B, RAG scored 81.4%. The closest memory system, A-Mem, scored 79.4%. OpenViking scored 58.8%.
The other two benchmarks were mixed rather than one-sided. On PersonaMem-v2, MemOS beat two-round RAG by 1.0 and 1.5 points on the two Qwen models, and Memory-R2 beat it by 2.0 points on the 4B model. MemoryOS and LightMem also edged it on the 4B model, by 0.9 and 0.2 points. On MINT-Eval, Mem-T and PlugMem came out ahead in a few cells, by up to 3.9 points with GPT-5.6 Luna. Most other cells in the main results table are below RAG, some by more than 10 points.
One more detail stands out. With GPT-5.6 Luna, two-round RAG (90.0%) scored higher than putting the full history in context (84.0%) on LongMemEval-s. For that model, the team ran 20% of the LongMemEval-s questions.
The paper's summary is careful: memory is "not uniformly worse." A query-level comparison shows memory systems do solve some questions RAG misses, which the authors call memory benefit. They lose others that RAG already solves. When the tax is bigger than the benefit, the system lands below RAG overall.
Where the tax comes from
The authors trace the losses to three causes.
Faithfulness: facts get lost while memory is built. The team sampled atomic facts from the raw history and traced each one into each system's store. Loss varied widely. In MemoryOS, 98.4% of sampled facts survived with no loss. In OpenViking, only 39.4% did, and 38.0% had no memory unit at all. Once a fact is dropped during construction, no retriever can bring it back.
Accessibility: facts survive but get harder to find. Even when the evidence is in the store, memory processing can fragment it across several units, dilute it with unrelated material, or rephrase it so it no longer matches the query well. When the authors replayed retrieval round by round, memory systems recovered fewer queries' evidence than RAG in the first round. Some systems, such as MAGMA and LightMem, scored close to their raw-evidence scores when handed the right memory units directly, yet far lower in normal use. That suggests the evidence was kept but became harder to reach.
Robustness: memory is more sensitive to the model. Long-lived agents may build memory with one model and read it with another. The team built each store with three backbones and queried each with different models. For many systems, switching the QA model caused a bigger accuracy drop than it did for RAG. Each LLM-driven memory operation adds another place where model differences show up.
A benchmark problem, too
The paper also argues that some widely used memory benchmarks are friendly to plain retrieval. Across a wider set of long-term memory benchmarks, an IDF keyword-matching baseline scored above 50% on some, and a hybrid RAG baseline reached about 70%. When simple retrieval already solves most questions, there is little room for memory to add value, and small losses become visible.
The authors also note that many memory papers compare mainly against other memory systems, and that when naive RAG is included its setup is not always fully specified. Their suggestion: report memory systems next to a naive RAG baseline, so readers can see how much room is left and what the memory adds.
The same pattern outside chat memory
The team checked two other settings. In experiential memory, where agents store past task traces to help on new tasks, results were mixed across ALFWorld, WebShop, Mind2Web, BigCodeBench, AppWorld and tau2-Bench with a Qwen3.5-4B agent. For video world models, directly retrieving and reinjecting past frames gave strong scene consistency, and latent or KV-cache memories did not consistently beat that simpler approach.
The practical fix: keep a way back to the source
The most useful part for engineers is short. The authors tested two changes that work on top of existing memory systems.
The first, called source augmentation, retrieves from the raw history alongside the normal memory lookup and reranks both together before answering. With Qwen3.6-27B on LongMemEval-s, it lifted SimpleMem from 73.6% to 82.8% and cut its measured memory tax from 12.0% to 5.2%. MemoryOS went from 73.4% to 76.4%, and Mem-T from 75.0% to 80.0%. On PersonaMem-v2, all three also improved, with Mem-T rising from 42.1% to 49.0%.
The second idea, recursive memory, writes past queries and the evidence found for them back into the store, so later similar questions find it faster. On PersonaMem-v2 its gains were smaller than those of source augmentation: SimpleMem moved from 46.7% to 46.8% (its measured tax rose from 11.2% to 12.1%), MemoryOS from 48.2% to 49.1%, and Mem-T from 42.1% to 45.7%. It does not apply to LongMemEval-s, which has one query per history.
The paper also sketches a middle path between building memory ahead of time and building it only when a query arrives: use a user's past interests and habits to predict which information will matter. That part is a proposal, not a tested result.
Why it matters
If you are adding memory to an agent, this paper gives you a concrete test to run first: compare your memory layer against plain chunk retrieval over the same history, on your own queries. If the memory layer does not win, the extra LLM calls, storage and complexity are a cost with no return. If you keep it, the cheapest insurance the authors found is to keep the raw history searchable and mix it into retrieval.
The same check applies to memory layers for coding agents, such as Beacon, which turns past sessions into reviewed skills.
Caveats
The authors list their own limits. The baselines are representative, not exhaustive. They used official configurations, and system-specific tuning might do better. The benchmarks are mainly static, offline and English-only. PersonaMem-v2 and MINT-Eval were run on 20% and 10% subsets, and the GPT-5.6 Luna runs used 20% of the LongMemEval-s questions. The paper does not test frontier models beyond GPT-5.6 Luna at low effort. Scores also come from an LLM judge. The project page is live. The GitHub repository marks the main results and analysis files as released on Hugging Face, and the per-system reproduction code as still to come.
