96.0% on LongMemEval: what retrieval does for long-term memory
Papr scores 480 of 500 on LongMemEval-S with the official judge, the highest published result we know of. Here is how the retrieval works, what 5 vs 20 sessions changes, and what it costs.
Papr
An assistant that remembers you has to find one sentence you said three months ago, inside hundreds of conversations, and then reason about it. LongMemEval is the benchmark most memory systems use to measure that. Each of its 500 questions comes with about 50 past chat sessions, around 112,000 tokens of history, and the right answer sits in one or a few of them.
We ran all 500 questions through Papr with the benchmark's own answer prompt and its own grader. Papr answered 480 correctly (96.0%). That is the highest published score we have found. Every answer and every grading decision is public, so you can check it yourself.
The number we care about more is what it took. Our 95.4% run sends the answer model 16.5k tokens per question, about a third of the next system that reports it. Accuracy, cost and speed together are the point of this post.
Where it lands
All rows use LongMemEval-S, 500 questions. Other rows are the numbers each team reported.
| # | System | Answer model | Judge | Overall | Context to answer model |
|---|---|---|---|---|---|
| 1 | Papr High, top 20 | Claude Opus 4.6 | gpt-4o | 96.0% | ~60k |
| 2 | Chronos High | Claude Opus 4.6 | gpt-4o | 95.6% | not reported |
| 3 | Papr High, top 5 | Claude Opus 4.6 | gpt-4o | 95.4% | 16.5k |
| 4 | Mastra Observational Memory | gpt-5-mini | gpt-4o | 94.9% | not reported |
| 5 | Hindsight | Gemini 3.1 Pro | Gemini 2.5 Flash Lite | 94.6% | 43.6k |
| 6 | Papr Rerank | gpt-5-mini | gpt-4o | 94.4% | 15k |
| 7 | Chronos Low | gpt-4o | gpt-4o | 92.6% | not reported |
| 8 | gbrain | Claude Sonnet 4.6 | gpt-4o | 90.6% | 15.8k |
| 9 | Honcho | Claude Haiku 4.5 | gpt-4o | 90.4% | not reported |
| 10 | Zep | gpt-4o | gpt-4o | 71.2% | not reported |
| 11 | Full history in the prompt | gpt-4o | gpt-4o | 60.2% | ~112k |
Two things stand out. With gpt-5-mini, a small and cheap model, Papr reaches 94.4%, close to systems running far larger models. And the 95.4% run sends the answer model 16.5k tokens per question, about a third of what Hindsight sends and a seventh of the full history.
The top three are within half a point of each other. The benchmark's variance from run to run is about 3 questions, so read 96.0, 95.6 and 95.4 as a tight group. We are not claiming a wide lead.
How Papr finds the right sessions
Most of the work is retrieval. If the right session never reaches the answer model, no model can answer. Our pipeline has three steps.
1. Embed with papr-embed. Every user turn and every question is embedded with papr-embed-schema-1-0.6b, our schema-aware embedding model, using the memory:personal schema. A general-purpose embedding treats text as text. A schema-aware one is trained to notice the things a memory question turns on: a person, a date, a purchase, a change of plan. A session scores as high as its best-matching turn, and we keep the top 20. Embedding a question takes about 170 ms and the search itself under 1 ms.
2. Rerank. Gemini 2.5 Flash reads those 20 sessions in one call and grades each one for whether it holds evidence the question needs. This takes 1.8 s at the median.
3. Answer. The answer model gets the top sessions, sorted by date, in the official LongMemEval prompt.
How often does retrieval find everything? For the 470 questions that have an answer, all of the evidence sessions are in the top 5 for 460 (97.9%), and in the top 20 for 467. So retrieval is close to solved here, and most remaining errors are the answer model reasoning wrongly over the right sessions.
Top 5 or top 20
Since the top 20 contains the evidence for 7 more questions than the top 5, we tried handing all 20 to the answer model.
It worked where we expected. Temporal reasoning, the category with questions like "how many weeks between the two trips", went from 123 to 130 of 133. Dates are scattered across sessions, and more sessions means more of them in view.
It also cost something. Twenty sessions gave the model more to get distracted by: 11 questions became right, 8 that had been right became wrong. The net gain is 3 questions, the same size as the run-to-run noise. Context grew from 16.5k to about 60k tokens per question.
Our read: top 5 is the better default. It is within 0.6 points of the best score at less than a third of the context. Top 20 is worth it when the question is about time and the user is willing to pay for it. We picked top 20 after looking at what top 5 got wrong, so treat 96.0% as a tuned result.
Speed and cost
Accuracy is only one axis. A memory system runs on every turn of a conversation, so latency and tokens add up.
- Retrieval latency: about 1.9 s per question with the reranker, which every config above uses. Without it, papr-embed alone takes 0.17 s and still scores 91.0% with gpt-5-mini.
- Context per question: 15k to 16.5k tokens for our top-5 configs, against 43.6k for Hindsight and about 112k for the full history. gbrain sends a similar 15.8k and scores 90.6%, so at the same budget Papr answers about 4 to 5 points more questions.
- Model size: 94.4% with gpt-5-mini means most teams do not need a frontier model to get strong memory.
Fewer tokens per answer is a direct cost saving on every call, and it leaves room in the context window for the actual task.
What we did not do
- No question labels are used anywhere in retrieval or answering. The type of question, and whether it has an answer, are only used for grading.
- The answer prompt and grader are the official ones, unmodified. The grader is gpt-4o-2024-08-06, as in the paper.
- We tuned prompts on half the questions and checked on the other half. There is no further held-out set, which is true of most published LongMemEval results.
Check it yourself
Everything is in github.com/Papr-ai/papr-longmemeval: every answer, every grading decision, the retrieved sessions for each question, and the code. The benchmark's own print_qa_metrics.py reproduces our scores from those files. We also posted the results to the LongMemEval repository.
papr-embed is available through the Papr Memory API. If you are building an agent that needs to remember, this is the retrieval behind these numbers.