memmesh-bench

Reproducible, apples-to-apples benchmarks for AI memory systems — how well does a memory layer let an agent recall the right thing from a long history?

Every system under test does the same job through the same interface: ingest a conversation/event history, then answer questions about it. Every adapter hands its retrieved context to the same answerer with the same prompt, so a score difference is the memory (what got retrieved), not the model or the prompt. Answers are graded by an LLM judge (paraphrase-tolerant), with correct handling of abstention questions.

Reproduce the published LOCOMO head-to-head (Aug 2026)

We claim MemMesh beats Mem0 and Zep on LOCOMO when scored by each competitor's own judge, at 13× fewer tokens (522/query vs Mem0's ~7,000). The step vendors fight over most is the judging, so we publish our exact per-question predictions and let you re-score them under all three judges — Mem0's partial-credit rule, Zep's "same-topic" rule, and our stricter internal bar — verbatim, in verify.py:

pip install openai
export OPENAI_API_KEY=sk-...
python verify.py            # scores our predictions under all three judges
python verify.py --judge mem0

Expected (all predictions in predictions.json, gpt-4o judge, answerable categories):

Judge	temporal	single	multi	open	overall
strict (our internal bar)	74	72	39	56	65.3
zep ("same-topic")	75	75	52	70	70.2
mem0 (partial-credit)	78	78	75	74	76.6

Compare like-for-like to the competitors' published LOCOMO numbers: MemMesh under Mem0's judge (76.6) vs Mem0's published (66.9); MemMesh under Zep's judge (70.2) vs Zep's published (66.0). We win overall against both, lead temporal / single-hop / multi-hop, sit at parity with Mem0 on open-domain, and Zep edges us on open-domain. A lenient judge inflates the lower scorer more, which is why the gap only widens under our strict bar. Predictions come from MemMesh's production extraction path (a further-tuned extractor is in progress); answers by gpt-4o-mini, judged by gpt-4o.

What it measures

Metric	Why
Accuracy	Did the agent recall the right fact? (overall + by category)
Avg latency	Retrieval + answer time per question
Avg tokens in	Context efficiency — a good memory sends less to the LLM
Cost	Blended $/run — the practical consequence of token bloat

See also: COMPARISON.md — a feature-by-feature matrix vs Mem0, Zep, Letta, OpenAI Memory, and LangMem (with MemMesh's own gaps marked honestly).

Systems

• memmesh — the system under test (hybrid retrieval + reranker + belief revision).
• no-memory — stuff the whole history into the prompt. The ceiling for short sessions; falls over (cost + lost-in-the-middle) as histories grow.
• rag — self-contained TF-IDF retrieval (no external embedding API). The honest "just do RAG" baseline.
• mem0 · zep · openai-memory — competitor adapters, gated behind their SDKs
  • keys for the full head-to-head.

Datasets

• smoke — tiny built-in set; runs today, no downloads. Exercises the core failure mode: a fact stated early, updated later, asked at the end.
• longmemeval — LongMemEval, 500 long-history QA across single/multi-session, temporal, knowledge-update, and abstention.
• locomo — LoCoMo, very long multi-session conversations.
• sec-finance-brain — grades a model against a pre-seeded specialized brain (SEC 10-K facts + reasoning) instead of a per-run conversation. Tests grounded domain answering, not recall. Ground truth from data.sec.gov. See SPECIALIZED-BRAINS.md.

Specialized brains — grounded domain answers

Beyond conversational recall, MemMesh publishes brains: domain-scoped memory (facts + induced reasoning) a model consumes over MCP. Every answer is retrieved from that brain's data only. On SEC-finance QA graded against real filings, a local 8B model + a brain matches GPT-4o + the same brain (85.3% each) — while both models score 0% without the brain, proving the answers are grounded in the brain, not the model. Full method + reproduction: SPECIALIZED-BRAINS.md.

Quickstart

pip install -e .                      # or: pip install memmesh-bench
export ANTHROPIC_API_KEY=sk-ant-...   # answerer + judge

# Runs immediately — no downloads, no third-party accounts:
memmesh-bench run --dataset smoke --adapters no-memory,rag

# Add MemMesh (fresh project id per run so sessions don't bleed):
export MEMMESH_API_KEY=sk-...  MEMMESH_PROJECT_ID=proj_...
memmesh-bench run --dataset smoke --adapters memmesh,rag,no-memory

# Full public benchmark, head-to-head:
pip install -e '.[datasets,mem0,zep]'
memmesh-bench run --dataset longmemeval --adapters memmesh,mem0,zep,rag \
  --limit 200 --markdown scorecard.md --out results.json

Notes on fairness

• Same answerer + prompt for every system (BENCH_ANSWER_MODEL, default claude-sonnet-5); the judge is BENCH_JUDGE_MODEL.
• One project/user id per run; the MemMesh adapter filters retrieved rows to the session under test.
• The full public sweep costs real LLM money (a few hundred $ across 5 systems on full LongMemEval) — start with --limit and the smoke set.
• Competitor adapters are thin wrappers over each vendor's SDK; if an API signature drifts, the two marked lines in adapters/competitors.py are the only things to adjust. PRs welcome.

Apache-2.0 · built by ThinkFleet.