PersonaMem benchmark: v1 or v2? Request for published results and reproduction steps

[dial481]

I'm trying to use your evaluation framework to run PersonaMem but there is some confusion regarding the version.

PersonaMem version

The benchmark chart in the README labels the rightmost panel "PersonaMem v2," but the evaluation code appears to support only PersonaMem v1:

• personamem_converter.py expects questions_32k.csv + shared_contexts_32k.jsonl (the v1 file format from bowen-upenn/PersonaMem)
• The converter parses v1-specific columns (user_question_or_message, shared_context_id, end_index_in_shared_context, etc.)
• PersonaMem v2 (bowen-upenn/PersonaMem-v2) uses a different schema (parquet format, user_query column, incorrect_answers instead of all_options, 5,000 benchmark questions vs 588 in v1-32k)
• The README links to the v1 HuggingFace dataset for download instructions

Could you clarify which version was used for the published scores (EverMemOS 53.2%, MemOS 50.7%, Mem0 43.9%, Zep 43.4%, memU 38.7% )? If v2 was used, is there a separate converter or branch we're missing?

Published results

For LoCoMo and LongMemEval, the detailed results table in figs/benchmark_2.png is very helpful. Is there a similar breakdown for PersonaMem (per-category accuracy, number of questions, etc.)?

More broadly, are the raw evaluation outputs (answer texts, search results, eval judgments) available for any of the benchmarked systems? Being able to inspect the actual answers would help the community:

1. Reproduce the published numbers independently
2. Do error analysis across systems
3. Ensure fair comparison when benchmarking new systems

Even just the eval_results.json files for each system/dataset combination would be valuable.

Reproduction

We ran PersonaMem through the framework as documented and want to make sure we're comparing apples to apples. A few specific questions:

1. Which LLM backbone was used for answer generation in the PersonaMem evaluation? (The LoCoMo/LongMemEval table shows GPT-4o-mini and GPT-4.1-mini backbones — was PersonaMem run with the same?)
2. 2. How many questions were evaluated: 588 (PersonaMem v1 32k) or 5,000 (PersonaMem v2 benchmark 32k, 128k, 1M)?
3. Were any category filters applied (as in LoCoMo category 5)?

Thanks for making the framework open source, it's been very useful. Just want to make sure we're benchmarking on the same footing.

Related: #73 #65 #31

[dial481]

Following up with some additional context from cross-referencing other published PersonaMem evaluations.

Cross-paper score comparison

I looked at every paper I could find that reports PersonaMem scores for memory systems. The same systems get very different numbers depending on who runs the evaluation:

System	BMAM (Table 14)	O-Mem (Table 2)	EverMemOS (README chart)
MemOS	34.0%	58.7%	50.7%
Mem0	53.9%	46.9%	43.9%
Zep	57.8%	—	43.4%
MemU	56.8%	—	38.7%

Sources: BMAM Table 14, O-Mem Table 2.

The spread is significant, MemOS ranges from 34% to 58.7%, Zep from 43.4% to 57.8%, MemU from 38.7% to 56.8%. This suggests PersonaMem scores are highly sensitive to evaluation setup (LLM backbone, answer prompt, context formatting, judging criteria), which makes reproduction details even more important.

Question count

The BMAM paper independently confirms 589 QA pairs across 20 users, which matches the v1-32k download from HuggingFace. (The original PersonaMem paper says 588, the actual CSV has 589 rows.)

Timeline

We also noticed that personamem_converter.py was added on Nov 5, 2025 (commit history), while PersonaMem v2 wasn't published until Dec 7, 2025 (arxiv 2512.06688).

I'm trying to run benchmarks on PersonaMem through your framework but I'd like to know what is an actual apples to apples comparison. I'd like to ensure the same dataset version, same question count, same LLM backbone, same answer prompt. Without that, any comparison would be apples to oranges, as the cross-paper table above illustrates.

Would it be possible to share the exact evaluation config (prompt, model, parameters) used for the PersonaMem scores in the README?