Tracking Issue: expand benchmark coverage

[PsiACE]

Goal

Track how PowerContext expands its benchmark coverage without creating a separate execution stack for every benchmark.

Existing E2E infrastructure, including Harbor, ACP, and containerized execution, should be the starting point where it fits. A benchmark may reuse upstream Harbor content, be represented as a Harbor task, or use another approach when its native structure calls for one. The integration should preserve the benchmark's original intent, inputs, and evaluation semantics.

Principles

We should prefer benchmarks that are recognized or used by comparable memory, context, or agent projects.

Each proposal should explain why the benchmark is relevant to PowerContext, whether the comparison can be fair, and whether the result is clear and measurable. The exact execution and evaluation shape may differ between benchmarks.

Community contributions should focus on a small, runnable validation rather than a complete evaluation. Contributors are not expected to cover the full runtime or inference cost.

After a benchmark is accepted, maintainers may run the complete evaluation or a useful subset. Published results should state what was run and should not present a partial run as a complete benchmark result.

Tracking

• Review candidate benchmarks and their use across comparable projects.
• Identify which parts of the existing E2E and Harbor infrastructure can be reused.
• Define a lightweight contribution path for adding a benchmark.
• Review fairness and quantitative evaluation for each proposal.
• Preserve the source benchmark's semantics when adapting it to PowerContext.
• Decide the appropriate full or partial evaluation scope after adoption.
• Add and evaluate accepted benchmarks.

This issue tracks benchmark adoption and shared infrastructure. Benchmark-specific design and execution details belong with each benchmark.

[knqiufan]

The evaluation boundary is written clearly. I learned a lot from it, then looked through the repo (e2e/bub, Harbor tasks, the LoCoMo runner). It seems interesting, so I would like to try a PR along these lines:

1. Add a short contributor guide on how to add one e2e sample: reuse the existing Harbor path and powercontext-e2e acceptance, and do not write a new runner.
2. Add 2–3 LoCoMo-derived samples, covering different question types if possible (for example temporal reasoning, cross-session / multi-hop, abstention or Unknown), with explicit provenance, and without putting gold answers into Agent-visible instructions.
3. On the LoCoMo docs: benchmark/locomo/README.md points to CONVERSATION_PROFILE_AB.md, which is not in the tree. Would removing this dead link be the right fix for now?

I also did a light pass over other candidate benchmarks. The principles are the ones above: prefer sets already used by memory / context / agent projects; reuse the existing E2E + Harbor + ACP + container path when it fits; preserve the source benchmark’s semantics; and start with a small, runnable validation from the community.

Candidate	What it actually measures	Fit for this repo	What would make a comparison fair	Suggestion
LongMemEval (ICLR 2025; 500 questions; _S ~115k tokens, _M ~500 sessions)	User–assistant memory: extraction, multi-session reasoning, temporal reasoning, knowledge update, abstention	The strongest candidate for a second memory exam. LoCoMo is two-speaker dialogue QA; this is closer to assistant memory and covers update / refuse, which we barely test today	Pin question_id and the dataset file (_s must not be reported as _m). A full run is too expensive for contributors. Use the paper’s judge; do not invent a substring score and call it LongMemEval	After the LoCoMo sample path is stable, add 1–3 Harbor smokes first. A full runner can wait, and must be labeled smoke / subset / full
BEAM (100 conversations, 2,000 questions; 128K–10M; 10 abilities)	Very long, coherent dialogues, including contradiction and instruction-following; 1M windows still drop	Better as a later high-pressure recall / update stress test, not as a coding-agent exam	Cost and protocol decide whether scores are comparable. A 128K slice is not a 10M result. Check license and release artifacts before any port	Research only for now
MemoryAgentBench (ICLR 2026; AR / TTL / LRU / CR)	Incremental multi-turn memory. Conflict resolution (FactConsolidation) is what really sets it apart; multi-hop CR is still close to unsolved in the paper	Useful as a source of regression cases (old fact vs new fact). Cloning the full numeric suite would also pull in ICL, recsys, detective QA, and ∞-Bench, most of which is not a PowerContext product claim	MAB accuracy is only fair if we keep the original splits and graders	Do not import the full suite. To cover conflict, a few hand-written Harbor positive samples are enough. That is not a MAB score
Terminal-Bench (already in-tree: terminal-bench@2.0 / db-wal-recovery)	Whether the agent finished the container task. Native score ≠ Memory acceptance (RFC 1229)	Already shows that “reuse upstream Harbor content” works. Adding one more task would only check that the path is repeatable	Keep Harbor rewards diagnostic. Do not put long-horizon tasks in default acceptance CI	Can wait. Should not go into the first contribution-path PR
SWE-bench Pro (~1,865 tasks; public / held-out / commercial; patch Pass@1)	Whether the agent fixed a real multi-file repo. Upstream Harbor has an adapter, but that scores patches, not memory	Only relevant if the claim is “did project context change the coding outcome?” The contract is different from LoCoMo	Public-set Pass@1 must not be compared with Memory probe_coverage. A community PR should not claim commercial-set results	There is already a separate evaluation path (#1247)

So the evaluation path I would expect is: contribution path + more LoCoMo samples → a few update / conflict e2e cases → LongMemEval smoke (or something else, depending on what people decide) → maintainers run a subset / full set → then publish an algorithm-regression baseline with a frozen protocol. Partial runs should be labeled as partial runs.