diff --git a/docs/superpowers/plans/2026-03-20-v2-r16-local-rerank-rare-anchor-implementation-plan.md b/docs/superpowers/plans/2026-03-20-v2-r16-local-rerank-rare-anchor-implementation-plan.md new file mode 100644 index 0000000..b7d6baa --- /dev/null +++ b/docs/superpowers/plans/2026-03-20-v2-r16-local-rerank-rare-anchor-implementation-plan.md @@ -0,0 +1,46 @@ +# V2-R16 Local Rerank Rare-Anchor Implementation Plan + +> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Improve local retrieval ranking by adding one rare-anchor-aware rerank signal. + +**Architecture:** Keep existing FTS and rerank pipeline; add one feature derived from per-query candidate token frequency. + +**Tech Stack:** Rust (`memory-core`) + existing LoCoMo scripts. + +--- + +### Task 1: Add Rare-Anchor Rerank Signal + +**Files:** +- Modify: `crates/memory-core/src/store.rs` + +- [ ] Build candidate sequences first, then compute query token document frequency. +- [ ] Derive rare-anchor token set and per-row rare-anchor coverage. +- [ ] Add rare-anchor coverage term into rerank score with minimal weighting. + +### Task 2: Add Regression Test + +**Files:** +- Modify: `crates/memory-core/tests/fts_fallback.rs` + +- [ ] Add one test for overlap tie where rare anchor should rank first. + +### Task 3: Verification + +**Files:** +- No source edits + +- [ ] `cargo test --workspace` +- [ ] `./.venv/bin/pytest python/tests -q` +- [ ] `corepack pnpm --dir packages/node test` +- [ ] `corepack pnpm --dir packages/mcp test` +- [ ] `REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5` + +### Task 4: Finish + +**Files:** +- Create: `docs/superpowers/releases/2026-03-20-v2-r16-local-rerank-rare-anchor-results.md` + +- [ ] Record score deltas and verification evidence. +- [ ] Commit, PR, merge. diff --git a/docs/superpowers/releases/2026-03-20-v2-r16-local-rerank-rare-anchor-results.md b/docs/superpowers/releases/2026-03-20-v2-r16-local-rerank-rare-anchor-results.md new file mode 100644 index 0000000..743d0b5 --- /dev/null +++ b/docs/superpowers/releases/2026-03-20-v2-r16-local-rerank-rare-anchor-results.md @@ -0,0 +1,47 @@ +# V2-R16 Local Rerank Rare-Anchor Results + +## Summary + +This round completed pure-local reranker design/implementation exploration with strict anti-overdesign gating. + +Outcome: + +- implemented and validated a minimal rare-anchor rerank feature in working branch +- ran full verification and LoCoMo gate +- measured no LoCoMo aggregate gain +- reverted retrieval code changes and closed as docs-only no-op + +## Why No-Op + +Measured K=5 remained unchanged after the candidate reranker: + +- overall: `0.5822` (no change) +- multi-hop: `0.4255` (no change) +- open-domain: `0.3370` (no change) + +Given no benchmark lift, keeping extra ranking logic would add complexity without measurable value. + +## Final Decision + +- Keep repository retrieval code unchanged in this round. +- Preserve design/plan/results docs for future reference. + +## Verification Evidence (final no-op state) + +Passed: + +1. `cargo test --workspace` +2. `./.venv/bin/pytest python/tests -q` +3. `corepack pnpm --dir packages/node test` +4. `corepack pnpm --dir packages/mcp test` +5. `REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5` + +LoCoMo final scores: + +- overall: `0.5822` +- multi-hop: `0.4255` +- open-domain: `0.3370` + +## Artifact + +- full-gate result: `/tmp/localmemos-v2-r16-final-gate/result_hit_at_5.json` diff --git a/docs/superpowers/specs/2026-03-20-v2-r16-local-rerank-rare-anchor-design.md b/docs/superpowers/specs/2026-03-20-v2-r16-local-rerank-rare-anchor-design.md new file mode 100644 index 0000000..3a82c03 --- /dev/null +++ b/docs/superpowers/specs/2026-03-20-v2-r16-local-rerank-rare-anchor-design.md @@ -0,0 +1,55 @@ +# V2-R16 Local Rerank Rare-Anchor Design + +## Goal + +Implement a pure-local, low-complexity reranker improvement that can lift retrieval quality without adding LLM dependency. + +## Baseline + +Current LoCoMo K=5 baseline: + +- overall: `0.5822` +- multi-hop: `0.4255` +- open-domain: `0.3370` + +## Problem + +When two candidates have similar token overlap, current scoring can over-favor phrase continuity/common-token matches and miss facts that contain rarer anchor clues. + +## Options Considered + +1. Keep current reranker +- zero risk, zero gain. + +2. Add one rare-anchor coverage feature (chosen) +- small local change; uses only per-query candidate statistics. + +3. Introduce external local model reranker +- higher complexity and dependency cost for this stage. + +## Chosen Design + +1. In `recall_with_fts`, compute per-query token document frequency across candidate rows. +2. Identify rare-anchor query tokens (minimum document frequency among matched query tokens). +3. Add one additional rerank feature: +- `rare_anchor_coverage = matched_rare_anchors / total_rare_anchors` +4. Add this feature into existing rerank score with small query-length-aware weight. + +No schema changes, no API changes, no external model dependency. + +## Test Strategy + +1. Add one regression test in `crates/memory-core/tests/fts_fallback.rs` for tie-like overlap where rare anchor should win. +2. Run required matrix: +- `cargo test --workspace` +- `./.venv/bin/pytest python/tests -q` +- `corepack pnpm --dir packages/node test` +- `corepack pnpm --dir packages/mcp test` +3. Run LoCoMo full gate: +- `REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5` + +## Success Criteria + +1. Pure-local rerank improvement implemented with one new feature. +2. Regression test passes. +3. Required matrix + full gate pass.