Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# V2-R16 Local Rerank Rare-Anchor Implementation Plan

> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Improve local retrieval ranking by adding one rare-anchor-aware rerank signal.

**Architecture:** Keep existing FTS and rerank pipeline; add one feature derived from per-query candidate token frequency.

**Tech Stack:** Rust (`memory-core`) + existing LoCoMo scripts.

---

### Task 1: Add Rare-Anchor Rerank Signal

**Files:**
- Modify: `crates/memory-core/src/store.rs`

- [ ] Build candidate sequences first, then compute query token document frequency.
- [ ] Derive rare-anchor token set and per-row rare-anchor coverage.
- [ ] Add rare-anchor coverage term into rerank score with minimal weighting.

### Task 2: Add Regression Test

**Files:**
- Modify: `crates/memory-core/tests/fts_fallback.rs`

- [ ] Add one test for overlap tie where rare anchor should rank first.

### Task 3: Verification

**Files:**
- No source edits

- [ ] `cargo test --workspace`
- [ ] `./.venv/bin/pytest python/tests -q`
- [ ] `corepack pnpm --dir packages/node test`
- [ ] `corepack pnpm --dir packages/mcp test`
- [ ] `REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5`

### Task 4: Finish

**Files:**
- Create: `docs/superpowers/releases/2026-03-20-v2-r16-local-rerank-rare-anchor-results.md`

- [ ] Record score deltas and verification evidence.
- [ ] Commit, PR, merge.
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# V2-R16 Local Rerank Rare-Anchor Results

## Summary

This round completed pure-local reranker design/implementation exploration with strict anti-overdesign gating.

Outcome:

- implemented and validated a minimal rare-anchor rerank feature in working branch
- ran full verification and LoCoMo gate
- measured no LoCoMo aggregate gain
- reverted retrieval code changes and closed as docs-only no-op

## Why No-Op

Measured K=5 remained unchanged after the candidate reranker:

- overall: `0.5822` (no change)
- multi-hop: `0.4255` (no change)
- open-domain: `0.3370` (no change)

Given no benchmark lift, keeping extra ranking logic would add complexity without measurable value.

## Final Decision

- Keep repository retrieval code unchanged in this round.
- Preserve design/plan/results docs for future reference.

## Verification Evidence (final no-op state)

Passed:

1. `cargo test --workspace`
2. `./.venv/bin/pytest python/tests -q`
3. `corepack pnpm --dir packages/node test`
4. `corepack pnpm --dir packages/mcp test`
5. `REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5`

LoCoMo final scores:

- overall: `0.5822`
- multi-hop: `0.4255`
- open-domain: `0.3370`

## Artifact

- full-gate result: `/tmp/localmemos-v2-r16-final-gate/result_hit_at_5.json`
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# V2-R16 Local Rerank Rare-Anchor Design

## Goal

Implement a pure-local, low-complexity reranker improvement that can lift retrieval quality without adding LLM dependency.

## Baseline

Current LoCoMo K=5 baseline:

- overall: `0.5822`
- multi-hop: `0.4255`
- open-domain: `0.3370`

## Problem

When two candidates have similar token overlap, current scoring can over-favor phrase continuity/common-token matches and miss facts that contain rarer anchor clues.

## Options Considered

1. Keep current reranker
- zero risk, zero gain.

2. Add one rare-anchor coverage feature (chosen)
- small local change; uses only per-query candidate statistics.

3. Introduce external local model reranker
- higher complexity and dependency cost for this stage.

## Chosen Design

1. In `recall_with_fts`, compute per-query token document frequency across candidate rows.
2. Identify rare-anchor query tokens (minimum document frequency among matched query tokens).
3. Add one additional rerank feature:
- `rare_anchor_coverage = matched_rare_anchors / total_rare_anchors`
4. Add this feature into existing rerank score with small query-length-aware weight.

No schema changes, no API changes, no external model dependency.

## Test Strategy

1. Add one regression test in `crates/memory-core/tests/fts_fallback.rs` for tie-like overlap where rare anchor should win.
2. Run required matrix:
- `cargo test --workspace`
- `./.venv/bin/pytest python/tests -q`
- `corepack pnpm --dir packages/node test`
- `corepack pnpm --dir packages/mcp test`
3. Run LoCoMo full gate:
- `REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5`

## Success Criteria

1. Pure-local rerank improvement implemented with one new feature.
2. Regression test passes.
3. Required matrix + full gate pass.
Loading