diff --git a/docs/superpowers/plans/2026-03-19-v2-r13-locomo-evidence-single-knob-implementation-plan.md b/docs/superpowers/plans/2026-03-19-v2-r13-locomo-evidence-single-knob-implementation-plan.md new file mode 100644 index 0000000..b5940ed --- /dev/null +++ b/docs/superpowers/plans/2026-03-19-v2-r13-locomo-evidence-single-knob-implementation-plan.md @@ -0,0 +1,45 @@ +# V2-R13 LoCoMo Evidence-Only Single-Knob Implementation Plan + +> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Run a constrained evidence check for three stopword candidates and avoid code change unless acceptance rule is met. + +**Architecture:** A/B evaluation only; repository code remains unchanged when criteria are not met. + +**Tech Stack:** Python offline evaluator + existing LoCoMo gate scripts. + +--- + +### Task 1: Candidate Evaluation + +**Files:** +- Create: `/tmp/v2_r13_candidate_eval.py` (temporary) + +- [ ] Evaluate `games`, `kind`, `names` against current baseline. +- [ ] Compare `overall` + `multi-hop` jointly. + +### Task 2: Execute Decision + +**Files:** +- No retrieval source edits if acceptance rule fails + +- [ ] Apply no-op decision if no candidate satisfies acceptance rule. + +### Task 3: Verification + +**Files:** +- No source edits + +- [ ] `cargo test --workspace` +- [ ] `./.venv/bin/pytest python/tests -q` +- [ ] `corepack pnpm --dir packages/node test` +- [ ] `corepack pnpm --dir packages/mcp test` +- [ ] `REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5` + +### Task 4: Finish Round + +**Files:** +- Create: `docs/superpowers/releases/2026-03-19-v2-r13-locomo-evidence-single-knob-results.md` + +- [ ] Record A/B evidence and no-op rationale. +- [ ] Commit docs-only round. diff --git a/docs/superpowers/releases/2026-03-19-v2-r13-locomo-evidence-single-knob-results.md b/docs/superpowers/releases/2026-03-19-v2-r13-locomo-evidence-single-knob-results.md new file mode 100644 index 0000000..ee1105f --- /dev/null +++ b/docs/superpowers/releases/2026-03-19-v2-r13-locomo-evidence-single-knob-results.md @@ -0,0 +1,35 @@ +# V2-R13 LoCoMo Evidence-Only Single-Knob Results + +## Summary + +This round intentionally avoids retrieval code changes unless one candidate gives clear dual gain. + +Candidates evaluated: `games`, `kind`, `names`. + +## Offline A/B Evidence + +Baseline: + +- overall: `0.5822` +- multi-hop: `0.4255` +- open-domain: `0.3370` +- single-hop: `0.6314` + +Candidates: + +- `games`: overall `0.5822`, multi-hop `0.4291`, single-hop `0.6302` +- `kind`: unchanged vs baseline +- `names`: overall `0.5817`, multi-hop `0.4220` + +Decision rule: + +- require both overall + multi-hop improvement with no obvious side effect + +Decision: + +- no candidate passed; no retrieval code change made + +## Outcome + +- V2-R13 is a docs-only, evidence-driven no-op round. +- over-design avoided by enforcing the single-knob acceptance gate. diff --git a/docs/superpowers/specs/2026-03-19-v2-r13-locomo-evidence-single-knob-design.md b/docs/superpowers/specs/2026-03-19-v2-r13-locomo-evidence-single-knob-design.md new file mode 100644 index 0000000..ff7704a --- /dev/null +++ b/docs/superpowers/specs/2026-03-19-v2-r13-locomo-evidence-single-knob-design.md @@ -0,0 +1,43 @@ +# V2-R13 LoCoMo Evidence-Only Single-Knob Design + +## Goal + +Avoid over-design by allowing at most one stopword tweak only when evidence shows clear gain. + +Round policy: + +- run offline A/B for exactly three candidates (`games`, `kind`, `names`) +- only change code if `overall` and `multi-hop` both improve +- if not, ship no retrieval code change + +## Baseline + +- overall: `0.5822` +- multi-hop: `0.4255` +- open-domain: `0.3370` + +## A/B Results + +- `games`: overall `0.5822`, multi-hop `0.4291`, single-hop `0.6302` +- `kind`: unchanged from baseline +- `names`: overall `0.5817`, multi-hop `0.4220` + +## Decision + +No candidate satisfies the acceptance rule (`overall` + `multi-hop` both up without side effects). + +Chosen action: no retrieval logic change in this round. + +## Test Strategy + +1. `cargo test --workspace` +2. `./.venv/bin/pytest python/tests -q` +3. `corepack pnpm --dir packages/node test` +4. `corepack pnpm --dir packages/mcp test` +5. `REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5` + +## Success Criteria + +1. no code changes in retrieval path +2. required 4-command matrix passes +3. full-gate remains PASS