Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# V2-R13 LoCoMo Evidence-Only Single-Knob Implementation Plan

> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Run a constrained evidence check for three stopword candidates and avoid code change unless acceptance rule is met.

**Architecture:** A/B evaluation only; repository code remains unchanged when criteria are not met.

**Tech Stack:** Python offline evaluator + existing LoCoMo gate scripts.

---

### Task 1: Candidate Evaluation

**Files:**
- Create: `/tmp/v2_r13_candidate_eval.py` (temporary)

- [ ] Evaluate `games`, `kind`, `names` against current baseline.
- [ ] Compare `overall` + `multi-hop` jointly.

### Task 2: Execute Decision

**Files:**
- No retrieval source edits if acceptance rule fails

- [ ] Apply no-op decision if no candidate satisfies acceptance rule.

### Task 3: Verification

**Files:**
- No source edits

- [ ] `cargo test --workspace`
- [ ] `./.venv/bin/pytest python/tests -q`
- [ ] `corepack pnpm --dir packages/node test`
- [ ] `corepack pnpm --dir packages/mcp test`
- [ ] `REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5`

### Task 4: Finish Round

**Files:**
- Create: `docs/superpowers/releases/2026-03-19-v2-r13-locomo-evidence-single-knob-results.md`

- [ ] Record A/B evidence and no-op rationale.
- [ ] Commit docs-only round.
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# V2-R13 LoCoMo Evidence-Only Single-Knob Results

## Summary

This round intentionally avoids retrieval code changes unless one candidate gives clear dual gain.

Candidates evaluated: `games`, `kind`, `names`.

## Offline A/B Evidence

Baseline:

- overall: `0.5822`
- multi-hop: `0.4255`
- open-domain: `0.3370`
- single-hop: `0.6314`

Candidates:

- `games`: overall `0.5822`, multi-hop `0.4291`, single-hop `0.6302`
- `kind`: unchanged vs baseline
- `names`: overall `0.5817`, multi-hop `0.4220`

Decision rule:

- require both overall + multi-hop improvement with no obvious side effect

Decision:

- no candidate passed; no retrieval code change made

## Outcome

- V2-R13 is a docs-only, evidence-driven no-op round.
- over-design avoided by enforcing the single-knob acceptance gate.
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# V2-R13 LoCoMo Evidence-Only Single-Knob Design

## Goal

Avoid over-design by allowing at most one stopword tweak only when evidence shows clear gain.

Round policy:

- run offline A/B for exactly three candidates (`games`, `kind`, `names`)
- only change code if `overall` and `multi-hop` both improve
- if not, ship no retrieval code change

## Baseline

- overall: `0.5822`
- multi-hop: `0.4255`
- open-domain: `0.3370`

## A/B Results

- `games`: overall `0.5822`, multi-hop `0.4291`, single-hop `0.6302`
- `kind`: unchanged from baseline
- `names`: overall `0.5817`, multi-hop `0.4220`

## Decision

No candidate satisfies the acceptance rule (`overall` + `multi-hop` both up without side effects).

Chosen action: no retrieval logic change in this round.

## Test Strategy

1. `cargo test --workspace`
2. `./.venv/bin/pytest python/tests -q`
3. `corepack pnpm --dir packages/node test`
4. `corepack pnpm --dir packages/mcp test`
5. `REBUILD_PYTHON=0 tools/locomo/run-full-gate.sh 5`

## Success Criteria

1. no code changes in retrieval path
2. required 4-command matrix passes
3. full-gate remains PASS
Loading