feat(replay): re-query action-match mode — the honest metric for the one nondeterministic stage (#9028) - #9586
Conversation
…one nondeterministic stage (#9028) Completes #9028. Requirement 2 (wall-clock capture) was verified already landed: staleness-clock.ts cites this issue in its header, decision_replay_inputs.replay_json carries the captured instant, and decision-replay.test.ts pins both arms per rule plus the CLI's clock-divergence exit. This PR is requirement 1, previously deferred on the prompt-persistence decision -- which is made here, not deferred again. WHAT IT IS. Everything downstream of the model is a pure function the replay harness pins bit-exactly, including time. The model call is the one stage that cannot be: hosted inference is not bit-deterministic even at temperature 0 (batching, kernel scheduling, silent model revisions). So `replay-decision.ts --requery <n>` re-runs the model n times against the EXACT persisted prompt and reports an ACTION-MATCH RATE -- across fresh runs, how often the verdict lands in the same CLASS the recorded decision acted on (defect vs clean, through the same parseModelReview the live pipeline uses, with the same ai_consensus_defect/ai_review_split boundary the gate acts on). The word "reproducibility" appears only inside the metric's own label as a negation -- "action-match-rate (NOT reproducibility)" -- so a pasted report cannot shed the caveat. THE PERSISTENCE DECISION. decision_replay_prompts (migration 0200): a private sibling of decision_replay_inputs with the identical posture and a deliberately SHORTER 30-day retention -- the prompt embeds the full diff plus contributor content (the largest, most sensitive artifact in the replay family), requery is a debugging tool for recent decisions, and the public promptDigest commitment outlives the text forever. Keyed by the BASE record id so a supersession's :rev<N> row still finds its head's prompt. Oversize prompts are SKIPPED, never truncated: a truncated prompt re-queried would report a rate for a prompt that was never sent. BOTH TURNS, because the first honest test caught the design flaw: the system prompt carries the rubric + config suffixes (what promptDigest commits to), but the DIFF travels in the USER turn -- re-querying with the system prompt alone would ask the model to review nothing. Captured at the orchestration site, never through findings (findings reach public render surfaces). The CLI replays the live call's own shape: a system turn plus a user turn at temperature 0, so the rate is a statement about the decision, not about a different way of asking. Denominator honesty: a transport failure is an UNUSABLE run counted in the denominator, never a silent skip -- shrinking it would inflate the rate exactly when the provider is flakiest. Unusable matches neither class, mirroring the live pipeline's fail-closed inconclusive routing. Provider config is explicit env only (REPLAY_AI_BASE_URL/REPLAY_AI_MODEL or ANTHROPIC_API_KEY/REPLAY_AI_MODEL) -- a tool that spends tokens never guesses which model to spend against. Exit 0 on a produced report (a low rate is a FINDING); 2 on unusable input or missing config. Pure replay mode remains network-free by construction, untouched. Tests: 21 across four suites -- the class boundaries through the live parser, both recorded-class codes, denominator invariants, zero-run degenerate arm, verbatim two-turn passthrough, bundle/env parsing with every named error (including the 30-day retention hint), base-id upsert semantics, oversize skip, swallowed persist failure, the retention rule at its own window, and an end-to-end pass proving a real AI review persists both turns with the diff in the user one. New module: 100% statements, branches, and functions. Full suite: 24,045 green.
|
Tip ✅ LoopOver review result - approve/merge recommendedReview updated: 2026-07-28 20:05:16 UTC
Review summary Nits — 6 non-blocking
Decision drivers
Context & advisory signals — never blocks the verdict
Linked issue satisfactionAddressed Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Logic backtestReplayed 0 historical case(s) for Backtest comparison:
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9586 +/- ##
=======================================
Coverage 89.84% 89.84%
=======================================
Files 878 879 +1
Lines 111036 111060 +24
Branches 26434 26438 +4
=======================================
+ Hits 99756 99780 +24
Misses 9992 9992
Partials 1288 1288
Flags with carried forward coverage won't be shown. Click here to find out more.
|
![loopover-orb[bot]](https://avatars.githubusercontent.com/in/4139483?s=60&v=4){kind=small}
1 participant
Closes #9028
Summary
Completes Replay v2. Requirement 2 (wall-clock capture) was verified already landed —
staleness-clock.tscites #9028 in its own header, the captured instant ridesdecision_replay_inputs.replay_json, and the seeded per-rule tests plus the CLI's clock-divergence exit are all in place. This PR is requirement 1, whose "blocked on" was a persistence decision — made here rather than deferred again.What it is
Everything downstream of the model is a pure function the replay harness pins bit-exactly, including time. The model call is the one stage that cannot be: hosted inference is not bit-deterministic even at temperature 0 (batching, kernel scheduling, silent model revisions).
So
replay-decision.ts --requery <n>re-runs the model n times against the exact persisted prompt and reports an action-match rate: across fresh runs, how often the verdict lands in the same class the recorded decision acted on — defect vs clean, through the sameparseModelReviewthe live pipeline uses, on the sameai_consensus_defect/ai_review_splitboundary the gate acts on. Two runs that phrase the same blocker differently are the same action; a defect ⇄ clean flip is a different action regardless of phrasing.Per the issue's hard requirement, it is never labeled reproducibility — the metric's own label is
"action-match-rate (NOT reproducibility)", so a pasted report cannot shed the caveat, and a test pins that the word appears nowhere else in the artifact.The persistence decision
decision_replay_prompts(migration 0200): a private sibling ofdecision_replay_inputswith the identical posture and a deliberately shorter 30-day retention — the prompt embeds the full diff plus contributor content (the largest, most sensitive artifact in the replay family), requery is a debugging tool for recent decisions, and the publicpromptDigestcommitment outlives the text forever. Keyed by the base record id, so a supersession's:rev<N>row still finds its head's prompt (the CLI header documents the extract join).Oversize prompts are skipped, never truncated: a truncated prompt re-queried would report a rate for a prompt that was never sent.
Both turns — the flaw my own first test caught
The first end-to-end assertion failed in the right way: the persisted system prompt did not contain the PR's diff, because the system prompt carries the rubric + config suffixes (what
promptDigestcommits to) while the diff travels in the user turn. Re-querying with the system prompt alone would ask the model to review nothing, and the rate would have been meaningless.Both turns are now captured — at the orchestration site, never through findings (findings reach public render surfaces) — and the CLI replays the live call's own shape: system turn + user turn at temperature 0. That is what makes the rate a statement about the decision, not about a different way of asking.
Denominator honesty
A transport failure is an unusable run counted in the denominator, never a silent skip — shrinking it would inflate the rate exactly when the provider is flakiest.
unusablematches neither class, mirroring the live pipeline's fail-closed inconclusive routing.Provider config is explicit env only (
REPLAY_AI_BASE_URL+REPLAY_AI_MODEL, orANTHROPIC_API_KEY+REPLAY_AI_MODEL): a tool that spends tokens never guesses which model to spend against. Exit 0 on a produced report (a low rate is a finding, not a failure); 2 on unusable input or missing config. Pure replay mode remains network-free by construction, untouched.Testing
21 tests across four suites: class boundaries through the live parser, both recorded-class codes, the denominator invariants, the zero-run arm, verbatim two-turn passthrough, bundle/env parsing with every named error (including the 30-day retention hint on the missing-prompt path), base-id upsert semantics, the oversize skip, the swallowed persist failure, the retention rule pruning at its own window, and an end-to-end pass proving a real AI review persists both turns with the diff in the user one.
New module: 100% statements, branches, functions. All changed lines in existing files: 0 uncovered. Full suite: 24,045 green.
db:migrations:check,db:schema-drift:check, and every drift checker pass.