feat(content-refinement): reviewer decision bands + Devil's Advocate concession guard#9
Open
Ar9av wants to merge 1 commit into
Open
feat(content-refinement): reviewer decision bands + Devil's Advocate concession guard#9Ar9av wants to merge 1 commit into
Ar9av wants to merge 1 commit into
Conversation
Make the simulated-reviewer halt logic more principled with two deterministic additions, both backward compatible with the existing score_delta exit codes. Decision bands (decision_band.py): - Map the 0-100 overall_score to a canonical band: Accept (>=80), Minor Revision (65-79), Major Revision (50-64), Reject (<50). Importable + CLI. - score_delta.py now emits decision_band_prev/curr on every comparison and adds a target-met halt (exit 5): when an accepted iteration reaches the Accept band, the loop promotes the current draft and stops rather than risk regressing it chasing marginal gains. Takes precedence over the plateau halt; disable with --no-target-halt. Existing exit codes 0/1/2/4 unchanged. DA concession guard (concession_guard.py): - Enforce the Devil's Advocate concession-threshold protocol deterministically so the simulated reviewer cannot sycophantically cave. Reads a concession log and rejects any concession made at rebuttal_score < 4 or in a round immediately following another concession, restoring the affected finding to 'standing'. A standing CRITICAL blocks acceptance (exit 1 -> host treats the iteration as REVERT); a rejected non-blocking concession is a WARN (exit 2). Docs: reviewer-rubric.md (bands + decision_band field), da-reviewer.md (concession-log schema + guard verdict table), halt-rules.md (exit 5 + guard override), and SKILL.md (wired into Steps 1/2/5/6, Resources, description).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Upgrades the
content-refinement-agentsimulated-reviewer loop with two deterministic mechanisms, both backward-compatible with the existingscore_delta.pyexit-code contract:This is item #4 from the study of academic-research-skills, reimplemented to fit this repo (concepts only — no files copied; that repo is CC-BY-NC).
Why
The loop already had a 6-axis 0-100 rubric, anti-inflation caps, score deltas, and a prose-only DA protocol. Two gaps:
decisionwas free-form and not tied to the numeric score — it could drift (e.g. "Accept" at overall 62), and the loop had no absolute quality target, only relative deltas.How
scripts/decision_band.pyoverall_score→ canonical band (Accept ≥80 / Minor 65–79 / Major 50–64 / Reject <50); importable + CLIscripts/score_delta.pydecision_band_prev/curr; adds target-met halt (exit 5)scripts/concession_guard.pyDecision bands + target-met halt. Bands give the loop an absolute target on top of the relative accept/revert logic. When an accepted iteration reaches the Accept band (overall ≥ 80),
score_delta.pyreturns the new exit code 5 (HALT_TARGET_MET) — promote the current draft and stop, rather than risk regressing it for marginal gains. Takes precedence over the plateau halt;--no-target-haltdisables it (band still reported). Exit codes 0/1/2/4 are unchanged, so existing callers keep working.DA concession guard. Reads a concession log (
da_concessions.json) and re-derives which concessions are legitimate:rebuttal_score ≥ 4. Caving at ≤3 is rejected; the finding is restored to standing.Testing
py_compileclean on all three scripts.decision_band.py: 81→Accept, 74.6→Minor, 58→Major, 42→Reject; readsoverall_scorefrom a realscore.json.score_delta.py: accept below band → exit 0 (Major→Minor); reaches band → HALT_TARGET_MET exit 5;--no-target-halt→ exit 0; regression → exit 1. Regression-tested on the real workspaceiter1→iter2files: still ACCEPT_IMPROVED, exit 0, behavior preserved.concession_guard.py: clear→0/PROCEED, caving-on-critical→1/REVERT (concession rejected, critical stands), consecutive-concessions-on-majors→2/DA_RESTATE, unresolved-critical→1/REVERT, bad severity→3.Docs
references/reviewer-rubric.md— decision-band table +decision_bandoutput fieldreferences/da-reviewer.md— concession-log schema + guard verdict→action tablereferences/halt-rules.md— exit 5, decision bands, guard override of ACCEPT→REVERTSKILL.md— wired into Steps 1/2/5/6, Resources, and frontmatter description