diff --git a/skills/content-refinement-agent/SKILL.md b/skills/content-refinement-agent/SKILL.md index fd58d28..679f7ee 100644 --- a/skills/content-refinement-agent/SKILL.md +++ b/skills/content-refinement-agent/SKILL.md @@ -1,6 +1,6 @@ --- name: content-refinement-agent -description: Step 5 of the PaperOrchestra pipeline (arXiv:2604.05018). Iteratively refine drafts/paper.tex by simulating peer review and applying targeted revisions, with strict accept/revert halt rules. Maintains a worklog and snapshots each iteration so revert is real, not symbolic. TRIGGER when the orchestrator delegates Step 5 or when the user asks to "refine the draft", "iterate on the paper", or "run peer review on this paper". +description: Step 5 of the PaperOrchestra pipeline (arXiv:2604.05018). Iteratively refine drafts/paper.tex by simulating peer review and applying targeted revisions, with strict accept/revert halt rules, deterministic 0-100 decision bands (Accept/Minor/Major/Reject) that drive a target-met early stop, and a Devil's Advocate concession-threshold guard that blocks acceptance on unresolved critical findings. Maintains a worklog and snapshots each iteration so revert is real, not symbolic. TRIGGER when the orchestrator delegates Step 5 or when the user asks to "refine the draft", "iterate on the paper", or "run peer review on this paper". data_access_level: verified_only --- @@ -127,6 +127,24 @@ issues a CRITICAL finding that remains unaddressed after all reviewers weigh in, that finding blocks the "refinement accepted" decision regardless of rubric scores. Log DA CRITICAL findings in worklog.json: `{da_critical: true, finding: "..."}`. +Record the DA's per-round findings and concession decisions in +`workspace/refinement/da_concessions.json` (schema in `references/da-reviewer.md`) +and enforce the concession-threshold protocol deterministically — this stops the +simulated DA from sycophantically caving: + +```bash +python skills/content-refinement-agent/scripts/concession_guard.py \ + --log workspace/refinement/da_concessions.json \ + --out workspace/refinement/iter/da_guard.json +# exit 0 = clear; exit 1 = standing CRITICAL → force REVERT this iteration; +# exit 2 = a concession was rejected (caving/consecutive) → DA must restate; +# exit 3 = schema error. +``` + +The guard rejects any concession made at `rebuttal_score < 4` or in a round +immediately following another concession, and restores the affected finding to +"standing". A standing CRITICAL (exit 1) overrides an ACCEPT into a REVERT. + Save to `workspace/refinement/iter/review.json`. ### 2. Score the draft @@ -144,6 +162,7 @@ The reviewer call produces both qualitative feedback and a per-axis score: "academic_style": {"score": 68, "justification": "..."} }, "overall_score": 64.5, + "decision_band": "Major Revision", "strengths": [...], "weaknesses": [...], "questions": [...] @@ -153,6 +172,12 @@ The reviewer call produces both qualitative feedback and a per-axis score: Save to `iter/score.json`. (Combined with `review.json` if your host emits one document; the schemas overlap.) +`decision_band` is derived deterministically from `overall_score` — Accept +(≥80) / Minor Revision (65–79) / Major Revision (50–64) / Reject (<50). Fill it +in with `python skills/content-refinement-agent/scripts/decision_band.py +--score-json iter/score.json` rather than by hand, so it can never disagree +with the number. The bands drive the target-met halt in Step 5. + ### 3. Apply revision Load the **verbatim Content Refinement Agent prompt** at `references/prompt.md`. @@ -195,6 +220,7 @@ python skills/content-refinement-agent/scripts/score_delta.py \ --curr workspace/refinement/iter/score.json \ --plateau-threshold 1.0 \ --plateau-streak 3 \ + --accept-threshold 80 \ --consecutive-small $CONSECUTIVE_SMALL \ > workspace/refinement/iter/delta.json @@ -208,10 +234,11 @@ print(d['consecutive_small']) ``` Exit codes: -- `0` — ACCEPT (overall improved or tied with non-negative net sub-axis, no plateau) +- `0` — ACCEPT (overall improved or tied with non-negative net sub-axis, below the Accept band, no plateau) - `1` — REVERT (overall decreased) - `2` — REVERT (tied overall, but net sub-axis change negative) - `4` — HALT_PLATEAU (accepted but N consecutive iterations below threshold — stop early) +- `5` — HALT_TARGET_MET (accepted AND reached the Accept band, overall ≥ 80 — stop) Behavior: @@ -221,6 +248,15 @@ Behavior: iterations are unlikely to yield meaningful gains. In practice ~85% of refinement gain comes in iteration 1; the plateau fires when subsequent iterations improve by less than 1 point for 3 consecutive rounds. +- **HALT_TARGET_MET (exit 5)**: keep current (it was accepted), but stop — the + paper has reached the Accept band (overall ≥ 80), so there is no reason to + keep iterating and risk a regression. The `delta.json` carries + `decision_band_prev` / `decision_band_curr` for the run report. + +**Override — DA CRITICAL.** If `concession_guard.py` (Step 1) returned exit 1 +for this iteration, treat the outcome as **REVERT** even when `score_delta.py` +says ACCEPT: roll back to `iter/paper.tex` and require the next revision to +address the standing CRITICAL finding. Always log the decision via `apply_worklog.py --decision ...`. @@ -229,10 +265,13 @@ Always log the decision via `apply_worklog.py --decision ...`. Halt the loop when ANY of these is true: 1. Iteration count reaches `ITER_CAP` (default 3). -2. `score_delta.py` returned exit code 1 or 2 (REVERT). +2. `score_delta.py` returned exit code 1 or 2 (REVERT), OR `concession_guard.py` + returned exit 1 (standing DA CRITICAL → forced REVERT). 3. The simulated reviewer's `weaknesses` list is empty (no actionable feedback to apply). 4. `score_delta.py` returned exit code 4 (HALT_PLATEAU — plateau early-stop). +5. `score_delta.py` returned exit code 5 (HALT_TARGET_MET — reached the Accept + band, overall ≥ 80; promote the current draft). ### 7. Promote the best snapshot @@ -247,9 +286,9 @@ cp workspace/refinement/iter/paper.pdf workspace/final/paper.pdf Then in the final report, tell the user: - How many iterations were run -- The final overall score -- The score trajectory (e.g., "iter0 64.5 → iter1 67.3 (accept) → iter2 69.1 (accept) → iter3 68.9 (revert, halt)") -- Which iteration was promoted +- The final overall score and its decision band (Accept / Minor / Major / Reject) +- The score trajectory with bands (e.g., "iter0 58.0 Major → iter1 67.3 Minor (accept) → iter2 81.0 Accept (halt: target met)") +- Which iteration was promoted, and the halt reason (revert / plateau / target met / iter cap / DA critical) ## Critical safety constraints (App. F.1 page 50–51) @@ -286,7 +325,9 @@ These rules prevent reward hacking and keep the refinement loop honest. - `references/writing-quality-check.md` — 5-category anti-AI-prose checklist (pointer to shared) - `references/ai-failure-modes.md` — 7-mode integrity gate run before first iteration (pointer to shared) - `references/da-reviewer.md` — Devil's Advocate reviewer protocol and concession rules -- `scripts/score_delta.py` — accept/revert decision from two score JSONs +- `scripts/score_delta.py` — accept/revert/halt decision from two score JSONs; emits decision bands + target-met halt (exit 5) +- `scripts/decision_band.py` — map an overall score to a canonical decision band (Accept/Minor/Major/Reject) +- `scripts/concession_guard.py` — enforce the DA concession-threshold protocol; blocks accept on a standing CRITICAL - `scripts/score_trajectory.py` — per-dimension score history, regression and plateau detection - `scripts/apply_worklog.py` — append iteration entries to worklog.json - `scripts/snapshot.py` — copy paper.tex/paper.pdf into iter/ for rollback diff --git a/skills/content-refinement-agent/references/da-reviewer.md b/skills/content-refinement-agent/references/da-reviewer.md index b3af40f..c363688 100644 --- a/skills/content-refinement-agent/references/da-reviewer.md +++ b/skills/content-refinement-agent/references/da-reviewer.md @@ -44,3 +44,60 @@ If the DA issues a CRITICAL finding, `score_delta.py` exit code is overridden to continuing. Log in worklog.json: `{da_critical: true, finding: "..."}` + +## Deterministic enforcement: `scripts/concession_guard.py` + +The concession threshold and the no-consecutive-concessions iron rule are easy +for a simulated reviewer to quietly relax — it caves. To make them +non-negotiable, record the DA's findings and concession decisions in a +**concession log** and run `concession_guard.py` each iteration. The script +re-derives which concessions are valid and whether any CRITICAL is still +standing; the host agent must obey its verdict over the LLM's prose. + +Concession log schema (`workspace/refinement/da_concessions.json`): + +```json +{ + "rounds": [ + { + "round": 1, + "findings": [ + { + "id": "F1", + "severity": "critical", + "attack": "Sec 4 claims X *causes* Y from correlation only.", + "rebuttal_score": 2, + "conceded": false, + "resolved": false + } + ] + } + ] +} +``` + +- `rebuttal_score` (1–5) — the DA's score of the author/revision rebuttal, + using the concession-threshold scale above. +- `conceded` — did the DA drop the attack this round? +- `resolved` — was the underlying issue actually fixed in the revision? + +```bash +python skills/content-refinement-agent/scripts/concession_guard.py \ + --log workspace/refinement/da_concessions.json \ + --out workspace/refinement/iter/da_guard.json +``` + +Verdict → loop action: + +| Guard exit | Meaning | Host action | +|---|---|---| +| 0 | CLEAR — no standing critical, no violations | accept may proceed | +| 1 | BLOCK — a critical is still standing | treat the iteration as **REVERT** (force `score_delta.py` outcome to exit 2) and require the next revision to address it | +| 2 | WARN — a concession was rejected (caving or consecutive) but no critical is blocked | the DA must restate the attack; do not let the rejected concession stand | +| 3 | input / schema error | fix the log | + +The guard rejects (does not honor) any concession made at `rebuttal_score < 4` +or in a round immediately following another conceding round, and restores the +affected finding to "standing". A standing CRITICAL blocks acceptance +regardless of rubric scores — this is the deterministic backstop behind the +prose rules above. diff --git a/skills/content-refinement-agent/references/halt-rules.md b/skills/content-refinement-agent/references/halt-rules.md index 7db93a1..15322a0 100644 --- a/skills/content-refinement-agent/references/halt-rules.md +++ b/skills/content-refinement-agent/references/halt-rules.md @@ -46,10 +46,32 @@ The script exits with: | 0 | ACCEPT_TIED_NON_NEGATIVE | keep new draft, continue loop | | 1 | REVERT_OVERALL_DECREASED | rollback to prev, halt loop | | 2 | REVERT_TIED_NEGATIVE_SUBAXIS | rollback to prev, halt loop | +| 4 | HALT_PLATEAU | keep new draft (accepted), halt loop | +| 5 | HALT_TARGET_MET | keep new draft (accepted), halt loop | The script also prints a one-line decision string and a JSON object on stdout for the host agent to log. +## Decision bands and the target-met halt + +`score_delta.py` annotates every comparison with the prev/curr **decision band** +(`decision_band.py`): Accept (≥80), Minor Revision (65–79), Major Revision +(50–64), Reject (<50). These give the loop an *absolute* quality target on top of +the *relative* delta rules. + +``` +if DECISION in {ACCEPT_IMPROVED, ACCEPT_TIED_NON_NEGATIVE}: + if curr.overall >= accept_threshold (default 80): + DECISION = HALT_TARGET_MET # exit 5 — keep current draft, stop + elif consecutive_small >= plateau_streak: + DECISION = HALT_PLATEAU # exit 4 — keep current draft, stop +``` + +Target-met takes precedence over plateau: once the paper reaches the Accept +band there is no reason to keep iterating and risk a regression. Both 4 and 5 +**keep** the just-accepted draft (unlike REVERT, which rolls back). Disable the +target-met halt with `--no-target-halt` (the band is still reported). + ## Loop-level halt conditions In addition to the per-iteration accept/revert decision, the loop halts @@ -66,6 +88,18 @@ when ANY of these is true: `overall_delta < threshold`. Default: threshold=1.0 points, N=3. Configurable via `--plateau-threshold` and `--plateau-streak`. +5. **Target met (exit code 5).** `score_delta.py` returns `HALT_TARGET_MET` + when an accepted iteration reaches the Accept band (overall ≥ 80). The + current draft is promoted; the loop stops rather than risk a regression. + +6. **DA CRITICAL standing (concession guard).** `concession_guard.py` exit 1 + means a Devil's Advocate CRITICAL finding is still standing (unresolved and + not validly conceded). This **overrides an ACCEPT into a REVERT**: roll back + and require the next revision to address the finding before continuing. Exit + 2 (a rejected concession with no blocked critical) is a WARN — the DA must + restate the attack, but it does not by itself force a revert. See + `da-reviewer.md`. + The calling loop must pass `--consecutive-small ` to `score_delta.py` to track the streak across iterations: diff --git a/skills/content-refinement-agent/references/reviewer-rubric.md b/skills/content-refinement-agent/references/reviewer-rubric.md index dda30d8..2f7467b 100644 --- a/skills/content-refinement-agent/references/reviewer-rubric.md +++ b/skills/content-refinement-agent/references/reviewer-rubric.md @@ -60,7 +60,8 @@ Then identify: - Questions: 2-4 specific questions the paper should answer for a reader to be convinced. - Decision: one of "Strong Accept", "Accept", "Borderline", "Reject", - "Strong Reject". + "Strong Reject". This is your qualitative judgment; it must be consistent + with the decision band the overall score falls into (see below). - Overall Score: weighted average 0-100. Use: overall = 0.20*depth + 0.20*execution + 0.15*flow + 0.15*clarity + 0.20*evidence + 0.10*style @@ -68,6 +69,25 @@ Then identify: Output STRICT JSON only. No prose outside the JSON. ``` +## Decision bands (canonical, derived from overall score) + +The free-form `decision` above is advisory. The refinement loop reasons about a +**canonical decision band** computed deterministically from `overall_score` by +`scripts/decision_band.py`, so the band can never drift from the number it +summarizes: + +| Overall score | Decision band | Loop meaning | +|---|---|---| +| ≥ 80 | **Accept** | Clears the acceptance bar — loop may stop (target met) | +| 65–79 | **Minor Revision** | Close; keep refining presentation | +| 50–64 | **Major Revision** | Substantive gaps remain | +| < 50 | **Reject** | Far from publishable | + +The reviewer's qualitative `decision` should agree with the band (e.g. don't +write "Accept" with an overall of 62). The thresholds are configurable on +`decision_band.py` / `score_delta.py` (`--accept-min` etc.) but default to the +table above. See `halt-rules.md` for how the Accept band triggers an early halt. + ## Output JSON schema ```json @@ -99,10 +119,17 @@ Output STRICT JSON only. No prose outside the JSON. "How does the temporal branch behave on videos longer than the training distribution?" ], "decision": "Borderline", + "decision_band": "Major Revision", "overall_score": 64.5 } ``` +`decision_band` is filled in deterministically — run +`python scripts/decision_band.py --score-json iter/score.json` and copy the +result, or let `score_delta.py` report it (it emits `decision_band_prev` / +`decision_band_curr` on every comparison). Never hand-set it inconsistently with +`overall_score`. + ## How the loop uses this output The `score_delta.py` script reads two consecutive score JSONs and applies diff --git a/skills/content-refinement-agent/scripts/concession_guard.py b/skills/content-refinement-agent/scripts/concession_guard.py new file mode 100644 index 0000000..746e178 --- /dev/null +++ b/skills/content-refinement-agent/scripts/concession_guard.py @@ -0,0 +1,189 @@ +#!/usr/bin/env python3 +""" +concession_guard.py — Enforce the Devil's Advocate concession-threshold protocol +deterministically (references/da-reviewer.md). + +The DA reviewer challenges the paper's core claims. A simulated reviewer left to +its own devices tends to *cave*: it concedes an attack as soon as the author +pushes back, even when the rebuttal is weak — sycophancy that defeats the point +of an adversarial reviewer. This script makes the protocol non-negotiable by +checking the DA's concession log against two hard rules, so the LLM cannot quietly +relax them: + + Rule 1 — Concession requires evidence. + A finding may be conceded only if its rebuttal_score is >= 4 (the protocol's + "rebuttal directly / strongly addresses the attack with evidence"). Conceding + at rebuttal_score <= 3 is caving: the concession is REJECTED and the finding + is restored to "standing". + + Rule 2 — No consecutive concessions (IRON RULE). + The DA may make at most one valid concession per two review rounds. + A concession in a round immediately following another conceding round is + REJECTED and the finding is restored to "standing". + +A CRITICAL finding that is still standing after these rules (not resolved by the +revision, and not validly conceded) BLOCKS the "refinement accepted" decision — +the host must treat the iteration as REVERT, regardless of rubric scores. + +Concession log schema (--log): + { + "rounds": [ + { + "round": 1, + "findings": [ + { + "id": "F1", + "severity": "critical" | "major" | "minor", + "attack": "Causal overclaiming: Sec 4 says X *causes* Y from corr only.", + "rebuttal_score": 2, # 1-5, DA's score of the author's rebuttal + "conceded": false, # did the DA drop the attack this round? + "resolved": false # was the underlying issue fixed in the revision? + } + ] + } + ] + } + +Usage: + python concession_guard.py --log workspace/refinement/da_concessions.json + python concession_guard.py --log da_concessions.json --out guard_report.json + +Exit codes: + 0 CLEAR — no standing critical, no protocol violations → accept may proceed + 1 BLOCK — a critical finding is still standing → host must REVERT this iteration + 2 WARN — protocol violation(s) found but no critical blocked → DA must restate; + the rejected concession does not by itself force a revert + 3 input / schema error +""" +import argparse +import json +import sys + +VALID_SEVERITY = {"critical", "major", "minor"} +CONCESSION_MIN_REBUTTAL = 4 # rebuttal_score >= this to allow a concession + + +def analyze(rounds: list) -> dict: + violations = [] + valid_concessions = [] + standing_criticals = [] + last_conceded_round = None # round index of the previous *valid* concession + + for r in rounds: + rnum = r.get("round") + conceded_this_round = False + for fnd in r.get("findings", []): + fid = fnd.get("id", "?") + sev = fnd.get("severity", "minor") + conceded = bool(fnd.get("conceded", False)) + resolved = bool(fnd.get("resolved", False)) + score = fnd.get("rebuttal_score") + + concession_valid = False + if conceded: + # Rule 1 — evidence threshold + if score is None or score < CONCESSION_MIN_REBUTTAL: + violations.append({ + "round": rnum, "id": fid, "type": "caving", + "detail": f"conceded at rebuttal_score={score} " + f"(< {CONCESSION_MIN_REBUTTAL}); concession rejected", + }) + # Rule 2 — no consecutive concessions + elif last_conceded_round is not None and rnum == last_conceded_round + 1: + violations.append({ + "round": rnum, "id": fid, "type": "consecutive_concession", + "detail": f"concession in round {rnum} immediately follows a " + f"concession in round {last_conceded_round}; " + f"rejected (max one per two rounds)", + }) + else: + concession_valid = True + conceded_this_round = True + valid_concessions.append({"round": rnum, "id": fid, "severity": sev}) + + # A critical is "standing" unless resolved OR validly conceded. + if sev == "critical" and not resolved and not concession_valid: + standing_criticals.append({ + "round": rnum, "id": fid, "attack": fnd.get("attack", ""), + }) + + if conceded_this_round: + last_conceded_round = rnum + + block = bool(standing_criticals) + if block: + action = "REVERT" + elif violations: + action = "DA_RESTATE" + else: + action = "PROCEED" + + return { + "rounds_analyzed": len(rounds), + "valid_concessions": valid_concessions, + "violations": violations, + "standing_criticals": standing_criticals, + "block_accept": block, + "recommended_action": action, + } + + +def main() -> int: + p = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("--log", required=True, help="Concession log JSON path") + p.add_argument("--out", default=None, help="Optional path to write the guard report JSON") + args = p.parse_args() + + try: + with open(args.log) as f: + log = json.load(f) + except (OSError, json.JSONDecodeError) as e: + print(f"ERROR: cannot read concession log: {e}", file=sys.stderr) + return 3 + + rounds = log.get("rounds") + if not isinstance(rounds, list) or not rounds: + print("ERROR: log['rounds'] must be a non-empty list", file=sys.stderr) + return 3 + + # Light schema validation — fail loudly rather than silently mis-classify. + for r in rounds: + for fnd in r.get("findings", []): + sev = fnd.get("severity", "minor") + if sev not in VALID_SEVERITY: + print(f"ERROR: finding {fnd.get('id','?')} has invalid severity " + f"'{sev}' (expected one of {sorted(VALID_SEVERITY)})", file=sys.stderr) + return 3 + score = fnd.get("rebuttal_score") + if score is not None and not (1 <= score <= 5): + print(f"ERROR: finding {fnd.get('id','?')} rebuttal_score={score} " + f"out of range 1-5", file=sys.stderr) + return 3 + + report = analyze(rounds) + + if args.out: + with open(args.out, "w") as f: + json.dump(report, f, indent=2, ensure_ascii=False) + + # Human-readable summary. + print(f"DA concession guard: {report['rounds_analyzed']} round(s) " + f"valid_concessions={len(report['valid_concessions'])} " + f"violations={len(report['violations'])} " + f"standing_criticals={len(report['standing_criticals'])}") + for v in report["violations"]: + print(f" VIOLATION [{v['type']}] round {v['round']} {v['id']}: {v['detail']}") + for c in report["standing_criticals"]: + print(f" STANDING CRITICAL round {c['round']} {c['id']}: {c['attack']}") + print(f" → recommended action: {report['recommended_action']}") + + if report["block_accept"]: + return 1 + if report["violations"]: + return 2 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/skills/content-refinement-agent/scripts/decision_band.py b/skills/content-refinement-agent/scripts/decision_band.py new file mode 100644 index 0000000..8ea41eb --- /dev/null +++ b/skills/content-refinement-agent/scripts/decision_band.py @@ -0,0 +1,107 @@ +#!/usr/bin/env python3 +""" +decision_band.py — Map a 0-100 overall score to a deterministic decision band. + +The reviewer rubric (references/reviewer-rubric.md) produces a weighted 0-100 +`overall_score` plus a free-form qualitative `decision`. That free-form label is +advisory; this script computes the *canonical* band the refinement loop reasons +about, so the band is reproducible and never drifts from the number it claims to +summarize. + +Default bands (override with flags): + >= 80 Accept + 65 .. 79 Minor Revision + 50 .. 64 Major Revision + < 50 Reject + +The bands give the loop an *absolute* quality target to complement the +*relative* accept/revert delta logic in score_delta.py: a paper can keep +improving relative to its previous self yet still sit in "Major Revision", and +conversely the loop can stop once it reaches "Accept" rather than burning +iterations chasing marginal gains. + +Usage: + python decision_band.py --score 74.6 + python decision_band.py --score-json workspace/refinement/iter2/score.json + python decision_band.py --score 81 --accept-min 80 --minor-min 65 --major-min 50 + +Exit codes: + 0 band computed (always, on valid input) + 2 usage / input error +""" +import argparse +import json +import sys + +DEFAULT_ACCEPT_MIN = 80 +DEFAULT_MINOR_MIN = 65 +DEFAULT_MAJOR_MIN = 50 + +ACCEPT = "Accept" +MINOR = "Minor Revision" +MAJOR = "Major Revision" +REJECT = "Reject" + +# Ordered worst -> best, so callers can compare band strength numerically. +BAND_RANK = {REJECT: 0, MAJOR: 1, MINOR: 2, ACCEPT: 3} + + +def band_for(score: float, + accept_min: float = DEFAULT_ACCEPT_MIN, + minor_min: float = DEFAULT_MINOR_MIN, + major_min: float = DEFAULT_MAJOR_MIN) -> str: + """Return the decision band for an overall score. Importable by other scripts.""" + if score >= accept_min: + return ACCEPT + if score >= minor_min: + return MINOR + if score >= major_min: + return MAJOR + return REJECT + + +def main() -> int: + p = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + src = p.add_mutually_exclusive_group(required=True) + src.add_argument("--score", type=float, help="Overall score 0-100") + src.add_argument("--score-json", help="Path to a score.json with an overall_score field") + p.add_argument("--accept-min", type=float, default=DEFAULT_ACCEPT_MIN) + p.add_argument("--minor-min", type=float, default=DEFAULT_MINOR_MIN) + p.add_argument("--major-min", type=float, default=DEFAULT_MAJOR_MIN) + args = p.parse_args() + + if not (args.accept_min > args.minor_min > args.major_min): + print("ERROR: thresholds must satisfy accept-min > minor-min > major-min", + file=sys.stderr) + return 2 + + if args.score is not None: + score = args.score + else: + try: + with open(args.score_json) as f: + data = json.load(f) + score = float(data["overall_score"]) + except (OSError, json.JSONDecodeError, KeyError, ValueError, TypeError) as e: + print(f"ERROR: cannot read overall_score from {args.score_json}: {e}", + file=sys.stderr) + return 2 + + band = band_for(score, args.accept_min, args.minor_min, args.major_min) + out = { + "overall_score": score, + "decision_band": band, + "band_rank": BAND_RANK[band], + "thresholds": { + "accept_min": args.accept_min, + "minor_min": args.minor_min, + "major_min": args.major_min, + }, + } + print(json.dumps(out, indent=2)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/skills/content-refinement-agent/scripts/score_delta.py b/skills/content-refinement-agent/scripts/score_delta.py index 367bc4c..841f637 100644 --- a/skills/content-refinement-agent/scripts/score_delta.py +++ b/skills/content-refinement-agent/scripts/score_delta.py @@ -19,12 +19,25 @@ Exit code 4. The loop should stop — further iterations are unlikely to yield meaningful gains. +And the decision-band target halt (see decision_band.py): + + - HALT_TARGET_MET if an accepted iteration reaches the "Accept" band + (overall >= --accept-threshold, default 80). Exit code 5. The paper now + clears the acceptance bar, so the loop stops on the current draft rather + than risk regressing it chasing marginal gains. Takes precedence over the + plateau halt. Disable with --no-target-halt. + +Every output JSON also carries the prev/curr decision band (Accept / Minor +Revision / Major Revision / Reject) so the run report can show an absolute +quality trajectory, not just relative deltas. + Exit codes: - 0 ACCEPT (improved or tied non-negative, and no plateau) + 0 ACCEPT (improved or tied non-negative; below Accept band; no plateau) 1 REVERT (overall decreased) 2 REVERT (tied with negative sub-axis delta) 3 argument or input error 4 HALT_PLATEAU (accepted but diminishing returns detected) + 5 HALT_TARGET_MET (accepted and reached the Accept band) Score JSON shape (see references/reviewer-rubric.md): { @@ -47,8 +60,12 @@ """ import argparse import json +import os import sys +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from decision_band import band_for, BAND_RANK # noqa: E402 + AXES = [ "scientific_depth", "technical_execution", @@ -89,6 +106,15 @@ def main() -> int: help="Number of consecutive small-delta accepted iterations so far " "(maintained by the calling loop; default: 0)", ) + p.add_argument( + "--accept-threshold", type=float, default=80.0, metavar="SCORE", + help="Overall score at/above which the paper is in the Accept band; " + "an accepted iteration here triggers HALT_TARGET_MET (default: 80)", + ) + p.add_argument( + "--no-target-halt", action="store_true", + help="Do not halt on reaching the Accept band (still reports the band)", + ) args = p.parse_args() try: @@ -126,12 +152,22 @@ def main() -> int: decision = "REVERT_OVERALL_DECREASED" exit_code = 1 + # --- Decision bands (absolute quality, independent of the delta) --- + band_prev = band_for(p_overall, args.accept_threshold) + band_curr = band_for(c_overall, args.accept_threshold) + # --- Plateau early-stop (only applies to accepted iterations) --- is_small_delta = overall_delta < args.plateau_threshold new_consecutive_small = (args.consecutive_small + 1) if is_small_delta else 0 plateau_triggered = False - if exit_code == 0 and new_consecutive_small >= args.plateau_streak: + # --- Target-met halt takes precedence over plateau --- + target_met = False + if exit_code == 0 and not args.no_target_halt and c_overall >= args.accept_threshold: + decision = "HALT_TARGET_MET" + exit_code = 5 + target_met = True + elif exit_code == 0 and new_consecutive_small >= args.plateau_streak: decision = "HALT_PLATEAU" exit_code = 4 plateau_triggered = True @@ -142,6 +178,11 @@ def main() -> int: "overall_prev": p_overall, "overall_curr": c_overall, "overall_delta": overall_delta, + "decision_band_prev": band_prev, + "decision_band_curr": band_curr, + "band_improved": BAND_RANK[band_curr] > BAND_RANK[band_prev], + "target_met": target_met, + "accept_threshold": args.accept_threshold, "subaxis_deltas": deltas, "net_subaxis": net_subaxis, "is_small_delta": is_small_delta,