feat(orb): salvageability axis — hold fixable, well-authored PRs instead of one-shot-closing - #8986
Conversation
…ead of one-shot-closing
The 2026-07-26 decision audit proved defect confidence alone tops out
around 96% close precision: 28 contributor PRs held on real high-
confidence findings later merged — the residual error class is 'real
defect, salvageable PR'. This adds the second axis as a deterministic,
explainable score (fixable defect class + the author's realized merged
history here + in-flight iteration; no AI call, every point named in
factors) and a manifest-only gate.aiReview.salvageabilityMinScore knob:
when set and an AT/ABOVE-floor AI-judgment close would fire, a clearing
score routes it to the existing held-for-review path with fix-it
guidance. Unset (the default) is byte-identical to today. Decision
records bump to v3 carrying {score, factors} whenever an AI judgment
shaped the decision, so the new boundary is auditable from day one; the
rubric gains the additive salvageable_close reason category.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | bc9e81a | Commit Preview URL Branch Preview URL |
Jul 26 2026, 03:39 PM |
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Logic backtestReplayed 0 historical case(s) for Backtest comparison:
|
❌ 1 Tests Failed:
View the top 1 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
|
Caution 🛑 LoopOver review result - fixes requiredReview updated: 2026-07-26 15:34:45 UTC
Review summary Nits — 5 non-blocking
CI checks failing
Decision drivers
Context & advisory signals — never blocks the verdict
Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. Decision record
Visual preview
Click any thumbnail to open the full-size screenshot. Before = production · After = this PR's preview deploy. Scroll preview
A short scroll-through clip (desktop) — click either thumbnail to open the full animation. Evidence for scroll-linked behavior a single screenshot can't show. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
Engine version 3.14.1 -> 3.15.0 (the manifest gate type gained aiReviewSalvageabilityMinScore — a public engine surface change — and the advisory twin-pair guard accepts a bump where the new resolver deliberately does not belong in gate-advisory.ts, matching its low-confidence sibling). Sync packages/loopover-miner/expected-engine .version, register the settings->yml alias row for salvageabilityMinScore in check-docs-drift, and update the two decision-record schemaVersion assertions to v3.


Closes #8962. Advances #8828.
The decision audit's deepest finding: defect-confidence measures whether the defect is REAL, but close-correctness depends on whether closing is the right ACTION — 28 audited holds carried real findings at confidence 0.55–0.97 and still merged (17 as-is, 11 after the author fixed the flagged defect), while acted closes at the same confidences ran 96.0% correct. Same confidence, opposite close-correctness, split entirely by salvageability. That class is the road from ~96% to the 99.5% target.
The score (
src/review/salvageability.ts, pure): deterministic and explainable — no AI call, every point named infactors:The wire (
salvageability-wire.ts): two cheap reads (merged count viareview_auditpr_outcome, cycles viadecision_records), fail-OPEN — a DB blip degrades to today's behavior, never blocks or unblocks.The gate seam (
resolveAiReviewSalvageableHold):resolveAiReviewLowConfidenceHold's sibling for the OTHER side of the floor — fires only when the knob is set, the failure is AI-judgment-only, every blocker sits at/above the close-confidence floor, and the score clearsgate.aiReview.salvageabilityMinScore. Routes into the existing held-for-review path with fix-it guidance; the low-confidence hold keeps precedence below the floor. Knob absent (default) short-circuits before any IO.Auditability: decision records bump to schema v3 carrying
salvageability: {score, factors}whenever an AI judgment shaped the decision (recomputed at finalize even when the knob is off, so the boundary accrues evidence BEFORE anyone enables it); the adjudication rubric gains the additivesalvageable_closereason category (correct/incorrect semantics unchanged — version 1 labels stay comparable).Config-as-code parity sweep: engine manifest type/parse/serializer/round-trip + src overlay + policy passthrough + OpenAPI schema (regenerated) + both example-YAML twins (the #8846 lesson).
Tests: 12 pure/resolver/wire tests (every factor tier, structural-wins, all stand-down arms, fail-open); manifest parse/round-trip/overlay; policy passthrough both arms; and an end-to-end queue test — an ABOVE-floor consensus defect on a salvageable PR (manifest knob 60, mechanical finding, 2 merged priors) is HELD not closed, with the v3 record carrying score 70 and its factors. Changed-line coverage: all statements and branches covered.