Skip to content

orb(gate): finding severity is decorative — 'warning' findings close PRs and a 'critical' one does nothing #9085

Description

@JSONbored

Severity never determines consequence anywhere on the decision path. severityToAnnotationLevel (src/rules/advisory.ts ~430-434) and formatCheckRunOutput (~573) are its only readers, both purely presentational. The two disagree in both directions:

finding severity actual consequence
slop_risk_above_threshold (~1300-1307) warning pushed straight into blockers (~681) → failureCLOSE
missing_linked_issue (~859-867) warning block mode → CLOSE, and breaker-exempt
self_authored_linked_issue (~895-901) warning breaker-exempt CLOSE
duplicate_pr_risk (~878-884) warning defaults to block → breaker-exempt CLOSE
ai_consensus_defect critical defaults to advisory (~1051) → no effect at all
oversized_pr / guardrail_hold (~602, ~620) warning hold only

A contributor reading a ⚠️ on the finding that is about to close their PR is being actively misinformed.

Taxonomy drift found alongside it

  • slop_risk_above_threshold is in neither CONFIGURED_GATE_BLOCKER_SIGNAL_CODES (~192-209) nor resolveConfiguredGateMode. It gates real closes but records reversals under a different id (slop_gate_score, ~1241), so the per-rule precision check in downgradeCloseToHold (Reason B) can never apply to it.
  • surface_lane_reject is in CONCRETE_EVIDENCE_BLOCKER_CODES but not in resolveConfiguredGateMode — live, but invisible to recordConfiguredGateBlockerSignals and therefore to the entire calibration corpus.
  • Near-miss pair: repo_not_registered holds the gate (~1010) while repo_unregistered (~786) does not. Two codes one character apart with opposite consequences.
  • Confirmed dead/inert: authorMergedPrCount, authorClosedUnmergedPrCount, confirmedContributor (~117-126, self-documented INERT), reconcileGateEvaluationForGreenCi (~281-285, voids both args), aiCiRefutationEnabled, ciRequiredContextsVerified, assignLinkedIssues.
  • Dead branch: conclusion === "action_required" in manualHoldReason (agent-actions.ts ~1131) is unreachable — the core only returns success/neutral/failure, and other callers pass "skipped" which returns earlier.

Also: confidence defaults to maximum on the gate side

advisory.ts ~242 (blocker.confidence ?? 1) and ~1055 finding.confidence ?? 1 default missing confidence to 1.0, suppressing the low-confidence hold and yielding a straight one-shot close. That directly contradicts CONFIDENCE_WHEN_UNSTATED = 0.5 (src/services/ai-review.ts ~756), whose own doc says "silence is not certainty". Currently masked because producers always set confidence — but the cache path (repositories.ts ~5098 parses AdvisoryFinding[] with no schema validation) would land straight on the wrong default.

Fix

  1. Make severity load-bearing (or rename it to something honestly presentational) — a warning must not be able to close a PR.
  2. Register slop_risk_above_threshold and surface_lane_reject in the blocker-signal taxonomy so the breaker and calibration can see them.
  3. Default absent confidence to CONFIDENCE_WHEN_UNSTATED, not 1.
  4. Rename one of repo_not_registered / repo_unregistered.

Refs #9074, #9040, #9003.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.orbGittensory Orb related - maintainer self-hosting analytics.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions