Skip to content

fix(orb): stop two wrong-close paths — superseded cancelled checks, permanent linked-issue violations#9113

Merged
JSONbored merged 1 commit into
mainfrom
fix/9029-9031-9053-wrong-close
Jul 26, 2026
Merged

fix(orb): stop two wrong-close paths — superseded cancelled checks, permanent linked-issue violations#9113
JSONbored merged 1 commit into
mainfrom
fix/9029-9031-9053-wrong-close

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

Two paths that auto-closed PRs that should not have been closed.

#9053 — upgraded from "suspected" to CONFIRMED. The issue asked for one empirical check; I ran it. On a real PR head, GET /commits/{sha}/check-runs returned 11 runs across 6 suites — 10 of them from older suites, even under GitHub's default filter=latest. Combined with ci.yml's concurrency: cancel-in-progress: true, a same-SHA re-trigger (reopened / ready_for_review / labeled — and ORB itself applies labels) starts a new suite and cancels the first, leaving conclusion: "cancelled" runs on the head SHA in the old suite.

cancelled is in CI_FAILING_CONCLUSIONS, and the dedupe key deliberately includes the suite id so a genuine failure can't be hidden by a later success — so those stale runs landed in failingDetails beside the new suite's green ones → ciState: "failed"willCloseone-shot auto-close of a PR whose CI is green. The executor's act-boundary recheck re-derives the same verdict, so it confirmed rather than caught it. CI is failing is the single most common close reason in the ledger, which makes the false-positive cost high.

The fix drops a cancelled run only when a strictly newer suite ran the same (name, app). A cancellation with no superseding attempt still fails; genuine failure/timed_out/startup_failure are never dropped; a non-numeric or absent suite id isn't comparable and is conservatively kept.

#9029 — a momentary issue state condemned a PR forever. linked_issue_hard_rule_violated_at was written the first time any pass saw a violation and never cleared. A maintainer who momentarily self-assigns the linked issue to triage it stamped a permanent one-shot-close marker on every PR linking it — and un-assigning did not exonerate. The only escape was disabling every hard rule repo-wide. That punishes a contributor for the maintainer's own action.

The persistence exists to stop a stamp-then-dodge (edit the body to unlink the ineligible issue after being caught), so it's kept and narrowed to exactly that. Migration 0184 records which issues were linked at violation time; the merge exonerates only when the same set is still linked and the live rules now pass — i.e. the improvement came from the issue side, which the author doesn't control. A changed linked set (swapped, unlinked, or extended) still stands, as do pre-#9029 rows with no snapshot where a dodge can't be ruled out.

Closes #9029
Closes #9053

⚠️ Deliberate policy reversal — please sanity-check

#9029 requires inverting an existing regression test that asserted an unchanged PR body whose linked issue merely lost its assignee still closed. That behavior is precisely the harm #9029 documents, so the test is rewritten to the new policy with the rationale inline. The sibling test covering the genuine dodge (body edited to unlink) is unchanged and still passes — that's the discrimination working, and it's what makes this a narrowing rather than a removal of the protection.

On #9031 (also in this branch name)

Not fixed here. Its core ask — a cron-rechecked deadline on the pending-closure flag — is a scheduling design I didn't want to bundle into a behavior-change PR. Worth noting its stated cause is now partly addressed: it says clearLinkedIssueFlag "can never fire" because the violation memory is permanent, and #9029 makes that memory clearable. I'll take the remaining deadline work as its own change.

Test plan

  • 4 CI regressions: superseded-cancelled ignored (including when the newer suite appears first in array order — GitHub documents no ordering contract, so supersession must be decided by suite id); a lone cancelled run still fails; a genuine older-suite failure still reports; non-comparable suite ids are kept
  • 5 merge-discriminator unit tests: same-set exonerates, order-insensitive, swapped/unlinked/extended all still violate, no-snapshot rows keep pre-orb(gate): a momentary linked-issue state permanently condemns a PR (linked_issue_hard_rule_violated_at never clears) #9029 behavior, live-violation never resurrected
  • 3 repository-level tests against the real schema: snapshot deduped+sorted, COALESCE never overwrites the first violation, default reason applied, clear removes all three fields
  • Exoneration-clear-failure test proving a failed D1 write never becomes a wrong close
  • 100% line and branch coverage on every added line; 1715/1715 across 12 suites; migrations/schema-drift/docs/engine-parity/cf-typegen/release-manifest gates all green

…d permanently-remembered linked-issue violations

#9053 -- CONFIRMED live, upgraded from the issue's "suspected". Verified on a
real PR head that GET /commits/{sha}/check-runs returns runs from MULTIPLE
suites even under GitHub's default filter=latest: 11 runs across 6 suites, 10
of them from older suites. Combined with ci.yml's `concurrency:
cancel-in-progress: true`, a same-SHA re-trigger (reopened / ready_for_review /
labeled -- and ORB ITSELF applies labels) starts a new suite and cancels the
first, leaving conclusion:"cancelled" runs on the head SHA in the OLD suite.
"cancelled" is in CI_FAILING_CONCLUSIONS and the dedupe key deliberately
includes the suite id (so a genuine failure is never hidden by a later
success), so those stale runs landed in failingDetails beside the new suite's
green ones -> ciState "failed" -> willClose on ciFailed -> one-shot auto-close
of a PR whose CI is green. The executor's recheck re-derived the same verdict,
confirming rather than catching it. "CI is failing" is the most common close
reason in the ledger, so the false-positive cost is high.

Fix drops a cancelled run only when a strictly NEWER suite ran the same
(name, app). A cancellation with no superseding attempt still fails; genuine
failure / timed_out / startup_failure conclusions are never dropped, so the
cross-suite non-collapse keeps doing its job; a non-numeric or absent suite id
is not comparable and is conservatively kept.

#9029 -- linked_issue_hard_rule_violated_at was written the first time ANY pass
observed a violation and NEVER cleared, so an ephemeral, benign issue state
condemned a PR for its whole lifetime: a maintainer who momentarily
self-assigns the linked issue to triage it stamped a permanent one-shot-close
marker on every PR linking it, and un-assigning did not exonerate. The only
escape was disabling every hard rule repo-wide.

The persistence exists to stop a stamp-then-dodge (edit the body to unlink the
ineligible issue after being caught), so it is kept and narrowed to exactly
that case. Migration 0184 records WHICH issues were linked at violation time;
the merge exonerates only when the SAME set is still linked and the live rules
now pass (the improvement came from the issue side, which the author does not
control), and the marker is then cleared. A changed linked set -- swapped,
unlinked, or extended -- still stands, as do pre-#9029 rows with no snapshot,
where a dodge cannot be ruled out.

POLICY REVERSAL, called out explicitly: this inverts an existing regression
test that asserted an unchanged body whose linked issue merely lost its
assignee STILL closed. That is precisely the harm #9029 documents. The sibling
test covering the real dodge (body edited to unlink) is unchanged and still
passes.

Closes #9029
Closes #9053

Tests: 4 CI regressions (superseded-cancelled is ignored including when the
newer suite appears FIRST in array order, a lone cancelled run still fails, a
genuine older-suite failure still reports, non-comparable suite ids are kept),
5 merge-discriminator unit tests, 3 repository-level tests for the snapshot
writer/clearer, and an exoneration-clear-failure test proving a failed D1 write
never becomes a wrong close. 100% line+branch coverage on all added lines;
1715/1715 across 12 suites; all drift gates green.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored JSONbored self-assigned this Jul 26, 2026
@JSONbored
JSONbored merged commit 60f0c69 into main Jul 26, 2026
4 checks passed
@JSONbored
JSONbored deleted the fix/9029-9031-9053-wrong-close branch July 26, 2026 20:59
@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 26, 2026
@codecov

codecov Bot commented Jul 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.89%. Comparing base (90ecd88) to head (60ab394).
⚠️ Report is 3 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #9113   +/-   ##
=======================================
  Coverage   93.88%   93.89%           
=======================================
  Files         813      813           
  Lines       80735    80768   +33     
  Branches    24503    24518   +15     
=======================================
+ Hits        75800    75834   +34     
  Misses       3563     3563           
+ Partials     1372     1371    -1     
Flag Coverage Δ
backend 95.17% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/db/repositories.ts 96.73% <100.00%> (+<0.01%) ⬆️
src/db/schema.ts 73.11% <ø> (ø)
src/github/backfill.ts 95.95% <100.00%> (+0.05%) ⬆️
src/queue/processors.ts 95.65% <100.00%> (+0.03%) ⬆️
src/review/linked-issue-hard-rules.ts 98.48% <100.00%> (+0.20%) ⬆️
src/types.ts 100.00% <ø> (ø)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

1 participant