fix(orb): stop two wrong-close paths — superseded cancelled checks, permanent linked-issue violations#9113
Merged
Merged
Conversation
…d permanently-remembered linked-issue violations #9053 -- CONFIRMED live, upgraded from the issue's "suspected". Verified on a real PR head that GET /commits/{sha}/check-runs returns runs from MULTIPLE suites even under GitHub's default filter=latest: 11 runs across 6 suites, 10 of them from older suites. Combined with ci.yml's `concurrency: cancel-in-progress: true`, a same-SHA re-trigger (reopened / ready_for_review / labeled -- and ORB ITSELF applies labels) starts a new suite and cancels the first, leaving conclusion:"cancelled" runs on the head SHA in the OLD suite. "cancelled" is in CI_FAILING_CONCLUSIONS and the dedupe key deliberately includes the suite id (so a genuine failure is never hidden by a later success), so those stale runs landed in failingDetails beside the new suite's green ones -> ciState "failed" -> willClose on ciFailed -> one-shot auto-close of a PR whose CI is green. The executor's recheck re-derived the same verdict, confirming rather than catching it. "CI is failing" is the most common close reason in the ledger, so the false-positive cost is high. Fix drops a cancelled run only when a strictly NEWER suite ran the same (name, app). A cancellation with no superseding attempt still fails; genuine failure / timed_out / startup_failure conclusions are never dropped, so the cross-suite non-collapse keeps doing its job; a non-numeric or absent suite id is not comparable and is conservatively kept. #9029 -- linked_issue_hard_rule_violated_at was written the first time ANY pass observed a violation and NEVER cleared, so an ephemeral, benign issue state condemned a PR for its whole lifetime: a maintainer who momentarily self-assigns the linked issue to triage it stamped a permanent one-shot-close marker on every PR linking it, and un-assigning did not exonerate. The only escape was disabling every hard rule repo-wide. The persistence exists to stop a stamp-then-dodge (edit the body to unlink the ineligible issue after being caught), so it is kept and narrowed to exactly that case. Migration 0184 records WHICH issues were linked at violation time; the merge exonerates only when the SAME set is still linked and the live rules now pass (the improvement came from the issue side, which the author does not control), and the marker is then cleared. A changed linked set -- swapped, unlinked, or extended -- still stands, as do pre-#9029 rows with no snapshot, where a dodge cannot be ruled out. POLICY REVERSAL, called out explicitly: this inverts an existing regression test that asserted an unchanged body whose linked issue merely lost its assignee STILL closed. That is precisely the harm #9029 documents. The sibling test covering the real dodge (body edited to unlink) is unchanged and still passes. Closes #9029 Closes #9053 Tests: 4 CI regressions (superseded-cancelled is ignored including when the newer suite appears FIRST in array order, a lone cancelled run still fails, a genuine older-suite failure still reports, non-comparable suite ids are kept), 5 merge-discriminator unit tests, 3 repository-level tests for the snapshot writer/clearer, and an exoneration-clear-failure test proving a failed D1 write never becomes a wrong close. 100% line+branch coverage on all added lines; 1715/1715 across 12 suites; all drift gates green.
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9113 +/- ##
=======================================
Coverage 93.88% 93.89%
=======================================
Files 813 813
Lines 80735 80768 +33
Branches 24503 24518 +15
=======================================
+ Hits 75800 75834 +34
Misses 3563 3563
+ Partials 1372 1371 -1
Flags with carried forward coverage won't be shown. Click here to find out more.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two paths that auto-closed PRs that should not have been closed.
#9053 — upgraded from "suspected" to CONFIRMED. The issue asked for one empirical check; I ran it. On a real PR head,
GET /commits/{sha}/check-runsreturned 11 runs across 6 suites — 10 of them from older suites, even under GitHub's defaultfilter=latest. Combined withci.yml'sconcurrency: cancel-in-progress: true, a same-SHA re-trigger (reopened/ready_for_review/labeled— and ORB itself applies labels) starts a new suite and cancels the first, leavingconclusion: "cancelled"runs on the head SHA in the old suite.cancelledis inCI_FAILING_CONCLUSIONS, and the dedupe key deliberately includes the suite id so a genuine failure can't be hidden by a later success — so those stale runs landed infailingDetailsbeside the new suite's green ones →ciState: "failed"→willClose→ one-shot auto-close of a PR whose CI is green. The executor's act-boundary recheck re-derives the same verdict, so it confirmed rather than caught it.CI is failingis the single most common close reason in the ledger, which makes the false-positive cost high.The fix drops a cancelled run only when a strictly newer suite ran the same
(name, app). A cancellation with no superseding attempt still fails; genuinefailure/timed_out/startup_failureare never dropped; a non-numeric or absent suite id isn't comparable and is conservatively kept.#9029 — a momentary issue state condemned a PR forever.
linked_issue_hard_rule_violated_atwas written the first time any pass saw a violation and never cleared. A maintainer who momentarily self-assigns the linked issue to triage it stamped a permanent one-shot-close marker on every PR linking it — and un-assigning did not exonerate. The only escape was disabling every hard rule repo-wide. That punishes a contributor for the maintainer's own action.The persistence exists to stop a stamp-then-dodge (edit the body to unlink the ineligible issue after being caught), so it's kept and narrowed to exactly that. Migration
0184records which issues were linked at violation time; the merge exonerates only when the same set is still linked and the live rules now pass — i.e. the improvement came from the issue side, which the author doesn't control. A changed linked set (swapped, unlinked, or extended) still stands, as do pre-#9029 rows with no snapshot where a dodge can't be ruled out.Closes #9029
Closes #9053
#9029 requires inverting an existing regression test that asserted an unchanged PR body whose linked issue merely lost its assignee still closed. That behavior is precisely the harm #9029 documents, so the test is rewritten to the new policy with the rationale inline. The sibling test covering the genuine dodge (body edited to unlink) is unchanged and still passes — that's the discrimination working, and it's what makes this a narrowing rather than a removal of the protection.
On #9031 (also in this branch name)
Not fixed here. Its core ask — a cron-rechecked deadline on the pending-closure flag — is a scheduling design I didn't want to bundle into a behavior-change PR. Worth noting its stated cause is now partly addressed: it says
clearLinkedIssueFlag"can never fire" because the violation memory is permanent, and #9029 makes that memory clearable. I'll take the remaining deadline work as its own change.Test plan