fix(selfhost): rescue orphaned dispositions, break the freeze-deadlock, and release AI-review locks before a hard kill can - #9174
Merged
Conversation
…k, and release AI-review locks before a hard kill can #8997 — a deploy restart can leave a PR wearing a decisive panel with no matching disposition. SIGTERM kills whatever pass is in flight at an arbitrary point, and the worst possible cut is between the public-surface publish (panel, gate check-run, CI aggregate — all committed) and maybeRunAgentMaintenance ever claiming its per-PR actuation lock for that head. Confirmed live on #8965: the panel published at 14:55:24Z from the dying container, and the matching close only landed at 15:07:53Z — ~12 minutes later, purely because an unrelated later sweep tick happened to re-run the whole pass from scratch. The existing outage-repair check (lastPublishedSurfaceSha !== headSha) cannot see this: the surface IS current, so it already looks healthy. maybeRunAgentMaintenance now records a marker (agent.maintenance.disposition_ considered) the moment it claims the actuation lock — the moment a real disposition attempt begins, independent of what it decides. A new standalone scan, reconcileSurfaceWithoutDisposition, rides the same sweep tick as the existing pr-outcome/pending-closure repair scans (#9026/#9031) and re-enqueues a regate for any open PR whose current-head surface carries no such marker. Deliberately NOT folded into the existing surfaceRepairPriorityPullNumbers: that function's return value drives the regular sweep's staleness-ordered fan-out and is asserted against by a large, already-green test surface (dispatch shape, ordering, backlog-restriction semantics) — an early attempt at injecting this check there flagged every "surface current, nothing else recorded" fixture in that file, which is a real, deliberate policy widening this fix makes, but one that belongs in its own independently-testable scan rather than retrofitted onto a heavily-relied-upon function blind. #8998 — an orphaned ai-review-lock (30-min TTL) starves every subsequent re-review, including an explicit maintainer re-run, with the "already in progress" placeholder for real time nobody is reviewing anything. Two of the three layers this needed were already shipped earlier this session: the boot-time flush (#9021, ORPHANED_LOCK_KEY_PATTERNS already includes ai-review-lock:*) and the explicit force-lock-steal for a maintainer's forced re-run (#9008, `steal: webhook.forceAiReview === true` already threaded into claimAiReviewLock). What was still missing: queue.stop() (#9007) lets an in-flight job finish naturally, releasing its own lock via its own finally block — but only if the orchestrator's SIGKILL grace period is long enough for that drain to complete, and a common 10-30s grace period is shorter than an AI-review LLM call legitimately runs. A new held-lock-registry tracks every real ai-review-lock claim this process currently holds; the shutdown handler now releases all of them FIRST, before anything else in the shutdown sequence, so the lock is gone the instant SIGTERM/SIGINT arrives regardless of whether the rest of shutdown gets time to finish. Complementary to the boot-time flush, not a replacement for it: a true `kill -9` delivers no signal at all, so that backstop still matters. #8999 — a lock-contended hold applied the sticky manual-review label, and the label then froze every LATER pass from running a FRESH AI review at all (only replay) — meaning the one thing that could produce the real verdict needed to auto-clear the label (already implemented, #9009) could never run, because the freeze it depends on being lifted is gated by the very label it's trying to clear. #9009 only handled the case where a fresh review DID run and showed contention had resolved; it could not reach that state on a repo requiring blocking AI review, because the freeze blocks the fresh attempt outright. Fixed by exempting the freeze specifically when the label's own provenance — per the same transient marker #9009 already reads/writes — is the lock-contention hold and nothing else. This does not reopen the gaming surface the freeze exists to close (a contributor iterating pushes to buy a fresh verdict): a lock-contention hold is an infra artifact of two ORB passes racing, not a contributor action, and if some OTHER reason also justifies the hold, that reason re-applies the label on this same pass's own disposition regardless of the exemption. #9013's largest remaining piece (a single per-PR mutex spanning the public-surface publish AND the disposition plan/execute, currently two separately-claimed critical sections) is NOT included here — an initial attempt showed it needs a moderate refactor of both giant call sites (the sweep/CI- completion path and the webhook path) with real risk of subtly changing span/catch/decisionOutcome semantics the existing test suite exercises extensively, and that deserves its own dedicated, carefully-tested pass rather than a rushed addition alongside three already-substantial fixes. Targeted tests only (no full local gate run this pass, per instruction): typecheck clean, and every touched/new test file passes locally — 39 new tests across surface-disposition-reconciler.test.ts, held-lock-registry.test.ts, and the extended job-dispatch.test.ts fan-out coverage — plus a broader targeted sweep of 967 tests across the queue/lifecycle/transient-lock/precision-breaker suites most likely to interact with the actuation-lock, freeze, and sweep- priority code paths this change touches. CI runs the full gate on push. Closes #8997 Closes #8998 Closes #8999
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #9174 +/- ##
==========================================
- Coverage 93.90% 92.68% -1.22%
==========================================
Files 817 819 +2
Lines 81039 81085 +46
Branches 24611 24617 +6
==========================================
- Hits 76096 75150 -946
- Misses 3568 4850 +1282
+ Partials 1375 1085 -290
Flags with carried forward coverage won't be shown. Click here to find out more.
|
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#8997 — a deploy restart can leave a PR wearing a decisive panel with no matching disposition
SIGTERM kills whatever pass is in flight at an arbitrary point, and the worst possible cut is between the public-surface publish (panel, gate check-run, CI aggregate — all committed) and
maybeRunAgentMaintenanceever claiming its per-PR actuation lock for that head. Confirmed live on #8965: the panel published at 14:55:24Z from the dying container, and the matching close only landed at 15:07:53Z — ~12 minutes later, purely because an unrelated later sweep tick happened to re-run the whole pass from scratch. The existing outage-repair check (lastPublishedSurfaceSha !== headSha) cannot see this: the surface is current, so it already looks healthy.maybeRunAgentMaintenancenow records a marker (agent.maintenance.disposition_considered) the moment it claims the actuation lock — the moment a real disposition attempt begins, independent of what it decides. A new standalone scan,reconcileSurfaceWithoutDisposition, rides the same sweep tick as the existing pr-outcome/pending-closure repair scans (#9026/#9031) and re-enqueues a regate for any open PR whose current-head surface carries no such marker.Deliberately not folded into the existing
surfaceRepairPriorityPullNumbers: that function's return value drives the regular sweep's staleness-ordered fan-out and is asserted against by a large, already-green test surface (dispatch shape, ordering, backlog-restriction semantics) — an early attempt at injecting this check there flagged every "surface current, nothing else recorded" fixture in that file, which is a real, deliberate policy widening this fix makes, but one that belongs in its own independently-testable scan rather than retrofitted onto a heavily-relied-upon function blind.#8998 — an orphaned ai-review-lock starves every subsequent re-review
A 30-min TTL lock left behind by a dead process starves every subsequent pass — including an explicit maintainer re-run — with the "already in progress" placeholder for real time nobody is reviewing anything. Two of the three layers this needed were already shipped earlier this session:
ORPHANED_LOCK_KEY_PATTERNSalready includesai-review-lock:*)steal: webhook.forceAiReview === truealready threaded intoclaimAiReviewLock)What was still missing:
queue.stop()(#9007) lets an in-flight job finish naturally, releasing its own lock via its ownfinallyblock — but only if the orchestrator's SIGKILL grace period is long enough for that drain to complete, and a common 10-30s grace period is shorter than an AI-review LLM call legitimately runs. A newheld-lock-registrytracks every realai-review-lockclaim this process currently holds; the shutdown handler now releases all of them first, before anything else in the shutdown sequence, so the lock is gone the instant SIGTERM/SIGINT arrives regardless of whether the rest of shutdown gets time to finish. Complementary to the boot-time flush, not a replacement for it: a truekill -9delivers no signal at all, so that backstop still matters.#8999 — the freeze-deadlock
A lock-contended hold applied the sticky manual-review label, and the label then froze every later pass from running a fresh AI review at all (only replay) — meaning the one thing that could produce the real verdict needed to auto-clear the label (already implemented, #9009) could never run, because the freeze it depends on being lifted is gated by the very label it's trying to clear.
#9009 only handled the case where a fresh review DID run and showed contention had resolved; it could not reach that state on a repo requiring blocking AI review, because the freeze blocks the fresh attempt outright. Fixed by exempting the freeze specifically when the label's own provenance — per the same transient marker #9009 already reads/writes — is the lock-contention hold and nothing else.
This does not reopen the gaming surface the freeze exists to close (a contributor iterating pushes to buy a fresh verdict): a lock-contention hold is an infra artifact of two ORB passes racing, not a contributor action, and if some other reason also justifies the hold, that reason re-applies the label on this same pass's own disposition regardless of the exemption.
Notes for review
#9013 is not closed here. Its largest remaining piece — a single per-PR mutex spanning the public-surface publish AND the disposition plan/execute, currently two separately-claimed critical sections — needs a moderate refactor of both giant call sites (the sweep/CI-completion path and the webhook path), with real risk of subtly changing span/catch/decisionOutcome semantics the existing test suite exercises extensively. That deserves its own dedicated, carefully-tested pass rather than a rushed addition alongside three already-substantial fixes in this one.
Verification
Per instruction, no full local gate run this pass — targeted tests only:
surface-disposition-reconciler.test.ts,held-lock-registry.test.ts, and the extendedjob-dispatch.test.tsfan-out coverage — all pass.CI runs the full gate on push.
Closes #8997
Closes #8998
Closes #8999