Skip to content

fix(selfhost): rescue orphaned dispositions, break the freeze-deadlock, and release AI-review locks before a hard kill can - #9174

Merged
JSONbored merged 1 commit into
mainfrom
fix/8997-8998-8999-9013-restart
Jul 27, 2026
Merged

fix(selfhost): rescue orphaned dispositions, break the freeze-deadlock, and release AI-review locks before a hard kill can#9174
JSONbored merged 1 commit into
mainfrom
fix/8997-8998-8999-9013-restart

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

#8997 — a deploy restart can leave a PR wearing a decisive panel with no matching disposition

SIGTERM kills whatever pass is in flight at an arbitrary point, and the worst possible cut is between the public-surface publish (panel, gate check-run, CI aggregate — all committed) and maybeRunAgentMaintenance ever claiming its per-PR actuation lock for that head. Confirmed live on #8965: the panel published at 14:55:24Z from the dying container, and the matching close only landed at 15:07:53Z — ~12 minutes later, purely because an unrelated later sweep tick happened to re-run the whole pass from scratch. The existing outage-repair check (lastPublishedSurfaceSha !== headSha) cannot see this: the surface is current, so it already looks healthy.

maybeRunAgentMaintenance now records a marker (agent.maintenance.disposition_considered) the moment it claims the actuation lock — the moment a real disposition attempt begins, independent of what it decides. A new standalone scan, reconcileSurfaceWithoutDisposition, rides the same sweep tick as the existing pr-outcome/pending-closure repair scans (#9026/#9031) and re-enqueues a regate for any open PR whose current-head surface carries no such marker.

Deliberately not folded into the existing surfaceRepairPriorityPullNumbers: that function's return value drives the regular sweep's staleness-ordered fan-out and is asserted against by a large, already-green test surface (dispatch shape, ordering, backlog-restriction semantics) — an early attempt at injecting this check there flagged every "surface current, nothing else recorded" fixture in that file, which is a real, deliberate policy widening this fix makes, but one that belongs in its own independently-testable scan rather than retrofitted onto a heavily-relied-upon function blind.

#8998 — an orphaned ai-review-lock starves every subsequent re-review

A 30-min TTL lock left behind by a dead process starves every subsequent pass — including an explicit maintainer re-run — with the "already in progress" placeholder for real time nobody is reviewing anything. Two of the three layers this needed were already shipped earlier this session:

What was still missing: queue.stop() (#9007) lets an in-flight job finish naturally, releasing its own lock via its own finally block — but only if the orchestrator's SIGKILL grace period is long enough for that drain to complete, and a common 10-30s grace period is shorter than an AI-review LLM call legitimately runs. A new held-lock-registry tracks every real ai-review-lock claim this process currently holds; the shutdown handler now releases all of them first, before anything else in the shutdown sequence, so the lock is gone the instant SIGTERM/SIGINT arrives regardless of whether the rest of shutdown gets time to finish. Complementary to the boot-time flush, not a replacement for it: a true kill -9 delivers no signal at all, so that backstop still matters.

#8999 — the freeze-deadlock

A lock-contended hold applied the sticky manual-review label, and the label then froze every later pass from running a fresh AI review at all (only replay) — meaning the one thing that could produce the real verdict needed to auto-clear the label (already implemented, #9009) could never run, because the freeze it depends on being lifted is gated by the very label it's trying to clear.

#9009 only handled the case where a fresh review DID run and showed contention had resolved; it could not reach that state on a repo requiring blocking AI review, because the freeze blocks the fresh attempt outright. Fixed by exempting the freeze specifically when the label's own provenance — per the same transient marker #9009 already reads/writes — is the lock-contention hold and nothing else.

This does not reopen the gaming surface the freeze exists to close (a contributor iterating pushes to buy a fresh verdict): a lock-contention hold is an infra artifact of two ORB passes racing, not a contributor action, and if some other reason also justifies the hold, that reason re-applies the label on this same pass's own disposition regardless of the exemption.

Notes for review

#9013 is not closed here. Its largest remaining piece — a single per-PR mutex spanning the public-surface publish AND the disposition plan/execute, currently two separately-claimed critical sections — needs a moderate refactor of both giant call sites (the sweep/CI-completion path and the webhook path), with real risk of subtly changing span/catch/decisionOutcome semantics the existing test suite exercises extensively. That deserves its own dedicated, carefully-tested pass rather than a rushed addition alongside three already-substantial fixes in this one.

Verification

Per instruction, no full local gate run this pass — targeted tests only:

  • Typecheck clean.
  • 39 new tests across surface-disposition-reconciler.test.ts, held-lock-registry.test.ts, and the extended job-dispatch.test.ts fan-out coverage — all pass.
  • A broader targeted sweep of 967 tests across the queue/lifecycle/transient-lock/precision-breaker suites most likely to interact with the actuation-lock, freeze, and sweep-priority code paths this change touches — all pass.

CI runs the full gate on push.

Closes #8997
Closes #8998
Closes #8999

…k, and release AI-review locks before a hard kill can

#8997 — a deploy restart can leave a PR wearing a decisive panel with no
matching disposition. SIGTERM kills whatever pass is in flight at an arbitrary
point, and the worst possible cut is between the public-surface publish (panel,
gate check-run, CI aggregate — all committed) and maybeRunAgentMaintenance ever
claiming its per-PR actuation lock for that head. Confirmed live on #8965: the
panel published at 14:55:24Z from the dying container, and the matching close
only landed at 15:07:53Z — ~12 minutes later, purely because an unrelated later
sweep tick happened to re-run the whole pass from scratch. The existing
outage-repair check (lastPublishedSurfaceSha !== headSha) cannot see this: the
surface IS current, so it already looks healthy.

maybeRunAgentMaintenance now records a marker (agent.maintenance.disposition_
considered) the moment it claims the actuation lock — the moment a real
disposition attempt begins, independent of what it decides. A new standalone
scan, reconcileSurfaceWithoutDisposition, rides the same sweep tick as the
existing pr-outcome/pending-closure repair scans (#9026/#9031) and re-enqueues a
regate for any open PR whose current-head surface carries no such marker.
Deliberately NOT folded into the existing surfaceRepairPriorityPullNumbers: that
function's return value drives the regular sweep's staleness-ordered fan-out and
is asserted against by a large, already-green test surface (dispatch shape,
ordering, backlog-restriction semantics) — an early attempt at injecting this
check there flagged every "surface current, nothing else recorded" fixture in
that file, which is a real, deliberate policy widening this fix makes, but one
that belongs in its own independently-testable scan rather than retrofitted onto
a heavily-relied-upon function blind.

#8998 — an orphaned ai-review-lock (30-min TTL) starves every subsequent
re-review, including an explicit maintainer re-run, with the "already in
progress" placeholder for real time nobody is reviewing anything. Two of the
three layers this needed were already shipped earlier this session: the
boot-time flush (#9021, ORPHANED_LOCK_KEY_PATTERNS already includes
ai-review-lock:*) and the explicit force-lock-steal for a maintainer's forced
re-run (#9008, `steal: webhook.forceAiReview === true` already threaded into
claimAiReviewLock). What was still missing: queue.stop() (#9007) lets an
in-flight job finish naturally, releasing its own lock via its own finally
block — but only if the orchestrator's SIGKILL grace period is long enough for
that drain to complete, and a common 10-30s grace period is shorter than an
AI-review LLM call legitimately runs. A new held-lock-registry tracks every real
ai-review-lock claim this process currently holds; the shutdown handler now
releases all of them FIRST, before anything else in the shutdown sequence, so
the lock is gone the instant SIGTERM/SIGINT arrives regardless of whether the
rest of shutdown gets time to finish. Complementary to the boot-time flush, not
a replacement for it: a true `kill -9` delivers no signal at all, so that
backstop still matters.

#8999 — a lock-contended hold applied the sticky manual-review label, and the
label then froze every LATER pass from running a FRESH AI review at all (only
replay) — meaning the one thing that could produce the real verdict needed to
auto-clear the label (already implemented, #9009) could never run, because the
freeze it depends on being lifted is gated by the very label it's trying to
clear. #9009 only handled the case where a fresh review DID run and showed
contention had resolved; it could not reach that state on a repo requiring
blocking AI review, because the freeze blocks the fresh attempt outright. Fixed
by exempting the freeze specifically when the label's own provenance — per the
same transient marker #9009 already reads/writes — is the lock-contention hold
and nothing else. This does not reopen the gaming surface the freeze exists to
close (a contributor iterating pushes to buy a fresh verdict): a lock-contention
hold is an infra artifact of two ORB passes racing, not a contributor action, and
if some OTHER reason also justifies the hold, that reason re-applies the label on
this same pass's own disposition regardless of the exemption.

#9013's largest remaining piece (a single per-PR mutex spanning the
public-surface publish AND the disposition plan/execute, currently two
separately-claimed critical sections) is NOT included here — an initial attempt
showed it needs a moderate refactor of both giant call sites (the sweep/CI-
completion path and the webhook path) with real risk of subtly changing
span/catch/decisionOutcome semantics the existing test suite exercises
extensively, and that deserves its own dedicated, carefully-tested pass rather
than a rushed addition alongside three already-substantial fixes.

Targeted tests only (no full local gate run this pass, per instruction):
typecheck clean, and every touched/new test file passes locally — 39 new tests
across surface-disposition-reconciler.test.ts, held-lock-registry.test.ts, and
the extended job-dispatch.test.ts fan-out coverage — plus a broader targeted
sweep of 967 tests across the queue/lifecycle/transient-lock/precision-breaker
suites most likely to interact with the actuation-lock, freeze, and sweep-
priority code paths this change touches. CI runs the full gate on push.

Closes #8997
Closes #8998
Closes #8999
@JSONbored JSONbored self-assigned this Jul 27, 2026
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored
JSONbored merged commit 70b60e7 into main Jul 27, 2026
4 checks passed
@JSONbored
JSONbored deleted the fix/8997-8998-8999-9013-restart branch July 27, 2026 05:35
@codecov

codecov Bot commented Jul 27, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.75000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.68%. Comparing base (fbd1ed1) to head (c3a9ee5).
⚠️ Report is 6 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/review/surface-disposition-reconciler.ts 90.47% 1 Missing and 1 partial ⚠️
src/queue/processors.ts 80.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9174      +/-   ##
==========================================
- Coverage   93.90%   92.68%   -1.22%     
==========================================
  Files         817      819       +2     
  Lines       81039    81085      +46     
  Branches    24611    24617       +6     
==========================================
- Hits        76096    75150     -946     
- Misses       3568     4850    +1282     
+ Partials     1375     1085     -290     
Flag Coverage Δ
backend 93.49% <93.75%> (-1.69%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/queue/ai-review-orchestration.ts 100.00% <100.00%> (ø)
src/queue/held-lock-registry.ts 100.00% <100.00%> (ø)
src/queue/job-dispatch.ts 99.44% <100.00%> (+<0.01%) ⬆️
src/queue/processors.ts 95.61% <80.00%> (-0.03%) ⬇️
src/review/surface-disposition-reconciler.ts 90.47% <90.47%> (ø)

... and 3 files with indirect coverage changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment