Skip to content

fix(selfhost): heal restart-orphaned active reviews at boot instead of 15-25 minutes later - #9802

Merged
JSONbored merged 1 commit into
mainfrom
fix/boot-orphan-reconciliation
Jul 29, 2026
Merged

fix(selfhost): heal restart-orphaned active reviews at boot instead of 15-25 minutes later#9802
JSONbored merged 1 commit into
mainfrom
fix/boot-orphan-reconciliation

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

The bug

A container recreation kills any in-flight review pass — the process, its provider subprocess, its locks — but the pass's active_review_tracking row stays active. While that orphan exists, every new pass for that head skips with "AI review already in progress" (aiReviewLockContendedResult) — including a maintainer's forced re-runs from the panel checkbox.

The existing self-heal (runActiveReviewReconciliation) only considers rows older than STALE_ACTIVE_REVIEW_MIN_AGE_MS (15 min) and runs on a 10-minute cron. So each deploy wedges every mid-review PR head for 15–25 minutes, and the redeploy-companion deploys automatically.

Observed live on the ORB (2026-07-29, five recreations in one day — two beta deploys, a secret rotation, a config change): one recreate landed 3 seconds after a review pass started for JSONbored/loopover#9793. The row sat active; the maintainer's re-ticks bounced off it; a background monitor caught it still wedged 10+ minutes later; it was terminalized by hand. Every "AI review already in progress" / "manual review recommended" flurry the maintainer saw today traces to this plus the deploys themselves.

The fix

Boot-time sweep: terminalize every active row with started_at before the process boot time, immediately after env construction and before the server listens.

At boot the 15-minute age heuristic is unnecessary — started_at < boot is exact, not a guess: no review survives a restart, and there are no live passes yet to protect. The cron sweep stays as the backstop for non-restart orphan shapes, unchanged. Fail-safe: a sweep failure logs active_review_boot_sweep_failed and boot proceeds.

Each healed row logs active_review_boot_orphan_terminalized at warn (same level discipline as the cron sweep's LOOPOVER-2K lesson: healing working is not an anomaly).

Tests

3 cases: the live regression (row written before boot → terminalized and reported), the safety invariant (a review started after boot is never touched), and idempotency (second sweep heals nothing, reports nothing). 37 pass in the file; tsc clean.

…5 minutes later

A container recreation kills any in-flight review pass -- the process, its
provider subprocess, its locks -- but the pass's active_review_tracking row stays
'active'. While it exists, every new pass for that head skips with "AI review
already in progress" (aiReviewLockContendedResult), INCLUDING a maintainer's
forced re-runs from the panel checkbox.

The existing reconciliation sweep only touches rows older than 15 minutes and
runs on a 10-minute cron, so each deploy wedged every mid-review PR head for
15-25 minutes. Observed live on the ORB (2026-07-29, five deploys in one day):
a recreate landed 3 seconds after a review pass started, and the maintainer's
re-ticks bounced off the orphan until it was terminalized by hand.

At boot the age heuristic is unnecessary: started_at < process boot time is
EXACT -- no review survives a restart -- so the sweep can heal immediately with
zero risk to a live pass. Runs right after env construction, before the server
listens; fail-safe (a failure logs and boot proceeds; the cron sweep remains the
backstop).
@loopover-orb

loopover-orb Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Important

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏳ LoopOver is waiting…

LoopOver has seen this pull request and is waiting on CI checks to finish before reviewing it. This comment will update once the review runs.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟨 Waiting

@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored JSONbored self-assigned this Jul 29, 2026
@JSONbored
JSONbored merged commit d80ec9f into main Jul 29, 2026
4 checks passed
@JSONbored
JSONbored deleted the fix/boot-orphan-reconciliation branch July 29, 2026 10:08
@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.52%. Comparing base (f5711a3) to head (70d21ab).
⚠️ Report is 6 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9802      +/-   ##
==========================================
- Coverage   90.33%   89.52%   -0.82%     
==========================================
  Files         918      918              
  Lines      113936   113937       +1     
  Branches    26975    26975              
==========================================
- Hits       102926   101997     -929     
- Misses       9681    10850    +1169     
+ Partials     1329     1090     -239     
Flag Coverage Δ
backend 94.09% <100.00%> (-1.48%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/db/repositories.ts 96.85% <100.00%> (+<0.01%) ⬆️

... and 3 files with indirect coverage changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant