This is the root cause behind #8997, #8998 and #8999
Every "ORB posted a verdict but didn't act" / "stuck on AI review already in progress" incident on 2026-07-26 traces back to one thing: a deploy cannot shut this process down cleanly, so every busy deploy ends in SIGKILL in the middle of a review pass. Three independent defects compose:
1. The drain loop keeps claiming NEW work. src/selfhost/pg-queue.ts — pump() is while (await processOne()) with no running check, so a pump already in flight when stop() sets running = false keeps claiming and starting fresh jobs until the due queue empties. Verified in source.
2. stop() has no deadline. async stop() { running = false; …; while (active > 0) await sleep(10); } — unbounded wait. Combined with (1), under load it may never return.
3. Docker gives it 10 seconds. /opt/loopover/docker-compose.yml has no stop_grace_period on any service (verified on edge-nl-01), so Docker's 10s default applies: SIGTERM → 10s → SIGKILL. A review pass runs minutes (LLM calls + GitHub round-trips), so the kill lands mid-pass essentially every time.
Observed: container replaced at 14:55:30Z; PR #8965's pass died ~6s after publishing its panel, stranding the close for 12 minutes; PR #8972's pass died mid-LLM-call, orphaning its AI-review lock for the full 30-minute TTL.
Fix
while (running && await processOne()) in pump() — stop claiming new work immediately on shutdown.- Bound
stop() with a deadline (~25s), and log any jobs still in flight when it expires.
- Add
stop_grace_period: 60s (or more) to the app service in docker-compose.yml — this is the one-line change with the largest immediate payoff and can ship independently.
- Flip readiness/health to failing before closing the HTTP server so in-flight webhook deliveries aren't lost at ingress (a delivery killed mid-request is never redelivered by GitHub).
The same shape exists in the sqlite queue twin — fix both.
Acceptance
docker compose restart under active load completes with zero jobs killed mid-execution (assert via a drain-complete log line).- No orphaned
ai-review-lock:* / pr-actuation-lock:* keys in Redis after a restart under load.
This is the root cause behind #8997, #8998 and #8999
Every "ORB posted a verdict but didn't act" / "stuck on AI review already in progress" incident on 2026-07-26 traces back to one thing: a deploy cannot shut this process down cleanly, so every busy deploy ends in SIGKILL in the middle of a review pass. Three independent defects compose:
1. The drain loop keeps claiming NEW work.
src/selfhost/pg-queue.ts—pump()iswhile (await processOne())with norunningcheck, so a pump already in flight whenstop()setsrunning = falsekeeps claiming and starting fresh jobs until the due queue empties. Verified in source.2.
stop()has no deadline.async stop() { running = false; …; while (active > 0) await sleep(10); }— unbounded wait. Combined with (1), under load it may never return.3. Docker gives it 10 seconds.
/opt/loopover/docker-compose.ymlhas nostop_grace_periodon any service (verified on edge-nl-01), so Docker's 10s default applies: SIGTERM → 10s → SIGKILL. A review pass runs minutes (LLM calls + GitHub round-trips), so the kill lands mid-pass essentially every time.Observed: container replaced at 14:55:30Z; PR #8965's pass died ~6s after publishing its panel, stranding the close for 12 minutes; PR #8972's pass died mid-LLM-call, orphaning its AI-review lock for the full 30-minute TTL.
Fix
while (running && await processOne())inpump()— stop claiming new work immediately on shutdown.stop()with a deadline (~25s), and log any jobs still in flight when it expires.stop_grace_period: 60s(or more) to the app service indocker-compose.yml— this is the one-line change with the largest immediate payoff and can ship independently.The same shape exists in the sqlite queue twin — fix both.
Acceptance
docker compose restartunder active load completes with zero jobs killed mid-execution (assert via a drain-complete log line).ai-review-lock:*/pr-actuation-lock:*keys in Redis after a restart under load.