locks: boot flush, SIGTERM ordering, registry keying and Redis LRU each free a live holder's lock

[JSONbored]

Parent: #9466

Summary

Three separate code paths free a per-PR lock while the job holding it is still running: the boot-time orphan flush, the SIGTERM release ordering, and the held-lock registry's key-only identity. A fourth, Redis's allkeys-lru eviction policy, can do the same thing without any code path at all.

All four are latent on today's single-instance deployment and become live the moment there are two processes — which docker-compose.yml:14 ("multi-instance capable") and the pg queue backend both anticipate. The SIGTERM one is live today, during every redeploy.

Defect 1 — boot flush deletes every live lock in shared Redis

src/selfhost/redis-cache.ts:111-137:

export const ORPHANED_LOCK_KEY_PATTERNS: readonly string[] = [
  "pr-actuation-lock:*", "ai-review-lock:*", "contributor-cap-wake:*", "contributor-cap-lock:*",
];
export async function flushOrphanedLocksAtBoot(redis: Redis): Promise<number> { /* SCAN + DEL */ }

Called unconditionally at src/server.ts:743.

The justification (#9021) is that "on a SINGLE-INSTANCE deployment any lock present at boot is provably orphaned" — but nothing enforces single-instance. There is no flag and no backend check. The same file wires the token cache as explicitly "shared across replicas" (server.ts:750), and the pg queue backend's lease heartbeat (#9023) exists precisely because siblings share the table.

Scenario: replica A is mid-LLM holding ai-review-lock:K and pr-actuation-lock:PR#N; replica B restarts (OOM, crash-loop, scale-out, or a start-before-stop rolling deploy) and its boot flush DELs A's live keys. The next sweep or webhook pass claims them fresh and duplicates the review and the actuation — with no TTL expiry required. A crash-looping replica becomes a periodic fleet-wide lock wipe.

Defect 2 — SIGTERM releases locks before the drain completes

src/server.ts:1415-1432: releaseAllHeldLocksAtShutdown() runs first; backend.shutdown() (which waits for in-flight jobs to finish — pg-queue.ts:1789-1799) runs after.

So SIGTERM at t0 frees the AI-review and actuation locks of a review that then keeps executing for minutes under the drain — that is stop()'s entire contract (#9007). Any concurrent claimant (a sibling replica, or the new container in an overlapped deploy) acquires the freed lock at t0+ε and duplicates the work while the old process is still writing cache rows and publishing.

The #8998/#9174 tradeoff is deliberate and correct for the case where SIGKILL lands before the drain ends. But the case where the drain completes re-opens exactly the duplication the locks exist to prevent. The token-checked release protects the new claim from corruption; it does not prevent double-running.

Defect 3 — the held-lock registry is keyed by lock key alone

src/queue/held-lock-registry.ts:28-37 keys entries by lock key with no owner token, and releaseAiReviewLock calls unregisterHeldLock(key) unconditionally (src/queue/ai-review-orchestration.ts:151), even when its own token is stale or null.

Scenario: pass A holds ai-review-lock:K (registered). A maintainer's forced re-run steals K in the same process (#9008, processors.ts:11107) and registerHeldLock overwrites the entry with B's release. A then finishes: its releaseTransientLockIfOwner(A-token) correctly no-ops, but its unregisterHeldLock(K) deletes B's entry. A subsequent SIGTERM now skips K, so after a hard kill it strands for the full 1800 s TTL — precisely the #8998 incident shape the registry was built to prevent. The same shape occurs when a fail-open claim (null token from a Redis blip) later "releases".

Defect 4 — Redis evicts locks under memory pressure

Verified live on edge-nl-01: maxmemory 256MB, maxmemory-policy allkeys-lru, current usage 1.64 MB across 1116 keys.

Lock keys share an LRU-evictable keyspace with caches (ci-pending-first-seen:* alone accounts for ~1100 keys with TTLs up to ~5.7 days). Under pressure, Redis may evict a held lock. Exclusivity is then lost silently — no expiry event, no log, no metric. Low likelihood at 0.6% of the cap, but the failure is invisible when it happens.

Requirements

1. No code path may free a lock whose owner is still running.
2. Lock keys must not be evictable while held.
3. Any "this deployment is single-instance" assumption must be explicit and enforced, not implied by a comment.

Deliverables

• Boot flush: gate on an explicit single-instance signal (env flag, or skip when the pg backend is active since that is the multi-instance backend), or restrict deletion to keys whose stored value carries this instance's own id prefix. TTL + steal (orb(review): forceAiReview does not bypass the AI-review lock — the re-run button is silently inert behind an orphaned lock (root cause of #9000) #9008) already cover the genuine-orphan case, so the flush can safely become conservative.
• Shutdown ordering: release exclusivity locks after the drain wait, or per-job in each job's own finally so a lock is freed exactly when its owner stops. Keep the pre-SIGKILL bulk release as a last-resort path tied to the drain deadline (coordinate with the bounded-shutdown work tracked in the runtime epic).
• Registry identity: store ownerToken in the registry and make unregisterHeldLock(key, token) conditional, mirroring the store-side compare-and-delete.
• Redis eviction: move correctness keys out of the LRU keyspace — either a dedicated Redis logical DB / instance with noeviction, or a key-prefix policy documented and asserted at boot. This is a config change on edge-nl-01 plus a boot-time assertion in code.
• Document, in transient-locks.ts's header, the guarantee under each of: TTL expiry, hard kill, redeploy, eviction, multi-instance.

Tests (must fail against current main)

• Boot flush with a live lock held by another instance-id ⇒ key survives.
• SIGTERM with a job still draining ⇒ its lock remains held until that job settles; a competing claim is refused meanwhile.
• Steal-then-original-release ⇒ the stealer's registry entry survives and is released at shutdown.
• Fail-open claim (null token) release ⇒ does not unregister a different live holder.
• Boot assertion fires when the configured Redis eviction policy can evict lock keys.

Expected outcome

The only ways a lock is released are: its owner finished, its owner was explicitly stolen from, or its TTL genuinely elapsed with no live owner. Restarts, redeploys and memory pressure stop being sources of silent double actuation.

[linear-code]

JSO-6532