queue(locks): register the actuation and contributor-cap locks in the shutdown held-lock registry - #10067
Conversation
… shutdown held-lock registry Fixes JSONbored#10021
|
Tip ✅ LoopOver review result - approve/merge recommendedReview updated: 2026-07-31 06:19:25 UTC
Review summary Nits — 4 non-blocking
Decision drivers
Context & advisory signals — never blocks the verdict
Linked issue satisfactionAddressed Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #10067 +/- ##
==========================================
+ Coverage 79.57% 79.60% +0.02%
==========================================
Files 282 283 +1
Lines 58664 58740 +76
Branches 6842 6860 +18
==========================================
+ Hits 46682 46758 +76
Misses 11694 11694
Partials 288 288
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Summary
src/queue/held-lock-registry.tsexists so a shutdown signal can release every transient lock thisprocess holds instead of letting each ride out its TTL. Its module doc
(
src/queue/held-lock-registry.ts:14-19) states the goal:registerHeldLockhas exactly one production call site:claimAiReviewLock(
src/queue/ai-review-orchestration.ts:135-138). A repo-wide grep forregisterHeldLockoutsidesrc/queue/held-lock-registry.tsreturns only that line plustest/unit/held-lock-registry.test.ts.The two other locks built on the same primitive are never registered:
claimPrActuationLock(src/queue/transient-locks.ts:210-220), TTLPR_ACTUATION_LOCK_TTL_SECONDS = 600(
src/queue/transient-locks.ts:203). Claimed atsrc/queue/processors.ts:4420and, throughwithPrActuationLock, by all five close-enforcement guards (src/queue/review-evasion.ts:74-89).claimContributorCapLock(src/queue/transient-locks.ts:264-274), TTL 600 seconds(
src/queue/transient-locks.ts:260).So
releaseAllHeldLocksAtShutdown()atsrc/server.ts:1590— including theLOOPOVER_SHUTDOWN_LOCK_RELEASE_AFTER_MScut-short-drain path added by #9468 (
src/server.ts:1579-1587) — can only ever releaseai-review-lockkeys.A hard kill during a publish-and-maintain pass strands the PR's actuation lock for its full 600 seconds.
The boot-time flush is not a backstop for this.
src/server.ts:773-786runsflushOrphanedLocksAtBootonly whenisSingleInstanceDeployment(process.env)is true(
LOOPOVER_SINGLE_INSTANCE), because on a shared-Redis multi-replica deployment the flush would delete asibling's live locks — the explicit #9468 reasoning at
src/server.ts:765-771. On any deployment that doesnot set that var (the default), the shutdown registry is the only proactive release path, and it covers one
of the three lock namespaces.
The concrete cost is documented in this repo:
src/queue/retryable.ts:44-50records that "the only jobs inthe dead-letter queue over a 7-day window were three actuation-lock contentions, one of which was a
reopen-reclose-- a policy enforcement with a single webhook-gated trigger and no reconciler, so thatenforcement was lost outright." A waiter on an orphaned actuation lock is bounded by
ATTEMPT_FREE_RETRY_DEADLINE_MS = 15 minutes(src/queue/retryable.ts:66), so a 600-second orphaned lockconsumes two-thirds of that budget before the waiter can make any progress.
Deliverables
claimPrActuationLockregisters the held lock on a real acquire;releasePrActuationLockunregistersit token-scoped.
claimContributorCapLockregisters the held lock on a real acquire;releaseContributorCapLockunregisters it token-scoped.
test/unit/transient-locks.test.tsasserting that after a successfulclaimPrActuationLockagainst a cache adapter with a workingclaim/releaseIfValue,heldLockCountForTest()increases by exactly 1, and thatreleaseAllHeldLocksAtShutdown()then issuesa
releaseIfValuefor thepr-actuation-lock:key.claimContributorCapLock/ thecontributor-cap-lock:key intest/unit/transient-locks.test.ts.claimPrActuationLock(adapter with noclaimprimitive, soownerTokenisnull) leavesheldLockCountForTest()unchanged.test/unit/transient-locks.test.tsnamed for this bug asserting thatreleasePrActuationLockwith a DIFFERENTownerTokenthan the one registered does not remove theregistry entry (the locks: boot flush, SIGTERM ordering, registry keying and Redis LRU each free a live holder's lock #9468 steal invariant, now applied to this lock too).
All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example
registering the actuation lock but not unregistering it on release, so the registry grows unboundedly across a
long-lived process and shutdown issues deletes against keys already handed to other passes — does not resolve
this issue.
Test plan
This repo enforces 99%+ Codecov patch coverage, branch-counted.
vitest.config.ts'scoverage.includecovers
src/**/*.ts, sosrc/queue/transient-locks.tsis measured and gated. Each added guard is a branchwith two real arms that both need a test:
claim.acquired && claim.ownerToken !== null(register) vs thefail-open claim (do not register), and
ownerToken !== null(unregister) vsnull(do not). Four new brancharms across the two claim/release pairs, eight in total.
Fixes #10021