Skip to content

queue(locks): register the actuation and contributor-cap locks in the shutdown held-lock registry - #10067

Merged
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
phamngocquy:miner/issue-10021
Jul 31, 2026
Merged

queue(locks): register the actuation and contributor-cap locks in the shutdown held-lock registry#10067
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
phamngocquy:miner/issue-10021

Conversation

@phamngocquy

Copy link
Copy Markdown
Contributor

Summary

src/queue/held-lock-registry.ts exists so a shutdown signal can release every transient lock this
process holds instead of letting each ride out its TTL. Its module doc
(src/queue/held-lock-registry.ts:14-19) states the goal:

 * This is the proactive half: a tiny process-local registry of "locks I currently hold and how to release
 * them", so the shutdown handler can best-effort release every one of them immediately on SIGTERM/SIGINT --
 * before, and independent of, whether the graceful drain itself has time to finish.

registerHeldLock has exactly one production call site: claimAiReviewLock
(src/queue/ai-review-orchestration.ts:135-138). A repo-wide grep for registerHeldLock outside
src/queue/held-lock-registry.ts returns only that line plus test/unit/held-lock-registry.test.ts.

The two other locks built on the same primitive are never registered:

  • claimPrActuationLock (src/queue/transient-locks.ts:210-220), TTL PR_ACTUATION_LOCK_TTL_SECONDS = 600
    (src/queue/transient-locks.ts:203). Claimed at src/queue/processors.ts:4420 and, through
    withPrActuationLock, by all five close-enforcement guards (src/queue/review-evasion.ts:74-89).
  • claimContributorCapLock (src/queue/transient-locks.ts:264-274), TTL 600 seconds
    (src/queue/transient-locks.ts:260).

So releaseAllHeldLocksAtShutdown() at src/server.ts:1590 — including the LOOPOVER_SHUTDOWN_LOCK_RELEASE_AFTER_MS
cut-short-drain path added by #9468 (src/server.ts:1579-1587) — can only ever release ai-review-lock keys.
A hard kill during a publish-and-maintain pass strands the PR's actuation lock for its full 600 seconds.

The boot-time flush is not a backstop for this. src/server.ts:773-786 runs
flushOrphanedLocksAtBoot only when isSingleInstanceDeployment(process.env) is true
(LOOPOVER_SINGLE_INSTANCE), because on a shared-Redis multi-replica deployment the flush would delete a
sibling's live locks — the explicit #9468 reasoning at src/server.ts:765-771. On any deployment that does
not set that var (the default), the shutdown registry is the only proactive release path, and it covers one
of the three lock namespaces.

The concrete cost is documented in this repo: src/queue/retryable.ts:44-50 records that "the only jobs in
the dead-letter queue over a 7-day window were three actuation-lock contentions, one of which was a
reopen-reclose -- a policy enforcement with a single webhook-gated trigger and no reconciler, so that
enforcement was lost outright." A waiter on an orphaned actuation lock is bounded by
ATTEMPT_FREE_RETRY_DEADLINE_MS = 15 minutes (src/queue/retryable.ts:66), so a 600-second orphaned lock
consumes two-thirds of that budget before the waiter can make any progress.

Deliverables

  • claimPrActuationLock registers the held lock on a real acquire; releasePrActuationLock unregisters
    it token-scoped.
  • claimContributorCapLock registers the held lock on a real acquire; releaseContributorCapLock
    unregisters it token-scoped.
  • A test in test/unit/transient-locks.test.ts asserting that after a successful
    claimPrActuationLock against a cache adapter with a working claim/releaseIfValue,
    heldLockCountForTest() increases by exactly 1, and that releaseAllHeldLocksAtShutdown() then issues
    a releaseIfValue for the pr-actuation-lock: key.
  • The same assertion for claimContributorCapLock / the contributor-cap-lock: key in
    test/unit/transient-locks.test.ts.
  • A test asserting that a FAIL-OPEN claimPrActuationLock (adapter with no claim primitive, so
    ownerToken is null) leaves heldLockCountForTest() unchanged.
  • A regression test at test/unit/transient-locks.test.ts named for this bug asserting that
    releasePrActuationLock with a DIFFERENT ownerToken than the one registered does not remove the
    registry entry (the locks: boot flush, SIGTERM ordering, registry keying and Redis LRU each free a live holder's lock #9468 steal invariant, now applied to this lock too).

All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example
registering the actuation lock but not unregistering it on release, so the registry grows unboundedly across a
long-lived process and shutdown issues deletes against keys already handed to other passes — does not resolve
this issue.

Test plan

This repo enforces 99%+ Codecov patch coverage, branch-counted. vitest.config.ts's coverage.include
covers src/**/*.ts, so src/queue/transient-locks.ts is measured and gated. Each added guard is a branch
with two real arms that both need a test: claim.acquired && claim.ownerToken !== null (register) vs the
fail-open claim (do not register), and ownerToken !== null (unregister) vs null (do not). Four new branch
arms across the two claim/release pairs, eight in total.

Fixes #10021

@phamngocquy
phamngocquy requested a review from JSONbored as a code owner July 31, 2026 06:09
@loopover-orb

loopover-orb Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Tip

✅ LoopOver review result - approve/merge recommended

Review updated: 2026-07-31 06:19:25 UTC

2 files · 1 AI reviewer · no blockers · readiness 100/100 · CI green · clean

✅ Suggested Action - Approve/Merge

  • safe to merge

Review summary
This PR closes the gap where claimPrActuationLock and claimContributorCapLock never registered with held-lock-registry, meaning releaseAllHeldLocksAtShutdown() could only ever release ai-review-lock keys, exactly as the PR description traces. The wiring mirrors the existing claimAiReviewLock shape (register on real acquisition, unregister on token-matched release), null-token fail-open claims are correctly excluded, and the test suite exercises registration, normal release, fail-open non-registration, null-token no-op, and mismatched-token no-op for both locks. The closure captures claim.ownerToken by reference inside registerHeldLock's release callback rather than binding it to a local const, which is safe here since ownerToken is never reassigned but is a subtle pattern worth a second look.

Nits — 4 non-blocking
  • src/queue/transient-locks.ts: the registerHeldLock callback closes over `claim.ownerToken` directly rather than destructuring/aliasing to a local const first — functionally fine since `claim` is never mutated, but matching the more defensive local-binding style used elsewhere would remove any doubt for a future reader.
  • The PR only wires claimPrActuationLock/claimContributorCapLock; if any other transient lock is added later on this same primitive, it would be easy to forget registration again — a lint rule or a single shared wrapper (e.g. `claimAndRegister`) could prevent recurrence, though that's arguably out of scope here.
  • Consider extracting a small `claimAndRegisterTransientLock(env, key, ttl)` helper in transient-locks.ts that both claimPrActuationLock and claimContributorCapLock could call, since the register/unregister pattern is now duplicated three times (ai-review, actuation, contributor-cap) across the codebase.
  • test/unit/transient-locks.test.ts: the two `cacheWithReleaseTracking` helper functions are identical between the two describe blocks — could be hoisted to a shared module-level helper to avoid duplication.

Decision drivers

  • ✅ Code review — No blockers (1 reviewer)
  • ✅ Gate result — Passing (No configured blocker found.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ✅ Linked #10021
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 68 registered-repo PR(s), 17 merged, 0 issue(s).
Contributor context ✅ Confirmed Gittensor contributor phamngocquy; Gittensor profile; 68 PR(s), 0 issue(s).
Improvement ✅ Minor risk: clean · value: minor · LLM: moderate
Linked issue satisfaction

Addressed
The diff adds registerHeldLock/unregisterHeldLock calls to claimPrActuationLock/releasePrActuationLock and claimContributorCapLock/releaseContributorCapLock guarded exactly by acquired+non-null ownerToken (register) and non-null ownerToken (unregister), matching the required claimAiReviewLock/releaseAiReviewLock shape without touching the registry primitives or ai-review lock functions.

Review context
  • Author: phamngocquy
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: not available
  • Official Gittensor activity: 68 PR(s), 0 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Keep the PR focused and include validation evidence before maintainer review.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 31, 2026
@codecov

codecov Bot commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.60%. Comparing base (a7673e2) to head (57840c6).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #10067      +/-   ##
==========================================
+ Coverage   79.57%   79.60%   +0.02%     
==========================================
  Files         282      283       +1     
  Lines       58664    58740      +76     
  Branches     6842     6860      +18     
==========================================
+ Hits        46682    46758      +76     
  Misses      11694    11694              
  Partials      288      288              
Flag Coverage Δ
backend 100.00% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/queue/transient-locks.ts 100.00% <100.00%> (ø)

@loopover-orb loopover-orb Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LoopOver approves — the gate is satisfied and CI is green.

@loopover-orb
loopover-orb Bot merged commit eca3b61 into JSONbored:main Jul 31, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

queue(locks): register the actuation and contributor-cap locks in the shutdown held-lock registry

1 participant