fix(server): hold per-project wave lock in worker thread, not async request#25
Open
apollion69 wants to merge 1 commit into
Open
Conversation
…equest advance_project runs the mission wave synchronously via asyncio.to_thread. A running thread's future cannot be cancelled, so when the MCP request is aborted (user interrupt or client idle-abort) the wave keeps running and mutating disk state while the in-process asyncio.Lock releases on cancellation. A retried call then executes a second wave concurrently, and _reconcile_pending_attempts stubs the still-in-flight validator with an empty verdict, failing its gate on missing items (the maintainers' own note at server.py:103-108 flags this exact race as undefined behavior). Wrap every mutating controller call with an advisory flock whose lifetime is tied to the worker thread: acquired inside the to_thread target, released only when the wave truly returns. A concurrent same-project call now gets a typed wave_in_progress error instead of racing. Adds test_server_wave_lock.py incl. a cancellation-survival regression.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(server): hold the per-project wave lock in the worker thread, not the async request
Problem
advance_project(and the other mutating tools) run the controller call synchronously viaasyncio.to_thread, guarded by an in-processasyncio.Lock(server.py:103-119). A runningthread's future cannot be cancelled. When the MCP request is aborted — a user interrupt or the
client's idle-abort — the awaiting coroutine is cancelled and the
asyncio.Lockreleases, butthe wave thread keeps running
controller.advance_project'swhile True: coordinator.step()loop to completion, now mutating on-disk state (
task-state.json,attempts/*.json) withoutholding any lock.
If the caller then retries
advance_project, two waves run concurrently on the same project.The new wave's
_reconcile_pending_attemptsfinds the validator stillrunningwith no attemptfile yet and writes an empty stub (
cleared,passed=false,items=[]); the gate evaluatesthat stub, fails every target as
missing, and goes terminal-failed. The original wave thenfinishes and silently overwrites the attempt file with the real (passing) verdicts — too late,
the gate is already failed and needs a manual patch to recover.
The existing comment at
server.py:103-108already flags concurrent same-project operations asundefined behavior; this is that race, reachable through ordinary request cancellation.
Fix
Add
_run_project_locked, wrapping every mutating controller call with an advisoryflockacquired inside the
to_threadtarget and released only when the call returns. Because thelock lives on the worker thread, async cancellation of the request can no longer free it while
the wave is still running. A concurrent same-project mutating call gets a typed
wave_in_progressToolError instead of racing on disk. The in-processasyncio.Lockis kept asfirst-line serialization.
Scope:
submit_plan,advance_project,end_mission,decide_attention,abort_project.start_project(no project id yet) andinspect_project(read-only) are unchanged.Tests
tests/test_server_wave_lock.py:wave_in_progresswhile the first holds the lock;test_lock_survives_async_cancellation— cancels the awaiting task and asserts theorphaned thread still holds the lock (concurrent call →
wave_in_progress) until the waveactually completes, then the lock frees. This is the regression for the reported race.
Full suite: 205 passed, 7 skipped.
Notes
flockis POSIX advisory locking (Linux/macOS). If Windows support is required, this can fallback to a lockfile-with-pid or
msvcrt.locking; happy to adjust to the project's platformpolicy.