Skip to content

fix(server): hold per-project wave lock in worker thread, not async request#25

Open
apollion69 wants to merge 1 commit into
Intelligent-Internet:mainfrom
apollion69:fix/advance-project-wave-flock
Open

fix(server): hold per-project wave lock in worker thread, not async request#25
apollion69 wants to merge 1 commit into
Intelligent-Internet:mainfrom
apollion69:fix/advance-project-wave-flock

Conversation

@apollion69

Copy link
Copy Markdown

fix(server): hold the per-project wave lock in the worker thread, not the async request

Problem

advance_project (and the other mutating tools) run the controller call synchronously via
asyncio.to_thread, guarded by an in-process asyncio.Lock (server.py:103-119). A running
thread's future cannot be cancelled. When the MCP request is aborted — a user interrupt or the
client's idle-abort — the awaiting coroutine is cancelled and the asyncio.Lock releases, but
the wave thread keeps running controller.advance_project's while True: coordinator.step()
loop to completion, now mutating on-disk state (task-state.json, attempts/*.json) without
holding any lock
.

If the caller then retries advance_project, two waves run concurrently on the same project.
The new wave's _reconcile_pending_attempts finds the validator still running with no attempt
file yet and writes an empty stub (cleared, passed=false, items=[]); the gate evaluates
that stub, fails every target as missing, and goes terminal-failed. The original wave then
finishes and silently overwrites the attempt file with the real (passing) verdicts — too late,
the gate is already failed and needs a manual patch to recover.

The existing comment at server.py:103-108 already flags concurrent same-project operations as
undefined behavior; this is that race, reachable through ordinary request cancellation.

Fix

Add _run_project_locked, wrapping every mutating controller call with an advisory flock
acquired inside the to_thread target and released only when the call returns. Because the
lock lives on the worker thread, async cancellation of the request can no longer free it while
the wave is still running. A concurrent same-project mutating call gets a typed
wave_in_progress ToolError instead of racing on disk. The in-process asyncio.Lock is kept as
first-line serialization.

Scope: submit_plan, advance_project, end_mission, decide_attention, abort_project.
start_project (no project id yet) and inspect_project (read-only) are unchanged.

Tests

tests/test_server_wave_lock.py:

  • runs the wrapped fn and returns its result;
  • a second same-project call is rejected with wave_in_progress while the first holds the lock;
  • test_lock_survives_async_cancellation — cancels the awaiting task and asserts the
    orphaned thread still holds the lock (concurrent call → wave_in_progress) until the wave
    actually completes, then the lock frees. This is the regression for the reported race.

Full suite: 205 passed, 7 skipped.

Notes

flock is POSIX advisory locking (Linux/macOS). If Windows support is required, this can fall
back to a lockfile-with-pid or msvcrt.locking; happy to adjust to the project's platform
policy.

…equest

advance_project runs the mission wave synchronously via asyncio.to_thread. A
running thread's future cannot be cancelled, so when the MCP request is aborted
(user interrupt or client idle-abort) the wave keeps running and mutating disk
state while the in-process asyncio.Lock releases on cancellation. A retried
call then executes a second wave concurrently, and _reconcile_pending_attempts
stubs the still-in-flight validator with an empty verdict, failing its gate on
missing items (the maintainers' own note at server.py:103-108 flags this exact
race as undefined behavior).

Wrap every mutating controller call with an advisory flock whose lifetime is
tied to the worker thread: acquired inside the to_thread target, released only
when the wave truly returns. A concurrent same-project call now gets a typed
wave_in_progress error instead of racing. Adds test_server_wave_lock.py incl. a
cancellation-survival regression.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant