Skip to content

fix(scheduler): reap subreaper children on an isolated loop on shutdown#15383

Open
rgrinberg wants to merge 1 commit into
mainfrom
fix-scheduler-subreaper-shutdown-isolate
Open

fix(scheduler): reap subreaper children on an isolated loop on shutdown#15383
rgrinberg wants to merge 1 commit into
mainfrom
fix-scheduler-subreaper-shutdown-isolate

Conversation

@rgrinberg

Copy link
Copy Markdown
Member

Summary

Fix internal errors during scheduler shutdown when the subreaper child-process
cleanup runs.

On shutdown, kill_and_wait_for_all_processes reaps any "escaped" subreaper
children in a second Fiber.run. By that point the main run has finished and its
current binding has been cleared, yet that second run reused the scheduler's
own event queue
. As a result it could:

  • resume fibers belonging to the finished run — a different fiber scheduler —
    giving advance: it's illegal to call advance with a fiber created in a different scheduler, and
  • run code (e.g. the build loop's end-of-build cleanup_subreaper_child_processes
    and the file-watcher's shutdown) with no current scheduler, giving
    Option.value_exn from t ().

This surfaced as a flaky failure in test/blackbox-tests/test-cases/watching/shutdown.t:
watch mode + an RPC server keeps fibers parked on Async_io timers/ivars, so on
shutdown there are stale fills and in-flight fibers for the cleanup run to trip
over (a plain build has none).

Fix

Run the subreaper cleanup on a fresh, isolated Event.Queue + Async_io
loop
instead of the scheduler's shared queue, so it only ever observes its own
async-io completions — never the finished run's leftover fills or fibers. The
reap logic is unchanged; it is just parameterized over the async-io loop it uses.

The public cleanup_subreaper_child_processes (used by the build loop within the
main run) keeps using the scheduler's own loop and is unaffected.

On shutdown, [kill_and_wait_for_all_processes] reaps escaped subreaper children in a second [Fiber.run]. By then the main run has finished and cleared [current], yet that run reused the scheduler's event queue, so it resumed the finished run's fibers (a different fiber scheduler -> "advance ... in a different scheduler") and ran code with no current scheduler ([Option.value_exn] in [t ()]). Give the cleanup its own [Event.Queue] + [Async_io] loop so it only ever observes its own async-io completions.

Signed-off-by: Rudi Grinberg <me@rgrinberg.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant