adriencaccia submission#43
Open
adriencaccia wants to merge 1 commit into
Open
Conversation
Parallel build scheduler using a persistent module-level worker pool plus a chain fast-path for graphs with no fan-out. Highlights: * NUM_WORKERS-1 worker threads are spawned at module import and parked on a condition variable; the main thread joins as the Nth worker per build_all call. Thread startup cost is paid outside the timed window. * Chain fast-path: when max_fan_out <= 1 and there is a single source, the graph is walked single-threaded with no locks or queues. * Inline-chain execution: after a target finishes, the worker keeps the heaviest newly-ready dependent for itself and pushes the rest to the shared queue. Dependents are pre-sorted by 'work' descending so the inlined target is on the critical path. * Integer indices throughout: results / pending_deps / dependents are plain lists indexed by the target's position, avoiding dict hashing in the hot loop. * Lazy results dict: the harness only consults Target._result for correctness, so the returned _LazyResults subclass defers materialising name -> result until first access (after the timer has stopped).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Submission Checklist
submissions/<github_username>.py.Approach Description
Parallel topological scheduler with a persistent module-level worker pool, designed for Python 3.14t free-threaded execution.
Architecture:
NUM_WORKERS - 1worker threads are spawned at module import and parked on athreading.Conditionuntil a build is handed to them via a fresh_RunState. The main thread joins as the Nth worker perbuild_all()call. This pays the thread-startup cost outside the timed window and is reused across every harness invocation in the same process.targetsis a tuple;results,pending_deps, anddependentsare plain lists indexed by the target's position. No dict hashing in the hot loop.threading.Lockguardspending_depsand theremainingcounter. The critical section is just a handful of integer decrements and one list write — much smaller than the build work itself.workdescending), keeps the heaviest newly-ready dependent for itself to execute inline, and pushes the rest onto the shared queue. On a deep chain this collapses the whole walk into a single worker with zero queue round-trips; on graphs with real parallelism the "push the rest" path keeps every worker fed._LazyResultsdict subclass defers materialisingname -> resultuntil the first access — which only happens after the harness timer has stopped, since validation readsTarget._resultdirectly.Chain fast-path: when
max_fan_out <= 1and there is exactly one source, the graph is walked single-threaded with no locks, queues, or threads.Tradeoffs: