perf(orchestrator): parallelize the per-language resolve phase (~50s wall) - #478
Draft
Disentinel wants to merge 1 commit into
Draft
perf(orchestrator): parallelize the per-language resolve phase (~50s wall)#478Disentinel wants to merge 1 commit into
Disentinel wants to merge 1 commit into
Conversation
The full-analyze path ran the per-language resolvers strictly sequentially (js -> haskell -> rust -> java/kotlin/python/go/swift/apple-cross/jvm-cross/ cpp -> beam), each building its own ProcessPool, resolving, committing, and shutting the pool down before the next language started. Profiled on the grafema monorepo: js 10.8s + haskell 24.4s + rust 53.6s + beam 12.9s = ~102s sequential, while the single longest resolver is ~54s. The language resolvers are independent: each filters to its own language's files and writes a disjoint output slice. Their compute lives in independent daemon SUBPROCESSES; the orchestrator side is async pipe + socket I/O. So we now run all twelve as concurrent futures joined with tokio::join!. RFDB-connection constraint: the orchestrator holds ONE RfdbClient (single UnixStream, &mut self) and the resolvers query rfdb mid-compute (collect their node set up front, commit incrementally), so the shared read inputs cannot be snapshotted once. Instead each concurrent resolver opens its OWN RfdbClient connection to the same socket. The rfdb-server spawns a dedicated thread per client and serializes all writes behind the engine RwLock, so concurrent connections are safe and disjoint-slice commits do not race. The orchestrator's main rfdb handle is not touched inside the parallel section. Each future returns its IMPORTS_FROM edges; they are merged into all_imports_from_edges in deterministic language order after the join, and the first resolver error is propagated (preserving the prior sequential ? abort). Per-language profile! events are unchanged (concurrent durations may overlap); a new parallel_resolve_complete event records the wall time of the section. Measured on a full self-analyze of the grafema monorepo: resolve_ms 102024 -> 56611 (-45.4s, -44.5%) total_ms 414788 -> 365690 (-49.1s, -11.8%) Graph equivalent: nodes ~503.6k, edges ~937k, errors 25 (within run-to-run resolver nondeterminism; js/haskell-import/runtime-globals slices bit-identical). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Disentinel
marked this pull request as draft
June 19, 2026 19:23
Owner
Author
|
⏸️ HOLD — do not merge to main. Per the engine zone canon (resolve / rfdb* / datalog-resolution-logic = do-not-touch, tied to the active |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Parallelize the resolve phase of the full-analyze path in
grafema-orchestrator/src/main.rs. Previously the twelve per-languageresolvers (js → haskell → rust → java → kotlin → python → go → swift →
apple-cross → jvm-cross → cpp → beam) ran strictly sequentially: each built
its own
ProcessPool, resolved, committed, and shut the pool down before thenext language started.
The language resolvers are independent — each filters to its own language's
files and writes a disjoint output slice — and the heavy compute lives in
independent daemon subprocesses. The orchestrator side is async pipe +
socket I/O, so all twelve now run as concurrent futures joined with
tokio::join!.The rfdb-connection constraint
The orchestrator holds one
RfdbClient(singleUnixStream,&mut self),and the resolvers query rfdb mid-compute (
resolve_per_file/stream_and_resolve_single_workercollect their node set up front and commitincrementally), so the shared read inputs can't simply be snapshotted once.
Design chosen: each concurrent resolver opens its own
RfdbClientconnection to the same socket.
rfdb-serverspawns a dedicated thread perclient and serializes all writes behind the engine
RwLock, so concurrentconnections are safe and disjoint-slice commits don't race. The orchestrator's
main
rfdbhandle is not touched inside the parallel section. Each futurereturns its
IMPORTS_FROMedges, merged intoall_imports_from_edgesindeterministic language order after the join; the first resolver error is
propagated (preserving the prior sequential
?abort).Per-language
profile!events are unchanged (concurrent durations may overlap);a new
parallel_resolve_completeevent records the section's wall time.Measured (full self-analyze of the grafema monorepo, fresh runs)
resolve_mstotal_msresolve_mscollapses to ≈max(single resolver)= rust (~60s under load),exactly as predicted by the profile (sequential ≈ Σ = js+haskell+rust+beam).
Correctness — graph equivalence
Node/edge totals across four full runs on identical source + resolvers (only
orchestrator scheduling differs):
The two sequential runs differ by 622 nodes / 1090 edges; the parallel runs
land squarely inside that band. The variance is pre-existing run-to-run
resolver/derive nondeterminism, not introduced here.
errors = 25is constantacross all runs. Per-resolver output is stable: js, rust-cross-method-calls
(4926), rust-runtime-globals (42365), haskell-import (772), haskell-runtime-globals
(5288) are bit-identical sequential vs parallel; only
haskell-cross-module-callsfluctuates (1524↔1562↔1549), and it fluctuates between the two sequential
runs too — its nondeterminism is intrinsic, independent of this change.
Tests
cargo test --lib(grafema-orchestrator): 467 passed, 0 failedcargo build --release: clean (only pre-existing warnings)Caveats
#[cfg(feature = "ruby")], embedded — no subprocess)stays sequential after the join on the main handle; it's feature-gated and
not part of the standard build.
code path without
phase_summary/ profile-subgraph; left for a follow-up.(rust 54→60s under concurrent load), but wall-clock still ~halves.
🤖 Generated with Claude Code