Skip to content

perf(orchestrator): parallelize the per-language resolve phase (~50s wall) - #478

Draft
Disentinel wants to merge 1 commit into
feat/enrich-profilingfrom
feat/resolve-parallel
Draft

perf(orchestrator): parallelize the per-language resolve phase (~50s wall)#478
Disentinel wants to merge 1 commit into
feat/enrich-profilingfrom
feat/resolve-parallel

Conversation

@Disentinel

Copy link
Copy Markdown
Owner

What

Parallelize the resolve phase of the full-analyze path in
grafema-orchestrator/src/main.rs. Previously the twelve per-language
resolvers (js → haskell → rust → java → kotlin → python → go → swift →
apple-cross → jvm-cross → cpp → beam) ran strictly sequentially: each built
its own ProcessPool, resolved, committed, and shut the pool down before the
next language started.

The language resolvers are independent — each filters to its own language's
files and writes a disjoint output slice — and the heavy compute lives in
independent daemon subprocesses. The orchestrator side is async pipe +
socket I/O, so all twelve now run as concurrent futures joined with
tokio::join!.

The rfdb-connection constraint

The orchestrator holds one RfdbClient (single UnixStream, &mut self),
and the resolvers query rfdb mid-compute (resolve_per_file /
stream_and_resolve_single_worker collect their node set up front and commit
incrementally), so the shared read inputs can't simply be snapshotted once.

Design chosen: each concurrent resolver opens its own RfdbClient
connection to the same socket. rfdb-server spawns a dedicated thread per
client and serializes all writes behind the engine RwLock, so concurrent
connections are safe and disjoint-slice commits don't race. The orchestrator's
main rfdb handle is not touched inside the parallel section. Each future
returns its IMPORTS_FROM edges, merged into all_imports_from_edges in
deterministic language order after the join; the first resolver error is
propagated (preserving the prior sequential ? abort).

Per-language profile! events are unchanged (concurrent durations may overlap);
a new parallel_resolve_complete event records the section's wall time.

Measured (full self-analyze of the grafema monorepo, fresh runs)

metric sequential (baseline) parallel delta
resolve_ms 104 625 60 483 -44.1s (-42%)
total_ms 430 072 379 604 -50.5s (-12%)

resolve_ms collapses to ≈ max(single resolver) = rust (~60s under load),
exactly as predicted by the profile (sequential ≈ Σ = js+haskell+rust+beam).

Correctness — graph equivalence

Node/edge totals across four full runs on identical source + resolvers (only
orchestrator scheduling differs):

run mode nodes edges errors
baseline (orig) sequential 503 595 936 955 25
baseline (fresh) sequential 504 217 938 045 25
parallel #1 parallel 504 157 937 985 25
parallel #2 (fresh) parallel 504 243 938 071 25

The two sequential runs differ by 622 nodes / 1090 edges; the parallel runs
land squarely inside that band. The variance is pre-existing run-to-run
resolver/derive nondeterminism
, not introduced here. errors = 25 is constant
across all runs. Per-resolver output is stable: js, rust-cross-method-calls
(4926), rust-runtime-globals (42365), haskell-import (772), haskell-runtime-globals
(5288) are bit-identical sequential vs parallel; only haskell-cross-module-calls
fluctuates (1524↔1562↔1549), and it fluctuates between the two sequential
runs too — its nondeterminism is intrinsic, independent of this change.

Tests

  • cargo test --lib (grafema-orchestrator): 467 passed, 0 failed
  • cargo build --release: clean (only pre-existing warnings)
  • pre-commit hook (lint + mcp regression tests): green

Caveats

  • Ruby resolution (8l, #[cfg(feature = "ruby")], embedded — no subprocess)
    stays sequential after the join on the main handle; it's feature-gated and
    not part of the standard build.
  • Block2 (the resolve-only path, ~3107+) is not changed — it's a separate
    code path without phase_summary / profile-subgraph; left for a follow-up.
  • Peak per-language CPU contention raises individual resolver durations slightly
    (rust 54→60s under concurrent load), but wall-clock still ~halves.

🤖 Generated with Claude Code

The full-analyze path ran the per-language resolvers strictly sequentially
(js -> haskell -> rust -> java/kotlin/python/go/swift/apple-cross/jvm-cross/
cpp -> beam), each building its own ProcessPool, resolving, committing, and
shutting the pool down before the next language started. Profiled on the
grafema monorepo: js 10.8s + haskell 24.4s + rust 53.6s + beam 12.9s = ~102s
sequential, while the single longest resolver is ~54s.

The language resolvers are independent: each filters to its own language's
files and writes a disjoint output slice. Their compute lives in independent
daemon SUBPROCESSES; the orchestrator side is async pipe + socket I/O. So we
now run all twelve as concurrent futures joined with tokio::join!.

RFDB-connection constraint: the orchestrator holds ONE RfdbClient (single
UnixStream, &mut self) and the resolvers query rfdb mid-compute (collect their
node set up front, commit incrementally), so the shared read inputs cannot be
snapshotted once. Instead each concurrent resolver opens its OWN RfdbClient
connection to the same socket. The rfdb-server spawns a dedicated thread per
client and serializes all writes behind the engine RwLock, so concurrent
connections are safe and disjoint-slice commits do not race. The orchestrator's
main rfdb handle is not touched inside the parallel section.

Each future returns its IMPORTS_FROM edges; they are merged into
all_imports_from_edges in deterministic language order after the join, and the
first resolver error is propagated (preserving the prior sequential ? abort).
Per-language profile! events are unchanged (concurrent durations may overlap);
a new parallel_resolve_complete event records the wall time of the section.

Measured on a full self-analyze of the grafema monorepo:
  resolve_ms  102024 -> 56611  (-45.4s, -44.5%)
  total_ms    414788 -> 365690 (-49.1s, -11.8%)
Graph equivalent: nodes ~503.6k, edges ~937k, errors 25 (within run-to-run
resolver nondeterminism; js/haskell-import/runtime-globals slices bit-identical).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Disentinel
Disentinel marked this pull request as draft June 19, 2026 19:23
@Disentinel

Copy link
Copy Markdown
Owner Author

⏸️ HOLD — do not merge to main. Per the engine zone canon (resolve / rfdb* / datalog-resolution-logic = do-not-touch, tied to the active feat/datalog rewrite), this is to be integrated into feat/datalog, not merged independently. The change is pure scheduling/connection-management (12 resolvers via tokio::join!, one RfdbClient each), verified graph-equivalent (resolve_ms 104.6s→60.5s, −42%). Kept open as the reviewable vehicle for that integration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant