perf(indexing): profile and parallelize CPU-bound graph construction

[styler-ai]

Goal

Measure the v0.4 indexing pipeline at intended repository scale and use Rust's existing data-parallel tooling to reduce CPU-bound indexing and graph-construction time without weakening determinism, cancellation, memory limits, freshness, or SQLite publication safety.

Current baseline

ProjectAtlas already uses:

• ignore's parallel walker for Git-aware repository traversal and source hashing;
• bounded Rayon pools for built-in symbol parsing, optional-parser job coordination, and structural-summary derivation;
• std::thread::available_parallelism() plus work cardinality, caller ceilings, host-wide controls, and a hard safety ceiling of 32 workers;
• off-writer staging followed by one atomic SQLite publication.

On a 16-logical-core Windows host, the v0.4 scan selected 16 symbol workers. Remaining visibly serial work includes persisted-text source reads/hash validation, substantial normalized graph/entity/resolution projection, stage-to-stage sequencing, and the intentionally single-writer SQLite publication path.

Requirements

• Profile first. Record per-stage wall time, CPU time/utilization, worker occupancy, disk throughput, allocations/RSS, staged bytes, SQLite time, and end-to-end time on representative small, medium, large, and huge repositories across 4/8/16/32 logical-core hosts where available.
• Detect the effective CPU capacity available to the running ProjectAtlas process automatically rather than relying on a fixed machine-wide core count. Respect OS CPU affinity, VM/container/job-object quotas, an explicit user ceiling, memory and workload size, then feed one shared bounded worker policy.
• Use dynamic scheduling/work stealing inside that budget so a host such as an 8-core Ryzen 7 3700X can process independent files across up to eight useful workers without static one-eighth shards waiting on an unusually slow file. Scale down automatically for small repositories, constrained runners, memory pressure, concurrent ProjectAtlas work, or interactive-agent fairness.
• Reuse the installed rayon and ignore crates before considering another dependency. Prefer thread-local par_iter / try_fold / try_reduce results with deterministic ordered merge over shared concurrent maps.
• Use safe Rust parallelism throughout every indexing mode where it produces a measured net win: first-project init, full scan, incremental watch --once, continuous watcher batches, built-in and optional symbol/deep-graph refresh, and affected-closure rebuilding. Cold first indexing of large repositories is the most visible case, not the only target.
• Measure discovery, hashing, decoding, parsing, summary generation, graph entity/relation projection, resolution, deterministic merge, and publication separately. Automatically choose serial versus parallel execution from effective CPU capacity, work cardinality, byte/memory cost, storage behavior, and measured crossover thresholds so small changes avoid parallel overhead while large work uses the available host budget.
• Keep memory proportional to the shared admission budget rather than multiplying the full corpus by worker count. Reuse thread-local buffers and capacity, stream or chunk inputs, use bounded per-worker try_fold results plus deterministic try_reduce/ordered merge, avoid redundant path/string/source clones and repeated serialization, and release stage-owned memory promptly.
• Evaluate bounded parallel text preparation while preserving exact aggregate byte admission, source-change detection, cancellation, deterministic row order, and sensible storage-device concurrency.
• Evaluate bounded parallel graph entity, export-key, resolution-candidate, and relation projection before publication. Preserve stable keys, identical ambiguity/resolution states, exact coverage, row/output budgets, and deterministic bytes.
• Evaluate safe overlap only between independent off-writer stages when measurements show a win and the combined memory/I/O budget remains bounded.
• Keep SQLite transaction/publication ownership single-writer unless concrete SQLite evidence proves another design safe and faster. Retain prepared statements, batching, indexes, WAL policy, rollback, and last-complete-generation visibility.
• Preserve one host-wide worker budget across scanning, built-in parsers, optional contained workers, graph work, concurrent MCP sessions, and other ProjectAtlas processes. Do not target 100% CPU at the expense of agent responsiveness or machine stability.
• Add configurable/calibrated worker policy only if measurements show the current host-core default and 32-worker ceiling are wrong on real workloads.
• Add no new concurrency, graph, or concurrent-map crate unless a benchmark demonstrates that the existing standard library, Rayon, and ignore cannot implement the winning design.

Acceptance checklist

• Add a reproducible stage-timing/CPU/RSS/I/O benchmark matrix with frozen corpora and current sequential baselines.
• Identify the measured critical path and publish a before/after flamegraph or equivalent profiler evidence on each supported OS family.
• Parallelize only the proven CPU-bound off-writer stages with one shared bounded worker policy.
• Prove deterministic graph keys, rows, ordering, coverage, summaries, and published database bytes across worker counts and repeated runs.
• Prove cancellation, source races, memory/row/output limits, parser failure, worker panic, and SQLite rollback preserve the last complete generation.
• Prove automatic effective-CPU detection and bounded self-tuning on 4/8/16/32 logical-core hosts plus affinity- or quota-constrained processes; show dynamic load balancing avoids static-shard tail latency.
• Prove multi-project/MCP concurrency does not oversubscribe the host or starve interactive queries.
• Prove peak RSS and in-flight bytes remain within an explicit host-wide budget as worker count grows; demonstrate that no worker duplicates the full source corpus or graph and that failure/cancellation releases staged memory.
• Report end-to-end and per-stage speedup, CPU utilization, worker occupancy, peak RSS, allocations/copies, I/O, persistent bytes, and regressions for cold init, full scan, incremental/watch, and deep-graph refresh on small through huge repositories; prove the serial/parallel crossover is automatic.
• Update architecture and performance documentation with measured limits and the final worker-ownership model.

Non-goals

• Do not delay or alter v0.4.0.
• Do not equate full-core saturation with success; end-to-end latency, bounded resource use, determinism, and host fairness are the product goals.
• Do not parallelize SQLite writes speculatively or add a second graph/database authority.
• Do not replace Git-aware ignore traversal, Rayon, or existing cancellation/resource contracts without measured evidence.