feat: intra-procedural CFG + reaching-definitions (def-use) overlay — 11 languages (TS/JS, Python, Go, Java, C++, Rust, Ruby, C, C#, PHP)

[clay-good]

Summary

Implements add-intraprocedural-cfg-dataflow-overlay end-to-end: a deterministic, per-function control-flow graph plus reaching-definitions (def-use) overlay computed purely from the AST (no LLM, no type-checker, no build) while the parse tree is live, stored as a compact per-function blob in a new DB-only table — and an opt-in value-level precision mode on analyze_impact/trace_execution_path built on it.

This follows the Joern overlay model: the call graph stays the base; CFG and reaching-defs are two additive, independently-storable overlays. The default behavior of every existing tool is byte-for-byte unchanged — value-level precision is strictly opt-in.

What's implemented

Phase 1 — CFG + storage (commit 1)

• src/core/analyzer/cfg.ts — the first statement-level AST visitor in the analyzer (modeled on the Elixir walk precedent). Builds basic blocks + control-flow edges (branch/loop/early-exit/back edges). Returns pure data — no AST node references survive, so it is WASM-tree-lifetime safe.
• call-graph.ts — the in-scope extractors (TS/JS, Python, Go) build the overlay inside the live-tree window; the extractor return contract gains an optional cfg; CallGraphResult carries a transient cfgs map that is not added to SerializedCallGraph/the resident graph. Other languages fail soft (no overlay, no error).
• edge-store.ts — new cfg_overlay table (one JSON blob per function id), SCHEMA_VERSION 6→7 (drop-and-rebuild, zero migration). Lazy getCfg + per-file deleteCfgForFile/insertCfgs.
• artifact-generator.ts — persists the overlay to SQLite and strips it from llm-context.json so it never becomes resident.
• mcp-watcher.ts — recomputes only the changed file's overlay rows inside the existing per-file swap transaction (intra-procedural ⇒ caller files untouched).

Phase 2 — reaching definitions (commit 1)

• A classical intra-procedural reaching-definitions fixpoint over the CFG produces precision-labeled def-use edges: exact for sound local-scalar def-use, may for conservatively over-approximated field/subscript/closure dependences.

Step 5 — value-level opt-in (commit 2)

• valueReachableLines() — a pure forward data-flow slice over the def-use edges (the value's impact set).
• analyze_impact gains valueLevel/valueParam: narrows the downstream blast radius to the direct callees whose argument lines are data-dependent on the targeted value (cross-call hop labeled may), expanding forward from them. trace_execution_path gains the same flags (restricts each entry's first hop). Both fall back to function granularity (no error) when the function has no overlay. Plumbed through dispatch + tool schemas; nav-preset payload ceiling bumped 11800→12300 (spec-28 precedent).

Hardening (dogfooding on real code)

After the initial implementation, the overlay was dogfooded on the real OpenLore corpus, which surfaced five soundness/precision bugs in the CFG/def-use builder — all now fixed and regression-tested:

1. try/catch treated as straight-line — a catch-body def spuriously killed the try-body def. Now modeled as alternative paths from a common predecessor (both reach the join; finally runs from the merge).
2. switch treated as straight-line — cases linearly killed each other. Now alternative branches with language-correct fall-through (switchFallsThrough: TS/JS yes, Go/Python no).
3. closure captures labeled exact — the spec requires may. Nested functions are no longer descended into as the outer CFG; their free-variable reads become may captures and their own params/locals no longer leak.
4. Python if/elif/else collapsed to a single alternative (childForFieldName returns only the first) — later elif/else branches were dropped. Now the full elif chain is built.
5. Destructuring (const { a, b } = obj, [x, y], { a: x }, { a = default }) produced no defs. Now each binding leaf is a definition.

Verified deterministic across repeated analyses of the real corpus (identical overlay hash), with branch/merge structure and may edges present on actual code.

Exhaustive all-language verification (every language OpenLore supports)

Before ship, every one of OpenLore's 19 detectable languages was verified to either produce a correct overlay or fail soft cleanly without breaking anything — proven by three parallel agents on ~30 real repos (~50,000 overlays) plus a CI-protected contract guard.

• 7 fail-soft code languages (Kotlin, Swift, Scala, Dart, Lua, Elixir, Bash) on real repos (okhttp, Alamofire, playframework, flutter/samples, lua-repl, phoenix, nvm): exit 0, no crash, full call graph (up to 7,094 nodes), 0 overlay rows for the target language, DB integrity OK, search intact. The Lua/Bash → extractByQueries → buildCfgFor path fails soft cleanly. 0 invariant violations across 5,449 incidental overlays from co-located supported-language files.
• 5 IaC types (Terraform, Kubernetes, Helm, CloudFormation, Ansible) + polyglot repos (apache/spark, kubernetes): analyze cleanly, 0 overlay rows from IaC (the no-overlay invariant holds even mixed into Go/Bash/Python trees), deterministic, 0 invariant violations, no resident-artifact leak.
• CI-protected language-contract guard: a test locks cfgSupportsLanguage to exactly the 11 overlay languages and asserts buildFunctionCfg returns undefined (never throws) for every other language — the fail-soft contract can't silently regress.

Watcher bug found and fixed (real "breaking existing tooling")

Being methodical surfaced a genuine pre-existing bug: .c/.cs/.php/.kt files are in the watcher's SOURCE_EXTENSIONS but were missing from its CALL_GRAPH_LANGS, so editing one made buildGraphSubset return empty and the per-file swap wiped that file's nodes/edges (and, with the overlay, its cfg_overlay rows) in watch mode until the next full analyze. Added C/C#/PHP/Kotlin to the watcher's graph set so edits re-graph and re-overlay instead of wiping (grammars are optional deps → fail soft to empty if absent, identical to full analyze). Verified + regression-tested.

Broad re-validation: 11 overlay languages on 11 fresh real repos

A third agent re-validated all 11 overlay languages on new repos (got, lodash, click, gin, commons-lang, fmt, serde, jekyll, curl, automapper, laravel-framework): ~46,700 overlays / ~190,000 def-use edges, 0 structural violations, determinism bit-identical (clean-rebuild hashes equal), and no unsound-exact found in source-level soundness spot-checks. The dangerous cases are correctly downgraded to may on real code — C++ &r passed to an intrinsic, JS closure rebinds, Ruby statement modifiers, C pointer arithmetic. laravel (28,276 overlays) is a clean large-scale stress test.

It surfaced one C/C++ imprecision, now fixed: a write through a pointer (*p = x) was recorded as an exact def of the binding p (the pointer_expression fell through to the inner identifier), mislabeling provenance and killing p's real def. A wrong exact provenance label is precisely the failure mode the overlay must never emit, so *p = x is now handled like a member/subscript write — base pointer is a use, the l-value *p a conservative may def, never an exact reassign of p. Regression-tested.

Language coverage (verified)

Overlay-supported (sound, adversarially validated)	Fail-soft (safe: full call graph, 0 overlay rows, no crash)
TypeScript, JavaScript, Python, Go, Java, C++, Rust, Ruby, C, C#, PHP	Kotlin, Scala, Swift, Lua, Bash, Elixir, Dart

A regression+fail-soft safety agent confirmed the overlay breaks nothing for any of OpenLore's 18 languages — analyze exits 0, the no-overlay languages keep their full call-graph nodes/edges with 0 overlay rows, no resident leak, 0 invariant violations. The 11 overlay languages are the ones whose grammars expose the fields the (field-centric) builder needs and that adversarial agents validated as sound on real repos. Kotlin/Scala/Swift have field-less/positional grammars (e.g. Kotlin if exposes no condition/consequence fields) — adapting the builder to those is bug-prone, so they (and the deferred Lua/Bash/Elixir/Dart) stay fail-soft rather than risk an unsound overlay. Covered safely — never wrong data.

Adversarial validation of C/C#/PHP on real repos (sds, cecil, FastRoute): scale, invariants, and determinism clean across 2,704 overlays (0 violations, byte-identical hashes). C was fully sound on every probe. Four unsound-exact issues in PHP/C# escape detection were found on real code and fixed: PHP anonymous closures leaked their body (stale grammar node name anonymous_function_creation_expression → anonymous_function), PHP $r = &$x reference aliasing, PHP use (&$x) by-ref capture, and C# ref/out arguments (pervasive in cecil's TryGet(out …) pattern). After the fix, cecil's may-rate rose 378→987 (the previously-unsound exacts are now correctly conservative) and the FastRoute closure leak is gone.

Language extension round (7 languages)

Extended the overlay beyond the v1 TS/JS·Python·Go set to Java, C++, Rust, and Ruby — the other native-grammar languages OpenLore covers. The builder is spec-driven (a per-language CfgLangSpec of node-type names), so each language is a verified spec plus the infrastructure it needed:

• callNameField — Java uses a name field for the callee (Ruby method), so the receiver/args are still read while the method name is skipped.
• Compound-operator detection by text — Java/C/C++ use one node type for = and +=; a compound assignment now correctly reads its target first.
• caseParts() — per-language switch/match/case decomposition: Rust match_arm and Ruby when/else carry the body in a field, Java groups statements under switch_label markers, C/C++ use case_statement. All modeled as parallel alternatives so cases never kill each other (verified sound on every language).
• Expression-statement unwrap for Rust's expression-oriented control flow (if/while/match are wrapped in expression_statement), plus else/then/do/body_statement block types and declarator/pattern/value field fallbacks for the new grammars.

Node types were determined empirically against each grammar (never guessed). All the soundness machinery carries over: closure mutation → may, member/subscript → may, escape analysis, scope resolution, do/while post-test, fail-soft caps. Verified end-to-end (overlays stored for all four), deterministic, full suite 3598 passing with per-language tests. Unsupported languages still fail soft (no overlay).

Adversarial validation on real repos. Two agents stress-tested the four new languages on gson, nlohmann/json, ripgrep, and sinatra (the same rigor applied to TS/Py/Go). Scale, structural invariants, and determinism were clean across 5,601 overlays (0 violations, byte-identical hashes). Rust came back fully sound (97.3% exact, correct on shadowing/re-let/if-as-expression/closures). Four unsound-exact issues found and fixed — Java 14+ arrow switch (case N -> {}), C++ reference aliasing (int& r = x) and &x address-of escape, and Ruby statement modifiers (x = 2 if c, live in sinatra) — plus two coverage gaps (C++ parameters were nested under function_declarator; Java enhanced-for loop var). All verified with per-language tests; may correctly rose where aliasing was newly detected.

Production-readiness round (guarantees proven, not asserted; scale-tested)

A final pass verified the guarantees we'd claimed but not measured, and stress-tested at scale via three more agents:

• Default output is byte-identical — diffed pre-feature main vs the feature branch on a real repo: analyze_impact/select_tests/get_subgraph/trace_execution_path/find_dead_code outputs are identical to the byte (35,621 bytes, zero diff). The overlay's presence changes nothing for an agent that doesn't opt in.
• Zero new resident memory — llm-context.json is byte-identical (171,899 bytes) with the feature; the resident callGraph carries no cfgs/defUse. The overlay is DB-only and lazily loaded.
• MCP protocol end-to-end — verified over real JSON-RPC stdio: tools/list exposes valueLevel/valueParam with correct schemas, value-level narrows over the wire, default calls omit the field, invalid input degrades gracefully, and the server exits clean.
• Scale & determinism — analyzed the TypeScript compiler (src/compiler, 11.2k fns), Django (12.1k), Kubernetes (pkg/, 37.9k Go fns), and a mixed TS+Py+Go tree. 56,441 overlays, 0 invariant violations, perfect determinism (identical content hashes across runs), no crash/hang. Overhead is linear ~0.7–0.9 ms/function (~17–34% of analyze wall-clock); the fail-soft caps never fired on real code. ~88% exact / 12% may at scale.

Two issues found this pass, both fixed:

• do/while unsound-exact — it was modeled as a pre-test loop, leaking a pre-loop def into the condition/post-loop as exact. Now post-test wired (body runs first), so the body's defs correctly kill the pre-loop def. (Plain while/for unchanged.)
• Decorated Python functions got no overlay — buildCfgFor's body-dig didn't accept Python's inner function_definition under a decorated_definition wrapper, so @property/@cached_property/Django-view functions were silently skipped. Fixed; verified end-to-end.

Wider audit round (3 agents: value-level, robustness, deep idioms)

A third wave of parallel agents stress-tested the consumer seam, robustness, and deep language idioms. Four issues found, all fixed:

• Determinism (critical): the callee-skip compared tree-sitter nodes by object identity (c === fn), but native tree-sitter returns fresh wrapper objects per access path — so the overlay flickered ~50% of builds. Fixed with a position-based sameNode() comparator. Now 25–30 identical builds and identical full-analyze hashes; a strong many-build determinism test guards the class (the old 2-build test missed it).
• False narrowing (value-level, agent-misleading): valueReachableLines chained taint by exact line co-location, so a parameter feeding a multi-line initializer (const ctx = {\n x: p \n}) dropped every downstream that object reached — telling an agent a change was safe when it wasn't (reproduced on real zod safeParseAsync). Fixed by tagging RHS uses with the fed def's start line (useInDefLine) and chaining on it; verified handleResult now surfaces on the real zod function.
• Unsound exact: Python global / same-level nonlocal were exact despite hidden mutation; now may.
• Robustness: the reaching-defs fixpoint was quadratic on deeply-nested loops (47s at 700-deep); now capped (128 sweeps / 4000 blocks) and fails soft to no overlay in <2s. Plus Python with ... as binding defs.

Also widened test coverage to the integration seams: a watcher edit recomputes the overlay byte-equal to a fresh build, and the cross-call may label is asserted. Final sweep across zod/cobra/requests: 1,596 overlays, 0 violations, 0 key leaks, ~91% exact / 9% may, deterministic.

Deep correctness round (adversarial audit + real-repo dogfooding)

Two parallel agents — a real-repo stress test and an independent soundness audit — hunted for the one thing that must never happen: an exact label that's actually wrong. They found a cluster of bugs, all now fixed and regression-tested:

• Lexical scope resolution (the big one): name-based reaching-defs conflated shadowed variables, so an inner-block let/const/:= (TS/Go) or a Python comprehension loop var reached an outer same-named use as exact and dropped the correct outer edge. A resolveScopes pre-pass now keys every identifier by name#scopeId; reaching-defs groups by the scoped key while emitting the bare name. Inner shadows no longer kill or link to outer variables.
• x++/x-- are now recorded as a use+def (were invisible) and detected as closure mutations.
• Go closure escape now descends expression_list LHS (x = 5), inc/dec, and address-of taken inside a closure — all correctly downgrade the outer var to may.
• Python try/except/else — the else runs on the no-exception path after the try body; previously ignored (lost its def, kept the overwritten try def).
• Labeled loops — labeled_statement is unwrapped so the loop keeps its back-edge and loop-carried dependence.
• Go loop headers — Go wraps the header in a for_clause/range_clause child, so loop counters and range vars were invisible (and a range var could alias an outer same-name as exact). A loopHeaderField helper now descends into the clause; counters/range vars are defined, loop-carried, and scoped.

A follow-up verification audit confirmed all fixes hold and TS/Python scope resolution is clean; it surfaced the Go loop-header gap above, now fixed. Validated on zod, requests, and cobra (1,596 overlays): 0 structural-invariant violations, ~90% exact / 10% may, 0 scope keys leaked into the output across 9,634 edges, deterministic across repeated runs. The earlier express/flask/mux/ky/type-fest sweep adds another 1,000+ overlays clean.

Correctness & real-world validation

The exact precision label is the load-bearing safety signal — a wrong exact is the one thing that taxes an agent with incorrect context, so it gets adversarial scrutiny:

• Escape analysis — a local scalar reassigned inside a nested closure, or a Go local whose address is taken (&x), can change out of band. Both were silently labeled exact; they are now downgraded to may. The escape set is over-approximated on purpose (a false inclusion only weakens precision, never yields an unsound exact); read-only closure captures are not downgraded, so precision is preserved where sound. exact now means soundly exact for the supported languages.
• Structural safety net — isStructurallyValid runs on every built CFG (one entry/exit, edge endpoints reference real blocks, positive def/use lines, valid labels). A violation makes the builder emit no overlay rather than wrong context — so future grammar drift or builder bugs degrade safely.
• Real-world corpus — validated on four diverse third-party repos (express/JS, flask/Python, gorilla-mux/Go, ky/TS): all analyzed with no crashes, and a structural-invariant sweep over all 457 produced overlays found 0 violations.

Design posture: the overlay is opt-in, DB-only, fail-soft, and label-gated — an agent that doesn't request value-level precision sees byte-identical output and zero new data; one that does gets an honest exact/may signal, never an over-claimed dependence.

Verification

• Unit: cfg.test.ts — every spec scenario (branch/join, loop back edge, early-return path termination, determinism, fail-soft; local-scalar def-use, reassignment-kills, both-branches-reach, field write labeled may, exact/may distinguishable) across TypeScript, Python, and Go.
• Storage: cfg-overlay-storage.test.ts — overlay loadable from store, not in the resident serialized graph, schema bump rebuilds without migration, per-file delete is isolated.
• Consumer: graph.test.ts value-level block — default unchanged, narrows to the data-dependent callee, falls back when no overlay.
• Full CI-mirror suite (vitest run src examples): 3555 passed, 2 skipped.
• End-to-end through the compiled CLI on a real TS/Python/Go repo: schema v7; overlay persisted for every in-scope-language function (542/650 non-external nodes — every node lacking one is a non-TS/Py/Go fixture = correct fail-soft); overlay ~1.1 KB median/function, absent from llm-context.json (resident memory unchanged); value-level analyze_impact on param a narrows downstream to exactly the data-dependent callee.

Spec / decisions

Spec delta merged into openspec/specs/analyzer/spec.md + mcp-handlers/spec.md via approved decisions c8f2b9bf (overlay-in-live-tree-extractors) and b6f04199 (value-level opt-in). ADR adr-0007.

Out of scope (as specified)

Inter-procedural/whole-program data-flow, sound aliasing/points-to, control-dependence/slicing as first-class outputs, statement-level nodes in the resident graph, and languages beyond TS/JS·Python·Go in v1.

🤖 Generated with Claude Code