feat(graph): true graph data model (nodes + edges) — closes #8#24
Merged
Conversation
Introduces a true node + edge graph data model backed by SQLite (issue #8). New module scripts/graph_model.py: - init_graph_schema(conn): CREATE TABLE IF NOT EXISTS for graph_nodes + graph_edges + 6 indexes (node_type+name, name, file, source_id+edge_type, target_id+edge_type, edge_type). - populate_graph_tables(workspace, db_path): reads the flat backend.json registry and bulk-inserts all nodes/edges into the graph tables in a single transaction. Clears stale rows first so re-scans don't duplicate. - query_callers(node_id, db_path, max_depth=1): BFS over CALLS edges in reverse (who calls this node). - query_callees(node_id, db_path, max_depth=1): BFS over CALLS edges forward (what this node calls). - clear_graph_tables(db_path): DELETE FROM both tables. - find_nodes_by_name, graph_tables_exist, graph_tables_populated, graph_stats: introspection helpers for engines and tests. Schema (additive, prefixed graph_ to avoid collisions): graph_nodes(id, node_id UNIQUE, node_type, name, file, line, extra_json) graph_edges(id, source_id, target_id, edge_type, file, line, confidence, extra_json) Node types: function|class|file|module|route|type|interface Edge types: CALLS|IMPORTS|DEFINES|INHERITS|IMPLEMENTS|USES_TYPE (Only CALLS is populated in this pilot; other types are reserved for future engine migrations.) persistent_registry.py: calls init_graph_schema(conn) during _init_schema so the graph tables always exist by the time any engine queries them. NON-BREAKING: existing flat tables (symbols, refs, files, analysis_cache, scan_metadata) and JSON registries are untouched. All 56 CLI commands continue to work unchanged.
After the flat backend registry is built and saved, call
graph_model.populate_graph_tables(workspace, db_path) to bulk-insert
all nodes + edges into graph_nodes + graph_edges in a single transaction.
The population is wrapped in try/except so scan never fails if the graph
layer has an issue — the graph is an additive optimization layer; engines
fall back to the flat registry when it's missing.
Scan output now includes a 'graph' field with node + edge counts for
transparency:
"graph": { "nodes": 31, "edges": 97 }
Performance: population is a single bulk INSERT (executemany) wrapped in
one transaction, then committed once. On the clean_app fixture (31 nodes,
97 edges) it completes in <5ms.
Pilot engine migration for issue #8: the trace command now queries the new graph_edges table (CALLS edges) by default, with the flat-registry path retained as fallback. trace_engine.py: - Renamed trace_symbol -> trace_via_flat (legacy backend, kept verbatim). - Added trace_via_graph: BFS over CALLS edges via graph_model.query_callers / query_callees. Produces the same result-dict shape as trace_via_flat so callers and formatters don't need to know which backend was used. - Added trace_symbol dispatcher: picks graph by default, falls back to flat when graph tables are empty (pre-8.2 db) or use_graph=False. - Extracted shared frontend tracing (_trace_frontend_into) and result assembly (_assemble_result) so both backends produce identical output. - Std-lib callees are filtered in the graph path to match flat behavior (the flat path's edge_resolver marks them resolved=True with no node_id, and _bfs_trace_indexed silently skips them). commands/trace.py: - Added --use-graph (default) and --no-graph flags for A/B testing. - execute() passes use_graph through to trace_symbol. A/B verified on benchmarks/fixtures/clean_app: trace --use-graph and trace --no-graph produce identical chain sets (by depth + fn) for both 'up' and 'down' directions at depth 2-3. NON-BREAKING: existing trace output shape is unchanged. Default behavior improves only in that it uses indexed SQLite queries instead of in-memory edge_resolver index.
Adds tests/test_graph_model.py with 20 test cases across 6 test classes: 1. TestSchemaInit (3 tests): - Creates graph_nodes + graph_edges tables - Creates required indexes (type+name, source+edge_type, target+edge_type) - Idempotent (calling twice doesn't error or duplicate) 2. TestPopulation (3 tests): - Population from benchmarks/fixtures/clean_app populates nodes + edges - graph_nodes/edges counts match flat registry counts - Populated nodes preserve name, file, line, node_type metadata 3. TestQueryCallers (3 tests): - query_callers(get_user_by_id) returns main() at depth 1 - Caller entries include file and line for the calling site - BFS never returns the start node as its own caller 4. TestQueryCallees (3 tests): - query_callees(main) includes all expected function calls - Resolved callees have resolved=True; unresolved have resolved=False - Each resolved callee has a non-null node_id 5. TestRepopulation (3 tests): - Re-population does not duplicate rows - clear_graph_tables empties both tables - Clear then re-populate restores original counts 6. TestTracePilot (5 tests): - trace_via_graph up matches trace_via_flat up (get_user_by_id) - trace_via_graph down matches trace_via_flat down (main) - trace_via_graph both matches trace_via_flat both (process_data) - trace_symbol dispatcher defaults to graph backend - Dispatcher falls back to flat when graph tables are empty - use_graph=False forces the flat backend All 20 tests pass in 0.62s. Test results: - New tests: 20 passed, 0 failed - Pre-existing unrelated failures (test_hybrid_engine.py): 4 failed (confidence field assertions — unrelated to graph model) - Pre-existing integration test (test_integration.py): fails due to OOM when scanning /home/z (environment issue, not a regression)
README.md: - Bump version header to v8.2. - Add 'Graph Data Model (v8.2)' to Features list. - Add graph_model.py to the architecture tree under scripts/. CHANGELOG.md: - Add [8.2.0] Unreleased section documenting the graph data model addition: - New graph_model.py module with public API. - Graph schema (graph_nodes + graph_edges tables + 6 indexes). - trace --use-graph / --no-graph flags. - Changes to persistent_registry, scan, trace_engine. - Non-breaking guarantees. - Migration notes for engine authors (impact, circular, dependents next). references/agent-integration.md: - Add codelens.db to the .codelens/ file locations listing. - Add section 10.4 'Graph Data Model (v8.2+)' with code examples for find_nodes_by_name, query_callers, query_callees, and a decision table for when to use the graph vs the flat registry.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
This was referenced Jun 28, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Summary
Implements issue #8 — replaces the ad-hoc flat-registry graph traversal with a true node + edge graph backed by SQLite. This unblocks structural queries like "who calls this function across the entire codebase", "blast radius if I rename this class", and "circular dependency chains" — engines no longer need to reimplement partial graph traversal logic.
Non-breaking by design. The flat registry tables and JSON files are untouched. All 56 existing CLI commands continue to work unchanged. The graph tables are additive (prefixed
graph_to avoid collisions) and derived from the flat registry during scan.Files Touched
Created:
scripts/graph_model.py— New module: schema init, population, BFS queries (query_callers, query_callees), introspection helpers.tests/test_graph_model.py— 20 test cases across 6 test classes.Modified:
scripts/persistent_registry.py— Callsinit_graph_schema(conn)during_init_schemaso graph tables always exist.scripts/commands/scan.py— Callspopulate_graph_tables(workspace, db_path)after the flat backend registry is built. Scan output now includes agraphfield.scripts/trace_engine.py— Pilot engine migration:trace_symbolis now a dispatcher that pickstrace_via_graph(default) ortrace_via_flat(fallback). Shared frontend tracing and result assembly extracted to helpers.scripts/commands/trace.py— Adds--use-graph(default) and--no-graphflags.README.md— Bumped to v8.2, added Graph Data Model to Features, added graph_model.py to architecture tree.CHANGELOG.md— Added[8.2.0]Unreleased section with full migration notes.references/agent-integration.md— Added section 10.4 "Graph Data Model (v8.2+)" with code examples and a decision table.Graph Schema
graph_nodes( id INTEGER PRIMARY KEY AUTOINCREMENT, node_id TEXT NOT NULL UNIQUE, -- matches flat registry node "id" (file:line:fn) node_type TEXT NOT NULL, -- function|class|file|module|route|type|interface name TEXT NOT NULL, -- symbol name (flat registry "fn") file TEXT, line INTEGER, extra_json TEXT -- preserves original "type", "status", etc. ) graph_edges( id INTEGER PRIMARY KEY AUTOINCREMENT, source_id TEXT NOT NULL, -- references graph_nodes.node_id target_id TEXT, -- NULL for unresolved external calls edge_type TEXT NOT NULL, -- CALLS|IMPORTS|DEFINES|INHERITS|IMPLEMENTS|USES_TYPE file TEXT, -- file where the edge originates line INTEGER, -- line where the edge originates confidence REAL NOT NULL DEFAULT 1.0, extra_json TEXT -- preserves "ipc", "via_self", "to_fn", etc. )Indexes (6 total):
idx_graph_nodes_type_name,idx_graph_nodes_name,idx_graph_nodes_file,idx_graph_edges_source_type,idx_graph_edges_target_type,idx_graph_edges_type.Node types:
function|class|file|module|route|type|interface.Edge types:
CALLS|IMPORTS|DEFINES|INHERITS|IMPLEMENTS|USES_TYPE. OnlyCALLSis populated in v8.2 (pilot); other types are reserved for future engine migrations.Test Results
All 20 new tests pass:
Pre-existing test failures (NOT caused by this PR):
tests/test_hybrid_engine.py— 4 failures (test_query_confidence_without_deep,test_impact_confidence_without_deep,test_ai_format_confidence_distribution,test_dead_code_confidence_fields). These assertconfidencefields in hybrid engine output that don't exist onmaineither — pre-existing issue unrelated to the graph model.tests/test_integration.py— Fails due to OOM (rc=137) when scanning/home/z. Pre-existing environment issue.Pilot Verification:
trace --use-graphmatches flat outputA/B verified on
benchmarks/fixtures/clean_app(31 nodes, 97 edges):The graph path filters std-lib callees the same way the flat path does (via
edge_resolver._is_std_lib_method), so chain sets match exactly by (depth, fn, resolved) tuples.Migration Notes for Engine Authors
The flat registry remains the source of truth during scan. The graph tables are a derived projection. To migrate an engine:
graph_model.graph_tables_populated(db_path)— if False, fall back to flat (don't hard-fail).graph_model.find_nodes_by_name(name, db_path)to find start nodes.graph_model.query_callers/query_calleesfor BFS traversal.trace_engine.trace_via_graphfor a reference implementation.Next engine migrations (post-merge):
impact,circular,dependents.Commits
feat(graph): add graph_model.py with schema + populationfeat(scan): populate graph tables during scanfeat(trace): migrate trace engine to graph queries (--use-graph flag)test(graph): add graph model + trace pilot testsdocs(graph): update README + CHANGELOG for graph data modelcloses #8