Skip to content

feat(graph): true graph data model (nodes + edges) — closes #8#24

Merged
Wolfvin merged 5 commits into
mainfrom
feat/issue-8-graph-data-model
Jun 28, 2026
Merged

feat(graph): true graph data model (nodes + edges) — closes #8#24
Wolfvin merged 5 commits into
mainfrom
feat/issue-8-graph-data-model

Conversation

@Wolfvin

@Wolfvin Wolfvin commented Jun 28, 2026

Copy link
Copy Markdown
Owner

Summary

Implements issue #8 — replaces the ad-hoc flat-registry graph traversal with a true node + edge graph backed by SQLite. This unblocks structural queries like "who calls this function across the entire codebase", "blast radius if I rename this class", and "circular dependency chains" — engines no longer need to reimplement partial graph traversal logic.

Non-breaking by design. The flat registry tables and JSON files are untouched. All 56 existing CLI commands continue to work unchanged. The graph tables are additive (prefixed graph_ to avoid collisions) and derived from the flat registry during scan.

Files Touched

Created:

  • scripts/graph_model.py — New module: schema init, population, BFS queries (query_callers, query_callees), introspection helpers.
  • tests/test_graph_model.py — 20 test cases across 6 test classes.

Modified:

  • scripts/persistent_registry.py — Calls init_graph_schema(conn) during _init_schema so graph tables always exist.
  • scripts/commands/scan.py — Calls populate_graph_tables(workspace, db_path) after the flat backend registry is built. Scan output now includes a graph field.
  • scripts/trace_engine.py — Pilot engine migration: trace_symbol is now a dispatcher that picks trace_via_graph (default) or trace_via_flat (fallback). Shared frontend tracing and result assembly extracted to helpers.
  • scripts/commands/trace.py — Adds --use-graph (default) and --no-graph flags.
  • README.md — Bumped to v8.2, added Graph Data Model to Features, added graph_model.py to architecture tree.
  • CHANGELOG.md — Added [8.2.0] Unreleased section with full migration notes.
  • references/agent-integration.md — Added section 10.4 "Graph Data Model (v8.2+)" with code examples and a decision table.

Graph Schema

graph_nodes(
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    node_id     TEXT NOT NULL UNIQUE,   -- matches flat registry node "id" (file:line:fn)
    node_type   TEXT NOT NULL,          -- function|class|file|module|route|type|interface
    name        TEXT NOT NULL,          -- symbol name (flat registry "fn")
    file        TEXT,
    line        INTEGER,
    extra_json  TEXT                    -- preserves original "type", "status", etc.
)

graph_edges(
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    source_id   TEXT NOT NULL,          -- references graph_nodes.node_id
    target_id   TEXT,                   -- NULL for unresolved external calls
    edge_type   TEXT NOT NULL,          -- CALLS|IMPORTS|DEFINES|INHERITS|IMPLEMENTS|USES_TYPE
    file        TEXT,                   -- file where the edge originates
    line        INTEGER,                -- line where the edge originates
    confidence  REAL NOT NULL DEFAULT 1.0,
    extra_json  TEXT                    -- preserves "ipc", "via_self", "to_fn", etc.
)

Indexes (6 total): idx_graph_nodes_type_name, idx_graph_nodes_name, idx_graph_nodes_file, idx_graph_edges_source_type, idx_graph_edges_target_type, idx_graph_edges_type.

Node types: function|class|file|module|route|type|interface.
Edge types: CALLS|IMPORTS|DEFINES|INHERITS|IMPLEMENTS|USES_TYPE. Only CALLS is populated in v8.2 (pilot); other types are reserved for future engine migrations.

Test Results

tests/test_graph_model.py — 20 passed in 0.62s

All 20 new tests pass:

  • TestSchemaInit (3) — tables + indexes + idempotency
  • TestPopulation (3) — population from clean_app fixture, count match, metadata preservation
  • TestQueryCallers (3) — direct callers, file/line inclusion, no self-calls
  • TestQueryCallees (3) — direct callees, resolved flag, expected callees of main()
  • TestRepopulation (3) — no duplicates on re-populate, clear empties both, clear+repopulate restores
  • TestTracePilot (5) — A/B parity (up/down/both), dispatcher default, fallback when empty, use_graph=False

Pre-existing test failures (NOT caused by this PR):

  • tests/test_hybrid_engine.py — 4 failures (test_query_confidence_without_deep, test_impact_confidence_without_deep, test_ai_format_confidence_distribution, test_dead_code_confidence_fields). These assert confidence fields in hybrid engine output that don't exist on main either — pre-existing issue unrelated to the graph model.
  • tests/test_integration.py — Fails due to OOM (rc=137) when scanning /home/z. Pre-existing environment issue.

Pilot Verification: trace --use-graph matches flat output

A/B verified on benchmarks/fixtures/clean_app (31 nodes, 97 edges):

trace_via_flat('get_user_by_id', direction='up',   max_depth=3) == trace_via_graph(...)  # identical chain sets
trace_via_flat('main',            direction='down', max_depth=2) == trace_via_graph(...)  # identical chain sets
trace_via_flat('process_data',    direction='both', max_depth=3) == trace_via_graph(...)  # identical chain sets

The graph path filters std-lib callees the same way the flat path does (via edge_resolver._is_std_lib_method), so chain sets match exactly by (depth, fn, resolved) tuples.

Migration Notes for Engine Authors

The flat registry remains the source of truth during scan. The graph tables are a derived projection. To migrate an engine:

  1. Check graph_model.graph_tables_populated(db_path) — if False, fall back to flat (don't hard-fail).
  2. Use graph_model.find_nodes_by_name(name, db_path) to find start nodes.
  3. Use graph_model.query_callers / query_callees for BFS traversal.
  4. Preserve the existing flat-path output shape so callers and formatters don't break. See trace_engine.trace_via_graph for a reference implementation.

Next engine migrations (post-merge): impact, circular, dependents.

Commits

  • feat(graph): add graph_model.py with schema + population
  • feat(scan): populate graph tables during scan
  • feat(trace): migrate trace engine to graph queries (--use-graph flag)
  • test(graph): add graph model + trace pilot tests
  • docs(graph): update README + CHANGELOG for graph data model

closes #8

Wolfvin added 5 commits June 28, 2026 05:28
Introduces a true node + edge graph data model backed by SQLite (issue #8).

New module scripts/graph_model.py:
- init_graph_schema(conn): CREATE TABLE IF NOT EXISTS for graph_nodes +
  graph_edges + 6 indexes (node_type+name, name, file, source_id+edge_type,
  target_id+edge_type, edge_type).
- populate_graph_tables(workspace, db_path): reads the flat backend.json
  registry and bulk-inserts all nodes/edges into the graph tables in a
  single transaction. Clears stale rows first so re-scans don't duplicate.
- query_callers(node_id, db_path, max_depth=1): BFS over CALLS edges in
  reverse (who calls this node).
- query_callees(node_id, db_path, max_depth=1): BFS over CALLS edges
  forward (what this node calls).
- clear_graph_tables(db_path): DELETE FROM both tables.
- find_nodes_by_name, graph_tables_exist, graph_tables_populated,
  graph_stats: introspection helpers for engines and tests.

Schema (additive, prefixed graph_ to avoid collisions):
  graph_nodes(id, node_id UNIQUE, node_type, name, file, line, extra_json)
  graph_edges(id, source_id, target_id, edge_type, file, line,
              confidence, extra_json)

Node types: function|class|file|module|route|type|interface
Edge types:  CALLS|IMPORTS|DEFINES|INHERITS|IMPLEMENTS|USES_TYPE
(Only CALLS is populated in this pilot; other types are reserved for
future engine migrations.)

persistent_registry.py: calls init_graph_schema(conn) during _init_schema
so the graph tables always exist by the time any engine queries them.

NON-BREAKING: existing flat tables (symbols, refs, files, analysis_cache,
scan_metadata) and JSON registries are untouched. All 56 CLI commands
continue to work unchanged.
After the flat backend registry is built and saved, call
graph_model.populate_graph_tables(workspace, db_path) to bulk-insert
all nodes + edges into graph_nodes + graph_edges in a single transaction.

The population is wrapped in try/except so scan never fails if the graph
layer has an issue — the graph is an additive optimization layer; engines
fall back to the flat registry when it's missing.

Scan output now includes a 'graph' field with node + edge counts for
transparency:

  "graph": { "nodes": 31, "edges": 97 }

Performance: population is a single bulk INSERT (executemany) wrapped in
one transaction, then committed once. On the clean_app fixture (31 nodes,
97 edges) it completes in <5ms.
Pilot engine migration for issue #8: the trace command now queries the
new graph_edges table (CALLS edges) by default, with the flat-registry
path retained as fallback.

trace_engine.py:
- Renamed trace_symbol -> trace_via_flat (legacy backend, kept verbatim).
- Added trace_via_graph: BFS over CALLS edges via graph_model.query_callers
  / query_callees. Produces the same result-dict shape as trace_via_flat
  so callers and formatters don't need to know which backend was used.
- Added trace_symbol dispatcher: picks graph by default, falls back to
  flat when graph tables are empty (pre-8.2 db) or use_graph=False.
- Extracted shared frontend tracing (_trace_frontend_into) and result
  assembly (_assemble_result) so both backends produce identical output.
- Std-lib callees are filtered in the graph path to match flat behavior
  (the flat path's edge_resolver marks them resolved=True with no node_id,
  and _bfs_trace_indexed silently skips them).

commands/trace.py:
- Added --use-graph (default) and --no-graph flags for A/B testing.
- execute() passes use_graph through to trace_symbol.

A/B verified on benchmarks/fixtures/clean_app: trace --use-graph and
trace --no-graph produce identical chain sets (by depth + fn) for both
'up' and 'down' directions at depth 2-3.

NON-BREAKING: existing trace output shape is unchanged. Default behavior
improves only in that it uses indexed SQLite queries instead of in-memory
edge_resolver index.
Adds tests/test_graph_model.py with 20 test cases across 6 test classes:

1. TestSchemaInit (3 tests):
   - Creates graph_nodes + graph_edges tables
   - Creates required indexes (type+name, source+edge_type, target+edge_type)
   - Idempotent (calling twice doesn't error or duplicate)

2. TestPopulation (3 tests):
   - Population from benchmarks/fixtures/clean_app populates nodes + edges
   - graph_nodes/edges counts match flat registry counts
   - Populated nodes preserve name, file, line, node_type metadata

3. TestQueryCallers (3 tests):
   - query_callers(get_user_by_id) returns main() at depth 1
   - Caller entries include file and line for the calling site
   - BFS never returns the start node as its own caller

4. TestQueryCallees (3 tests):
   - query_callees(main) includes all expected function calls
   - Resolved callees have resolved=True; unresolved have resolved=False
   - Each resolved callee has a non-null node_id

5. TestRepopulation (3 tests):
   - Re-population does not duplicate rows
   - clear_graph_tables empties both tables
   - Clear then re-populate restores original counts

6. TestTracePilot (5 tests):
   - trace_via_graph up matches trace_via_flat up (get_user_by_id)
   - trace_via_graph down matches trace_via_flat down (main)
   - trace_via_graph both matches trace_via_flat both (process_data)
   - trace_symbol dispatcher defaults to graph backend
   - Dispatcher falls back to flat when graph tables are empty
   - use_graph=False forces the flat backend

All 20 tests pass in 0.62s.

Test results:
- New tests: 20 passed, 0 failed
- Pre-existing unrelated failures (test_hybrid_engine.py): 4 failed
  (confidence field assertions — unrelated to graph model)
- Pre-existing integration test (test_integration.py): fails due to
  OOM when scanning /home/z (environment issue, not a regression)
README.md:
- Bump version header to v8.2.
- Add 'Graph Data Model (v8.2)' to Features list.
- Add graph_model.py to the architecture tree under scripts/.

CHANGELOG.md:
- Add [8.2.0] Unreleased section documenting the graph data model addition:
  - New graph_model.py module with public API.
  - Graph schema (graph_nodes + graph_edges tables + 6 indexes).
  - trace --use-graph / --no-graph flags.
  - Changes to persistent_registry, scan, trace_engine.
  - Non-breaking guarantees.
  - Migration notes for engine authors (impact, circular, dependents next).

references/agent-integration.md:
- Add codelens.db to the .codelens/ file locations listing.
- Add section 10.4 'Graph Data Model (v8.2+)' with code examples for
  find_nodes_by_name, query_callers, query_callees, and a decision table
  for when to use the graph vs the flat registry.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@Wolfvin Wolfvin merged commit b77bf49 into main Jun 28, 2026
2 of 8 checks passed
@Wolfvin Wolfvin deleted the feat/issue-8-graph-data-model branch June 28, 2026 05:42
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
3.3% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ARCH] Replace flat registry with true graph data model (nodes + edges)

1 participant