Skip to content

feat(perf): RAM-first indexing — batch SQLite write at end of scan (closes #10 phase-1)#111

Merged
Wolfvin merged 1 commit into
mainfrom
feat/10-ram-first-indexing
Jun 30, 2026
Merged

feat(perf): RAM-first indexing — batch SQLite write at end of scan (closes #10 phase-1)#111
Wolfvin merged 1 commit into
mainfrom
feat/10-ram-first-indexing

Conversation

@Wolfvin

@Wolfvin Wolfvin commented Jun 30, 2026

Copy link
Copy Markdown
Owner

Summary

Implements issue #10: RAM-first indexing with batch SQLite write.

What changed

  • Collect graph_nodes / graph_edges in Python lists during scan (already in-memory from parsers; this PR makes the batch-write boundary explicit)
  • Single batch write at end of scan: one executemany() for nodes + one for edges, wrapped in a BEGIN EXCLUSIVE transaction (isolation_level=None + explicit BEGIN EXCLUSIVE / COMMIT)
  • Refactored populate_graph_tables (full scan) and incremental_graph_update (incremental scan) to collect all rows in memory BEFORE opening the EXCLUSIVE lock, minimizing lock-hold duration
  • New --scan-stats flag prints a two-line timing breakdown to stderr:
    Scan stats: 1240 files, 3091 nodes, 29285 edges
    Index time: 2.3s (parse: 1.8s, write: 0.5s)
    
  • --incremental still works — partial update path is preserved; the EXCLUSIVE-lock refactor applies to both full and incremental writes
  • Output to stderr so default scan output (stdout JSON / formatted text) is byte-identical to the pre-[PERF] RAM-first indexing pipeline: index in-memory, flush once to SQLite #10 behavior (backward compat)

Verification

  • Tests: 1008 passed, 87 skipped (12 new tests added in tests/test_scan_stats.py covering the --scan-stats flag, the BEGIN EXCLUSIVE batch-write atomicity, and the incremental-scan interaction)
  • Backward compat: default codelens scan output unchanged (verified by subprocess test that asserts no Scan stats: / Index time: lines on stderr without --scan-stats)
  • --incremental still works: existing tests/test_graph_incremental.py (18 tests) all pass
  • File headers preserved on all touched files (scan.py, graph_model.py) and added on the new test file
  • Pre-existing unrelated failure: tests/test_hybrid_engine.py::TestHybridIntegration::test_query_confidence_without_deep — verified to fail on main (commit 470e602) before this PR

Closes #10

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@sonarqubecloud

Copy link
Copy Markdown

@Wolfvin Wolfvin merged commit 85029a0 into main Jun 30, 2026
1 of 7 checks passed
@Wolfvin Wolfvin deleted the feat/10-ram-first-indexing branch July 1, 2026 06:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[PERF] RAM-first indexing pipeline: index in-memory, flush once to SQLite

1 participant