feat(git): git-diff aware incremental re-index — closes #14#26
Merged
Conversation
Adds scripts/git_aware.py with: - get_current_sha(workspace) / get_current_branch(workspace) -> HEAD SHA / branch - get_changed_files(workspace, since_sha) -> git diff --name-only (HEAD or <sha>) - get_last_indexed_sha / set_last_indexed_sha -> registry_meta key/value - detect_branch_switch(workspace, db_path) -> True when HEAD moved AND branch name changed (catches git checkout, not same-branch commits) - rescan_recommended(workspace, db_path) -> True when branch switch OR changed files since last index (used by git-status command) - init_registry_meta(conn) -> creates registry_meta(key TEXT PK, value TEXT) All functions gracefully degrade to None / [] / False when git is unavailable or the workspace is not a git repo (no exceptions raised). persistent_registry.py: calls init_registry_meta(conn) during _init_schema so the table always exists by the time any git-aware function tries to read or write a bookmark. Additive — no existing table or column modified. Refs #14.
After a successful scan (full or incremental), if git is available and the workspace is a git repo, record the current HEAD SHA + branch in the registry_meta table via set_last_indexed_sha(). This is the 'bookmark' that the next 'scan --incremental' will diff against. Additive and fail-soft: if git_aware import fails, git is unavailable, or the workspace is not a git repo, the scan still succeeds — the bookmark is simply left as None. Scan output now includes a 'git' field with last_indexed_sha + last_indexed_branch so agents can verify the bookmark was recorded. Refs #14.
…lback) incremental.find_changed_files now tries the git-aware path FIRST: if a last_indexed_sha bookmark exists in registry_meta, uses 'git diff <sha> --name-only' + 'git ls-files --others' to enumerate exactly the files git knows changed (tracked changes + untracked new files). Deleted files (in diff but not on disk) are returned in the deleted slot so scan.py's existing deletion cleanup path runs. Fallback to the existing mtime-based detection is preserved for: - non-git workspaces - git unavailable - first scan (no bookmark yet) - any unexpected error in the git path Signature is backward-compatible: db_path is a new optional kwarg. scan.py passes db_path explicitly so the bookmark lookup is cheap. Also adds git_aware.get_untracked_files(workspace) helper used by the git path to catch newly-created files that haven't been git-added. Does NOT fix issue #25 (incremental graph population) — incremental scans still don't populate graph tables, only full scans do. Existing behavior preserved. Refs #14.
New 'git-status' command (auto-registered) shows the git-aware scan state in one call: - current HEAD SHA + branch - last-indexed SHA + branch (from registry_meta) - changed files count since last index - whether a branch switch was detected - whether a re-scan is recommended Designed for AI agents that need a 'do I need to re-scan?' check without running a full diff themselves. All fields degrade gracefully to None / 0 / False when git is unavailable or the workspace is not a git repo — status is always 'ok' in those cases (git-unavailable is reported via git_available=False, not an error). Refs #14.
Diff command now supports '--git-aware' which produces a single-call 'what changed + what's affected' view for AI agents: - changed_files: paths git knows changed since last_indexed_sha (tracked) + untracked new files - symbols: backend flat-registry nodes whose 'file' is in changed_files - impact: for each changed symbol, the direct callers from the graph tables (graph_model.query_callers max_depth=1). Empty when graph tables aren't populated (e.g. incremental-only scan — issue #25); that's a known gap, not a bug in this command. Default snapshot-diff behavior is unchanged — --git-aware is purely additive. Falls back to git_available=False when git is unavailable or the workspace is not a git repo (status stays 'ok'). Refs #14.
Watch command now supports --git-mode (default off) which switches from watchdog file events to git-diff polling. Every --interval seconds (default 2.0), runs 'git diff --name-only' + 'git ls-files --others' and re-indexes only the files git knows changed. Behavior matches the BOS decision: keep watchdog as the default real-time path (the issue title says 'replace file-watcher polling' but ripping out watchdog is risky), ADD git-awareness as an alternative/complement. Falls back to mtime polling when git is unavailable or the workspace is not a git repo. The new --interval flag is shared between git-mode and any future polling modes; --debounce remains for coalescing rapid changes. Refs #14.
Adds tests/test_git_aware.py with 32 test cases across 9 classes:
- TestCurrentSha: get_current_sha / get_current_branch in & out of git
- TestChangedFiles: get_changed_files / get_untracked_files (modified,
untracked, since-sha diff)
- TestRegistryMeta: set/get_last_indexed_sha round-trip, branch
persistence, None handling, clear-via-None
- TestBranchSwitch: false on no-bookmark, false on same-branch commit,
true after checkout, false when back on original, false outside git
- TestRescanRecommended: true when bookmark unset + in git, false when
bookmark matches HEAD, true when working tree dirty
- TestGitStatusCommand: status=ok with all fields, ok outside git,
changed_files_count after edit
- TestDiffGitAware: ok outside git, changed files in git repo, symbols
extracted from changed files (real scan), impact empty without graph
- TestIncrementalGitPath: uses git when bookmark set, mtime fallback
when no bookmark, mtime fallback outside git
- TestScanStoresBookmark: scan writes bookmark in git repo, no bookmark
outside git
All git operations use a temp directory + 'git init' so tests don't
depend on the CodeLens repo's git state. If git isn't installed, every
test is skipped via pytest.mark.skipif('git not available') (NOT failed).
Test results:
- test_git_aware.py: 32/32 pass in 1.75s
- broader suite: 568 pass, 4 pre-existing failures in test_hybrid_engine.py
(confidence-field assertions, NOT caused by this PR), 12 skipped
Refs #14.
README.md: - Add 'Git-Aware Re-Index (v8.2)' to Features section - Add 'git-status' and 'watch --git-mode/--interval' to Setup & Lifecycle command table - Add git_aware.py to the scripts/ architecture tree SKILL-QUICK.md: - Add 'git-status' to Setup & Lifecycle category (now 8+ commands) - Add 'watch [--git-mode] [--interval SECS]' to watch entry - Add 'what changed?' -> 'diff --git-aware' to trigger map - Add 'do I need to re-scan?' -> 'git-status' to trigger map CHANGELOG.md ([8.2.0] section): - Add 'Git-Aware Incremental Re-Index (issue #14)' subsection with Added / Changed / Non-Breaking / Known Gaps entries - Note that issue #25 (incremental graph population) is NOT made worse references/agent-integration.md (Section 9): - Add '9.4 Git-Aware Workflow (v8.2+)' with 5-step flow diagram covering git-status -> scan --incremental -> diff --git-aware -> impact review -> branch-switch full-scan - Document branch-switch detection semantics - Document watch --git-mode behavior - Note issue #25 known gap All doc changes are ADDITIVE — no existing sections rewritten. Other parallel workers (2-a, 2-b) may also add their own sections to the same files. Refs #14.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Wolfvin
pushed a commit
that referenced
this pull request
Jun 28, 2026
… (architecture) Navigation section: 10 → 11 commands (added architecture from #28) Pagination flags [--limit N] [--offset N] from #27 preserved on trace/search/symbols/outline/list diff gains [--git-aware] flag from #26 Total commands: 57 → 58 (graph-schema #17 + architecture #19) MCP tools: 55 → 56 (51 static + 5 dynamic, added codelens_architecture)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements issue #14 — git-diff aware incremental re-index. Adds optional git-diff awareness so incremental scans target exactly the files git knows changed (tracked + untracked), instead of relying solely on filesystem mtime polling. All features gracefully degrade to None / [] / False / mtime-fallback when git is unavailable or the workspace is not a git repo.
Closes #14.
Files Touched
Created
scripts/git_aware.py— new module: SHA tracking, changed-file detection, registry_meta bookmark, branch-switch detection, rescan predicatescripts/commands/git_status.py— newgit-statuscommand (auto-registered)tests/test_git_aware.py— 32 test cases across 9 classesModified
scripts/persistent_registry.py— callsinit_registry_meta(conn)during_init_schema(additive — only adds registry_meta table init block)scripts/incremental.py—find_changed_filestries git-diff path FIRST, falls back to mtime (signature backward-compatible; new optionaldb_pathkwarg)scripts/commands/scan.py— after successful scan, persistslast_indexed_sha+last_indexed_branch; scan output now includesgitfield (only my additions — no other logic touched)scripts/commands/diff.py— new--git-awareflag; default snapshot-diff behavior unchanged (purely additive)scripts/commands/watch.py— new--git-mode+--intervalflags; default watchdog behavior preserved (BOS decision: keep watchdog as default, ADD git-awareness as alternative)Docs (additive only — no existing sections rewritten)
README.md— new Features bullet + new command table rows +git_aware.pyin architecture treeSKILL-QUICK.md—git-statusin Setup category + new trigger map entriesCHANGELOG.md— newGit-Aware Incremental Re-Index (issue #14)subsection under[8.2.0]references/agent-integration.md— new9.4 Git-Aware Workflowsubsection with 5-step flowDesign Decisions (per BOS spec)
scripts/git_aware.pymodule with:get_current_sha,get_current_branch,get_changed_files(workspace, since_sha=None),get_untracked_files,get_last_indexed_sha,set_last_indexed_sha,get_last_indexed_branch,detect_branch_switch,rescan_recommended,init_registry_meta.registry_meta(key TEXT PRIMARY KEY, value TEXT)table for the SHA + branch bookmark — created byinit_registry_meta(conn), called byPersistentRegistry._init_schema.incremental.find_changed_filestries git FIRST — iflast_indexed_shais set, usesgit diff <sha> --name-only+git ls-files --others(catches tracked changes AND untracked new files). Falls back to existing mtime path when git unavailable, no bookmark yet, or any error. Mtime fallback stays.scan.pystores the bookmark after every successful scan — both full and incremental. The bookmark is the diff base for the next incremental.diff --git-awarereturns changed_files + symbols + impact in one call. Impact usesgraph_model.query_callers(empty when graph tables aren't populated — see Known Gaps).git-statuscommand — single-call 'do I need to re-scan?' check.watch --git-mode— pollsgit diff --name-onlyinstead of watchdog. Default stays watchdog (BOS decision).subprocessto callgit, no GitPython.from __future__ import annotationsfor forward refs.Test Results
tests/test_git_aware.py: 32/32 pass in 1.65spytest tests/ --ignore=tests/test_integration.py): 568 pass, 4 fail (pre-existing), 12 skipped in 10.20stest_hybrid_engine.py(confidence-field assertions) — pre-existing on main, NOT caused by this PR. Documented in worklog from worker-1.test_integration.pyis excluded (pre-existing OOM scanning/home/z).Graceful Degradation
Every git-aware feature degrades cleanly when git is unavailable or the workspace is not a git repo:
get_current_sha/get_current_branchNoneget_changed_files/get_untracked_files[]get_last_indexed_shaNone(db missing)set_last_indexed_shadetect_branch_switchFalserescan_recommendedFalsegit-statuscommandstatus=ok, git_available=Falsediff --git-awarestatus=ok, git_available=False, changed_files=[], symbols=[], impact=[]watch --git-modefind_changed_files(incremental)scan(bookmark write)Tests skip (not fail) when git is missing:
pytest.mark.skipif(not _git_available(), reason='git not available').Verified on a Temp Git Repo Fixture
The test suite creates a fresh
git inittemp directory per test and verifies end-to-end:git-statusreturnsstatus=okwith all fields populated on a real git repo,git_available=Falseoutside git.diff --git-awarereturns the changed file (module.py), the symbol defined in it (hellofromapp.py), and an emptyimpactarray (graph not populated by the test's incremental scan).find_changed_filesuses the git path when a bookmark is set, falls back to mtime when no bookmark.scanwrites the bookmark that matches the current HEAD SHA.detect_branch_switchreturns True aftergit checkout -b feature/x+ commit, False on same-branch commit, False when back on the original branch.Known Gaps (NOT made worse by this change)
graph_nodes+graph_edges); only full scans do.diff --git-awarereports an emptyimpactarray when graph tables aren't populated. This is a pre-existing gap tracked in [BUG] Incremental scan does not populate graph tables (only full scan does) #25 and is NOT made worse by this change — the incremental scan path is untouched, only the changed-file DETECTION layer (git vs mtime) is new.git-statuscommand'schanged_filesfield is currently NOT capped — for very dirty working trees this could be large. Future enhancement: add a--top Nflag (out of scope for [FEATURE] Git-diff aware incremental re-index (replace file-watcher polling) #14).Coordination with Parallel Workers
Two other workers are running in parallel on separate branches:
feat/issue-17-compact-output(compact format + pagination +graph-schemacommand) — file ownership respected: I did NOT touchscripts/formatters/or add--limit/--offsetto commands. I did NOT creategraph-schema.feat/issue-19-get-architecture(architecture command +architecture_engine.py) — file ownership respected: I did NOT createarchitecturecommand orarchitecture_engine.py.Shared docs (README, SKILL-QUICK, CHANGELOG, agent-integration.md) were edited additively — I only ADDED my own sections; existing content was not rewritten. Other workers can add their sections in parallel without conflict.
Commits
8 logical commits following the BOS workflow:
feat(git): add git_aware.py with SHA tracking + changed file detectionfeat(scan): store last_indexed_sha after scanfeat(incremental): use git-diff for changed file detection (mtime fallback)feat(cmd): add git-status commandfeat(diff): add --git-aware flag for changed symbols + impactfeat(watch): add --git-mode flag for git-diff pollingtest(git): add git_aware + git-status + diff --git-aware testsdocs(git): update README, SKILL-QUICK, CHANGELOG, agent-integration