Skip to content
104 changes: 104 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,110 @@ graph traversal logic.
population, query_callers, query_callees, re-population idempotency, and
the trace pilot A/B comparison.

### Git-Aware Incremental Re-Index (issue #14)

Replaces the file-watcher-only change detection with optional git-diff
awareness so incremental scans target exactly the files git knows changed
(tracked + untracked), instead of relying solely on filesystem mtimes.
All features gracefully degrade to None / [] / False / mtime-fallback
when git is unavailable or the workspace is not a git repo.

### Added (git-aware)

- **`scripts/git_aware.py`** — New module implementing the git-aware
change-detection layer:
- `get_current_sha(workspace)` / `get_current_branch(workspace)` —
HEAD SHA + branch (None when not a git repo).
- `get_changed_files(workspace, since_sha=None)` — `git diff
--name-only` (HEAD or `<sha>`).
- `get_untracked_files(workspace)` — `git ls-files --others
--exclude-standard` so newly-created (not-yet-added) files are
visible to incremental scans.
- `get_last_indexed_sha` / `set_last_indexed_sha` — registry_meta
key/value bookmark of the HEAD SHA + branch at the time of the last
successful scan.
- `detect_branch_switch(workspace, db_path)` — True when HEAD moved
AND the branch name changed (catches `git checkout`, not same-branch
commits).
- `rescan_recommended(workspace, db_path)` — True when a branch
switch is detected OR any changed files exist since the last index.
- `init_registry_meta(conn)` — creates the `registry_meta(key TEXT
PRIMARY KEY, value TEXT)` table (idempotent).

- **`scripts/commands/git_status.py`** — New `git-status` command
(auto-registered). Single-call "do I need to re-scan?" check for AI
agents. Reports: current_sha, current_branch, last_indexed_sha,
last_indexed_branch, changed_files_count, branch_switch_detected,
rescan_recommended. Always returns status=ok; git-unavailable is
reported via git_available=False (not an error).

- **`tests/test_git_aware.py`** — 32 test cases across 9 classes
(TestCurrentSha, TestChangedFiles, TestRegistryMeta, TestBranchSwitch,
TestRescanRecommended, TestGitStatusCommand, TestDiffGitAware,
TestIncrementalGitPath, TestScanStoresBookmark). All git operations
use a temp directory + `git init` — no dependency on the CodeLens
repo's git state. Tests skip with `pytest.skip('git not available')`
when git is missing.

### Changed (git-aware)

- **`scripts/incremental.py`** — `find_changed_files` now tries the
git-aware path FIRST: if a `last_indexed_sha` bookmark exists in
`registry_meta`, uses `git diff <sha> --name-only` + `git ls-files
--others` to enumerate exactly the files git knows changed. Deleted
files (in diff but not on disk) are returned in the deleted slot so
`scan.py`'s existing deletion-cleanup path runs unchanged. Falls back
to the existing mtime-based detection when git is unavailable, no
bookmark is stored, or any unexpected error occurs. Signature is
backward-compatible — `db_path` is a new optional kwarg.
- **`scripts/commands/scan.py`** — After a successful scan (full or
incremental), if git is available, persists `last_indexed_sha` +
`last_indexed_branch` via `set_last_indexed_sha()`. Scan output now
includes a `git` field with `{last_indexed_sha, last_indexed_branch}`
so agents can verify the bookmark was recorded. Fail-soft: if the
bookmark write fails, the scan still succeeds.
- **`scripts/commands/diff.py`** — New `--git-aware` flag. When set,
the diff command produces a single-call "what changed + what's
affected" view: changed_files (from git), symbols (from flat backend
registry, filtered to changed files), impact (callers from
`graph_model.query_callers` when graph tables are populated). Default
snapshot-diff behavior is unchanged — `--git-aware` is purely
additive. Falls back to `git_available=False` when git is unavailable
(status stays "ok").
- **`scripts/commands/watch.py`** — New `--git-mode` flag (default
off). When set, switches from watchdog file events to git-diff
polling: every `--interval` seconds (default 2.0), runs `git diff
--name-only` + `git ls-files --others` and re-indexes only the files
git knows changed. Falls back to mtime polling when git is
unavailable or the workspace is not a git repo. Default watchdog
behavior is preserved (BOS decision: keep watchdog as default, ADD
git-awareness as alternative).
- **`scripts/persistent_registry.py`** — Calls `init_registry_meta(conn)`
during `_init_schema` so the `registry_meta` table always exists by
the time any git-aware function tries to read or write a bookmark.
Additive — no existing table or column modified.

### Non-Breaking (git-aware)

- All 56 existing CLI commands continue to work unchanged.
- The git-aware layer is purely additive — when git is unavailable, all
functions return None / [] / False and the existing mtime path runs.
- The `registry_meta` table is additive — no existing table or column
was modified.
- Scan output gained a new top-level `git` field; existing fields are
untouched.
- The `diff --git-aware` flag is opt-in; default `diff` behavior is
unchanged.

### Known Gaps (NOT made worse by this change)

- **Issue #25 (incremental graph population)** — Incremental scans
still don't populate the graph tables (`graph_nodes` + `graph_edges`);
only full scans do. `diff --git-aware` reports an empty `impact` array
when graph tables aren't populated (e.g. after an incremental-only
scan). This is a pre-existing gap tracked in #25 and is NOT made
worse by this change.

### Changed

- **`scripts/persistent_registry.py`** — Calls `init_graph_schema(conn)`
Expand Down
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ CodeLens is an AI-native code intelligence platform that gives AI agents **full
- **Regex Fallback Parsers** — 28 additional languages supported via regex-based parsers (C, C++, Go, Java, Kotlin, Swift, Ruby, PHP, Scala, Dart, Elixir, Lua, R, Haskell, Nim, Objective-C, GDScript, Shell, Vim, Zig, and more)
- **Framework Auto-Detection** — React/Next.js, Vue, Svelte, Tailwind CSS, Express, Fastify, Koa, Hono, Django, Flask, FastAPI, Tauri, and more
- **Incremental Scanning** — Only re-parse changed files for speed, with SQLite persistent registry storage
- **Git-Aware Re-Index (v8.2)** — `scan --incremental` uses `git diff <last-indexed-sha> --name-only` to enumerate exactly the files git knows changed (mtime fallback when git unavailable). `git-status` reports the HEAD/last-indexed SHA + branch + changed-files count + re-scan recommendation in one call. `diff --git-aware` shows changed files + symbols + downstream caller impact. `watch --git-mode` polls `git diff --name-only` instead of watchdog file events. All features gracefully degrade when git is unavailable
- **Workspace Auto-Detect** — No need to specify workspace path if you're already in the project
- **AI-Optimized Output** — `--format ai` and `--lite` flags for token-efficient AI agent consumption
- **Auto-Fix Engine** — Confidence-scored auto-fixes with dry-run-by-default safety
Expand Down Expand Up @@ -82,7 +83,8 @@ python3 scripts/codelens.py query "myFunction" --lite
| `scan [workspace] [--incremental] [--full] [--max-files N]` | Scan workspace and build registry |
| `validate [workspace]` | Validate registry vs file system |
| `detect [workspace]` | Detect frameworks and show recommended config |
| `watch [workspace]` | Start file watcher for real-time registry updates |
| `watch [workspace] [--git-mode] [--interval SECS]` | Start file watcher (default: watchdog; `--git-mode` polls `git diff --name-only`) |
| `git-status [workspace]` | Show git-aware scan state: HEAD SHA, last-indexed SHA, changed files, re-scan recommendation |
| `migrate [workspace]` | Migrate JSON registry to SQLite persistent database |
| `serve` | Start MCP server for AI agent integration (JSON-RPC over stdio) |
| `lsp-status` | Check which LSP servers are available for `--deep` analysis |
Expand Down Expand Up @@ -229,6 +231,7 @@ codelens/
│ ├── incremental.py # Incremental scan support
│ ├── edge_resolver.py # Cross-file edge resolution
│ ├── graph_model.py # Graph data model (nodes + edges) — issue #8
│ ├── git_aware.py # Git-diff aware incremental re-index — issue #14
│ ├── search_engine.py # Regex code search
│ ├── trace_engine.py # Call chain tracing
│ ├── impact_engine.py # Change impact analysis
Expand Down
6 changes: 4 additions & 2 deletions SKILL-QUICK.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,8 @@ $CLI complexity --top 5 --lite # → Top 5 most complex, minimal output
| Intent | Command |
|--------|---------|
| Create/edit/delete code | `query` → write → `scan --incremental` |
| "what changed?" | `diff --git-aware` |
| "do I need to re-scan?" | `git-status` |
| "does this exist?" | `query --lite` |
| "who calls this?" | `trace --direction up` |
| "safe to delete?" | `impact` → `dead-code` |
Expand Down Expand Up @@ -106,8 +108,8 @@ $CLI complexity --top 5 --lite # → Top 5 most complex, minimal output

## All 56 Commands

### Setup & Lifecycle (8)
`init` · `scan [--incremental] [--max-files N] [--full]` · `validate` · `detect` · `watch [--debounce SECS]` · `migrate` · `serve` · `lsp-status`
### Setup & Lifecycle (8+)
`init` · `scan [--incremental] [--max-files N] [--full]` · `validate` · `detect` · `watch [--debounce SECS] [--git-mode] [--interval SECS]` · `git-status` · `migrate` · `serve` · `lsp-status`

### Pre-Write Safety (5)
`query "name" [--domain ...] [--fuzzy]` · `impact "name" [--action modify|delete]` · `refactor-safe "name" [--action rename|move]` · `guard (--pre|--post) --file PATH` · `check [--severity ...] [--max-findings N]`
Expand Down
59 changes: 59 additions & 0 deletions references/agent-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -1155,6 +1155,65 @@ Step 5: Final audit:
"After cleanup: 0 dead entries remaining."
```

### 9.4 Git-Aware Workflow (v8.2+)

CodeLens v8.2 adds optional git-diff awareness to incremental scans so
agents can target exactly the files git knows changed, instead of
re-parsing every file with a bumped mtime. All git-aware features
degrade gracefully to None / mtime-fallback when git is unavailable or
the workspace is not a git repo — no special-casing required in agent
code.

```
Step 1: codelens_git_status(workspace)
├── rescan_recommended: false ──► skip scan, registry is current
└── rescan_recommended: true ──► proceed to Step 2
Step 2: codelens_scan(workspace, incremental=True)
↑ uses `git diff <last_indexed_sha> --name-only` to pick files;
mtime fallback runs automatically when git unavailable
Step 3: codelens_diff(workspace, git_aware=True)
├── changed_files[] ← files git knows changed since last index
├── symbols[] ← backend nodes in those files
└── impact[] ← callers of those symbols (graph BFS)
Step 4: For each impact[].callers[] entry:
- Read the caller's file + line to understand the call site
- Decide if the change is safe (signature preserved) or
breaking (signature changed → run `refactor-safe` next)
Step 5: If branch switch detected (git checkout rewrote many files):
codelens_scan(workspace, incremental=False)
↑ full scan recommended — incremental may miss cross-file
edges that a checkout rewrote atomically
```

**Branch switch detection**: `git-status` reports `branch_switch_detected: true`
when the current HEAD SHA differs from `last_indexed_sha` AND the branch
name differs from `last_indexed_branch`. Same-branch commits do NOT
trigger this (SHA moves but branch doesn't) — only `git checkout` to a
different branch does.

**Watch mode**: `watch --git-mode` polls `git diff --name-only` every
`--interval` seconds (default 2.0) instead of using watchdog file
events. Useful when watchdog is unavailable or when the agent wants
git-aware delta instead of mtime-based change detection. Falls back to
mtime polling when git is unavailable.

**Known gap (issue #25)**: Incremental scans do NOT populate the graph
tables (`graph_nodes` + `graph_edges`) — only full scans do. When the
graph is empty, `diff --git-aware` returns an empty `impact` array.
This is a pre-existing gap tracked in #25 and is NOT made worse by the
git-aware layer.

---

## 10. Programmatic Registry File Access
Expand Down
Loading
Loading