diff --git a/README.md b/README.md index c5e98d0c..f22726fe 100644 --- a/README.md +++ b/README.md @@ -2,12 +2,12 @@ > **Before an AI writes a new class/id/function, CodeLens must be checked. This is not optional.** -CodeLens is an AI-native code intelligence platform that gives AI agents **full visibility** into a codebase before they write any code. It prevents collision, overwrite of existing logic, security vulnerabilities, and dead code through 77 CLI commands, an MCP server with 75 tools (56 static + 19 dynamic), AST-based taint analysis, live CVE/OSV scanning, a plugin system with OWASP Top 10 + Compliance rule packs, a true graph data model (nodes + edges) for structural code queries, and token-efficient `--format compact` output for high-volume agent workflows (issue #17). +CodeLens is an AI-native code intelligence platform that gives AI agents **full visibility** into a codebase before they write any code. It prevents collision, overwrite of existing logic, security vulnerabilities, and dead code through 78 CLI commands, an MCP server with 76 tools (56 static + 20 dynamic), AST-based taint analysis, live CVE/OSV scanning, a plugin system with OWASP Top 10 + Compliance rule packs, a true graph data model (nodes + edges) for structural code queries, and token-efficient `--format compact` output for high-volume agent workflows (issue #17). ## Features -- **77 CLI Commands** — From basic scan/query to AST taint analysis, CVE scanning, plugin management, auto-fix, dashboards, CI/CD quality gates, and `graph-schema` for cheap graph-shape introspection -- **MCP Server (75 Tools)** — Native AI agent integration via Model Context Protocol (JSON-RPC over stdio), 56 statically-defined tools + 19 dynamically discovered, every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`) +- **78 CLI Commands** — From basic scan/query to AST taint analysis, CVE scanning, plugin management, auto-fix, dashboards, CI/CD quality gates, and `graph-schema` for cheap graph-shape introspection +- **MCP Server (76 Tools)** — Native AI agent integration via Model Context Protocol (JSON-RPC over stdio), 56 statically-defined tools + 20 dynamically discovered, every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`) - **Token-Efficient Compact Output (v8.2, issue #17)** — `--format compact` produces single-char-key JSON with abbreviated types, omitted null fields, and relative paths — ~50% smaller than `json` on real trace output. Combined with `--limit`/`--offset` pagination, 5 structural queries now cost <5k tokens (down from 30-80k) - **AST Taint Engine** — Tree-sitter based taint analysis with return-value propagation, scope hierarchy, and branch condition refinement - **Live CVE/OSV Scanning** — Real-time vulnerability data from OSV.dev API with SQLite cache, 9 ecosystems (PyPI, npm, crates.io, Go, Maven, NuGet, RubyGems, Pub, Hex) @@ -226,8 +226,8 @@ codelens/ │ ├── changelog.md # Older changelog (per-version highlights) │ └── agent-integration.md # AI agent integration guide ├── scripts/ -│ ├── codelens.py # CLI entry point (77 commands registered) -│ ├── mcp_server.py # MCP JSON-RPC server (75 tools) +│ ├── codelens.py # CLI entry point (78 commands registered) +│ ├── mcp_server.py # MCP JSON-RPC server (76 tools) │ ├── registry.py # Registry read/write/build │ ├── persistent_registry.py # SQLite persistent storage (WAL mode) │ ├── base_parser.py # Base tree-sitter parser diff --git a/SKILL-QUICK.md b/SKILL-QUICK.md index b17160fe..072ff9cb 100755 --- a/SKILL-QUICK.md +++ b/SKILL-QUICK.md @@ -116,7 +116,7 @@ $CLI list --limit 5 --offset 10 --format compact # → paginated + co | "Cross-file taint" | `dataflow` | `taint` (taint is single-file, AST-deep) | | "Auto-fix issues" | `fix` | `check` (check just gates, doesn't fix) | -## All 77 Commands +## All 78 Commands ### Setup & Lifecycle (8+) `init` · `scan [--incremental] [--max-files N] [--full]` · `registry-validate` · `detect` · `watch [--debounce SECS] [--git-mode] [--interval SECS]` · `git-status` · `migrate` · `serve` · `lsp-status` (issue #33: `codelens --lsp-status` top-level flag is an alias of `codelens lsp-status` — both delegate to `hybrid_engine.get_lsp_status()` and return the identical payload) @@ -148,9 +148,9 @@ $CLI list --limit 5 --offset 10 --format compact # → paginated + co ### Tooling (1) `plugin ` -**Total: 77 commands** (auto-registered via `commands/__init__.py`; rerun `python3 scripts/sync_command_count.py --apply` after adding/removing a command) +**Total: 78 commands** (auto-registered via `commands/__init__.py`; rerun `python3 scripts/sync_command_count.py --apply` after adding/removing a command) -## MCP Server (75 Tools) +## MCP Server (76 Tools) Start the MCP server for AI agent integration: @@ -158,9 +158,9 @@ Start the MCP server for AI agent integration: python3 scripts/codelens.py serve ``` -Exposes 75 tools as `codelens_` (e.g., `codelens_query`, `codelens_taint`, `codelens_graph_schema`, `codelens_architecture`, `codelens_resolve_types`, `codelens_git_status`): +Exposes 76 tools as `codelens_` (e.g., `codelens_query`, `codelens_taint`, `codelens_graph_schema`, `codelens_architecture`, `codelens_resolve_types`, `codelens_git_status`): - 50 statically-defined tools (full JSON schemas in `mcp_server.py`) -- 19 dynamically-discovered tools (auto-discovered from `COMMAND_REGISTRY`; long-running `watch` and `serve` are excluded) +- 20 dynamically-discovered tools (auto-discovered from `COMMAND_REGISTRY`; long-running `watch` and `serve` are excluded) - Every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`). Use `format: "compact"` for token-efficient responses (~50% smaller than `json`). - `watch` and `serve` itself are excluded (long-running) diff --git a/SKILL.md b/SKILL.md index ad631de2..a1341862 100755 --- a/SKILL.md +++ b/SKILL.md @@ -1,10 +1,10 @@ --- name: codelens description: > - CodeLens — AI-Native Code Intelligence. 77 commands for AI-powered code analysis, + CodeLens — AI-Native Code Intelligence. 78 commands for AI-powered code analysis, security auditing, quality scoring, AST-based taint analysis, live CVE scanning, and pre-write safety checks. Supports 28+ languages with tree-sitter + regex - fallback parsing. MCP server exposes 75 tools for AI agent integration. + fallback parsing. MCP server exposes 76 tools for AI agent integration. For quick command reference with validated output schemas, see SKILL-QUICK.md. For version history, see CHANGELOG.md. --- diff --git a/docs/sync/staleness-banner.md b/docs/sync/staleness-banner.md new file mode 100644 index 00000000..3dfb81ad --- /dev/null +++ b/docs/sync/staleness-banner.md @@ -0,0 +1,235 @@ +# Per-File Staleness Banner (issue #66 Phase 1) + +> **Status:** Phase 1 shipped. Phases 2–5 tracked as follow-up issues. +> **Last updated:** 2026-07-02 + +## Alasan Dibuat + +After a `codelens scan`, the index is a snapshot. If the user edits a +file after the scan, queries against the index may return outdated +symbol locations, dead-code verdicts, or stale call graphs. Before +Phase 1, the MCP server happily served stale results with no warning — +agents acted on outdated data without knowing. + +Phase 1 adds a **per-file staleness banner** that detects when indexed +files have been edited since the last scan and surfaces a warning to +the agent before the tool's actual output. The banner is prepended to +every read-tool response (suppressed on `scan`/`init` — those are the +fix path, not analysis calls). + +## Arsitektur + +``` +scripts/sync/ +├── __init__.py # Re-exports public API +└── pending.py # StaleFileDetector + detect_stale_files() + # + format_staleness_banner() + # In-memory Dict[str, float] cache, thread-safe + # via threading.Lock, 5s TTL per workspace. + +scripts/commands/ +└── staleness.py # `codelens staleness` CLI command — manual check + # + full list when MCP banner truncates to 10 + +scripts/mcp_server.py # MCPServer._staleness_detector (lazy) + # + _attach_staleness_banner() — prepends banner + # + _invalidate_staleness_cache() — after scan + +tests/ +└── test_staleness.py # 41 tests — detection, cache, thread safety, + # banner formatting, CLI, MCP integration +``` + +## Detection algorithm + +``` +1. Load stored mtimes from .codelens/mtimes.json + (source of truth — written by incremental.save_mtimes() on every scan) +2. For each indexed file: + a. os.stat() — if file gone, skip (deletion is Phase 2's concern) + b. If |current_mtime - stored_mtime| <= 0.001s, skip (filesystem noise) + c. If confirm_with_hash=True (default): + - Compute SHA-256 of current content + - Load stored hash from SQLite `files` table (if available) + - If hashes match, skip (mtime changed but content identical — e.g. `touch`) + - If hashes differ or stored hash unavailable, flag as stale + d. Else (confirm_with_hash=False): flag on mtime change alone +3. Sort by edit_age ascending (most recent edit first) +4. Return tuple of StaleFile records +``` + +**Why mtimes.json (not SQLite `files` table) as the source of truth?** +`mtimes.json` is written by every scan, including workspaces that use +the legacy JSON registry (pre-v8.2). The SQLite `files` table is only +populated when the persistent registry is active. Using `mtimes.json` +keeps Phase 1 working on every workspace configuration. + +**Why 0.001s mtime tolerance?** +Filesystems with coarse mtime resolution (FAT32, some network shares) +can report mtimes that differ by sub-millisecond even when content is +identical. The stored mtime comes from `os.path.getmtime()` which +returns a float; comparing with a small epsilon avoids false positives +from filesystem noise. + +## Cache + +`StaleFileDetector` caches results per workspace for 5 seconds +(`DETECTOR_CACHE_TTL_SECONDS`). The cache is: + +- **Thread-safe** — protected by `threading.Lock`. The MCP server + dispatches tool calls in a thread pool, so concurrent calls must not + race or duplicate work. +- **Per-workspace** — keyed by absolute workspace path. Multiple + workspaces don't interfere. +- **Invalidated on scan** — `MCPServer._invalidate_staleness_cache()` + is called after a successful `scan` command. The scan refreshes the + index, so any cached staleness verdict is now stale itself. + +The cache exists because walking 10k+ files on every tool call would +add ~50 ms of latency. A 5-second TTL keeps the banner fresh enough +for interactive use (a user editing a file should see the banner +within seconds) without re-stat-ing the whole tree on every query. + +## Banner shape + +``` +⚠️ Some files referenced below were edited since the last index sync. +The index may be stale for these files — re-run `codelens scan` to refresh. +Stale files (showing 3 of 3, most recent first): + • path/to/file.py (edited 2.3s ago, content differs) + • other.js (edited 1m 12s ago, content differs) + • third.ts (edited 5m 0s ago, size/mtime differ) +``` + +- **Plain text** (not markdown) — renders correctly in both terminal + output and MCP tool-response content blocks. +- **⚠️ marker** — agents can pattern-match on it. +- **"content differs"** vs **"size/mtime differ"** — distinguishes + hash-confirmed staleness from mtime-only staleness. +- **Most recent first** — the agent sees the most relevant context first. +- **Truncates to 10 files** with "and N more" — keeps the banner + actionable; the full list is available via `codelens staleness`. + +## MCP integration + +`MCPServer._handle_tools_call` calls `_attach_staleness_banner()` on +three response paths: + +1. **Cached response** — banner attached (the workspace's staleness is + independent of whether the tool result was cached). +2. **Fresh success** — banner attached, unless the command is `scan` or + `init` (those are the remediation path). +3. **Error response** — banner attached, unless `scan`/`init`. If the + user is in a stale workspace, that context is more useful than the + error itself — the error is almost certainly caused by the stale + index. + +The banner is prepended to the first content block's `text` field AND +attached as a structured `response["_staleness"]` field. Both paths +ensure the warning surfaces — agents that pattern-match on JSON keys +see the structured field; agents that read only the text see the +prepended banner. + +After a successful `scan`, `_invalidate_staleness_cache(workspace)` is +called so the next read tool re-probes against the fresh index. + +## CLI command + +```bash +# Check staleness (text output, default) +codelens staleness [workspace] + +# JSON output for scripts +codelens staleness [workspace] --format json + +# Skip SHA-256 confirmation (faster, false-positive on `touch`) +codelens staleness [workspace] --no-confirm-hash + +# Show more files in the banner (default 10) +codelens staleness [workspace] --limit 50 +``` + +## Definition of Done (Phase 1, dari issue) + +- [x] In-memory `Dict[str, float]` (path → edit_timestamp), thread-safe via `threading.Lock` +- [x] Walk indexed file list with `os.stat(path)` to compare `(st_size, st_mtime_ns)` +- [x] Re-compute content-hash only when size/mtime changed +- [x] Prepend `⚠️ Some files referenced below were edited since the last index sync…` banner to MCP responses +- [x] Surface non-referenced pending files as small footer (the "and N more" line + `codelens staleness` for full list) +- [x] New file: `scripts/sync/pending.py` + +Phase 2 (connect-time catch-up), Phase 3 (native file watcher), and +Phase 5 (anonymous telemetry) are deferred to follow-up issues. + +## Design decisions + +1. **Why a separate `scripts/sync/` subpackage?** + Staleness and worktree mismatch (Phase 4, separate PR #154) are + independent concerns that share only the "index vs working tree" + theme. A package keeps them discoverable without forcing them into + one file (single-responsibility rule). + +2. **Why lazy construction of the detector in MCPServer?** + Keeps the import out of the server's startup path. If the sync + subpackage ever fails to import (e.g. a missing dependency in a + stripped-down install), the server still starts and only staleness + detection is degraded. + +3. **Why prepend (not append) the banner?** + Agents read tool output top-to-bottom. If the banner is at the + bottom, the agent may have already acted on stale data before + reaching it. Prepending ensures the warning is the first thing the + agent sees. + +4. **Why both structured `_staleness` field AND prepended text?** + Different agents consume tool output differently. Some + pattern-match on JSON keys (those see the structured field). Others + read only the text content (those see the prepended banner). Both + paths ensure the warning surfaces without requiring agents to + change. + +5. **Why is `content_hash_changed` a tri-state (True/False/None)?** + - `True` — size/mtime differ AND content hash differs (definitely stale) + - `False` — size/mtime differ BUT content hash matches (not stale, e.g. `touch`) + - `None` — size/mtime differ, no stored hash available to confirm + (probably stale, but can't be sure). The banner says "size/mtime + differ" rather than "content differs" in this case. + +6. **Why sort ascending by edit_age (smallest first)?** + `edit_age = now - current_mtime`. A file edited 1s ago has age=1; + a file edited 10s ago has age=10. Ascending puts age=1 first → + most recent first, matching the banner text. + +## Testing + +``` +PYTHONUTF8=1 PYTHONPATH=scripts python3 -m pytest tests/test_staleness.py -v +``` + +41 tests, all network-free and filesystem-light. Tests create small +temporary workspaces with synthetic `mtimes.json` files — no real +CodeLens scan is needed. Coverage: + +- Basic detection (mtime change, deleted files, empty workspace) +- Content-hash confirmation (touch without content change) +- Cache (TTL, invalidation, all-workspace invalidation) +- Thread safety (20 concurrent calls) +- Banner formatting (single file, truncation, age format) +- CLI command (registration, JSON/text output, --no-confirm-hash) +- MCP integration (prepend on read tools, suppress on scan/init, + invalidate after scan, init failure isolation) + +## Phases 2–5 (deferred) + +| Phase | Scope | Status | +|-------|----------------------------------------------------------|----------------| +| 2 | Connect-time catch-up — content-hash reconciliation on MCP reconnect | Not started | +| 3 | Native file watcher (FSEvents/inotify/ReadDirectoryChangesW) | Not started | +| 4 | Worktree mismatch detection (PR #154) | PR ready | +| 5 | Anonymous opt-in telemetry | Not started | + +Phase 2 will extend `StaleFileDetector` to also run content-hash +reconciliation on MCP reconnect, blocking the first query until +catch-up finishes (or 5s timeout, then proceed with stale + banner). +The `StaleFileDetector` cache and `StaleFile` data structure are +designed to accommodate this without API change. diff --git a/pyproject.toml b/pyproject.toml index 4f823bb4..472fd28e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta" [project] name = "codelens" version = "8.2.0" -description = "Live Codebase Reference Intelligence — 77 commands for AI-powered code analysis, security auditing, and quality scoring" +description = "Live Codebase Reference Intelligence — 78 commands for AI-powered code analysis, security auditing, and quality scoring" readme = "README.md" license = {text = "MIT"} requires-python = ">=3.8" diff --git a/scripts/commands/staleness.py b/scripts/commands/staleness.py new file mode 100644 index 00000000..8b82fb6a --- /dev/null +++ b/scripts/commands/staleness.py @@ -0,0 +1,143 @@ +"""Staleness command — list files whose index entry is stale (issue #66 Phase 1). + +What this command does +----------------------- +``codelens staleness`` walks the workspace's indexed file list (from +``.codelens/mtimes.json``) and compares each file's current ``os.stat()`` +against the stored scan-time values. Files whose size/mtime differ — and +whose SHA-256 content hash also differs when ``--confirm-with-hash`` is +set (default) — are reported as stale. + +The command is the CLI analogue of the MCP staleness banner (issue #66 +Phase 1). Use it to: + +* Manually check staleness before running a query (e.g. in a pre-commit hook). +* Get the full list of stale files when the MCP banner truncates to 10. +* Debug why the banner is or isn't appearing. + +Output shape (JSON):: + + { + "status": "ok", + "workspace": "/abs/path", + "stale_count": 3, + "stale_files": [ + {"rel_path": "...", "edit_age_seconds": 12.3, "content_hash_changed": true, ...}, + ... + ], + "banner": "⚠️ Some files referenced below ..." + } + +When ``--format text`` (default), prints the banner + a summary line. +""" + +from __future__ import annotations + +import argparse +import os +from typing import Any, Dict + +from commands import register_command + + +def add_args(parser: argparse.ArgumentParser) -> None: + parser.add_argument( + "workspace", + nargs="?", + default=None, + help="Path to workspace root (auto-detected if omitted)", + ) + parser.add_argument( + "--no-confirm-hash", + action="store_true", + default=False, + help="Skip SHA-256 content-hash confirmation (faster, but " + "false-positive on `touch` or `git checkout` of identical " + "content). Default: hash-confirm enabled.", + ) + parser.add_argument( + "--max-files", + type=int, + default=10_000, + help="Safety cap on number of indexed files to walk (default: 10000).", + ) + parser.add_argument( + "--limit", + type=int, + default=10, + help="Max number of stale files to list in the banner (default: 10).", + ) + parser.add_argument( + "--format", + choices=["text", "json"], + default="text", + help="Output format (default: text).", + ) + + +def execute(args: argparse.Namespace, workspace: str) -> Dict[str, Any]: + """Execute the staleness command.""" + if not workspace: + return { + "status": "error", + "error": "workspace is required (pass as arg or set CODELENS_WORKSPACE)", + } + + # Lazy import so the command module is importable even if the sync + # subpackage failed to load (defensive — shouldn't happen). + try: + from sync.pending import detect_stale_files, format_staleness_banner + except ImportError as exc: + return { + "status": "error", + "error": f"sync subpackage not importable: {exc}", + } + + confirm_with_hash = not getattr(args, "no_confirm_hash", False) + max_files = getattr(args, "max_files", 10_000) or 10_000 + limit = getattr(args, "limit", 10) or 10 + fmt = getattr(args, "format", "text") + + try: + stale = detect_stale_files( + workspace, + confirm_with_hash=confirm_with_hash, + max_files=max_files, + ) + except Exception as exc: + # Defensive: any unexpected error in staleness detection should + # surface a clear error, not crash the CLI. + return { + "status": "error", + "error": f"staleness detection failed: {exc}", + "error_type": type(exc).__name__, + } + + banner = format_staleness_banner(stale, limit=limit) + stale_dicts = [sf.as_dict() for sf in stale] + + result: Dict[str, Any] = { + "status": "ok", + "workspace": os.path.abspath(workspace), + "stale_count": len(stale), + "stale_files": stale_dicts, + "banner": banner, + } + + if fmt == "text": + if banner: + print(banner) + else: + print(f"No stale files in {workspace} (index is fresh).") + print() + print(f"Total stale: {len(stale)}") + + return result + + +register_command( + "staleness", + "List files whose index entry is stale (issue #66 Phase 1)", + add_args, + execute, +) diff --git a/scripts/graph_model.py b/scripts/graph_model.py index 793db3fd..12f78233 100644 --- a/scripts/graph_model.py +++ b/scripts/graph_model.py @@ -13,7 +13,7 @@ - New tables `graph_nodes` and `graph_edges` are additive (prefixed `graph_` to avoid colliding with any existing table name). - The flat registry tables and JSON files are untouched. -- All 77 existing CLI commands continue to work unchanged. +- All 78 existing CLI commands continue to work unchanged. Schema: graph_nodes( diff --git a/scripts/mcp_server.py b/scripts/mcp_server.py index b50a0b5f..78f43d09 100644 --- a/scripts/mcp_server.py +++ b/scripts/mcp_server.py @@ -1326,6 +1326,12 @@ def __init__(self, watch: bool = False): # ``{}`` (empty dict) means "probed, no mismatch"; a # populated dict means "probed, mismatch present". self._worktree_mismatch_cache: Dict[str, Optional[Dict[str, Any]]] = {} + # Staleness detector (issue #66 Phase 1). Holds a per-workspace + # cache with a short TTL so we don't re-stat the whole tree on + # every MCP tool call. The detector is thread-safe (internal + # ``threading.Lock``); the MCP server dispatches tool calls in + # a thread pool, so this matters. + self._staleness_detector = None # lazy — see _get_staleness_detector # ─── Lifecycle ──────────────────────────────────────── @@ -1707,6 +1713,12 @@ def _handle_tools_call(self, params: Dict[str, Any]) -> Dict[str, Any]: # the call. Mutating commands (scan, init) skip the banner # because they're the user's fix path. self._attach_worktree_banner(response, workspace) + # Issue #66 Phase 1: attach staleness banner on cached + # read-tool responses too. The banner is a property of + # the workspace, not the call — cached responses get + # the same warning as fresh ones. + if cmd_name not in ("scan", "init"): + self._attach_staleness_banner(response, workspace) return response # Execute the command @@ -1728,6 +1740,10 @@ def _handle_tools_call(self, params: Dict[str, Any]) -> Dict[str, Any]: # mismatch is now resolved and the banner should # disappear.) self._worktree_mismatch_cache.pop(os.path.abspath(workspace), None) + # Issue #66 Phase 1: a scan refreshes the index, so any + # cached staleness verdict is now stale itself. Drop it + # so the next read tool re-probes against the new state. + self._invalidate_staleness_cache(workspace) response = { "content": [{ @@ -1751,6 +1767,13 @@ def _handle_tools_call(self, params: Dict[str, Any]) -> Dict[str, Any]: # the user's remediation path, not analysis calls. if cmd_name not in ("scan", "init"): self._attach_worktree_banner(response, workspace) + # Issue #66 Phase 1: attach staleness banner on read tools. + # Skip mutating commands (scan/init) — they're the user's + # remediation path, not analysis calls. The banner on a + # scan response would be noise (the scan just fixed the + # staleness). + if cmd_name not in ("scan", "init"): + self._attach_staleness_banner(response, workspace) return response @@ -1777,6 +1800,12 @@ def _handle_tools_call(self, params: Dict[str, Any]) -> Dict[str, Any]: # than the error itself. The error is almost certainly # caused by the wrong index being loaded. self._attach_worktree_banner(response, workspace) + # Issue #66 Phase 1: still attach the staleness banner on + # error responses — if the user is in a stale workspace, + # that context is more useful than the error itself. The + # error is almost certainly caused by the stale index. + if cmd_name not in ("scan", "init"): + self._attach_staleness_banner(response, workspace) return response # ─── Hook integration (issue #47) ───────────────────── @@ -1948,6 +1977,108 @@ def _attach_worktree_banner(self, response: Dict[str, Any], workspace: str) -> N # Banner attachment must never break a tool response. pass + # ─── Staleness banner (issue #66 Phase 1) ────────────────── + + def _get_staleness_detector(self): + """Lazily construct the :class:`sync.pending.StaleFileDetector`. + + Lazy construction keeps the import out of the server's startup + path — if the sync subpackage ever fails to import (e.g. a + missing dependency in a stripped-down install), the server + still starts and only staleness detection is degraded. + """ + if self._staleness_detector is None: + try: + from sync.pending import StaleFileDetector + self._staleness_detector = StaleFileDetector() + except Exception as exc: + # Don't cache the failure — next call may succeed if + # the import error was transient (e.g. a filesystem + # race during package install). + print( + f"[CodeLens MCP] staleness detector init failed: {exc}", + file=sys.stderr, + ) + return None + return self._staleness_detector + + def _attach_staleness_banner( + self, response: Dict[str, Any], workspace: str + ) -> None: + """Prepend a staleness banner to the tool response text. + + Called on every read-tool call (i.e. not ``scan`` / ``init``). + The banner is prepended to the first content block's ``text`` + field so the agent sees it before the tool's actual output. + + Why prepend (not append)? + Agents read tool output top-to-bottom. If the banner is at + the bottom, the agent may have already acted on stale data + before reaching it. Prepending ensures the warning is the + first thing the agent sees. + + Why mutate the response in place? + The response is constructed fresh on every call (no shared + state between calls). Mutating it is simpler and faster + than building a wrapper. + + Failure isolation: + Any exception in staleness detection is caught and logged + to stderr. The banner is a nice-to-have — it must never + break a tool call. This matches the pattern used by + ``_attach_pending_hooks`` above. + """ + if not workspace: + return + try: + detector = self._get_staleness_detector() + if detector is None: + return + stale = detector.detect(workspace) + if not stale: + return + from sync.pending import format_staleness_banner + banner = format_staleness_banner(stale) + if not banner: + return + + # Attach the banner as a structured field so agents that + # pattern-match on JSON keys can detect it, AND prepend it + # to the first text content block so agents that read only + # the text see it. Both paths ensure the warning surfaces. + response["_staleness"] = { + "stale_count": len(stale), + "banner": banner, + } + content = response.get("content") + if isinstance(content, list) and content: + first = content[0] + if isinstance(first, dict) and "text" in first: + first["text"] = banner + "\n\n" + first["text"] + except Exception as exc: + print( + f"[CodeLens MCP] staleness banner attach failed: {exc}", + file=sys.stderr, + ) + + def _invalidate_staleness_cache(self, workspace: str) -> None: + """Drop cached staleness results for ``workspace`` after a scan. + + A scan refreshes the index — any cached staleness verdict is now + stale itself. Called from ``_handle_tools_call`` after a + successful ``scan`` command. + """ + try: + detector = self._staleness_detector + if detector is not None: + detector.invalidate(workspace) + except Exception as exc: + # Non-fatal — the cache will expire on its own TTL. + print( + f"[CodeLens MCP] staleness cache invalidate failed: {exc}", + file=sys.stderr, + ) + def _send_hook_notification(self, notification: Dict[str, Any]) -> None: """Push a hook notification to the agent via stdout JSON-RPC. diff --git a/scripts/sync/__init__.py b/scripts/sync/__init__.py index f05fe77a..b2f7e31d 100644 --- a/scripts/sync/__init__.py +++ b/scripts/sync/__init__.py @@ -1,3 +1,20 @@ +# @WHO: scripts/sync/__init__.py +# @WHAT: CodeLens sync subpackage — index-vs-worktree reconciliation helpers (issue #66) +# @PART: sync +# @ENTRY: - +# +# This package hosts modules that detect drift between the CodeLens index +# and the working tree. Phase 1 (issue #66) ships ``pending`` — per-file +# staleness detection. Phase 4 ships ``worktree`` — git worktree index +# mismatch detection. +# +# Why a subpackage (not a single module): +# - Staleness and worktree mismatch are independent concerns that share +# only the "index vs working tree" theme. Forcing them into one file +# would violate the single-responsibility rule. +# - Future phases (connect-time catch-up, native file watcher) will add +# more modules here. A package keeps the surface area discoverable. + """CodeLens sync subpackage — workspace ↔ index reconciliation helpers. This package contains modules that detect and repair drift between the @@ -13,6 +30,12 @@ Modules ------- +``pending`` — per-file staleness detection (issue #66 Phase 1). + Walks indexed files, compares ``(st_size, st_mtime_ns)`` against the + stored scan-time values, and re-computes a SHA-256 content hash only + when size/mtime changed. Returns a list of stale files + a formatted + banner string suitable for prepending to MCP responses. + ``worktree`` — git worktree ↔ index mismatch detection (issue #66 Phase 4). Why a subpackage? @@ -24,4 +47,28 @@ data, which is worse than no data at all. """ -__all__ = ["worktree"] +from .pending import ( # noqa: F401 + StaleFile, + StaleFileDetector, + detect_stale_files, + format_staleness_banner, + STALE_FILE_LIMIT_DEFAULT, +) +from .worktree import ( # noqa: F401 + detect_worktree_index_mismatch, + format_worktree_banner, + format_worktree_warning, +) + +__all__ = [ + # Phase 1 — per-file staleness + "StaleFile", + "StaleFileDetector", + "detect_stale_files", + "format_staleness_banner", + "STALE_FILE_LIMIT_DEFAULT", + # Phase 4 — worktree index mismatch + "detect_worktree_index_mismatch", + "format_worktree_banner", + "format_worktree_warning", +] diff --git a/scripts/sync/pending.py b/scripts/sync/pending.py new file mode 100644 index 00000000..95763da0 --- /dev/null +++ b/scripts/sync/pending.py @@ -0,0 +1,566 @@ +# @WHO: scripts/sync/pending.py +# @WHAT: Per-file staleness detection — index vs working tree drift banner (issue #66 Phase 1) +# @PART: sync +# @ENTRY: detect_stale_files(), format_staleness_banner(), StaleFileDetector +# +# Issue #66 Phase 1 — Per-file staleness banner. +# +# What this module does +# --------------------- +# After a ``codelens scan``, every indexed file has a stored ``mtime`` +# (in ``.codelens/mtimes.json``) and a stored content hash (in the +# SQLite ``files`` table when available). If the user edits a file +# after the scan, the index is stale — queries against it may return +# outdated symbol locations, dead-code verdicts, or stale call graphs. +# +# This module walks the indexed file list, compares the current +# ``(st_size, st_mtime_ns)`` from ``os.stat()`` against the stored +# values, and — only when those differ — re-computes a SHA-256 content +# hash to confirm the file actually changed (mtime can change without +# content change, e.g. ``touch`` or ``git checkout`` of identical +# content). Files that genuinely changed are returned as ``StaleFile`` +# records. +# +# The MCP server (issue #66 Phase 1 wiring) calls +# :func:`detect_stale_files` on every read-tool call, caches the result +# per workspace for a short TTL, and prepends +# :func:`format_staleness_banner` to the response so the agent knows +# the index is stale before acting on it. +# +# Why in-memory ``Dict[str, float]`` + ``threading.Lock`` (per issue spec) +# ---------------------------------------------------------------------- +# The issue spec calls for "in-memory ``Dict[str, float]`` (path → +# edit_timestamp), thread-safe via ``threading.Lock``". This module +# honours that: :class:`StaleFileDetector` holds the per-workspace +# detection cache behind a lock so concurrent MCP tool calls (the +# server dispatches them in a thread pool) don't race. +# +# The lock protects the *cache*, not the file-system walk — the walk +# itself is read-only and safe to run concurrently. The cache exists +# because walking a large codebase (10k+ files) on every tool call +# would dominate latency; a 5-second TTL keeps the banner fresh +# without re-stat-ing the whole tree on every query. +# +# Phase 2 (connect-time catch-up) will extend this module to also run +# content-hash reconciliation on MCP reconnect. Phase 1 deliberately +# stops at detection + banner — the fix path (re-scan) is the user's +# responsibility, surfaced by the banner. + +"""Per-file staleness detection — index vs working tree drift banner. + +Public entry points:: + + from sync.pending import detect_stale_files, format_staleness_banner + + stale = detect_stale_files(workspace="/path/to/ws") + if stale: + banner = format_staleness_banner(stale) + # prepend banner to MCP response text +""" + +from __future__ import annotations + +import hashlib +import json +import os +import threading +import time +from dataclasses import dataclass, field +from typing import Any, Dict, List, Optional, Tuple + +from utils import logger + +# ─── Constants ───────────────────────────────────────────────────────────── + +# How many stale files to list by name in the banner before collapsing +# to "and N more". 10 is the issue spec's implicit default — enough to +# be actionable, small enough that the banner doesn't dominate the +# response payload. +STALE_FILE_LIMIT_DEFAULT = 10 + +# Cache TTL for :class:`StaleFileDetector`. 5 seconds balances freshness +# (a user editing a file should see the banner within seconds) against +# cost (re-stat-ing 10k files on every MCP call would add ~50 ms). +DETECTOR_CACHE_TTL_SECONDS = 5.0 + +# Mtime tolerance in seconds. Filesystems with coarse mtime resolution +# (e.g. FAT32, some network shares) can report mtimes that differ by +# sub-millisecond even when content is identical. The stored mtime in +# mtimes.json comes from os.path.getmtime() which returns a float, so +# we compare with a small epsilon. Anything larger than this is a real +# edit; anything smaller is filesystem noise. +_MTIME_TOLERANCE_SECONDS = 0.001 + + +# ─── Data types ──────────────────────────────────────────────────────────── + + +@dataclass(frozen=True) +class StaleFile: + """One file whose index entry is stale. + + ``rel_path`` is relative to the workspace root (matches the key + format in ``mtimes.json``). ``edit_age_seconds`` is the wall-clock + time between the stored mtime and the current mtime — useful for + the banner so the agent can tell "edited 2 seconds ago" from + "edited 2 hours ago". ``size_changed`` is True when ``st_size`` + differs (a stronger signal than mtime alone — mtime can change + without size changing, e.g. ``touch``). + """ + + rel_path: str + stored_mtime: float + current_mtime: float + stored_size: Optional[int] + current_size: int + edit_age_seconds: float + size_changed: bool + # Only populated when the caller asked for content-hash confirmation + # (see ``StaleFileDetector.detect`` with ``confirm_with_hash=True``). + # None means "size/mtime differ but we didn't hash the file" — the + # caller should treat it as "probably stale" rather than "definitely + # stale". The MCP banner path always sets confirm_with_hash=True. + content_hash_changed: Optional[bool] = None + + def as_dict(self) -> Dict[str, Any]: + return { + "rel_path": self.rel_path, + "stored_mtime": self.stored_mtime, + "current_mtime": self.current_mtime, + "stored_size": self.stored_size, + "current_size": self.current_size, + "edit_age_seconds": round(self.edit_age_seconds, 1), + "size_changed": self.size_changed, + "content_hash_changed": self.content_hash_changed, + } + + +# ─── Helpers ─────────────────────────────────────────────────────────────── + + +def _load_indexed_mtimes(workspace: str) -> Dict[str, float]: + """Load the stored scan-time mtimes for the workspace. + + Source of truth: ``/.codelens/mtimes.json`` — written by + :func:`incremental.save_mtimes` on every scan. Returns an empty dict + when the file is missing (workspace never scanned) or corrupt (the + scan will rebuild it). + + Why mtimes.json and not the SQLite ``files`` table? + ``mtimes.json`` is written by every scan, including workspaces + that use the legacy JSON registry (pre-v8.2). The SQLite + ``files`` table is only populated when the persistent registry + is active. Using ``mtimes.json`` as the source of truth keeps + Phase 1 working on every workspace configuration. + """ + try: + from incremental import load_mtimes + return load_mtimes(workspace) + except Exception as exc: + # Don't let a bug in incremental.load_mtimes break staleness + # detection — log and treat as "no indexed files yet". + logger.warning(f"[sync.pending] load_mtimes failed for {workspace!r}: {exc}") + return {} + + +def _stat_file(abs_path: str) -> Optional[Tuple[int, int, float]]: + """Stat a file and return ``(st_size, st_mtime_ns, st_mtime_float)``. + + Returns ``None`` when the file doesn't exist (deleted since scan) + or can't be stat'd (permission denied, broken symlink, etc.). The + caller treats ``None`` as "file is gone" — a separate signal from + "file is stale". + """ + try: + st = os.stat(abs_path) + return (st.st_size, st.st_mtime_ns, st.st_mtime) + except OSError: + return None + + +def _compute_sha256(abs_path: str) -> Optional[str]: + """Compute the SHA-256 hash of a file's content. + + Returns ``None`` on any I/O error — the caller treats ``None`` as + "couldn't confirm content change, treat as stale". Reads in 64 KB + chunks so large files don't blow memory. + """ + h = hashlib.sha256() + try: + with open(abs_path, "rb") as f: + while True: + chunk = f.read(64 * 1024) + if not chunk: + break + h.update(chunk) + return h.hexdigest() + except OSError as exc: + logger.debug(f"[sync.pending] sha256 failed for {abs_path!r}: {exc}") + return None + + +def _load_indexed_content_hashes(workspace: str) -> Dict[str, str]: + """Load stored content hashes from the SQLite ``files`` table. + + Returns an empty dict when: + - SQLite is not available (legacy install) + - The DB file doesn't exist (workspace never scanned with v8.2+) + - The ``files`` table doesn't exist (pre-v8.2 DB) + - Any other error occurs + + When this returns empty, :class:`StaleFileDetector` falls back to + size/mtime-only detection — still useful, just less precise. The + banner will say "size/mtime differ" rather than "content differs". + """ + try: + from persistent_registry import PersistentRegistry + from utils import default_db_path + db_path = default_db_path(workspace) + if not os.path.exists(db_path): + return {} + reg = PersistentRegistry(workspace, db_path=db_path) + conn = reg._connect() + rows = conn.execute( + "SELECT file_path, content_hash FROM files WHERE content_hash IS NOT NULL" + ).fetchall() + # file_path in the DB is absolute; convert to workspace-relative + # to match the mtimes.json key format. + result: Dict[str, str] = {} + ws_abs = os.path.abspath(workspace) + for row in rows: + abs_path = row["file_path"] + try: + rel = os.path.relpath(abs_path, ws_abs) + except ValueError: + # Different drive on Windows — skip. + continue + result[rel] = row["content_hash"] + return result + except Exception as exc: + logger.debug(f"[sync.pending] content-hash load failed for {workspace!r}: {exc}") + return {} + + +# ─── Detector ────────────────────────────────────────────────────────────── + + +class StaleFileDetector: + """Thread-safe per-workspace stale-file detection with short TTL cache. + + Usage:: + + detector = StaleFileDetector() + stale = detector.detect(workspace="/path/to/ws") + if stale: + banner = format_staleness_banner(stale) + + The detector caches results per workspace for + :data:`DETECTOR_CACHE_TTL_SECONDS` seconds. The cache is keyed by + absolute workspace path and protected by a :class:`threading.Lock` + so concurrent MCP tool calls (dispatched in a thread pool) don't + race or duplicate work. + + The cache exists because walking 10k+ files on every tool call + would add ~50 ms of latency. A 5-second TTL keeps the banner fresh + enough for interactive use (a user editing a file should see the + banner within seconds) without re-stat-ing the whole tree on every + query. Phase 2 (connect-time catch-up) will invalidate the cache + on MCP reconnect. + """ + + def __init__(self, cache_ttl_seconds: float = DETECTOR_CACHE_TTL_SECONDS) -> None: + self._cache_ttl = cache_ttl_seconds + # ``_cache`` maps abs workspace path → (detection_time, result). + # ``result`` is the tuple returned by ``detect()`` (possibly empty). + self._cache: Dict[str, Tuple[float, Tuple[StaleFile, ...]]] = {} + self._lock = threading.Lock() + + def detect( + self, + workspace: str, + *, + confirm_with_hash: bool = True, + max_files: int = 10_000, + ) -> Tuple[StaleFile, ...]: + """Detect stale files in ``workspace``. + + Args: + workspace: Absolute path to the workspace root. + confirm_with_hash: When True (default), re-compute SHA-256 + for files whose size/mtime differ and only flag them + as stale when the content hash also differs. When + False, flag on size/mtime difference alone (faster but + false-positive on ``touch``). + max_files: Safety cap on the number of files to walk. + 10k is enough for most projects; larger codebases + should run a scan to refresh the index instead of + relying on the staleness banner. + + Returns: + Tuple of :class:`StaleFile` records, sorted by edit age + (most recent first). Empty tuple when the workspace has + no indexed files or no stale files. + + Why a tuple (not a list)? + The result is immutable from the caller's perspective — + the banner formatter shouldn't accidentally mutate it. + Tuples also hash, which simplifies testing. + """ + if not workspace: + return () + + ws_abs = os.path.abspath(workspace) + now = time.monotonic() + + # Cache check — fast path. We hold the lock only for the dict + # lookup, not for the walk below. + with self._lock: + cached = self._cache.get(ws_abs) + if cached is not None: + cached_at, cached_result = cached + if now - cached_at < self._cache_ttl: + return cached_result + + # Cache miss or expired — do the walk. This is read-only and + # safe to run without the lock; concurrent calls may duplicate + # the walk but won't corrupt the cache. + result = self._walk_and_detect( + ws_abs, + confirm_with_hash=confirm_with_hash, + max_files=max_files, + ) + + # Store in cache. We re-acquire the lock only for the write. + with self._lock: + self._cache[ws_abs] = (now, result) + return result + + def invalidate(self, workspace: Optional[str] = None) -> None: + """Drop cached results for ``workspace`` (or all workspaces). + + Called by the MCP server after a ``scan`` command — the scan + refreshes the index, so any cached staleness verdict is now + stale itself. Passing ``None`` drops everything (used in + tests). + """ + with self._lock: + if workspace is None: + self._cache.clear() + else: + self._cache.pop(os.path.abspath(workspace), None) + + def _walk_and_detect( + self, + ws_abs: str, + *, + confirm_with_hash: bool, + max_files: int, + ) -> Tuple[StaleFile, ...]: + """Walk indexed files, compare stats, return stale records. + + This is the uncached path. Separated from :meth:`detect` so + tests can call it directly without polluting the cache. + """ + stored_mtimes = _load_indexed_mtimes(ws_abs) + if not stored_mtimes: + # Workspace never scanned (or mtimes.json missing/corrupt). + # Nothing to compare against — return empty. + return () + + stored_hashes: Dict[str, str] = {} + if confirm_with_hash: + stored_hashes = _load_indexed_content_hashes(ws_abs) + + now_time = time.time() + stale: List[StaleFile] = [] + walked = 0 + + for rel_path, stored_mtime in stored_mtimes.items(): + walked += 1 + if walked > max_files: + # Safety cap — don't let a pathological mtimes.json + # (10M entries from a misbehaving scan) lock up the + # MCP server. The banner will say "and possibly more". + break + + abs_path = os.path.join(ws_abs, rel_path) + stat = _stat_file(abs_path) + if stat is None: + # File deleted since scan — not "stale" in the edit + # sense; the index is wrong but the fix is a re-scan, + # not a banner. Skip (Phase 2 may surface deletions + # separately). + continue + + current_size, _current_mtime_ns, current_mtime = stat + + # Quick check: mtime within tolerance → not stale. + if abs(current_mtime - stored_mtime) <= _MTIME_TOLERANCE_SECONDS: + continue + + # mtime differs — check size. We don't have a stored size + # in mtimes.json (only mtime), so we use the stored content + # hash's size if available, otherwise None. + stored_size = None + size_changed = True # conservative default + + # Slow path: confirm with content hash if requested. + content_hash_changed: Optional[bool] = None + if confirm_with_hash: + current_hash = _compute_sha256(abs_path) + stored_hash = stored_hashes.get(rel_path) + if current_hash is not None and stored_hash is not None: + content_hash_changed = current_hash != stored_hash + if not content_hash_changed: + # mtime changed but content identical — skip. + # (e.g. ``touch`` or ``git checkout`` of same content.) + continue + # If we couldn't get a hash (file vanished mid-walk, + # permission error, no stored hash), content_hash_changed + # stays None — the banner will say "size/mtime differ" + # rather than "content differs". + + stale.append(StaleFile( + rel_path=rel_path, + stored_mtime=stored_mtime, + current_mtime=current_mtime, + stored_size=stored_size, + current_size=current_size, + edit_age_seconds=max(0.0, now_time - current_mtime), + size_changed=size_changed, + content_hash_changed=content_hash_changed, + )) + + # Sort by edit age ascending (smallest age = most recent edit + # first) — the banner shows the most recently edited files at + # the top so the agent sees the most relevant context first. + # edit_age = now - current_mtime, so a file edited 1s ago has + # age=1, a file edited 10s ago has age=10. Ascending puts age=1 + # first → most recent first. + stale.sort(key=lambda s: s.edit_age_seconds, reverse=False) + return tuple(stale) + + +# ─── Banner formatting ──────────────────────────────────────────────────── + + +def format_staleness_banner( + stale_files: Tuple[StaleFile, ...] | List[StaleFile], + *, + limit: int = STALE_FILE_LIMIT_DEFAULT, +) -> str: + """Format stale files into a human/agent-readable banner string. + + The banner is plain text (no markdown) so it renders correctly in + both terminal output and MCP tool-response content blocks. It + starts with a ⚠️ marker so agents can pattern-match on it. + + Args: + stale_files: Sequence of :class:`StaleFile` records (typically + the return value of :meth:`StaleFileDetector.detect`). + limit: Max number of file names to list before collapsing to + "and N more". Default 10. + + Returns: + Multi-line banner string, or empty string when ``stale_files`` + is empty (caller should not prepend anything in that case). + + The banner shape:: + + ⚠️ Some files referenced below were edited since the last index sync. + The index may be stale for these files — re-run `codelens scan` to refresh. + Stale files (showing 3 of 3, most recent first): + • path/to/file.py (edited 2.3s ago, content differs) + • other.js (edited 1m 12s ago, content differs) + • third.ts (edited 5m 0s ago, size/mtime differ) + """ + if not stale_files: + return "" + + total = len(stale_files) + shown = list(stale_files[:limit]) + + lines: List[str] = [ + "⚠️ Some files referenced below were edited since the last index sync.", + "The index may be stale for these files — re-run `codelens scan` to refresh.", + f"Stale files (showing {len(shown)} of {total}, most recent first):", + ] + for sf in shown: + age = _format_age(sf.edit_age_seconds) + # Distinguish "content differs" (hash confirmed) from + # "size/mtime differ" (hash not confirmed or not available). + if sf.content_hash_changed is True: + change_desc = "content differs" + elif sf.content_hash_changed is False: + # This shouldn't happen — we filter these out in detect(). + # But be defensive: if a caller passes a pre-filter list, + # don't claim the file is stale. + change_desc = "content identical (mtime-only)" + else: + change_desc = "size/mtime differ" + lines.append(f" • {sf.rel_path} (edited {age} ago, {change_desc})") + + if total > limit: + lines.append(f" … and {total - limit} more (run `codelens staleness` for the full list)") + + return "\n".join(lines) + + +def _format_age(seconds: float) -> str: + """Format an age in seconds as a human-readable string. + + Examples: ``2.3s``, ``1m 12s``, ``5m 0s``, ``1h 23m``, ``2d 4h``. + """ + if seconds < 60: + return f"{seconds:.1f}s" + if seconds < 3600: + m = int(seconds // 60) + s = int(seconds % 60) + return f"{m}m {s}s" + if seconds < 86400: + h = int(seconds // 3600) + m = int((seconds % 3600) // 60) + return f"{h}h {m}m" + d = int(seconds // 86400) + h = int((seconds % 86400) // 3600) + return f"{d}d {h}h" + + +# ─── Module-level convenience function ───────────────────────────────────── + +# Module-level singleton detector — the MCP server uses this to avoid +# constructing a new detector on every tool call. Tests can call +# ``_default_detector.invalidate()`` to reset state between cases. +_default_detector = StaleFileDetector() + + +def detect_stale_files( + workspace: str, + *, + confirm_with_hash: bool = True, + max_files: int = 10_000, +) -> Tuple[StaleFile, ...]: + """Module-level convenience wrapper around the singleton detector. + + Equivalent to:: + + _default_detector.detect(workspace, confirm_with_hash=..., max_files=...) + + Most callers should use this. Construct a :class:`StaleFileDetector` + directly only when you need a separate cache (e.g. tests). + """ + return _default_detector.detect( + workspace, + confirm_with_hash=confirm_with_hash, + max_files=max_files, + ) + + +__all__ = [ + "StaleFile", + "StaleFileDetector", + "detect_stale_files", + "format_staleness_banner", + "STALE_FILE_LIMIT_DEFAULT", + "DETECTOR_CACHE_TTL_SECONDS", +] diff --git a/skill.json b/skill.json index 9b0ea517..cf72f846 100755 --- a/skill.json +++ b/skill.json @@ -1,7 +1,7 @@ { "name": "codelens", "version": "8.2.0", - "description": "Live Codebase Reference Intelligence. 77 commands for AI-powered code analysis, security auditing, quality scoring, and pre-write safety checks. Supports 28+ languages with regex+AST hybrid parsing. Must activate before writing/editing/deleting any class, id, or function.", + "description": "Live Codebase Reference Intelligence. 78 commands for AI-powered code analysis, security auditing, quality scoring, and pre-write safety checks. Supports 28+ languages with regex+AST hybrid parsing. Must activate before writing/editing/deleting any class, id, or function.", "author": "codelens", "command_categories": { "setup": [ diff --git a/tests/test_staleness.py b/tests/test_staleness.py new file mode 100644 index 00000000..cfc99d93 --- /dev/null +++ b/tests/test_staleness.py @@ -0,0 +1,690 @@ +"""Tests for the per-file staleness banner (issue #66 Phase 1). + +Scope: + +* :mod:`sync.pending` — :class:`StaleFileDetector`, :func:`detect_stale_files`, + :func:`format_staleness_banner`. Covers: mtime detection, content-hash + confirmation, cache TTL, thread safety, banner formatting, edge cases + (no mtimes.json, deleted files, permission errors). +* :class:`mcp_server.MCPServer` staleness integration — banner is + prepended to read-tool responses, suppressed on ``scan``/``init``, + cache invalidated after ``scan``. +* ``codelens staleness`` CLI command — registration, JSON/text output, + ``--no-confirm-hash`` flag. + +All tests are **network-free** and **filesystem-light** — they create +small temporary workspaces with synthetic ``mtimes.json`` files. No +real CodeLens scan is needed; the staleness module reads ``mtimes.json`` +directly. +""" + +from __future__ import annotations + +import json +import os +import subprocess +import sys +import tempfile +import threading +import time +from typing import Any, Dict, List, Optional, Tuple +from unittest import mock + +import pytest + +# ─── Path setup (mirror other tests) ─────────────────────────────────────── + +_THIS_DIR = os.path.dirname(os.path.abspath(__file__)) +_SCRIPTS_DIR = os.path.join(os.path.dirname(_THIS_DIR), "scripts") +if _SCRIPTS_DIR not in sys.path: + sys.path.insert(0, _SCRIPTS_DIR) + +from commands import COMMAND_REGISTRY # noqa: E402 +from sync.pending import ( # noqa: E402 + DETECTOR_CACHE_TTL_SECONDS, + STALE_FILE_LIMIT_DEFAULT, + StaleFile, + StaleFileDetector, + detect_stale_files, + format_staleness_banner, + _default_detector, + _format_age, +) + + +# ─── Fixtures ────────────────────────────────────────────────────────────── + + +@pytest.fixture +def fresh_detector(): + """Return a brand-new StaleFileDetector (no shared state with the module singleton).""" + return StaleFileDetector(cache_ttl_seconds=0.0) # 0 TTL = always re-walk + + +@pytest.fixture +def reset_default_detector(): + """Reset the module-level singleton detector before and after the test.""" + _default_detector.invalidate() + yield + _default_detector.invalidate() + + +def _make_workspace(files: Dict[str, str]) -> str: + """Create a temp workspace with the given files. + + ``files`` maps relative path → content. Returns the workspace root + path. Caller is responsible for cleanup (use ``tmp_path`` fixture + in real tests; this helper is for ad-hoc use). + """ + ws = tempfile.mkdtemp(prefix="codelens-staleness-test-") + for rel, content in files.items(): + abs_path = os.path.join(ws, rel) + os.makedirs(os.path.dirname(abs_path), exist_ok=True) + with open(abs_path, "w", encoding="utf-8") as f: + f.write(content) + return ws + + +def _write_mtimes(workspace: str, mtimes: Dict[str, float]) -> None: + """Write a synthetic ``.codelens/mtimes.json`` for the workspace.""" + codelens_dir = os.path.join(workspace, ".codelens") + os.makedirs(codelens_dir, exist_ok=True) + with open(os.path.join(codelens_dir, "mtimes.json"), "w", encoding="utf-8") as f: + json.dump(mtimes, f) + + +def _get_mtime(path: str) -> float: + return os.path.getmtime(path) + + +# ─── StaleFileDetector — basic detection ────────────────────────────────── + + +class TestStaleFileDetectorBasic: + """Core detection logic — mtime comparison, file walking.""" + + def test_no_mtimes_file_returns_empty(self, tmp_path, fresh_detector): + """Workspace with no .codelens/mtimes.json → no stale files.""" + # Create a file but no mtimes.json. + (tmp_path / "app.py").write_text("print('hello')") + stale = fresh_detector.detect(str(tmp_path)) + assert stale == () + + def test_fresh_index_no_stale(self, tmp_path, fresh_detector): + """All files match stored mtimes → no stale files.""" + f1 = tmp_path / "app.py" + f2 = tmp_path / "other.js" + f1.write_text("print('hello')") + f2.write_text("console.log('hi')") + _write_mtimes(str(tmp_path), { + "app.py": _get_mtime(str(f1)), + "other.js": _get_mtime(str(f2)), + }) + stale = fresh_detector.detect(str(tmp_path)) + assert stale == () + + def test_edited_file_detected_as_stale(self, tmp_path, fresh_detector): + """File edited after scan → detected as stale.""" + f = tmp_path / "app.py" + f.write_text("print('hello')") + stored_mtime = _get_mtime(str(f)) + _write_mtimes(str(tmp_path), {"app.py": stored_mtime}) + + # Edit the file (must wait >1ms so mtime actually changes). + time.sleep(0.05) + f.write_text("print('hello world')") + + stale = fresh_detector.detect(str(tmp_path), confirm_with_hash=False) + assert len(stale) == 1 + assert stale[0].rel_path == "app.py" + assert stale[0].stored_mtime == stored_mtime + assert stale[0].current_mtime > stored_mtime + assert stale[0].edit_age_seconds >= 0.0 + + def test_deleted_file_skipped_not_stale(self, tmp_path, fresh_detector): + """File deleted since scan → not reported as stale (it's gone).""" + f = tmp_path / "app.py" + f.write_text("print('hello')") + _write_mtimes(str(tmp_path), {"app.py": _get_mtime(str(f))}) + + # Delete the file. + f.unlink() + + stale = fresh_detector.detect(str(tmp_path), confirm_with_hash=False) + assert stale == () + + def test_empty_workspace_returns_empty(self, tmp_path, fresh_detector): + """Workspace with no files at all → no stale files.""" + _write_mtimes(str(tmp_path), {}) + stale = fresh_detector.detect(str(tmp_path)) + assert stale == () + + def test_empty_workspace_path_returns_empty(self, fresh_detector): + """Empty string workspace → no crash, empty result.""" + assert fresh_detector.detect("") == () + + def test_results_sorted_by_edit_age_most_recent_first(self, tmp_path, fresh_detector): + """Most recently edited file should appear first in the result. + + edit_age = now - current_mtime. A file edited 1s ago has age=1; + a file edited 10s ago has age=10. The banner says "most recent + first", so the file with the SMALLEST edit_age should be first. + """ + f1 = tmp_path / "old.py" + f2 = tmp_path / "new.py" + f1.write_text("a") + f2.write_text("b") + # Store mtimes as-of-write. + m1 = _get_mtime(str(f1)) + m2 = _get_mtime(str(f2)) + _write_mtimes(str(tmp_path), {"old.py": m1, "new.py": m2}) + + # Edit old.py first, then new.py — new.py is "more recent" + # (smaller edit_age). + time.sleep(0.05) + f1.write_text("a-edited") + time.sleep(0.05) + f2.write_text("b-edited") + + stale = fresh_detector.detect(str(tmp_path), confirm_with_hash=False) + assert len(stale) == 2 + # new.py was edited last → smaller edit_age → should be first. + assert stale[0].rel_path == "new.py" + assert stale[0].edit_age_seconds < stale[1].edit_age_seconds + assert stale[1].rel_path == "old.py" + + +# ─── StaleFileDetector — content-hash confirmation ──────────────────────── + + +class TestStaleFileDetectorHashConfirmation: + """The ``confirm_with_hash`` path — only flag when content actually changed.""" + + def test_touch_without_content_change_skipped(self, tmp_path, fresh_detector): + """``touch`` updates mtime but not content → not stale (with hash confirm).""" + f = tmp_path / "app.py" + f.write_text("print('hello')") + stored_mtime = _get_mtime(str(f)) + + # We need a stored content hash to compare against. The hash comes + # from the SQLite ``files`` table, which we don't have in this test. + # Without a stored hash, the detector falls back to "size/mtime differ" + # and reports the file as stale. So this test verifies the fallback + # behavior: when no stored hash is available, mtime change → stale. + time.sleep(0.05) + # Re-write the SAME content (simulates touch). + os.utime(str(f), None) + + _write_mtimes(str(tmp_path), {"app.py": stored_mtime}) + stale = fresh_detector.detect(str(tmp_path), confirm_with_hash=True) + # Without stored hash, we can't confirm content identical → reported as stale. + assert len(stale) == 1 + assert stale[0].content_hash_changed is None + + def test_no_confirm_hash_flags_on_mtime_only(self, tmp_path, fresh_detector): + """``confirm_with_hash=False`` flags on mtime change alone.""" + f = tmp_path / "app.py" + f.write_text("print('hello')") + stored_mtime = _get_mtime(str(f)) + _write_mtimes(str(tmp_path), {"app.py": stored_mtime}) + + time.sleep(0.05) + os.utime(str(f), None) # touch — mtime changes, content same + + stale = fresh_detector.detect(str(tmp_path), confirm_with_hash=False) + assert len(stale) == 1 + assert stale[0].content_hash_changed is None # not checked + + +# ─── StaleFileDetector — cache ───────────────────────────────────────────── + + +class TestStaleFileDetectorCache: + """The per-workspace TTL cache.""" + + def test_cache_returns_same_result_within_ttl(self, tmp_path): + """Within TTL, detect() returns the cached result without re-walking.""" + f = tmp_path / "app.py" + f.write_text("print('hello')") + _write_mtimes(str(tmp_path), {"app.py": _get_mtime(str(f))}) + + detector = StaleFileDetector(cache_ttl_seconds=10.0) + stale1 = detector.detect(str(tmp_path), confirm_with_hash=False) + + # Edit the file AFTER the first detect() call. + time.sleep(0.05) + f.write_text("print('changed')") + + # Second call within TTL should return the cached (empty) result. + stale2 = detector.detect(str(tmp_path), confirm_with_hash=False) + assert stale1 == stale2 == () + + def test_cache_expires_after_ttl(self, tmp_path): + """After TTL, detect() re-walks and picks up changes.""" + f = tmp_path / "app.py" + f.write_text("print('hello')") + _write_mtimes(str(tmp_path), {"app.py": _get_mtime(str(f))}) + + detector = StaleFileDetector(cache_ttl_seconds=0.05) + stale1 = detector.detect(str(tmp_path), confirm_with_hash=False) + assert stale1 == () + + # Edit + wait for TTL to expire. + time.sleep(0.05) + f.write_text("print('changed')") + time.sleep(0.05) + + stale2 = detector.detect(str(tmp_path), confirm_with_hash=False) + assert len(stale2) == 1 + + def test_invalidate_drops_cache(self, tmp_path): + """invalidate() forces the next detect() to re-walk.""" + f = tmp_path / "app.py" + f.write_text("print('hello')") + _write_mtimes(str(tmp_path), {"app.py": _get_mtime(str(f))}) + + detector = StaleFileDetector(cache_ttl_seconds=10.0) + stale1 = detector.detect(str(tmp_path), confirm_with_hash=False) + assert stale1 == () + + # Edit + invalidate. + time.sleep(0.05) + f.write_text("print('changed')") + detector.invalidate(str(tmp_path)) + + stale2 = detector.detect(str(tmp_path), confirm_with_hash=False) + assert len(stale2) == 1 + + def test_invalidate_all_workspaces(self, tmp_path): + """invalidate(None) drops cache for ALL workspaces.""" + f = tmp_path / "app.py" + f.write_text("print('hello')") + _write_mtimes(str(tmp_path), {"app.py": _get_mtime(str(f))}) + + detector = StaleFileDetector(cache_ttl_seconds=10.0) + detector.detect(str(tmp_path), confirm_with_hash=False) + + # invalidate(None) clears everything. + detector.invalidate(None) + + # Internal cache should be empty. + assert detector._cache == {} + + +# ─── StaleFileDetector — thread safety ───────────────────────────────────── + + +class TestStaleFileDetectorThreadSafety: + """Concurrent detect() calls must not corrupt the cache.""" + + def test_concurrent_calls_same_workspace(self, tmp_path): + """20 threads calling detect() on the same workspace don't crash.""" + f = tmp_path / "app.py" + f.write_text("print('hello')") + _write_mtimes(str(tmp_path), {"app.py": _get_mtime(str(f))}) + + detector = StaleFileDetector(cache_ttl_seconds=1.0) + results: List[Tuple[StaleFile, ...]] = [] + errors: List[Exception] = [] + + def worker(): + try: + r = detector.detect(str(tmp_path), confirm_with_hash=False) + results.append(r) + except Exception as e: + errors.append(e) + + threads = [threading.Thread(target=worker) for _ in range(20)] + for t in threads: + t.start() + for t in threads: + t.join() + + assert errors == [] + assert len(results) == 20 + # All results should be equal (cache hit or consistent walk). + assert all(r == results[0] for r in results) + + +# ─── format_staleness_banner ─────────────────────────────────────────────── + + +class TestFormatStalenessBanner: + """Banner string formatting.""" + + def test_empty_input_returns_empty_string(self): + assert format_staleness_banner([]) == "" + assert format_staleness_banner(()) == "" + + def test_single_file_banner(self): + sf = StaleFile( + rel_path="app.py", + stored_mtime=1000.0, + current_mtime=1001.0, + stored_size=None, + current_size=21, + edit_age_seconds=2.5, + size_changed=True, + content_hash_changed=True, + ) + banner = format_staleness_banner([sf]) + assert "⚠️" in banner + assert "app.py" in banner + assert "content differs" in banner + assert "2.5s" in banner + assert "showing 1 of 1" in banner + + def test_size_mtime_only_when_hash_not_confirmed(self): + sf = StaleFile( + rel_path="app.py", + stored_mtime=1000.0, + current_mtime=1001.0, + stored_size=None, + current_size=21, + edit_age_seconds=2.5, + size_changed=True, + content_hash_changed=None, + ) + banner = format_staleness_banner([sf]) + assert "size/mtime differ" in banner + + def test_limit_truncates_with_and_n_more(self): + files = [ + StaleFile( + rel_path=f"file_{i}.py", + stored_mtime=1000.0, + current_mtime=1001.0, + stored_size=None, + current_size=10, + edit_age_seconds=float(i), + size_changed=True, + content_hash_changed=True, + ) + for i in range(15) + ] + banner = format_staleness_banner(files, limit=10) + assert "showing 10 of 15" in banner + assert "and 5 more" in banner + + def test_banner_mentions_rescan_command(self): + sf = StaleFile( + rel_path="app.py", + stored_mtime=1000.0, + current_mtime=1001.0, + stored_size=None, + current_size=21, + edit_age_seconds=2.5, + size_changed=True, + content_hash_changed=True, + ) + banner = format_staleness_banner([sf]) + assert "codelens scan" in banner + + +class TestFormatAge: + """The _format_age helper.""" + + @pytest.mark.parametrize( + "seconds,expected", + [ + (0.1, "0.1s"), + (1.0, "1.0s"), + (59.9, "59.9s"), + (60.0, "1m 0s"), + (72.0, "1m 12s"), + (3599.0, "59m 59s"), + (3600.0, "1h 0m"), + (3661.0, "1h 1m"), + (86400.0, "1d 0h"), + (90000.0, "1d 1h"), + ], + ) + def test_format_age(self, seconds, expected): + assert _format_age(seconds) == expected + + +# ─── detect_stale_files module-level function ────────────────────────────── + + +class TestDetectStaleFilesModuleFunction: + """The module-level detect_stale_files() uses the singleton detector.""" + + def test_module_function_returns_tuple(self, tmp_path, reset_default_detector): + f = tmp_path / "app.py" + f.write_text("print('hello')") + _write_mtimes(str(tmp_path), {"app.py": _get_mtime(str(f))}) + + result = detect_stale_files(str(tmp_path), confirm_with_hash=False) + assert isinstance(result, tuple) + + def test_module_function_caches(self, tmp_path, reset_default_detector): + """Two calls within TTL return the same cached result.""" + f = tmp_path / "app.py" + f.write_text("print('hello')") + _write_mtimes(str(tmp_path), {"app.py": _get_mtime(str(f))}) + + r1 = detect_stale_files(str(tmp_path), confirm_with_hash=False) + r2 = detect_stale_files(str(tmp_path), confirm_with_hash=False) + assert r1 is r2 or r1 == r2 # cached or equal + + +# ─── CLI command ─────────────────────────────────────────────────────────── + + +class TestStalenessCommand: + """The ``codelens staleness`` CLI command.""" + + def test_command_is_registered(self): + assert "staleness" in COMMAND_REGISTRY + info = COMMAND_REGISTRY["staleness"] + assert callable(info["execute"]) + assert callable(info["add_args"]) + + def test_no_workspace_returns_error(self): + from commands import staleness as cmd + args = mock.MagicMock() + args.workspace = None + args.no_confirm_hash = False + args.max_files = 10000 + args.limit = 10 + args.format = "json" + result = cmd.execute(args, "") + assert result["status"] == "error" + + def test_json_output_has_required_fields(self, tmp_path, reset_default_detector): + f = tmp_path / "app.py" + f.write_text("print('hello')") + _write_mtimes(str(tmp_path), {"app.py": _get_mtime(str(f))}) + + from commands import staleness as cmd + args = mock.MagicMock() + args.workspace = str(tmp_path) + args.no_confirm_hash = True + args.max_files = 10000 + args.limit = 10 + args.format = "json" + result = cmd.execute(args, str(tmp_path)) + assert result["status"] == "ok" + assert "stale_count" in result + assert "stale_files" in result + assert "banner" in result + assert result["workspace"] == os.path.abspath(str(tmp_path)) + + def test_stale_file_appears_in_output(self, tmp_path, reset_default_detector): + f = tmp_path / "app.py" + f.write_text("print('hello')") + stored = _get_mtime(str(f)) + _write_mtimes(str(tmp_path), {"app.py": stored}) + + time.sleep(0.05) + f.write_text("print('changed')") + + from commands import staleness as cmd + args = mock.MagicMock() + args.workspace = str(tmp_path) + args.no_confirm_hash = True # skip hash, mtime-only + args.max_files = 10000 + args.limit = 10 + args.format = "json" + result = cmd.execute(args, str(tmp_path)) + assert result["stale_count"] == 1 + assert result["stale_files"][0]["rel_path"] == "app.py" + assert "⚠️" in result["banner"] + + +# ─── CLI subprocess smoke test ───────────────────────────────────────────── + + +class TestCLISmoke: + """End-to-end: invoke ``codelens staleness`` as a real subprocess.""" + + def _run_cli(self, workspace, *extra_args): + env = os.environ.copy() + env["PYTHONPATH"] = _SCRIPTS_DIR + env["PYTHONUTF8"] = "1" + return subprocess.run( + [ + sys.executable, + os.path.join(_SCRIPTS_DIR, "codelens.py"), + "staleness", + workspace, + "--format", + "json", + *extra_args, + ], + capture_output=True, + text=True, + env=env, + timeout=30, + ) + + def test_staleness_runs_cleanly_on_empty_workspace(self, tmp_path): + _write_mtimes(str(tmp_path), {}) + result = self._run_cli(str(tmp_path)) + assert result.returncode == 0, ( + f"exit={result.returncode}\nstdout={result.stdout}\nstderr={result.stderr}" + ) + # Parse the JSON payload from stdout (skip the auto-detect line). + out = result.stdout + start = out.find("{") + assert start >= 0 + payload = json.loads(out[start:]) + assert payload["status"] == "ok" + assert payload["stale_count"] == 0 + + +# ─── MCP server integration ──────────────────────────────────────────────── + + +class TestMCPServerStalenessIntegration: + """The MCPServer prepends the staleness banner to read-tool responses.""" + + def _make_server(self): + """Construct an MCPServer without starting the JSON-RPC loop.""" + from mcp_server import MCPServer + return MCPServer() + + def test_staleness_banner_prepended_on_read_tool(self, tmp_path, reset_default_detector): + """A read-tool response gets the banner prepended when files are stale.""" + f = tmp_path / "app.py" + f.write_text("print('hello')") + stored = _get_mtime(str(f)) + _write_mtimes(str(tmp_path), {"app.py": stored}) + + time.sleep(0.05) + f.write_text("print('changed')") + + server = self._make_server() + response = { + "content": [{"type": "text", "text": "original output"}], + "isError": False, + } + server._attach_staleness_banner(response, str(tmp_path)) + + # Banner should be prepended to the text. + text = response["content"][0]["text"] + assert "⚠️" in text + assert "app.py" in text + assert "original output" in text + # And the structured field should be set. + assert response["_staleness"]["stale_count"] == 1 + + def test_no_banner_when_index_fresh(self, tmp_path, reset_default_detector): + """No stale files → no banner, no _staleness field.""" + f = tmp_path / "app.py" + f.write_text("print('hello')") + _write_mtimes(str(tmp_path), {"app.py": _get_mtime(str(f))}) + + server = self._make_server() + response = { + "content": [{"type": "text", "text": "original output"}], + "isError": False, + } + server._attach_staleness_banner(response, str(tmp_path)) + + assert "_staleness" not in response + assert response["content"][0]["text"] == "original output" + + def test_scan_invalidates_staleness_cache(self, tmp_path, reset_default_detector): + """After a scan, the staleness cache is dropped so the next read re-probes.""" + f = tmp_path / "app.py" + f.write_text("print('hello')") + stored = _get_mtime(str(f)) + _write_mtimes(str(tmp_path), {"app.py": stored}) + + time.sleep(0.05) + f.write_text("print('changed')") + + server = self._make_server() + + # First detect — should find the stale file. + detector = server._get_staleness_detector() + assert detector is not None + stale1 = detector.detect(str(tmp_path), confirm_with_hash=False) + assert len(stale1) == 1 + + # Simulate a scan: rewrite mtimes.json with the current mtime, + # then invalidate the cache. + _write_mtimes(str(tmp_path), {"app.py": _get_mtime(str(f))}) + server._invalidate_staleness_cache(str(tmp_path)) + + # Second detect — cache was invalidated, mtimes now match → no stale. + stale2 = detector.detect(str(tmp_path), confirm_with_hash=False) + assert stale2 == () + + def test_empty_workspace_no_crash(self, reset_default_detector): + """Empty workspace string → no banner, no crash.""" + server = self._make_server() + response = { + "content": [{"type": "text", "text": "original"}], + "isError": False, + } + server._attach_staleness_banner(response, "") + assert response["content"][0]["text"] == "original" + assert "_staleness" not in response + + def test_detector_init_failure_isolated(self, tmp_path, reset_default_detector, monkeypatch): + """If the sync subpackage fails to import, the banner is silently skipped.""" + # Force the import to fail. + import builtins + real_import = builtins.__import__ + + def fake_import(name, *args, **kwargs): + if name == "sync.pending" or name == "sync": + raise ImportError("simulated failure") + return real_import(name, *args, **kwargs) + + monkeypatch.setattr(builtins, "__import__", fake_import) + + server = self._make_server() + # Reset the cached detector so the import runs again. + server._staleness_detector = None + + response = { + "content": [{"type": "text", "text": "original"}], + "isError": False, + } + # Should not raise. + server._attach_staleness_banner(response, str(tmp_path)) + assert response["content"][0]["text"] == "original" + assert "_staleness" not in response