diff --git a/README.md b/README.md index b90f8c93..7ebed3bc 100644 --- a/README.md +++ b/README.md @@ -2,12 +2,12 @@ > **Before an AI writes a new class/id/function, CodeLens must be checked. This is not optional.** -CodeLens is an AI-native code intelligence platform that gives AI agents **full visibility** into a codebase before they write any code. It prevents collision, overwrite of existing logic, security vulnerabilities, and dead code through 68 CLI commands, an MCP server with 66 tools (54 static + 12 dynamic), AST-based taint analysis, live CVE/OSV scanning, a plugin system with OWASP Top 10 + Compliance rule packs, a true graph data model (nodes + edges) for structural code queries, and token-efficient `--format compact` output for high-volume agent workflows (issue #17). +CodeLens is an AI-native code intelligence platform that gives AI agents **full visibility** into a codebase before they write any code. It prevents collision, overwrite of existing logic, security vulnerabilities, and dead code through 70 CLI commands, an MCP server with 68 tools (54 static + 14 dynamic), AST-based taint analysis, live CVE/OSV scanning, a plugin system with OWASP Top 10 + Compliance rule packs, a true graph data model (nodes + edges) for structural code queries, and token-efficient `--format compact` output for high-volume agent workflows (issue #17). ## Features -- **68 CLI Commands** — From basic scan/query to AST taint analysis, CVE scanning, plugin management, auto-fix, dashboards, CI/CD quality gates, and `graph-schema` for cheap graph-shape introspection -- **MCP Server (66 Tools)** — Native AI agent integration via Model Context Protocol (JSON-RPC over stdio), 54 statically-defined tools + 12 dynamically discovered, every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`) +- **70 CLI Commands** — From basic scan/query to AST taint analysis, CVE scanning, plugin management, auto-fix, dashboards, CI/CD quality gates, and `graph-schema` for cheap graph-shape introspection +- **MCP Server (68 Tools)** — Native AI agent integration via Model Context Protocol (JSON-RPC over stdio), 54 statically-defined tools + 14 dynamically discovered, every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`) - **Token-Efficient Compact Output (v8.2, issue #17)** — `--format compact` produces single-char-key JSON with abbreviated types, omitted null fields, and relative paths — ~50% smaller than `json` on real trace output. Combined with `--limit`/`--offset` pagination, 5 structural queries now cost <5k tokens (down from 30-80k) - **AST Taint Engine** — Tree-sitter based taint analysis with return-value propagation, scope hierarchy, and branch condition refinement - **Live CVE/OSV Scanning** — Real-time vulnerability data from OSV.dev API with SQLite cache, 9 ecosystems (PyPI, npm, crates.io, Go, Maven, NuGet, RubyGems, Pub, Hex) @@ -225,8 +225,8 @@ codelens/ │ ├── changelog.md # Older changelog (per-version highlights) │ └── agent-integration.md # AI agent integration guide ├── scripts/ -│ ├── codelens.py # CLI entry point (68 commands registered) -│ ├── mcp_server.py # MCP JSON-RPC server (66 tools) +│ ├── codelens.py # CLI entry point (70 commands registered) +│ ├── mcp_server.py # MCP JSON-RPC server (68 tools) │ ├── registry.py # Registry read/write/build │ ├── persistent_registry.py # SQLite persistent storage (WAL mode) │ ├── base_parser.py # Base tree-sitter parser diff --git a/SKILL-QUICK.md b/SKILL-QUICK.md index 5764ffd7..f7944813 100755 --- a/SKILL-QUICK.md +++ b/SKILL-QUICK.md @@ -114,7 +114,7 @@ $CLI list --limit 5 --offset 10 --format compact # → paginated + co | "Cross-file taint" | `dataflow` | `taint` (taint is single-file, AST-deep) | | "Auto-fix issues" | `fix` | `check` (check just gates, doesn't fix) | -## All 68 Commands +## All 70 Commands ### Setup & Lifecycle (8+) `init` · `scan [--incremental] [--max-files N] [--full]` · `registry-validate` · `detect` · `watch [--debounce SECS] [--git-mode] [--interval SECS]` · `git-status` · `migrate` · `serve` · `lsp-status` (issue #33: `codelens --lsp-status` top-level flag is an alias of `codelens lsp-status` — both delegate to `hybrid_engine.get_lsp_status()` and return the identical payload) @@ -146,9 +146,9 @@ $CLI list --limit 5 --offset 10 --format compact # → paginated + co ### Tooling (1) `plugin ` -**Total: 68 commands** (auto-registered via `commands/__init__.py`; rerun `python3 scripts/sync_command_count.py --apply` after adding/removing a command) +**Total: 70 commands** (auto-registered via `commands/__init__.py`; rerun `python3 scripts/sync_command_count.py --apply` after adding/removing a command) -## MCP Server (66 Tools) +## MCP Server (68 Tools) Start the MCP server for AI agent integration: @@ -156,9 +156,9 @@ Start the MCP server for AI agent integration: python3 scripts/codelens.py serve ``` -Exposes 66 tools as `codelens_` (e.g., `codelens_query`, `codelens_taint`, `codelens_graph_schema`, `codelens_architecture`, `codelens_resolve_types`, `codelens_git_status`): +Exposes 68 tools as `codelens_` (e.g., `codelens_query`, `codelens_taint`, `codelens_graph_schema`, `codelens_architecture`, `codelens_resolve_types`, `codelens_git_status`): - 50 statically-defined tools (full JSON schemas in `mcp_server.py`) -- 12 dynamically-discovered tools (auto-discovered from `COMMAND_REGISTRY`; long-running `watch` and `serve` are excluded) +- 14 dynamically-discovered tools (auto-discovered from `COMMAND_REGISTRY`; long-running `watch` and `serve` are excluded) - Every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`). Use `format: "compact"` for token-efficient responses (~50% smaller than `json`). - `watch` and `serve` itself are excluded (long-running) diff --git a/SKILL.md b/SKILL.md index bd91f902..77b8be13 100755 --- a/SKILL.md +++ b/SKILL.md @@ -1,10 +1,10 @@ --- name: codelens description: > - CodeLens — AI-Native Code Intelligence. 68 commands for AI-powered code analysis, + CodeLens — AI-Native Code Intelligence. 70 commands for AI-powered code analysis, security auditing, quality scoring, AST-based taint analysis, live CVE scanning, and pre-write safety checks. Supports 28+ languages with tree-sitter + regex - fallback parsing. MCP server exposes 66 tools for AI agent integration. + fallback parsing. MCP server exposes 68 tools for AI agent integration. For quick command reference with validated output schemas, see SKILL-QUICK.md. For version history, see CHANGELOG.md. --- diff --git a/pyproject.toml b/pyproject.toml index 51b37456..cfb46efa 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta" [project] name = "codelens" version = "8.2.0" -description = "Live Codebase Reference Intelligence — 68 commands for AI-powered code analysis, security auditing, and quality scoring" +description = "Live Codebase Reference Intelligence — 70 commands for AI-powered code analysis, security auditing, and quality scoring" readme = "README.md" license = {text = "MIT"} requires-python = ">=3.8" diff --git a/scripts/codelens.py b/scripts/codelens.py index 652aaddf..3aa7b3b2 100755 --- a/scripts/codelens.py +++ b/scripts/codelens.py @@ -873,9 +873,12 @@ def main(): # Detect which args the command already defines existing_dests = set() + existing_option_strings = set() for action in sub._actions: if hasattr(action, 'dest'): existing_dests.add(action.dest) + if hasattr(action, 'option_strings'): + existing_option_strings.update(action.option_strings) _existing_subparser_args[cmd_name] = existing_dests # Add --format to each subparser (UNLESS the command defines its own). @@ -884,7 +887,14 @@ def main(): # a text table, not JSON. Skipping the global add here lets that # command-specific format work without an argparse conflict. if "format" not in existing_dests: - sub.add_argument("--format", "-f", choices=["json", "markdown", "ai", "sarif", "compact"], default=None, + # Issue #62 Phase 1: ``affected`` command uses ``-f`` for + # ``--filter``. Avoid the ``-f`` shortcut clash by only adding + # the global ``-f`` shortcut when the command doesn't already + # claim it. The long form ``--format`` is always safe. + format_args = ["--format"] + if "-f" not in existing_option_strings: + format_args.append("-f") + sub.add_argument(*format_args, choices=["json", "markdown", "ai", "sarif", "compact"], default=None, help="Output format: json, markdown, ai (normalized schema), sarif (GitHub/VS Code), or compact (token-efficient single-char keys)") # Add AI-optimized flags to subparser ONLY if the command doesn't already have them diff --git a/scripts/commands/affected.py b/scripts/commands/affected.py new file mode 100644 index 00000000..237deda5 --- /dev/null +++ b/scripts/commands/affected.py @@ -0,0 +1,151 @@ +"""Affected command — identify test files affected by source changes. + +Issue #62 Phase 1: quick-win CI/CD tool. Given a list of changed source +files (positional args, or piped via ``--stdin`` from ``git diff --name-only``), +walks the reverse import-dependency graph and prints every test file that +transitively depends on any of the changed files. + +The classic CI pattern is:: + + AFFECTED=$(git diff --name-only HEAD | codelens affected --stdin --quiet) + pytest $AFFECTED + +By default only test files are returned (so ``pytest $AFFECTED`` doesn't try +to run non-test modules). Pass ``--include-source`` to also list non-test +dependents — useful for impact analysis in code review. + +Resolution of changed-file paths is forgiving: absolute paths, workspace- +relative paths, and bare basenames are all accepted. Ambiguous basenames +(files with the same name in multiple directories) are skipped rather +than silently picking the wrong one. + +Backed by :func:`dependents_engine.get_affected_files`. +""" + +from __future__ import annotations + +import sys + +from commands import register_command + + +def add_args(parser): + """Add `affected` command arguments.""" + parser.add_argument( + "files", + nargs="*", + default=None, + help="Changed file paths (absolute, relative, or bare basenames). " + "If omitted, reads from stdin — pipe `git diff --name-only`.", + ) + parser.add_argument( + "workspace", + nargs="?", + default=None, + help="Path to workspace root (auto-detected if omitted).", + ) + parser.add_argument( + "--stdin", + action="store_true", + default=False, + help="Read changed file paths from stdin (one per line). " + "Useful for piping: `git diff --name-only HEAD | codelens affected --stdin`.", + ) + parser.add_argument( + "-d", "--depth", + type=int, + default=5, + help="BFS depth cap (default 5). -1 = unlimited (with a 5000-node safety cap).", + ) + parser.add_argument( + "-f", "--filter", + default=None, + metavar="", + help="Only return affected files matching this glob (e.g. 'tests/*.py'). " + "Does not affect traversal. Default: no filter.", + ) + parser.add_argument( + "-j", "--json", + dest="as_json", + action="store_true", + default=False, + help="Emit full result dict as JSON instead of plain file list.", + ) + parser.add_argument( + "-q", "--quiet", + action="store_true", + default=False, + help="Only print affected file paths, one per line. No stats, no headers. " + "Default when piping to another command.", + ) + parser.add_argument( + "--include-source", + action="store_true", + default=False, + help="Also return non-test dependents. By default only test files are returned.", + ) + + +def execute(args, workspace): + """Run the affected-files analysis. + + Returns a dict (when ``--json`` or default) or prints paths to stdout + (when ``--quiet``). Reads from stdin when ``--stdin`` is set. + """ + # Gather changed files from args + stdin + changed_files: list[str] = list(args.files or []) + + if getattr(args, "stdin", False): + try: + stdin_text = sys.stdin.read() + except (KeyboardInterrupt, OSError): + stdin_text = "" + for line in stdin_text.splitlines(): + line = line.strip() + if line and not line.startswith("#"): + changed_files.append(line) + + if not changed_files: + return { + "status": "error", + "error": ( + "no changed files provided. Pass file paths as args, or use " + "--stdin to read from stdin (e.g. `git diff --name-only | " + "codelens affected --stdin`)." + ), + "workspace": workspace, + } + + # Lazy import — keeps the command light if only --help is invoked + from dependents_engine import get_affected_files + + result = get_affected_files( + changed_files=changed_files, + workspace=workspace, + depth=args.depth, + file_filter=args.filter, + include_source=args.include_source, + ) + + # --quiet mode: print only file paths, one per line, to stdout + if getattr(args, "quiet", False): + for f in result.get("affected", []): + print(f) + # Return a minimal dict so the CLI wrapper doesn't double-print + return { + "status": "ok", + "quiet": True, + "affected_count": len(result.get("affected", [])), + "workspace": workspace, + } + + # Default + --json: return the full result dict (formatter handles output) + return result + + +register_command( + "affected", + "Identify test files affected by source changes (issue #62 Phase 1)", + add_args, + execute, +) diff --git a/scripts/dependents_engine.py b/scripts/dependents_engine.py index a72e8b35..abddc329 100755 --- a/scripts/dependents_engine.py +++ b/scripts/dependents_engine.py @@ -11,6 +11,262 @@ from utils import DEFAULT_IGNORE_DIRS, logger +# ─── Issue #62 Phase 1: `affected` command helpers ──────────────────────── +# +# Test-file detection heuristic. Returns True if the file path looks like a +# test file in any of the common conventions across ecosystems. Conservative +# by design — better to over-report (run an extra test) than to silently +# miss a regression. +# +# Patterns matched (case-insensitive basename or path segment): +# *test*.py, *spec*.py — pytest / unittest / nose +# *test.js, *spec.js — Jest / Mocha / Jasmine +# *.test.ts(x), *.spec.ts(x) — Jest / Vitest with TypeScript +# test_*.py, *_test.py — Python unittest conventions +# *_test.go — Go testing +# *Test.java, *Tests.java — JUnit +# *Test.cs — NUnit / xUnit +# *.rs (with #[test]) — detected at content level, not here +# tests/ directory — common layout +# __tests__/ directory — Jest convention +# spec/ directory — RSpec / Jasmine convention + +_TEST_BASENAME_RE = re.compile( + r'^(' + r'test_.+|.+_test' # test_foo.py, foo_test.py + r'|.+tests?|.+specs?' # FooTest.java, FooTests.java, foo.spec.ts, foo.specs.js + r'|tests?|specs?' # test.py, tests.py, spec.py, specs.py (exact stem) + r')$', + re.IGNORECASE, +) + +_TEST_DIR_SEGMENTS = {'tests', 'test', '__tests__', '__test__', 'spec', 'specs'} + +_TEST_EXTENSIONS = { + '.py', '.pyw', + '.js', '.mjs', '.cjs', '.ts', '.tsx', '.jsx', + '.go', '.rs', '.java', '.kt', '.cs', '.rb', '.php', + '.dart', '.swift', +} + + +def is_test_file(path: str) -> bool: + """Heuristic: return True if ``path`` looks like a test file. + + Conservative — over-reports rather than misses. Matches common test + naming conventions across Python, JS/TS, Go, Rust, Java, C#, Ruby, + PHP, Dart, Swift. Path can be absolute, relative, or just a basename. + """ + if not path: + return False + # Normalize separators + norm = path.replace('\\', '/') + basename = norm.rsplit('/', 1)[-1] + if not basename: + return False + stem, ext = os.path.splitext(basename) + ext_lower = ext.lower() + + # 1. Skip non-source files (.md, .json, .txt, .gitignore, etc.) + # Empty extension is allowed (we'll gate it later). + if ext_lower and ext_lower not in _TEST_EXTENSIONS: + return False + + # 2. Path contains a test directory segment (tests/, __tests__/, spec/) + # This is the strongest signal — a file inside tests/ is a test file + # regardless of its basename. Check BEFORE the extension gate so + # that a file like `tests/conftest.py` (no test_ prefix) is caught. + segments = [s.lower() for s in norm.split('/') if s] + if any(seg in _TEST_DIR_SEGMENTS for seg in segments[:-1]): + return True + if '__tests__' in norm.lower(): + return True + + # 3. Require an extension for the basename-pattern match — a bare + # `test` or `tests` (no extension) is more likely a directory or + # symlink than an actual test file. + if not ext_lower: + return False + + # 4. Basename pattern: test_*, *_test, *test, *tests, *spec, *specs, + # exact `test`/`tests`/`spec`/`specs` stems + if _TEST_BASENAME_RE.match(stem): + return True + if stem.lower().startswith(('test_', 'spec_')): + return True + + return False + + +def get_affected_files( + changed_files: List[str], + workspace: str, + depth: int = 5, + file_filter: Optional[str] = None, + include_source: bool = False, +) -> Dict[str, Any]: + """Issue #62 Phase 1 — transitive affected-files analysis. + + Given a list of changed source files, walk the reverse dependency graph + (depth-controlled BFS) and return every file that transitively imports + one of the changed files. By default, only test files are returned + (the CI use-case: ``pytest $AFFECTED``); pass ``include_source=True`` + to also return non-test dependents. + + Args: + changed_files: list of file paths (absolute, relative, or bare + basenames — resolution is forgiving). Files that cannot be + resolved against the workspace are silently skipped. + workspace: absolute workspace path. + depth: BFS depth cap (default 5). ``0`` means only the changed + files themselves; ``-1`` means unlimited (with a 5000-node + safety cap to prevent runaway traversal on cyclic graphs). + file_filter: optional glob; if given, only files matching this + glob are returned in ``affected``. Does not affect traversal. + include_source: if True, also return non-test dependents. If + False (default), only test files are returned. + + Returns: + Dict with keys: + * ``status`` — "ok" + * ``workspace`` — absolute workspace path + * ``changed_files`` — resolved changed files that were found + * ``unresolved`` — changed files that could not be resolved + * ``affected`` — sorted list of affected test (or all) files + * ``affected_by_source`` — dict {changed_file: [affected tests]} + * ``depth`` — actual depth used + * ``stats`` — counts + """ + import fnmatch + + workspace = os.path.abspath(workspace) + import_graph, reverse_graph = _build_import_graph(workspace) + + # Normalize changed files to workspace-relative paths + all_known_files = set(import_graph.keys()) | set(reverse_graph.keys()) + resolved_changed: List[str] = [] + unresolved: List[str] = [] + for cf in changed_files: + cf = cf.strip() + if not cf: + continue + # Strip leading ./ and normalize + cf_norm = cf.replace('\\', '/').lstrip('./') + # Try: absolute path -> relative + if os.path.isabs(cf): + try: + cf_rel = os.path.relpath(cf, workspace) + except ValueError: + cf_rel = cf_norm + else: + cf_rel = cf_norm + # Try direct match + if cf_rel in all_known_files: + resolved_changed.append(cf_rel) + continue + # Try: maybe user passed full path that's already relative + if os.path.isfile(os.path.join(workspace, cf_rel)): + resolved_changed.append(cf_rel) + continue + # Try basename match (last resort — common in `git diff --name-only` + # output where the user is in a subdirectory) + basename = os.path.basename(cf_rel) + candidates = [f for f in all_known_files if os.path.basename(f) == basename] + if len(candidates) == 1: + resolved_changed.append(candidates[0]) + elif len(candidates) > 1: + # Ambiguous — pick the one whose path-suffix matches cf_rel + # (e.g., "scripts/foo.py" matches "scripts/foo.py" out of + # multiple "foo.py" files). If still ambiguous, skip. + suffix_match = [c for c in candidates if c.endswith(cf_rel) or cf_rel.endswith(c)] + if len(suffix_match) == 1: + resolved_changed.append(suffix_match[0]) + else: + unresolved.append(cf) + else: + unresolved.append(cf) + + # BFS over reverse_graph from each resolved changed file + # Track depth per node so we can respect the depth cap + visited: Dict[str, int] = {} # file -> depth-at-which-first-reached + queue: List[Tuple[str, int]] = [(cf, 0) for cf in resolved_changed] + safety_cap = 5000 + while queue and len(visited) < safety_cap: + current, d = queue.pop(0) + if current in visited and visited[current] <= d: + continue + visited[current] = d + if depth >= 0 and d >= depth: + continue + for dep in reverse_graph.get(current, set()): + if dep not in visited or visited[dep] > d + 1: + queue.append((dep, d + 1)) + + # Build affected_by_source mapping for traceability + affected_by_source: Dict[str, List[str]] = defaultdict(list) + affected_set: Set[str] = set() + for node, node_depth in visited.items(): + if node_depth == 0: + # The changed file itself — only include if include_source + # AND it passes the filter (test file or include_source). + if include_source and (not file_filter or fnmatch.fnmatch(node, file_filter)): + affected_set.add(node) + continue + # Apply filter + if file_filter and not fnmatch.fnmatch(node, file_filter): + continue + # Apply test-file gate + if not include_source and not is_test_file(node): + continue + affected_set.add(node) + + # Build affected_by_source: for each changed file, which affected files + # are reachable from it? We re-walk per-source to keep this accurate. + for cf in resolved_changed: + per_visited: Set[str] = set() + per_queue: List[Tuple[str, int]] = [(cf, 0)] + while per_queue: + current, d = per_queue.pop(0) + if current in per_visited: + continue + per_visited.add(current) + if depth >= 0 and d >= depth: + continue + for dep in reverse_graph.get(current, set()): + if dep not in per_visited: + per_queue.append((dep, d + 1)) + # Filter per_visited to affected-only + for v in per_visited: + if v == cf: + continue + if file_filter and not fnmatch.fnmatch(v, file_filter): + continue + if not include_source and not is_test_file(v): + continue + affected_by_source[cf].append(v) + + affected_sorted = sorted(affected_set) + test_count = sum(1 for f in affected_sorted if is_test_file(f)) + + return { + "status": "ok", + "workspace": workspace, + "changed_files": resolved_changed, + "unresolved": unresolved, + "affected": affected_sorted, + "affected_by_source": {k: sorted(v) for k, v in affected_by_source.items()}, + "depth": depth, + "stats": { + "changed_count": len(resolved_changed), + "unresolved_count": len(unresolved), + "affected_count": len(affected_sorted), + "affected_test_count": test_count, + "affected_source_count": len(affected_sorted) - test_count, + "visited_total": len(visited), + }, + } + + def get_dependents( file_path: str, workspace: str, @@ -482,24 +738,69 @@ def _resolve_path_alias(raw_import: str, workspace: str) -> Optional[str]: def _resolve_python_module(module_path: str, from_dir: str, workspace: str) -> Optional[str]: - """Resolve a Python module to an actual file.""" - # Try as a file + """Resolve a Python module to an actual file. + + Resolution order: + 1. Relative to the importing file's directory (``from_dir``). + 2. As a package (``__init__.py``) relative to ``from_dir``. + 3. From workspace root. + 4. As a package from workspace root. + 5. **Issue #62 Phase 1**: from ``scripts/`` — CodeLens convention is + ``sys.path.insert(0, SCRIPT_DIR)`` in tests and commands, so + ``from utils import X`` from ``tests/foo.py`` resolves to + ``scripts/utils.py``. Without this fallback, the reverse-dependency + graph misses test→script edges and ``codelens affected`` returns + false negatives. + 6. **Issue #62 Phase 1**: from ``scripts/`` — handles the + case where a test in ``tests/`` imports a sibling-named module + that lives in ``scripts//``. + """ + # 1. Try as a file, relative to from_dir full_path = os.path.join(workspace, from_dir, module_path + '.py') if os.path.isfile(full_path): return os.path.relpath(full_path, workspace) - # Try as a package + # 2. Try as a package, relative to from_dir init_path = os.path.join(workspace, from_dir, module_path, '__init__.py') if os.path.isfile(init_path): return os.path.relpath(init_path, workspace) - # Try from workspace root + # 3. Try from workspace root full_path = os.path.join(workspace, module_path + '.py') if os.path.isfile(full_path): return os.path.relpath(full_path, workspace) + # 4. Try as a package from workspace root init_path = os.path.join(workspace, module_path, '__init__.py') if os.path.isfile(init_path): return os.path.relpath(init_path, workspace) + # 5. Issue #62 Phase 1: try from scripts/ (CodeLens convention) + # Tests and commands do `sys.path.insert(0, SCRIPT_DIR)` then + # `from utils import X` — which resolves to scripts/utils.py. + scripts_dir = os.path.join(workspace, 'scripts') + if os.path.isdir(scripts_dir): + full_path = os.path.join(scripts_dir, module_path + '.py') + if os.path.isfile(full_path): + return os.path.relpath(full_path, workspace) + init_path = os.path.join(scripts_dir, module_path, '__init__.py') + if os.path.isfile(init_path): + return os.path.relpath(init_path, workspace) + + # 6. Issue #62 Phase 1: try from scripts/ — + # handles `from commands.scan import X` from tests/, which should + # resolve to scripts/commands/scan.py. + # module_path here may be 'commands/scan' (after .replace('.','/')) + if '/' in module_path: + first_seg = module_path.split('/')[0] + rest = module_path[len(first_seg) + 1:] + candidate_dir = os.path.join(scripts_dir, first_seg) + if os.path.isdir(candidate_dir): + full_path = os.path.join(candidate_dir, rest + '.py') + if os.path.isfile(full_path): + return os.path.relpath(full_path, workspace) + init_path = os.path.join(candidate_dir, rest, '__init__.py') + if os.path.isfile(init_path): + return os.path.relpath(init_path, workspace) + return None diff --git a/scripts/graph_model.py b/scripts/graph_model.py index 1790fb7a..a3cc6a08 100644 --- a/scripts/graph_model.py +++ b/scripts/graph_model.py @@ -13,7 +13,7 @@ - New tables `graph_nodes` and `graph_edges` are additive (prefixed `graph_` to avoid colliding with any existing table name). - The flat registry tables and JSON files are untouched. -- All 68 existing CLI commands continue to work unchanged. +- All 70 existing CLI commands continue to work unchanged. Schema: graph_nodes( diff --git a/skill.json b/skill.json index 20d1a64d..76595063 100755 --- a/skill.json +++ b/skill.json @@ -1,7 +1,7 @@ { "name": "codelens", "version": "8.2.0", - "description": "Live Codebase Reference Intelligence. 68 commands for AI-powered code analysis, security auditing, quality scoring, and pre-write safety checks. Supports 28+ languages with regex+AST hybrid parsing. Must activate before writing/editing/deleting any class, id, or function.", + "description": "Live Codebase Reference Intelligence. 70 commands for AI-powered code analysis, security auditing, quality scoring, and pre-write safety checks. Supports 28+ languages with regex+AST hybrid parsing. Must activate before writing/editing/deleting any class, id, or function.", "author": "codelens", "command_categories": { "setup": [ diff --git a/tests/test_affected_command.py b/tests/test_affected_command.py new file mode 100644 index 00000000..087422e6 --- /dev/null +++ b/tests/test_affected_command.py @@ -0,0 +1,460 @@ +""" +Tests for the `affected` command and `get_affected_files` engine (issue #62 Phase 1). + +Covers: +- test-file heuristic (is_test_file) +- transitive BFS resolution +- --stdin pipe behavior +- --depth / --filter / --include-source / --quiet / --json flags +- ambiguous basename handling +- cycle safety + +Run with:: + + python -m pytest tests/test_affected_command.py -v +""" + +from __future__ import annotations + +import io +import os +import sys +import tempfile +import textwrap + +import pytest + +SCRIPT_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "scripts") +sys.path.insert(0, SCRIPT_DIR) + +from dependents_engine import get_affected_files, is_test_file # noqa: E402 + + +# ─── is_test_file heuristic ─────────────────────────────────────────────── + + +@pytest.mark.parametrize("path,expected", [ + # Python conventions + ("tests/test_foo.py", True), + ("test_foo.py", True), + ("foo_test.py", True), + ("tests/foo.py", True), # tests/ directory + ("spec/foo.py", True), # spec/ directory + ("foo.py", False), # bare source + ("src/foo.py", False), + # JS/TS + ("foo.test.js", True), + ("foo.spec.ts", True), + ("foo.spec.tsx", True), + ("__tests__/foo.js", True), + ("foo.js", False), + # Go + ("foo_test.go", True), + ("foo.go", False), + # Java + ("FooTest.java", True), + ("FooTests.java", True), + ("Foo.java", False), + # C# + ("FooTest.cs", True), + ("Foo.cs", False), + # Non-source files + ("README.md", False), + ("package.json", False), + (".gitignore", False), + # Edge cases + ("", False), + ("test", False), # no extension, no test dir + ("tests", False), # directory-only path (no basename file) + # Path with multiple test segments + ("tests/unit/test_foo.py", True), + ("app/__tests__/foo.test.js", True), +]) +def test_is_test_file(path, expected): + assert is_test_file(path) is expected, f"is_test_file({path!r}) should be {expected}" + + +# ─── Fixture: small workspace with imports ────────────────────────────── + + +@pytest.fixture +def small_workspace(tmp_path): + """Build a small workspace with known import structure: + + src/ + models.py <- leaf, no imports + services.py <- imports models + api.py <- imports services + utils.py <- leaf + tests/ + test_models.py <- imports src.models + test_services.py <- imports src.services + test_api.py <- imports src.api + test_utils.py <- imports src.utils + conftest.py <- leaf + """ + ws = tmp_path / "ws" + ws.mkdir() + (ws / "src").mkdir() + (ws / "tests").mkdir() + + (ws / "src" / "models.py").write_text( + "class Model: pass\n", encoding="utf-8" + ) + (ws / "src" / "services.py").write_text( + "from src.models import Model\n\nclass Service: pass\n", + encoding="utf-8", + ) + (ws / "src" / "api.py").write_text( + "from src.services import Service\n\nclass API: pass\n", + encoding="utf-8", + ) + (ws / "src" / "utils.py").write_text( + "def helper(): pass\n", encoding="utf-8" + ) + (ws / "tests" / "test_models.py").write_text( + "from src.models import Model\n\ndef test_model(): assert Model\n", + encoding="utf-8", + ) + (ws / "tests" / "test_services.py").write_text( + "from src.services import Service\n\ndef test_service(): assert Service\n", + encoding="utf-8", + ) + (ws / "tests" / "test_api.py").write_text( + "from src.api import API\n\ndef test_api(): assert API\n", + encoding="utf-8", + ) + (ws / "tests" / "test_utils.py").write_text( + "from src.utils import helper\n\ndef test_helper(): assert helper\n", + encoding="utf-8", + ) + (ws / "conftest.py").write_text( + "# pytest conftest\n", encoding="utf-8" + ) + return str(ws) + + +# ─── get_affected_files — core BFS ────────────────────────────────────── + + +def test_affected_finds_direct_test_dependents(small_workspace): + """Changing models.py should affect test_models.py.""" + result = get_affected_files( + changed_files=["src/models.py"], + workspace=small_workspace, + ) + assert result["status"] == "ok" + affected = set(result["affected"]) + assert "tests/test_models.py" in affected + + +def test_affected_finds_transitive_test_dependents(small_workspace): + """Changing models.py should ALSO affect test_services.py and test_api.py + via the transitive chain models -> services -> api.""" + result = get_affected_files( + changed_files=["src/models.py"], + workspace=small_workspace, + depth=5, + ) + affected = set(result["affected"]) + # Direct: test_models.py imports models + assert "tests/test_models.py" in affected + # Transitive: test_services.py imports services which imports models + assert "tests/test_services.py" in affected + # Transitive: test_api.py imports api which imports services which imports models + assert "tests/test_api.py" in affected + + +def test_affected_does_not_include_unrelated_tests(small_workspace): + """Changing models.py should NOT affect test_utils.py (separate chain).""" + result = get_affected_files( + changed_files=["src/models.py"], + workspace=small_workspace, + ) + affected = set(result["affected"]) + assert "tests/test_utils.py" not in affected + + +def test_affected_depth_zero_returns_nothing(small_workspace): + """depth=0 means only the changed file itself, and since it's not a test, + nothing is returned.""" + result = get_affected_files( + changed_files=["src/models.py"], + workspace=small_workspace, + depth=0, + ) + assert result["affected"] == [] + + +def test_affected_depth_one_finds_direct_only(small_workspace): + """depth=1 means direct dependents only — no transitive propagation.""" + result = get_affected_files( + changed_files=["src/models.py"], + workspace=small_workspace, + depth=1, + ) + affected = set(result["affected"]) + assert "tests/test_models.py" in affected # direct + assert "tests/test_services.py" not in affected # transitive — too deep + assert "tests/test_api.py" not in affected # transitive — too deep + + +def test_affected_include_source_returns_non_test_dependents(small_workspace): + """With include_source=True, services.py should be returned as an + affected source file when models.py changes.""" + result = get_affected_files( + changed_files=["src/models.py"], + workspace=small_workspace, + include_source=True, + ) + affected = set(result["affected"]) + assert "src/services.py" in affected # non-test dependent + assert "src/api.py" in affected # transitive non-test dependent + assert "tests/test_models.py" in affected + assert result["stats"]["affected_source_count"] >= 2 + + +def test_affected_filter_glob(small_workspace): + """--filter 'tests/test_s*' should only return test_services.py.""" + result = get_affected_files( + changed_files=["src/models.py"], + workspace=small_workspace, + file_filter="tests/test_s*", + ) + affected = set(result["affected"]) + assert "tests/test_services.py" in affected + assert "tests/test_models.py" not in affected + assert "tests/test_api.py" not in affected + + +def test_affected_accepts_absolute_path(small_workspace): + abs_path = os.path.join(small_workspace, "src", "models.py") + result = get_affected_files( + changed_files=[abs_path], + workspace=small_workspace, + ) + assert result["stats"]["changed_count"] == 1 + assert "tests/test_models.py" in result["affected"] + + +def test_affected_accepts_bare_basename(small_workspace): + """Bare basename 'models.py' should resolve to src/models.py since it's + unique in this workspace.""" + result = get_affected_files( + changed_files=["models.py"], + workspace=small_workspace, + ) + assert result["stats"]["changed_count"] == 1 + assert "src/models.py" in result["changed_files"] + + +def test_affected_unresolved_files_listed(small_workspace): + """Non-existent files should be reported in `unresolved`, not crash.""" + result = get_affected_files( + changed_files=["src/nonexistent.py", "src/models.py"], + workspace=small_workspace, + ) + assert "src/nonexistent.py" in result["unresolved"] + assert result["stats"]["changed_count"] == 1 # only models.py resolved + + +def test_affected_empty_input(small_workspace): + """Empty changed_files list should return empty affected.""" + result = get_affected_files( + changed_files=[], + workspace=small_workspace, + ) + assert result["affected"] == [] + assert result["stats"]["affected_count"] == 0 + + +def test_affected_multiple_changed_files(small_workspace): + """Multiple changed files should union their affected sets.""" + result = get_affected_files( + changed_files=["src/models.py", "src/utils.py"], + workspace=small_workspace, + ) + affected = set(result["affected"]) + assert "tests/test_models.py" in affected + assert "tests/test_utils.py" in affected + assert "tests/test_services.py" in affected # transitive from models + + +def test_affected_by_source_mapping(small_workspace): + """affected_by_source should map each changed file to its affected tests.""" + result = get_affected_files( + changed_files=["src/models.py", "src/utils.py"], + workspace=small_workspace, + ) + by_src = result["affected_by_source"] + assert "src/models.py" in by_src + assert "src/utils.py" in by_src + assert "tests/test_models.py" in by_src["src/models.py"] + assert "tests/test_utils.py" in by_src["src/utils.py"] + # utils.py shouldn't pull in test_models.py + assert "tests/test_models.py" not in by_src["src/utils.py"] + + +def test_affected_whitespace_only_input_skipped(small_workspace): + """Whitespace-only lines should be skipped, not crash.""" + result = get_affected_files( + changed_files=[" ", "", "src/models.py", " "], + workspace=small_workspace, + ) + assert result["stats"]["changed_count"] == 1 + + +# ─── Cycle safety ──────────────────────────────────────────────────────── + + +def test_affected_handles_cycles(tmp_path): + """Two modules importing each other should not infinite-loop.""" + ws = tmp_path / "ws" + ws.mkdir() + (ws / "a.py").write_text("import b\n", encoding="utf-8") + (ws / "b.py").write_text("import a\n", encoding="utf-8") + (ws / "test_a.py").write_text("import a\n", encoding="utf-8") + + result = get_affected_files( + changed_files=["a.py"], + workspace=str(ws), + depth=10, + ) + assert result["status"] == "ok" + assert "test_a.py" in result["affected"] + # Safety cap should prevent runaway + assert result["stats"]["visited_total"] < 100 + + +def test_affected_safety_cap_on_large_graph(tmp_path): + """Even with depth=-1, the 5000-node safety cap should kick in.""" + # Build a graph that would explode without a cap: a chain of N files + # each importing the next, plus tests at each level. + ws = tmp_path / "ws" + ws.mkdir() + n = 200 + for i in range(n): + if i == 0: + (ws / f"mod_{i}.py").write_text("# root\n", encoding="utf-8") + else: + (ws / f"mod_{i}.py").write_text( + f"import mod_{i - 1}\n", encoding="utf-8" + ) + (ws / f"test_mod_{i}.py").write_text( + f"import mod_{i}\n", encoding="utf-8" + ) + result = get_affected_files( + changed_files=["mod_0.py"], + workspace=str(ws), + depth=-1, # unlimited + ) + assert result["status"] == "ok" + assert result["stats"]["visited_total"] <= 5000 + + +# ─── Command-level tests (add_args / execute wiring) ───────────────────── + + +class _Args: + """Minimal args stub mimicking argparse.Namespace.""" + def __init__(self, **kw): + for k, v in kw.items(): + setattr(self, k, v) + + +def test_command_add_args_registers_all_flags(): + from commands.affected import add_args + import argparse + + p = argparse.ArgumentParser() + add_args(p) + # Parse with typical flags + ns = p.parse_args(["--stdin", "-d", "3", "-f", "tests/*.py", "-j", "-q", + "--include-source", "src/foo.py"]) + assert ns.stdin is True + assert ns.depth == 3 + assert ns.filter == "tests/*.py" + assert ns.as_json is True + assert ns.quiet is True + assert ns.include_source is True + assert ns.files == ["src/foo.py"] + + +def test_command_execute_with_no_files_returns_error(small_workspace): + from commands.affected import execute + + args = _Args(files=None, stdin=False, depth=5, filter=None, + as_json=False, quiet=False, include_source=False) + result = execute(args, small_workspace) + assert result["status"] == "error" + assert "no changed files" in result["error"].lower() + + +def test_command_execute_with_files_returns_result(small_workspace): + from commands.affected import execute + + args = _Args(files=["src/models.py"], stdin=False, depth=5, filter=None, + as_json=False, quiet=False, include_source=False) + result = execute(args, small_workspace) + assert result["status"] == "ok" + assert "tests/test_models.py" in result["affected"] + + +def test_command_execute_quiet_mode_prints_paths(small_workspace, capsys): + from commands.affected import execute + + args = _Args(files=["src/models.py"], stdin=False, depth=5, filter=None, + as_json=False, quiet=True, include_source=False) + result = execute(args, small_workspace) + captured = capsys.readouterr() + # Quiet mode prints paths to stdout, one per line + assert "tests/test_models.py" in captured.out + assert "status" not in captured.out # no JSON noise + assert result["quiet"] is True + + +def test_command_execute_reads_stdin(small_workspace, monkeypatch): + from commands.affected import execute + + # Simulate `git diff --name-only | codelens affected --stdin` + fake_stdin = io.StringIO("src/models.py\nsrc/utils.py\n") + monkeypatch.setattr("sys.stdin", fake_stdin) + + args = _Args(files=None, stdin=True, depth=5, filter=None, + as_json=False, quiet=False, include_source=False) + result = execute(args, small_workspace) + assert result["status"] == "ok" + assert result["stats"]["changed_count"] == 2 + affected = set(result["affected"]) + assert "tests/test_models.py" in affected + assert "tests/test_utils.py" in affected + + +def test_command_execute_stdin_with_comments_and_blanks(small_workspace, monkeypatch): + """stdin input should skip blank lines and comments.""" + from commands.affected import execute + + fake_stdin = io.StringIO( + "# changed files from git diff\n" + "\n" + "src/models.py\n" + " \n" + "# another comment\n" + ) + monkeypatch.setattr("sys.stdin", fake_stdin) + + args = _Args(files=None, stdin=True, depth=5, filter=None, + as_json=False, quiet=False, include_source=False) + result = execute(args, small_workspace) + assert result["stats"]["changed_count"] == 1 + + +def test_command_registered_in_registry(): + """The command should be auto-registered via commands/__init__.py.""" + from commands import COMMAND_REGISTRY + + assert "affected" in COMMAND_REGISTRY + entry = COMMAND_REGISTRY["affected"] + assert entry["help"] + assert callable(entry["add_args"]) + assert callable(entry["execute"])