diff --git a/README.md b/README.md index 8396bec9..eb8ea12e 100644 --- a/README.md +++ b/README.md @@ -2,12 +2,12 @@ > **Before an AI writes a new class/id/function, CodeLens must be checked. This is not optional.** -CodeLens is an AI-native code intelligence platform that gives AI agents **full visibility** into a codebase before they write any code. It prevents collision, overwrite of existing logic, security vulnerabilities, and dead code through 70 CLI commands, an MCP server with 68 tools (55 static + 13 dynamic), AST-based taint analysis, live CVE/OSV scanning, a plugin system with OWASP Top 10 + Compliance rule packs, a true graph data model (nodes + edges) for structural code queries, and token-efficient `--format compact` output for high-volume agent workflows (issue #17). +CodeLens is an AI-native code intelligence platform that gives AI agents **full visibility** into a codebase before they write any code. It prevents collision, overwrite of existing logic, security vulnerabilities, and dead code through 72 CLI commands, an MCP server with 70 tools (54 static + 16 dynamic), AST-based taint analysis, live CVE/OSV scanning, a plugin system with OWASP Top 10 + Compliance rule packs, a true graph data model (nodes + edges) for structural code queries, and token-efficient `--format compact` output for high-volume agent workflows (issue #17). ## Features -- **70 CLI Commands** — From basic scan/query to AST taint analysis, CVE scanning, plugin management, auto-fix, dashboards, CI/CD quality gates, and `graph-schema` for cheap graph-shape introspection -- **MCP Server (68 Tools)** — Native AI agent integration via Model Context Protocol (JSON-RPC over stdio), 55 statically-defined tools + 13 dynamically discovered, every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`) +- **72 CLI Commands** — From basic scan/query to AST taint analysis, CVE scanning, plugin management, auto-fix, dashboards, CI/CD quality gates, and `graph-schema` for cheap graph-shape introspection +- **MCP Server (70 Tools)** — Native AI agent integration via Model Context Protocol (JSON-RPC over stdio), 54 statically-defined tools + 16 dynamically discovered, every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`) - **Token-Efficient Compact Output (v8.2, issue #17)** — `--format compact` produces single-char-key JSON with abbreviated types, omitted null fields, and relative paths — ~50% smaller than `json` on real trace output. Combined with `--limit`/`--offset` pagination, 5 structural queries now cost <5k tokens (down from 30-80k) - **AST Taint Engine** — Tree-sitter based taint analysis with return-value propagation, scope hierarchy, and branch condition refinement - **Live CVE/OSV Scanning** — Real-time vulnerability data from OSV.dev API with SQLite cache, 9 ecosystems (PyPI, npm, crates.io, Go, Maven, NuGet, RubyGems, Pub, Hex) @@ -225,8 +225,8 @@ codelens/ │ ├── changelog.md # Older changelog (per-version highlights) │ └── agent-integration.md # AI agent integration guide ├── scripts/ -│ ├── codelens.py # CLI entry point (70 commands registered) -│ ├── mcp_server.py # MCP JSON-RPC server (68 tools) +│ ├── codelens.py # CLI entry point (72 commands registered) +│ ├── mcp_server.py # MCP JSON-RPC server (70 tools) │ ├── registry.py # Registry read/write/build │ ├── persistent_registry.py # SQLite persistent storage (WAL mode) │ ├── base_parser.py # Base tree-sitter parser diff --git a/SKILL-QUICK.md b/SKILL-QUICK.md index 79bb01fb..a59bc713 100755 --- a/SKILL-QUICK.md +++ b/SKILL-QUICK.md @@ -115,7 +115,7 @@ $CLI list --limit 5 --offset 10 --format compact # → paginated + co | "Cross-file taint" | `dataflow` | `taint` (taint is single-file, AST-deep) | | "Auto-fix issues" | `fix` | `check` (check just gates, doesn't fix) | -## All 70 Commands +## All 72 Commands ### Setup & Lifecycle (8+) `init` · `scan [--incremental] [--max-files N] [--full]` · `registry-validate` · `detect` · `watch [--debounce SECS] [--git-mode] [--interval SECS]` · `git-status` · `migrate` · `serve` · `lsp-status` (issue #33: `codelens --lsp-status` top-level flag is an alias of `codelens lsp-status` — both delegate to `hybrid_engine.get_lsp_status()` and return the identical payload) @@ -147,9 +147,9 @@ $CLI list --limit 5 --offset 10 --format compact # → paginated + co ### Tooling (1) `plugin ` -**Total: 70 commands** (auto-registered via `commands/__init__.py`; rerun `python3 scripts/sync_command_count.py --apply` after adding/removing a command) +**Total: 72 commands** (auto-registered via `commands/__init__.py`; rerun `python3 scripts/sync_command_count.py --apply` after adding/removing a command) -## MCP Server (68 Tools) +## MCP Server (70 Tools) Start the MCP server for AI agent integration: @@ -157,9 +157,9 @@ Start the MCP server for AI agent integration: python3 scripts/codelens.py serve ``` -Exposes 68 tools as `codelens_` (e.g., `codelens_query`, `codelens_taint`, `codelens_graph_schema`, `codelens_architecture`, `codelens_resolve_types`, `codelens_git_status`): +Exposes 70 tools as `codelens_` (e.g., `codelens_query`, `codelens_taint`, `codelens_graph_schema`, `codelens_architecture`, `codelens_resolve_types`, `codelens_git_status`): - 50 statically-defined tools (full JSON schemas in `mcp_server.py`) -- 13 dynamically-discovered tools (auto-discovered from `COMMAND_REGISTRY`; long-running `watch` and `serve` are excluded) +- 16 dynamically-discovered tools (auto-discovered from `COMMAND_REGISTRY`; long-running `watch` and `serve` are excluded) - Every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`). Use `format: "compact"` for token-efficient responses (~50% smaller than `json`). - `watch` and `serve` itself are excluded (long-running) diff --git a/SKILL.md b/SKILL.md index 5ccdfd59..c3be3fdd 100755 --- a/SKILL.md +++ b/SKILL.md @@ -1,10 +1,10 @@ --- name: codelens description: > - CodeLens — AI-Native Code Intelligence. 70 commands for AI-powered code analysis, + CodeLens — AI-Native Code Intelligence. 72 commands for AI-powered code analysis, security auditing, quality scoring, AST-based taint analysis, live CVE scanning, and pre-write safety checks. Supports 28+ languages with tree-sitter + regex - fallback parsing. MCP server exposes 68 tools for AI agent integration. + fallback parsing. MCP server exposes 70 tools for AI agent integration. For quick command reference with validated output schemas, see SKILL-QUICK.md. For version history, see CHANGELOG.md. --- diff --git a/pyproject.toml b/pyproject.toml index cfb46efa..616c2983 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta" [project] name = "codelens" version = "8.2.0" -description = "Live Codebase Reference Intelligence — 70 commands for AI-powered code analysis, security auditing, and quality scoring" +description = "Live Codebase Reference Intelligence — 72 commands for AI-powered code analysis, security auditing, and quality scoring" readme = "README.md" license = {text = "MIT"} requires-python = ">=3.8" diff --git a/scripts/commands/query.py b/scripts/commands/query.py index 8099cc4c..16cd2e13 100644 --- a/scripts/commands/query.py +++ b/scripts/commands/query.py @@ -39,11 +39,42 @@ def add_args(parser): help="Return all callers/callees (no limit)") parser.add_argument("--fuzzy", action="store_true", help="Enable fuzzy/substring matching (case-insensitive)") + parser.add_argument( + "--additional-paths", + default=None, + metavar="PATHS", + help=( + "Comma-separated list of additional repo root paths to search " + "across (issue #15 cross-repo intelligence). When provided, the " + "query loads and merges registries from all repos and resolves " + "cross-repo call edges. Example: " + "'--additional-paths ../lib/shared,../services/worker'." + ), + ) def execute(args, workspace): limit = None if getattr(args, 'all', False) else getattr(args, 'limit', 20) fuzzy = getattr(args, 'fuzzy', False) + + # ── Cross-repo path (issue #15) ── + # When --additional-paths is provided, delegate to the cross-repo + # engine which merges registries from all repos and resolves + # cross-repo call edges. The result shape is compatible with the + # single-repo query result, so downstream formatting (compact, ai, + # etc.) works unchanged. + additional_paths_raw = getattr(args, 'additional_paths', None) + if additional_paths_raw: + from crossrepo_engine import query_cross_repo, _parse_additional_paths + additional_paths = _parse_additional_paths(additional_paths_raw) + result = query_cross_repo( + args.name, workspace, additional_paths, + fuzzy=fuzzy, limit=limit if limit else 20, + ) + _attach_baseline_confidence(result, args.name, workspace) + return result + + # ── Single-repo path (default, unchanged) ── result = cmd_query(args.name, workspace, args.domain, args.file, limit=limit, fuzzy=fuzzy) _attach_baseline_confidence(result, args.name, workspace) return result diff --git a/scripts/crossrepo_engine.py b/scripts/crossrepo_engine.py new file mode 100644 index 00000000..9d767f91 --- /dev/null +++ b/scripts/crossrepo_engine.py @@ -0,0 +1,392 @@ +""" +Cross-Repo Intelligence Engine for CodeLens (issue #15 MVP). + +Provides multi-repo symbol table merging and cross-repo edge resolution so +agents can trace calls that cross repository boundaries — e.g. a call from +``services/api`` into ``lib/shared``. + +Implements the Minimum Viable Version from issue #15: + Just support multiple ``repo_path`` entries — resolve imports against + the combined symbol table from all indexed repos. + +Design +------ +This module is **non-breaking and additive**. It does NOT modify: + +- ``scan`` — each repo is scanned independently as before. +- ``init`` — no config schema changes required (additional paths are + passed via the ``--additional-paths`` CLI flag). +- ``load_backend_registry`` — the single-repo loader is untouched. +- ``resolve_edges`` — the existing resolver is reused as-is. + +Instead, it sits *on top* of the existing registry layer: + +1. Loads the backend registry from the primary workspace. +2. Loads backend registries from each additional repo path. +3. Tags every node with a ``repo`` field (the workspace path it came + from) so downstream consumers can tell which repo a symbol lives in. +4. Merges all nodes + raw edges into combined lists. +5. Re-runs :func:`edge_resolver.resolve_edges` on the combined set. + Because the resolver matches ``to_fn`` (function name) against the + full node list, a call from repo A to a function defined in repo B + is now resolved automatically — the resolver doesn't care which repo + a node came from. +6. Post-processes the resolved edges: any edge whose ``from`` node and + ``to`` node are in different repos gets ``cross_repo: True`` and a + ``target_repo`` field, so agents can distinguish intra-repo calls + from cross-repo calls. + +The merged registry has the same shape as a single-repo backend +registry (``{"nodes": [...], "edges": [...]}``), so it can be fed +directly into :func:`edge_resolver.get_callers`, +:func:`edge_resolver.get_callees`, and the query logic in +:func:`commands.query.cmd_query`. + +Public surface +-------------- +- :func:`load_merged_registry` — merge multiple repos into one registry. +- :func:`query_cross_repo` — multi-repo symbol query (drop-in + replacement for single-repo query when additional paths are given). +""" + +from __future__ import annotations + +import os +from typing import Any, Dict, List, Optional, Tuple + +from registry import load_backend_registry +from edge_resolver import resolve_edges +from utils import logger + + +# ─── Path Normalization ─────────────────────────────────────── + +def _norm_path(p: str) -> str: + """Normalize a workspace path for comparison (absolute, no trailing /).""" + return os.path.abspath(p).rstrip(os.sep) if p else "" + + +def _parse_additional_paths(raw: Optional[str]) -> List[str]: + """Parse the ``--additional-paths`` CLI value into a list of paths. + + Accepts comma-separated paths (``"path1,path2,path3"``) and ignores + empty entries. Whitespace around each path is stripped. + + Args: + raw: The raw string from ``--additional-paths``, or ``None``. + + Returns: + List of absolute path strings. Empty if ``raw`` is ``None`` or + all entries are empty/whitespace. + """ + if not raw: + return [] + parts = [p.strip() for p in raw.split(",")] + return [os.path.abspath(p) for p in parts if p] + + +# ─── Registry Merging ───────────────────────────────────────── + +def load_merged_registry( + primary_workspace: str, + additional_paths: Optional[List[str]] = None, +) -> Dict[str, Any]: + """Load and merge backend registries from multiple repos. + + Loads the primary workspace's backend registry plus each additional + repo's registry, tags every node with its source ``repo`` path, and + re-resolves edges against the combined symbol table so cross-repo + calls are resolved. + + Args: + primary_workspace: Absolute path to the primary workspace root. + additional_paths: List of additional repo root paths to merge. + May be ``None`` or empty (returns just the primary registry + with ``repo`` tags added — still a valid merged registry). + + Returns: + Dict with the same shape as a single-repo backend registry:: + + { + "workspace": "", + "repos": ["", "", ...], + "nodes": [...], # merged, each with "repo" field + "edges": [...], # re-resolved, cross-repo flagged + "stats": { + "total_nodes": , + "total_edges": , + "cross_repo_edges": , + "repos_merged": , + }, + } + + If the primary workspace has no registry (never scanned), returns + an empty merged registry with ``repos_merged: 0``. If additional + repos can't be loaded, they're silently skipped (counted in + ``repos_merged`` only if they contributed at least one node). + """ + primary_workspace = os.path.abspath(primary_workspace) + additional_paths = additional_paths or [] + + # ── Load primary ── + primary_reg = load_backend_registry(primary_workspace) + primary_nodes = primary_reg.get("nodes", []) + primary_edges = primary_reg.get("edges", []) + + # Tag primary nodes with repo + for node in primary_nodes: + node.setdefault("repo", primary_workspace) + + # Collect raw edges — we need `from` and `to_fn` for re-resolution. + # Already-resolved edges still carry `to_fn`, so we can pass them + # through resolve_edges which will re-match against the merged node set. + all_nodes: List[Dict[str, Any]] = list(primary_nodes) + all_raw_edges: List[Dict[str, Any]] = list(primary_edges) + + repos_merged = 1 if primary_nodes else 0 + + # ── Load additionals ── + for repo_path in additional_paths: + repo_path = os.path.abspath(repo_path) + if repo_path == primary_workspace: + # Skip if same as primary (user accidentally listed it twice) + continue + try: + reg = load_backend_registry(repo_path) + nodes = reg.get("nodes", []) + edges = reg.get("edges", []) + if not nodes: + logger.debug(f"crossrepo: additional repo {repo_path} has no nodes, skipping") + continue + # Tag with repo + for node in nodes: + node.setdefault("repo", repo_path) + all_nodes.extend(nodes) + all_raw_edges.extend(edges) + repos_merged += 1 + except Exception as e: + logger.warning( + f"crossrepo: failed to load additional repo {repo_path}: {e}", + exc_info=True, + ) + + # ── Re-resolve edges against the merged node set ── + # resolve_edges returns (resolved_nodes, resolved_edges). The + # resolved_nodes are the same nodes we passed in (possibly with + # `duplicate_define` flags set). The resolved_edges have `to` + # (target node_id) filled in where a match was found. + if all_nodes and all_raw_edges: + resolved_nodes, resolved_edges = resolve_edges(all_nodes, all_raw_edges) + else: + resolved_nodes = all_nodes + resolved_edges = [] + + # ── Tag cross-repo edges ── + # Build a node_id → repo map for O(1) lookup + node_repo: Dict[str, str] = {} + for node in resolved_nodes: + node_id = node.get("id", "") + repo = node.get("repo", "") + if node_id and repo: + node_repo[node_id] = repo + + cross_repo_count = 0 + for edge in resolved_edges: + from_id = edge.get("from", "") + to_id = edge.get("to", "") + from_repo = node_repo.get(from_id, "") + to_repo = node_repo.get(to_id, "") + if from_repo and to_repo and from_repo != to_repo: + edge["cross_repo"] = True + edge["target_repo"] = to_repo + edge["source_repo"] = from_repo + cross_repo_count += 1 + else: + edge["cross_repo"] = False + + return { + "workspace": primary_workspace, + "repos": [primary_workspace] + [ + p for p in additional_paths if os.path.abspath(p) != primary_workspace + ], + "nodes": resolved_nodes, + "edges": resolved_edges, + "stats": { + "total_nodes": len(resolved_nodes), + "total_edges": len(resolved_edges), + "cross_repo_edges": cross_repo_count, + "repos_merged": repos_merged, + }, + } + + +# ─── Multi-Repo Query ───────────────────────────────────────── + +def query_cross_repo( + name: str, + primary_workspace: str, + additional_paths: Optional[List[str]] = None, + fuzzy: bool = False, + limit: int = 20, +) -> Dict[str, Any]: + """Query a symbol across multiple repos. + + This is a multi-repo version of :func:`commands.query.cmd_query`. It + loads the merged registry, finds matching nodes across all repos, + and returns callers/callees with ``repo`` and ``cross_repo`` + annotations so agents can see which calls cross repo boundaries. + + Args: + name: Symbol name to query. + primary_workspace: Primary workspace root path. + additional_paths: Additional repo paths to search across. + fuzzy: Enable fuzzy/substring matching (case-insensitive). + limit: Max callers/callees to return per result. + + Returns: + Dict with the same shape as ``cmd_query`` output, plus: + + * Each node in the result has a ``repo`` field. + * Each caller/callee that crosses a repo boundary has + ``cross_repo: True`` and ``target_repo`` / ``source_repo``. + * ``stats`` includes ``repos_searched`` and ``cross_repo_edges``. + + If the name is not found in any repo, returns a not-found result + with ``action: "CREATE"`` (same as single-repo query). + """ + from edge_resolver import get_callers, get_callees + from utils import deduplicate_callers + + primary_workspace = os.path.abspath(primary_workspace) + additional_paths = additional_paths or [] + + merged = load_merged_registry(primary_workspace, additional_paths) + nodes = merged.get("nodes", []) + edges = merged.get("edges", []) + repos_searched = merged.get("stats", {}).get("repos_merged", 0) + + # ── Exact + fuzzy match across all repos ── + name_lower = name.lower() + exact_matches = [] + fuzzy_matches = [] + + for node in nodes: + fn = node.get("fn", "") + if fn == name: + exact_matches.append(node) + elif fuzzy and name_lower in fn.lower() and fn.lower() != name_lower: + fuzzy_matches.append(node) + + # If we have exact matches, use those; otherwise fall back to fuzzy + matches = exact_matches if exact_matches else fuzzy_matches + + if not matches: + return { + "status": "ok", + "found": False, + "query": name, + "action": "CREATE", + "action_reason": "Name does not exist in any searched repo. Safe to create.", + "repos_searched": repos_searched, + "stats": merged.get("stats", {}), + } + + # ── Single match: return detailed result with callers/callees ── + if len(matches) == 1: + node = matches[0] + node_repo = node.get("repo", primary_workspace) + + all_callers = deduplicate_callers(get_callers(node["id"], edges, nodes)) + all_callees = get_callees(node["id"], edges, nodes) + + # Annotate callers/callees with cross-repo info + node_repo_map = {n.get("id", ""): n.get("repo", "") for n in nodes} + + for caller in all_callers: + from_id = caller.get("from", "") + caller_repo = node_repo_map.get(from_id, "") + caller["repo"] = caller_repo + caller["cross_repo"] = bool( + caller_repo and caller_repo != node_repo + ) + + for callee in all_callees: + to_id = callee.get("to", "") + callee_repo = node_repo_map.get(to_id, "") + callee["repo"] = callee_repo + callee["cross_repo"] = bool( + callee_repo and callee_repo != node_repo + ) + + total_callers = len(all_callers) + total_callees = len(all_callees) + callers = all_callers[:limit] if limit and limit > 0 else all_callers + callees = all_callees[:limit] if limit and limit > 0 else all_callees + + return { + "status": "ok", + "found": True, + "type": node.get("type", "function"), + "query": name, + "repo": node_repo, + "cross_repo_search": repos_searched > 1, + "repos_searched": repos_searched, + "node": { + "id": node["id"], + "fn": node["fn"], + "ref_count": node.get("ref_count", 0), + "status": node.get("status", "active"), + "file": node.get("file", ""), + "line": node.get("line", 0), + "repo": node_repo, + }, + "callers": callers, + "callees": callees, + "pagination": { + "callers_total": total_callers, + "callees_total": total_callees, + "callers_shown": len(callers), + "callees_shown": len(callees), + "has_more_callers": total_callers > len(callers), + "has_more_callees": total_callees > len(callees), + }, + "stats": merged.get("stats", {}), + } + + # ── Multiple matches: return summary list ── + matches_summary = [] + for node in matches[:limit] if limit and limit > 0 else matches: + node_repo = node.get("repo", primary_workspace) + matches_summary.append({ + "id": node["id"], + "fn": node["fn"], + "file": node.get("file", ""), + "line": node.get("line", 0), + "status": node.get("status", "active"), + "ref_count": node.get("ref_count", 0), + "repo": node_repo, + }) + + total_matches = len(matches) + return { + "status": "ok", + "found": True, + "type": "multi_match", + "query": name, + "match_type": "exact" if exact_matches else "fuzzy", + "cross_repo_search": repos_searched > 1, + "repos_searched": repos_searched, + "match_count": total_matches, + "action": "LIST_FIRST", + "action_reason": ( + f"Found {total_matches} definitions across {repos_searched} repo(s). " + "List all before making changes." + ), + "matches": matches_summary, + "pagination": { + "total_matches": total_matches, + "shown": len(matches_summary), + "has_more": total_matches > len(matches_summary), + }, + "stats": merged.get("stats", {}), + } diff --git a/scripts/graph_model.py b/scripts/graph_model.py index a3cc6a08..3c73c31c 100644 --- a/scripts/graph_model.py +++ b/scripts/graph_model.py @@ -13,7 +13,7 @@ - New tables `graph_nodes` and `graph_edges` are additive (prefixed `graph_` to avoid colliding with any existing table name). - The flat registry tables and JSON files are untouched. -- All 70 existing CLI commands continue to work unchanged. +- All 72 existing CLI commands continue to work unchanged. Schema: graph_nodes( diff --git a/scripts/mcp_server.py b/scripts/mcp_server.py index b1bc7c76..588974ce 100644 --- a/scripts/mcp_server.py +++ b/scripts/mcp_server.py @@ -78,7 +78,7 @@ } }, "query": { - "description": "Look up a symbol (function/class/variable/CSS class/ID) in the codebase registry. Returns action recommendation (CREATE/EXTEND/ASK/STOP).", + "description": "Look up a symbol (function/class/variable/CSS class/ID) in the codebase registry. Returns action recommendation (CREATE/EXTEND/ASK/STOP). Supports cross-repo search via additional_paths (issue #15).", "parameters": { "type": "object", "properties": { @@ -108,6 +108,11 @@ "type": "integer", "description": "Max callers/callees to return (default: 20)", "default": 20 + }, + "additional_paths": { + "type": "string", + "description": "Comma-separated additional repo root paths for cross-repo search (issue #15). When provided, merges registries from all repos and resolves cross-repo call edges. Example: '../lib/shared,../services/worker'", + "default": None } }, "required": ["name", "workspace"] diff --git a/skill.json b/skill.json index 76595063..213f5bca 100755 --- a/skill.json +++ b/skill.json @@ -1,7 +1,7 @@ { "name": "codelens", "version": "8.2.0", - "description": "Live Codebase Reference Intelligence. 70 commands for AI-powered code analysis, security auditing, quality scoring, and pre-write safety checks. Supports 28+ languages with regex+AST hybrid parsing. Must activate before writing/editing/deleting any class, id, or function.", + "description": "Live Codebase Reference Intelligence. 72 commands for AI-powered code analysis, security auditing, quality scoring, and pre-write safety checks. Supports 28+ languages with regex+AST hybrid parsing. Must activate before writing/editing/deleting any class, id, or function.", "author": "codelens", "command_categories": { "setup": [ diff --git a/tests/test_crossrepo_engine.py b/tests/test_crossrepo_engine.py new file mode 100644 index 00000000..be903d7b --- /dev/null +++ b/tests/test_crossrepo_engine.py @@ -0,0 +1,410 @@ +""" +Tests for the cross-repo intelligence engine (issue #15 MVP). + +Verifies that :func:`crossrepo_engine.load_merged_registry` correctly +merges symbol tables from multiple repos and that cross-repo call edges +are resolved and flagged. + +Test strategy: +- Create two temp workspaces, each with a ``backend.json`` registry + containing nodes and raw edges. +- Repo A has a function ``caller_fn`` that calls ``shared_helper``. +- Repo B defines ``shared_helper``. +- After merging, the cross-repo edge from ``caller_fn`` to + ``shared_helper`` should be resolved (``resolved: True``) and flagged + with ``cross_repo: True``. +- :func:`query_cross_repo` should find ``shared_helper`` when queried + from repo A with repo B as an additional path, and the caller list + should show ``caller_fn`` with ``cross_repo: True``. +""" + +import json +import os +import shutil +import sys +import tempfile + +import pytest + +# Make scripts/ importable +_SCRIPT_DIR = os.path.join( + os.path.dirname(os.path.dirname(os.path.abspath(__file__))), + "scripts", +) +if _SCRIPT_DIR not in sys.path: + sys.path.insert(0, _SCRIPT_DIR) + +from crossrepo_engine import ( + _parse_additional_paths, + load_merged_registry, + query_cross_repo, +) + + +# ─── Fixtures ───────────────────────────────────────────────── + +def _make_backend_json(workspace: str, nodes: list, edges: list) -> None: + """Write a ``backend.json`` registry into ``workspace/.codelens/``.""" + codelens_dir = os.path.join(workspace, ".codelens") + os.makedirs(codelens_dir, exist_ok=True) + data = { + "last_updated": "2026-07-01T00:00:00Z", + "workspace": workspace, + "nodes": nodes, + "edges": edges, + } + with open(os.path.join(codelens_dir, "backend.json"), "w", encoding="utf-8") as f: + json.dump(data, f, ensure_ascii=False) + + +def _make_node(fn: str, file: str, line: int, repo: str, node_id: str = None) -> dict: + """Build a minimal backend node dict.""" + if node_id is None: + node_id = f"{file}:{line}:{fn}" + return { + "id": node_id, + "fn": fn, + "file": file, + "line": line, + "type": "function", + "status": "active", + "ref_count": 0, + "repo": repo, + } + + +def _make_raw_edge(from_id: str, to_fn: str) -> dict: + """Build a minimal raw (unresolved) edge dict.""" + return { + "from": from_id, + "to_fn": to_fn, + "resolved": False, + "type": "call", + } + + +@pytest.fixture +def two_repos(): + """Create two temp workspaces with cross-repo call relationships. + + Repo A (primary): + - caller_fn in services/api/handler.py calls shared_helper + - local_fn in services/api/utils.py (no cross-repo calls) + + Repo B (additional): + - shared_helper in lib/shared/util.py (the cross-repo target) + - another_helper in lib/shared/util.py (not called from A) + """ + repo_a = tempfile.mkdtemp(prefix="codelens_xrepo_a_") + repo_b = tempfile.mkdtemp(prefix="codelens_xrepo_b_") + + # Repo A nodes + a_nodes = [ + _make_node("caller_fn", "services/api/handler.py", 10, repo_a), + _make_node("local_fn", "services/api/utils.py", 20, repo_a), + ] + # Repo A edges: caller_fn calls shared_helper (cross-repo) + local_fn + a_edges = [ + _make_raw_edge("services/api/handler.py:10:caller_fn", "shared_helper"), + _make_raw_edge("services/api/handler.py:10:caller_fn", "local_fn"), + ] + _make_backend_json(repo_a, a_nodes, a_edges) + + # Repo B nodes + b_nodes = [ + _make_node("shared_helper", "lib/shared/util.py", 5, repo_b), + _make_node("another_helper", "lib/shared/util.py", 15, repo_b), + ] + # Repo B edges: shared_helper calls another_helper (intra-repo) + b_edges = [ + _make_raw_edge("lib/shared/util.py:5:shared_helper", "another_helper"), + ] + _make_backend_json(repo_b, b_nodes, b_edges) + + yield repo_a, repo_b + + shutil.rmtree(repo_a, ignore_errors=True) + shutil.rmtree(repo_b, ignore_errors=True) + + +@pytest.fixture +def single_repo(): + """A single workspace with a small registry (for non-cross-repo tests).""" + ws = tempfile.mkdtemp(prefix="codelens_xrepo_single_") + nodes = [ + _make_node("foo", "main.py", 1, ws), + _make_node("bar", "main.py", 5, ws), + ] + edges = [_make_raw_edge("main.py:1:foo", "bar")] + _make_backend_json(ws, nodes, edges) + yield ws + shutil.rmtree(ws, ignore_errors=True) + + +# ─── Path parsing tests ─────────────────────────────────────── + +class TestParseAdditionalPaths: + """Verify --additional-paths string parsing.""" + + def test_none_returns_empty(self): + assert _parse_additional_paths(None) == [] + + def test_empty_string_returns_empty(self): + assert _parse_additional_paths("") == [] + + def test_single_path(self): + result = _parse_additional_paths("/tmp/repo") + assert result == ["/tmp/repo"] + + def test_multiple_paths_comma_separated(self): + result = _parse_additional_paths("/tmp/repo1,/tmp/repo2,/tmp/repo3") + assert result == ["/tmp/repo1", "/tmp/repo2", "/tmp/repo3"] + + def test_whitespace_stripped(self): + result = _parse_additional_paths(" /tmp/a , /tmp/b ") + assert result == ["/tmp/a", "/tmp/b"] + + def test_empty_entries_ignored(self): + result = _parse_additional_paths("/tmp/a,,/tmp/b,") + assert result == ["/tmp/a", "/tmp/b"] + + def test_relative_paths_made_absolute(self): + result = _parse_additional_paths("../sibling") + assert os.path.isabs(result[0]) + + +# ─── Merge tests ────────────────────────────────────────────── + +class TestLoadMergedRegistry: + """Verify multi-repo registry merging.""" + + def test_merges_nodes_from_all_repos(self, two_repos): + repo_a, repo_b = two_repos + merged = load_merged_registry(repo_a, [repo_b]) + assert merged["stats"]["repos_merged"] == 2 + assert merged["stats"]["total_nodes"] == 4 # 2 from A + 2 from B + node_fns = {n["fn"] for n in merged["nodes"]} + assert node_fns == {"caller_fn", "local_fn", "shared_helper", "another_helper"} + + def test_nodes_have_repo_tag(self, two_repos): + repo_a, repo_b = two_repos + merged = load_merged_registry(repo_a, [repo_b]) + for node in merged["nodes"]: + assert "repo" in node + assert node["repo"] in (repo_a, repo_b) + + def test_cross_repo_edge_resolved(self, two_repos): + """The edge from caller_fn (repo A) to shared_helper (repo B) + should be resolved after merging — the resolver finds + shared_helper in the combined node set.""" + repo_a, repo_b = two_repos + merged = load_merged_registry(repo_a, [repo_b]) + # Find the cross-repo edge + cross_edges = [e for e in merged["edges"] if e.get("cross_repo")] + assert len(cross_edges) == 1, f"Expected 1 cross-repo edge, got {len(cross_edges)}" + edge = cross_edges[0] + # The cross_repo flag is only set when BOTH from and to nodes + # were found in different repos — so its presence proves the + # edge was resolved. We also check the `to` field points to + # shared_helper's node_id. (Note: resolve_edges may not carry + # `resolved` or `to_fn` forward on resolved edges, so we rely + # on `to` and `cross_repo` instead.) + assert edge.get("cross_repo") is True + assert edge.get("target_repo") == repo_b + assert edge.get("source_repo") == repo_a + to_id = edge.get("to", "") + assert "shared_helper" in to_id, ( + f"Expected 'to' to contain 'shared_helper', got: {to_id}" + ) + + def test_intra_repo_edges_not_flagged_cross_repo(self, two_repos): + """Edges within the same repo should NOT have cross_repo=True.""" + repo_a, repo_b = two_repos + merged = load_merged_registry(repo_a, [repo_b]) + intra_edges = [e for e in merged["edges"] if not e.get("cross_repo")] + # caller_fn -> local_fn (intra A) + shared_helper -> another_helper (intra B) + assert len(intra_edges) >= 2 + for edge in intra_edges: + assert edge.get("cross_repo") is False + + def test_cross_repo_edge_count(self, two_repos): + repo_a, repo_b = two_repos + merged = load_merged_registry(repo_a, [repo_b]) + assert merged["stats"]["cross_repo_edges"] == 1 + + def test_no_additional_paths_returns_primary_only(self, single_repo): + """When additional_paths is empty, merged = primary only.""" + merged = load_merged_registry(single_repo, []) + assert merged["stats"]["repos_merged"] == 1 + assert merged["stats"]["total_nodes"] == 2 + assert merged["stats"]["cross_repo_edges"] == 0 + + def test_missing_primary_returns_empty(self): + """If the primary workspace has no registry, return empty merge.""" + ws = tempfile.mkdtemp(prefix="codelens_xrepo_empty_") + try: + merged = load_merged_registry(ws, []) + assert merged["stats"]["repos_merged"] == 0 + assert merged["stats"]["total_nodes"] == 0 + assert merged["nodes"] == [] + assert merged["edges"] == [] + finally: + shutil.rmtree(ws, ignore_errors=True) + + def test_duplicate_primary_in_additional_ignored(self, two_repos): + """If the primary path is also listed in additional_paths, it + should not be loaded twice.""" + repo_a, repo_b = two_repos + merged = load_merged_registry(repo_a, [repo_a, repo_b]) + # Should still be 2 repos, not 3 + assert merged["stats"]["repos_merged"] == 2 + assert merged["stats"]["total_nodes"] == 4 + + def test_unloadable_additional_repo_skipped(self, two_repos): + """If an additional repo path doesn't exist or has no registry, + it should be silently skipped.""" + repo_a, repo_b = two_repos + nonexistent = tempfile.mkdtemp(prefix="codelens_xrepo_nonexist_") + try: + merged = load_merged_registry(repo_a, [nonexistent, repo_b]) + # Only repo_a and repo_b contributed nodes + assert merged["stats"]["repos_merged"] == 2 + assert merged["stats"]["total_nodes"] == 4 + finally: + shutil.rmtree(nonexistent, ignore_errors=True) + + def test_repos_list_in_result(self, two_repos): + repo_a, repo_b = two_repos + merged = load_merged_registry(repo_a, [repo_b]) + assert repo_a in merged["repos"] + assert repo_b in merged["repos"] + + +# ─── Query tests ────────────────────────────────────────────── + +class TestQueryCrossRepo: + """Verify multi-repo symbol query.""" + + def test_finds_symbol_in_additional_repo(self, two_repos): + """Querying for shared_helper from repo A (where it's NOT defined) + should find it in repo B when repo B is an additional path.""" + repo_a, repo_b = two_repos + result = query_cross_repo("shared_helper", repo_a, [repo_b]) + assert result["status"] == "ok" + assert result["found"] is True + assert result["node"]["fn"] == "shared_helper" + assert result["node"]["repo"] == repo_b + + def test_caller_in_different_repo_flagged(self, two_repos): + """When querying shared_helper (repo B), the caller caller_fn + (repo A) should appear in callers with cross_repo=True.""" + repo_a, repo_b = two_repos + result = query_cross_repo("shared_helper", repo_a, [repo_b]) + assert result["found"] is True + callers = result.get("callers", []) + assert len(callers) >= 1 + # Find the cross-repo caller + cross_callers = [c for c in callers if c.get("cross_repo")] + assert len(cross_callers) >= 1, ( + f"Expected at least 1 cross-repo caller, got {callers}" + ) + assert cross_callers[0]["fn"] == "caller_fn" + assert cross_callers[0]["repo"] == repo_a + + def test_query_local_symbol_still_works(self, two_repos): + """Querying for a symbol in the primary repo should still work.""" + repo_a, repo_b = two_repos + result = query_cross_repo("local_fn", repo_a, [repo_b]) + assert result["found"] is True + assert result["node"]["fn"] == "local_fn" + assert result["node"]["repo"] == repo_a + + def test_not_found_returns_create_action(self, two_repos): + """Querying for a non-existent symbol returns CREATE action.""" + repo_a, repo_b = two_repos + result = query_cross_repo("nonexistent_xyz", repo_a, [repo_b]) + assert result["found"] is False + assert result["action"] == "CREATE" + assert result["repos_searched"] == 2 + + def test_multi_match_when_same_name_in_both_repos(self, two_repos): + """If the same function name exists in both repos, return a + multi_match summary.""" + repo_a, repo_b = two_repos + # Add a "shared_helper" to repo_a as well + a_backend = os.path.join(repo_a, ".codelens", "backend.json") + with open(a_backend) as f: + data = json.load(f) + data["nodes"].append(_make_node("shared_helper", "local_copy.py", 1, repo_a)) + with open(a_backend, "w") as f: + json.dump(data, f) + + result = query_cross_repo("shared_helper", repo_a, [repo_b]) + assert result["found"] is True + assert result["type"] == "multi_match" + assert result["match_count"] == 2 + # Each match should have a repo field + for m in result["matches"]: + assert "repo" in m + + def test_fuzzy_match_across_repos(self, two_repos): + """Fuzzy matching should find symbols across repos.""" + repo_a, repo_b = two_repos + result = query_cross_repo("shared", repo_a, [repo_b], fuzzy=True) + assert result["found"] is True + # Should find shared_helper + if result["type"] == "multi_match": + fns = [m["fn"] for m in result["matches"]] + assert "shared_helper" in fns + else: + assert result["node"]["fn"] == "shared_helper" + + def test_repos_searched_in_result(self, two_repos): + repo_a, repo_b = two_repos + result = query_cross_repo("local_fn", repo_a, [repo_b]) + assert result["repos_searched"] == 2 + + def test_cross_repo_search_flag(self, two_repos): + """cross_repo_search should be True when >1 repos searched.""" + repo_a, repo_b = two_repos + result = query_cross_repo("local_fn", repo_a, [repo_b]) + assert result["cross_repo_search"] is True + + def test_cross_repo_search_false_for_single_repo(self, single_repo): + """cross_repo_search should be False when only 1 repo searched.""" + result = query_cross_repo("foo", single_repo, []) + assert result["cross_repo_search"] is False + assert result["repos_searched"] == 1 + + +# ─── CLI registration smoke test ────────────────────────────── + +class TestCommandIntegration: + """Verify the --additional-paths flag is wired into the query command.""" + + def test_query_command_has_additional_paths_arg(self): + """The query command should accept --additional-paths.""" + from commands import COMMAND_REGISTRY + assert "query" in COMMAND_REGISTRY + info = COMMAND_REGISTRY["query"] + # Build a parser and check the arg is registered + import argparse + parser = argparse.ArgumentParser() + info["add_args"](parser) + # Parse with --additional-paths + args = parser.parse_args(["test_fn", "/tmp", "--additional-paths", "/tmp/a,/tmp/b"]) + assert args.additional_paths == "/tmp/a,/tmp/b" + + def test_query_command_without_additional_paths(self): + """Without --additional-paths, the arg should default to None.""" + from commands import COMMAND_REGISTRY + info = COMMAND_REGISTRY["query"] + import argparse + parser = argparse.ArgumentParser() + info["add_args"](parser) + args = parser.parse_args(["test_fn", "/tmp"]) + assert args.additional_paths is None + + +if __name__ == "__main__": + pytest.main([__file__, "-v", "--tb=short"])