diff --git a/README.md b/README.md index 162b1d96..c39ca14f 100644 --- a/README.md +++ b/README.md @@ -2,12 +2,12 @@ > **Before an AI writes a new class/id/function, CodeLens must be checked. This is not optional.** -CodeLens is an AI-native code intelligence platform that gives AI agents **full visibility** into a codebase before they write any code. It prevents collision, overwrite of existing logic, security vulnerabilities, and dead code through 71 CLI commands, an MCP server with 69 tools (54 static + 15 dynamic), AST-based taint analysis, live CVE/OSV scanning, a plugin system with OWASP Top 10 + Compliance rule packs, a true graph data model (nodes + edges) for structural code queries, and token-efficient `--format compact` output for high-volume agent workflows (issue #17). +CodeLens is an AI-native code intelligence platform that gives AI agents **full visibility** into a codebase before they write any code. It prevents collision, overwrite of existing logic, security vulnerabilities, and dead code through 73 CLI commands, an MCP server with 71 tools (55 static + 16 dynamic), AST-based taint analysis, live CVE/OSV scanning, a plugin system with OWASP Top 10 + Compliance rule packs, a true graph data model (nodes + edges) for structural code queries, and token-efficient `--format compact` output for high-volume agent workflows (issue #17). ## Features -- **71 CLI Commands** — From basic scan/query to AST taint analysis, CVE scanning, plugin management, auto-fix, dashboards, CI/CD quality gates, and `graph-schema` for cheap graph-shape introspection -- **MCP Server (69 Tools)** — Native AI agent integration via Model Context Protocol (JSON-RPC over stdio), 54 statically-defined tools + 15 dynamically discovered, every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`) +- **73 CLI Commands** — From basic scan/query to AST taint analysis, CVE scanning, plugin management, auto-fix, dashboards, CI/CD quality gates, and `graph-schema` for cheap graph-shape introspection +- **MCP Server (71 Tools)** — Native AI agent integration via Model Context Protocol (JSON-RPC over stdio), 55 statically-defined tools + 16 dynamically discovered, every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`) - **Token-Efficient Compact Output (v8.2, issue #17)** — `--format compact` produces single-char-key JSON with abbreviated types, omitted null fields, and relative paths — ~50% smaller than `json` on real trace output. Combined with `--limit`/`--offset` pagination, 5 structural queries now cost <5k tokens (down from 30-80k) - **AST Taint Engine** — Tree-sitter based taint analysis with return-value propagation, scope hierarchy, and branch condition refinement - **Live CVE/OSV Scanning** — Real-time vulnerability data from OSV.dev API with SQLite cache, 9 ecosystems (PyPI, npm, crates.io, Go, Maven, NuGet, RubyGems, Pub, Hex) @@ -225,8 +225,8 @@ codelens/ │ ├── changelog.md # Older changelog (per-version highlights) │ └── agent-integration.md # AI agent integration guide ├── scripts/ -│ ├── codelens.py # CLI entry point (71 commands registered) -│ ├── mcp_server.py # MCP JSON-RPC server (69 tools) +│ ├── codelens.py # CLI entry point (73 commands registered) +│ ├── mcp_server.py # MCP JSON-RPC server (71 tools) │ ├── registry.py # Registry read/write/build │ ├── persistent_registry.py # SQLite persistent storage (WAL mode) │ ├── base_parser.py # Base tree-sitter parser diff --git a/SKILL-QUICK.md b/SKILL-QUICK.md index e73a1f98..fcb6a574 100755 --- a/SKILL-QUICK.md +++ b/SKILL-QUICK.md @@ -115,7 +115,7 @@ $CLI list --limit 5 --offset 10 --format compact # → paginated + co | "Cross-file taint" | `dataflow` | `taint` (taint is single-file, AST-deep) | | "Auto-fix issues" | `fix` | `check` (check just gates, doesn't fix) | -## All 71 Commands +## All 73 Commands ### Setup & Lifecycle (8+) `init` · `scan [--incremental] [--max-files N] [--full]` · `registry-validate` · `detect` · `watch [--debounce SECS] [--git-mode] [--interval SECS]` · `git-status` · `migrate` · `serve` · `lsp-status` (issue #33: `codelens --lsp-status` top-level flag is an alias of `codelens lsp-status` — both delegate to `hybrid_engine.get_lsp_status()` and return the identical payload) @@ -147,9 +147,9 @@ $CLI list --limit 5 --offset 10 --format compact # → paginated + co ### Tooling (1) `plugin ` -**Total: 71 commands** (auto-registered via `commands/__init__.py`; rerun `python3 scripts/sync_command_count.py --apply` after adding/removing a command) +**Total: 73 commands** (auto-registered via `commands/__init__.py`; rerun `python3 scripts/sync_command_count.py --apply` after adding/removing a command) -## MCP Server (69 Tools) +## MCP Server (71 Tools) Start the MCP server for AI agent integration: @@ -157,9 +157,9 @@ Start the MCP server for AI agent integration: python3 scripts/codelens.py serve ``` -Exposes 69 tools as `codelens_` (e.g., `codelens_query`, `codelens_taint`, `codelens_graph_schema`, `codelens_architecture`, `codelens_resolve_types`, `codelens_git_status`): +Exposes 71 tools as `codelens_` (e.g., `codelens_query`, `codelens_taint`, `codelens_graph_schema`, `codelens_architecture`, `codelens_resolve_types`, `codelens_git_status`): - 50 statically-defined tools (full JSON schemas in `mcp_server.py`) -- 15 dynamically-discovered tools (auto-discovered from `COMMAND_REGISTRY`; long-running `watch` and `serve` are excluded) +- 16 dynamically-discovered tools (auto-discovered from `COMMAND_REGISTRY`; long-running `watch` and `serve` are excluded) - Every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`). Use `format: "compact"` for token-efficient responses (~50% smaller than `json`). - `watch` and `serve` itself are excluded (long-running) diff --git a/SKILL.md b/SKILL.md index a22843c1..11b3359c 100755 --- a/SKILL.md +++ b/SKILL.md @@ -1,10 +1,10 @@ --- name: codelens description: > - CodeLens — AI-Native Code Intelligence. 71 commands for AI-powered code analysis, + CodeLens — AI-Native Code Intelligence. 73 commands for AI-powered code analysis, security auditing, quality scoring, AST-based taint analysis, live CVE scanning, and pre-write safety checks. Supports 28+ languages with tree-sitter + regex - fallback parsing. MCP server exposes 69 tools for AI agent integration. + fallback parsing. MCP server exposes 71 tools for AI agent integration. For quick command reference with validated output schemas, see SKILL-QUICK.md. For version history, see CHANGELOG.md. --- diff --git a/pyproject.toml b/pyproject.toml index 975ec607..1ae515c0 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta" [project] name = "codelens" version = "8.2.0" -description = "Live Codebase Reference Intelligence — 71 commands for AI-powered code analysis, security auditing, and quality scoring" +description = "Live Codebase Reference Intelligence — 73 commands for AI-powered code analysis, security auditing, and quality scoring" readme = "README.md" license = {text = "MIT"} requires-python = ">=3.8" diff --git a/scripts/commands/semantic_query.py b/scripts/commands/semantic_query.py new file mode 100644 index 00000000..de363926 --- /dev/null +++ b/scripts/commands/semantic_query.py @@ -0,0 +1,66 @@ +"""Semantic query command — TF-IDF symbol search (issue #11, Option A).""" + +import os +from typing import Any, Dict + +from semantic_search_engine import semantic_query +from commands import register_command + + +def add_args(parser): + parser.add_argument( + "query", + help=( + "Natural-language or code-fragment query " + "(e.g. 'user authentication flow', 'parse jwt', 'error handler'). " + "Symbol names, signatures, kinds, and file paths are all searched." + ), + ) + parser.add_argument( + "workspace", + nargs="?", + default=None, + help="Path to workspace root (auto-detected if omitted).", + ) + parser.add_argument( + "--top", + type=int, + default=10, + metavar="N", + help="Maximum number of results to return (default: 10; use 0 for all).", + ) + parser.add_argument( + "--db-path", + default=None, + metavar="PATH", + help="Custom path for the SQLite registry database " + "(default: /.codelens/codelens.db).", + ) + + +def execute(args, workspace) -> Dict[str, Any]: + """Dispatch to :func:`semantic_engine.semantic_query`. + + The CLI ``--top`` flag is the user-facing name; the engine takes the + same value as ``top_k``. We pass ``workspace`` and ``db_path`` straight + through so users can override the database location for ad-hoc queries + against a snapshot. + """ + top_k = getattr(args, "top", 10) + if top_k is None: + top_k = 10 + db_path = getattr(args, "db_path", None) + return semantic_query( + workspace=workspace, + query=args.query, + top_k=top_k, + db_path=db_path, + ) + + +register_command( + "semantic-query", + "Semantic symbol search via TF-IDF (find symbols by meaning, not just name)", + add_args, + execute, +) diff --git a/scripts/graph_model.py b/scripts/graph_model.py index 8b2e8cd5..0fb0b6cf 100644 --- a/scripts/graph_model.py +++ b/scripts/graph_model.py @@ -13,7 +13,7 @@ - New tables `graph_nodes` and `graph_edges` are additive (prefixed `graph_` to avoid colliding with any existing table name). - The flat registry tables and JSON files are untouched. -- All 71 existing CLI commands continue to work unchanged. +- All 73 existing CLI commands continue to work unchanged. Schema: graph_nodes( diff --git a/scripts/mcp_server.py b/scripts/mcp_server.py index ce3f2b27..54786e59 100644 --- a/scripts/mcp_server.py +++ b/scripts/mcp_server.py @@ -470,6 +470,40 @@ "required": ["workspace"] } }, + "semantic-query": { + "description": ( + "Semantic symbol search via TF-IDF (issue #11). Finds symbols by " + "meaning, not just by name — e.g. querying 'user authentication flow' " + "can surface a function named verify_jwt_claims. Returns ranked symbols " + "with cosine-similarity scores. Pure-Python, zero dependencies; reads " + "from the existing SQLite registry so the index is always in sync with " + "the last 'scan' result." + ), + "parameters": { + "type": "object", + "properties": { + "query": { + "type": "string", + "description": ( + "Natural-language or code-fragment query, e.g. " + "'user authentication flow', 'parse jwt', 'error handler'. " + "Symbol names, signatures, kinds, and file paths are all " + "included in the TF-IDF document for each symbol." + ) + }, + "workspace": { + "type": "string", + "description": "Path to workspace root directory" + }, + "top": { + "type": "integer", + "description": "Maximum number of results to return (default: 10; use 0 for all).", + "default": 10 + } + }, + "required": ["query", "workspace"] + } + }, "vuln-scan": { "description": "Scan dependencies for known CVEs and security vulnerabilities.", "parameters": { diff --git a/scripts/semantic_search_engine.py b/scripts/semantic_search_engine.py new file mode 100644 index 00000000..1d349959 --- /dev/null +++ b/scripts/semantic_search_engine.py @@ -0,0 +1,667 @@ +""" +Semantic Search Engine for CodeLens (issue #11, Option A — TF-IDF). + +Provides semantic symbol search over the indexed codebase so AI agents can +locate "authentication logic" even when the actual function is named +``verify_jwt_claims``. Implements the zero-dependency Option A from the issue: +TF-IDF over symbol names + signatures + kinds + file paths, ranked by cosine +similarity. + +Design choices +-------------- +* **Pure Python, no model files.** Avoids bundling ``sentence-transformers`` + or any ~80 MB embedding model. Fast to import, deterministic, no native + deps. Good enough for the majority of agent "find the right file" queries. +* **Reads from the existing SQLite registry.** Symbol text is built from + fields already populated by ``persistent_registry`` — ``name``, + ``signature``, ``kind``, ``file_path``, ``language`` — so the index is + always in sync with the last ``scan`` result. No separate index file to + keep consistent. +* **In-memory index, cached per (db_path, mtime).** Building the vocabulary + is O(N_symbols) — for CodeLens's own 3000-node graph this is <100 ms. + Cached so repeated ``semantic_query`` calls within a session are ~1 ms. +* **Robust tokenization.** Splits ``snake_case``, ``camelCase``, ``kebab-case`` + and path segments so ``verifyJwtClaims`` and ``verify_jwt_claims`` produce + the same token stream. + +Naming note +----------- +This module is named ``semantic_search_engine`` to avoid colliding with the +pre-existing ``scripts/semantic_engine.py`` (a taint-analysis rules engine +that is itself deprecated as of PR #140 / issue #49 phase-1). The two +modules have no relationship — one is IR-style semantic search, the other +is taint-flow semantic rules. The naming overlap is unfortunate but +unavoidable without renaming the older module. + +Public surface +-------------- +- :func:`semantic_query` — main entry point used by the CLI command and the + ``codelens_semantic_query`` MCP tool. +- :func:`build_index` — exposed for tests and callers that want to inspect + the vocabulary / IDF weights directly. +- :func:`clear_cache` — used by tests to force a rebuild between cases. + +The non-breaking nature of this module is what makes it a safe MVP: it adds +a new engine + command without touching any existing scan flow, schema, or +output shape. Existing commands and tests continue to work unchanged. +""" + +from __future__ import annotations + +import math +import os +import re +import sqlite3 +import threading +from collections import Counter +from typing import Any, Dict, List, Optional, Tuple + +from utils import default_db_path, logger + + +# ─── Stopwords ──────────────────────────────────────────────── + +# Small, conservative English stopword list. We intentionally do NOT pull in +# NLTK or any external dependency for this — the goal is zero-dep TF-IDF. +# These are tokens that carry almost no discriminative signal across a +# codebase (e.g. ``the``, ``a``, ``of``) and would otherwise inflate +# document vectors with noise. +_STOPWORDS = frozenset({ + "a", "an", "the", "and", "or", "not", "is", "are", "was", "were", + "be", "been", "being", "to", "of", "in", "on", "at", "by", "for", + "with", "from", "as", "this", "that", "these", "those", "it", "its", + "into", "than", "then", "so", "if", "but", "do", "does", "did", + "has", "have", "had", "can", "could", "should", "would", "will", + "shall", "may", "might", "must", "i", "you", "he", "she", "we", + "they", "me", "him", "her", "us", "them", "my", "your", "his", + "our", "their", +}) + +# Minimum token length. Single-character tokens are almost never meaningful +# for code search (``x``, ``y``, ``i``, ``j``) and just add noise. +_MIN_TOKEN_LEN = 2 + +# Maximum tokens per document. Guards against pathological inputs (e.g. an +# enormous generated file with thousands of symbols in one path). 4096 is +# well above any realistic CodeLens symbol entry. +_MAX_TOKENS_PER_DOC = 4096 + + +# ─── Tokenization ───────────────────────────────────────────── + +# CamelCase boundary: split "verifyJwtClaims" -> ["verify", "Jwt", "Claims"] +_CAMEL_BOUNDARY_RE = re.compile(r"(?<=[a-z0-9])(?=[A-Z])") +# Split on any run of non-alphanumeric characters. This catches whitespace, +# underscores, hyphens, slashes, dots, AND signature punctuation like +# ``()``, ``,``, ``:``, ``[]``, ``<>``, ``=``, etc. — so ``def f(x, y)`` +# tokenizes to ``["def", "f", "x", "y"]`` instead of leaking ``"f(x"`` as +# a single token. +_SPLIT_RE = re.compile(r"[^A-Za-z0-9]+") +# Strip leading/trailing non-alphanumerics from a token. With the split +# pattern above this is mostly a no-op, but kept as a defensive measure +# in case a caller feeds in a string that begins/ends with a symbol. +_TRIM_RE = re.compile(r"^[^A-Za-z0-9]+|[^A-Za-z0-9]+$") + + +def tokenize(text: str) -> List[str]: + """Tokenize a string into normalized lowercase terms. + + Splits on whitespace, underscores, hyphens, slashes, dots, signature + punctuation (``()``, ``,``, ``:``), AND camelCase boundaries — so both + ``verify_jwt_claims`` and ``verifyJwtClaims`` produce the same token + list ``["verify", "jwt", "claims"]``, and ``def f(x, y)`` produces + ``["def", "f", "x", "y"]``. Single-character tokens and stopwords are + filtered out. + + Args: + text: Input string (symbol name, signature, file path, etc.). + + Returns: + List of lowercase tokens, in their original order. Empty list if + the input is empty or all-stopword. + """ + if not text: + return [] + + tokens: List[str] = [] + # First split on any non-alphanumeric run + for raw in _SPLIT_RE.split(text): + if not raw: + continue + # Then split camelCase boundaries within each segment + for sub in _CAMEL_BOUNDARY_RE.split(raw): + sub = _TRIM_RE.sub("", sub) + if not sub: + continue + # Lowercase for case-insensitive matching + low = sub.lower() + if len(low) < _MIN_TOKEN_LEN: + continue + if low in _STOPWORDS: + continue + tokens.append(low) + if len(tokens) >= _MAX_TOKENS_PER_DOC: + break + return tokens + + +# ─── Symbol Document Building ───────────────────────────────── + +def _build_symbol_text(symbol: Dict[str, Any]) -> str: + """Build a text blob from a symbol row for TF-IDF tokenization. + + Concatenates the symbol name (highest signal), signature (function + parameter and return type info), kind (function/class/id), language, + and file path components (so directory names like ``auth/`` contribute + signal). The relative weights are handled downstream by TF-IDF — the + name field naturally gets higher weight because it's a short string + where each token has a large TF contribution. + + Args: + symbol: Dict from ``persistent_registry.get_all_symbols()``. + Required keys: ``name``, ``kind``. Optional: ``signature``, + ``file_path``, ``language``, ``extra_json``. + + Returns: + Space-joined text blob. Never empty — at minimum contains the + symbol name and kind. + """ + parts: List[str] = [] + # Name — highest signal + name = symbol.get("name") or "" + if name: + parts.append(name) + # Signature — function parameter/return types + sig = symbol.get("signature") or "" + if sig: + parts.append(sig) + # Kind — function / class / id / etc. + kind = symbol.get("kind") or "" + if kind: + parts.append(kind) + # Language — "python", "js", "rust", ... + lang = symbol.get("language") or "" + if lang: + parts.append(lang) + # File path — directory names + filename carry semantic signal + # (e.g. "auth/login.py" -> "auth login py") + fp = symbol.get("file_path") or "" + if fp: + parts.append(fp) + # Extra JSON — may contain status, exported flag, etc. We only extract + # scalar string values; list/dict values are skipped to avoid + # bloating the document with reference locations. + extra_raw = symbol.get("extra_json") + if extra_raw: + try: + import json + extra = json.loads(extra_raw) + if isinstance(extra, dict): + for v in extra.values(): + if isinstance(v, str) and v: + parts.append(v) + except (ValueError, TypeError): + # Malformed JSON — ignore, the text blob already has the + # primary fields. + pass + return " ".join(parts) + + +# ─── Index ──────────────────────────────────────────────────── + +class SemanticIndex: + """In-memory TF-IDF index over the CodeLens symbol registry. + + A new index is built by :func:`build_index` and cached per + ``(db_path, db_mtime)``. Each instance is immutable after construction + and safe to share across threads (read-only). + + Attributes: + db_path: Absolute path to the SQLite database the index was built + from. + symbols: The raw symbol rows (dicts) the index covers. Used by + :meth:`query` to populate result entries. + vocabulary: Mapping ``term -> index`` into the IDF vector. + idf: List of inverse-document-frequency weights, parallel to + ``vocabulary`` values. ``idf[vocabulary[term]]`` is the IDF + of ``term``. + doc_vectors: List of TF-IDF vectors, one per symbol. Each vector + is a ``{term_index: weight}`` dict (sparse representation). + doc_norms: L2 norm of each document vector (precomputed for + cosine similarity). + """ + + def __init__( + self, + db_path: str, + symbols: List[Dict[str, Any]], + vocabulary: Dict[str, int], + idf: List[float], + doc_vectors: List[Dict[int, float]], + doc_norms: List[float], + ) -> None: + self.db_path = db_path + self.symbols = symbols + self.vocabulary = vocabulary + self.idf = idf + self.doc_vectors = doc_vectors + self.doc_norms = doc_norms + + # ─── Query ────────────────────────────────────────────── + + def query(self, query_text: str, top_k: int = 10) -> List[Dict[str, Any]]: + """Rank symbols by cosine similarity to ``query_text``. + + Args: + query_text: Natural-language or code-fragment query + (e.g. ``"user authentication flow"``). + top_k: Maximum number of results to return. ``0`` returns all + symbols sorted by similarity (still bounded by the index + size). + + Returns: + List of result dicts, sorted by descending similarity. Each + dict has: + + * ``symbol``: the original symbol row (name, kind, file_path, + line_start, language, signature, ...). + * ``score``: cosine similarity in ``[0.0, 1.0]``. ``0.0`` means + no shared terms with the query. + * ``matched_terms``: list of query terms that appeared in the + document (useful for explaining the ranking to a user). + + Symbols with zero similarity are excluded from the results. + """ + if not self.symbols: + return [] + if top_k < 0: + top_k = 0 + + # Tokenize the query and build a sparse query vector. + q_tokens = tokenize(query_text) + if not q_tokens: + return [] + + # Term frequency in the query + q_tf = Counter(q_tokens) + total_q = sum(q_tf.values()) + # Build sparse query vector: {term_index: tf * idf} + q_vec: Dict[int, float] = {} + q_matched_terms: List[str] = [] + for term, count in q_tf.items(): + idx = self.vocabulary.get(term) + if idx is None: + # Term not in any document — skip; contributes nothing to + # cosine similarity because the document side is 0. + continue + tf = count / total_q + q_vec[idx] = tf * self.idf[idx] + q_matched_terms.append(term) + + if not q_vec: + # Query had tokens but none appeared in any document + return [] + + # L2 norm of the query vector + q_norm = math.sqrt(sum(w * w for w in q_vec.values())) + if q_norm == 0.0: + return [] + + # Score every document. With ~3000 symbols this loop is <1 ms. + results: List[Tuple[int, float, List[str]]] = [] + # Build a term -> index map view so we can list which query terms + # matched in each document. + for doc_idx, doc_vec in enumerate(self.doc_vectors): + if not doc_vec: + continue + doc_norm = self.doc_norms[doc_idx] + if doc_norm == 0.0: + continue + # Dot product over the smaller vector + if len(q_vec) < len(doc_vec): + small, large = q_vec, doc_vec + else: + small, large = doc_vec, q_vec + dot = 0.0 + for term_idx, w in small.items(): + other = large.get(term_idx) + if other is not None: + dot += w * other + if dot == 0.0: + continue + sim = dot / (q_norm * doc_norm) + # Sanity: cosine similarity should be in [-1, 1]; for TF-IDF + # vectors (non-negative weights) it's in [0, 1]. Clamp to + # guard against floating-point drift just above 1.0. + if sim > 1.0: + sim = 1.0 + elif sim < 0.0: + sim = 0.0 + # Determine which query terms actually appeared in this doc + matched_here = [ + t for t in q_matched_terms + if self.vocabulary[t] in doc_vec + ] + results.append((doc_idx, sim, matched_here)) + + # Sort by similarity descending, then by name for stable ordering + results.sort(key=lambda r: (-r[1], self.symbols[r[0]].get("name", ""))) + + if top_k > 0: + results = results[:top_k] + + return [ + { + "symbol": self.symbols[doc_idx], + "score": round(sim, 4), + "matched_terms": matched, + } + for doc_idx, sim, matched in results + ] + + +# ─── Index Cache ────────────────────────────────────────────── + +# Module-level cache: {(db_path, mtime): SemanticIndex}. +# The cache key includes the db file's mtime so a re-scan automatically +# invalidates the cached index. This is the same pattern used by +# ``persistent_registry`` for its connection cache. +_INDEX_CACHE: Dict[Tuple[str, float], SemanticIndex] = {} +_INDEX_CACHE_LOCK = threading.Lock() + + +def clear_cache() -> None: + """Drop all cached indices. Primarily for tests.""" + with _INDEX_CACHE_LOCK: + _INDEX_CACHE.clear() + + +def _load_symbols_from_db(db_path: str) -> List[Dict[str, Any]]: + """Load all symbols from the SQLite registry at ``db_path``. + + Returns an empty list if the database doesn't exist, the ``symbols`` + table is missing, or SQLite is unavailable. Never raises — semantic + search is a non-breaking add-on and should degrade gracefully to + "no results" rather than crash the host command. + + Args: + db_path: Absolute path to ``.codelens/codelens.db``. + + Returns: + List of symbol dicts. Each dict has keys: ``id``, ``name``, + ``kind``, ``file_path``, ``line_start``, ``line_end``, + ``language``, ``signature``, ``hash``, ``extra_json``. + """ + if not os.path.exists(db_path): + return [] + try: + conn = sqlite3.connect(db_path) + try: + conn.row_factory = sqlite3.Row + # Sanity-check that the symbols table exists. If the workspace + # has been initialized but never scanned, the db file exists + # but has no tables. + table = conn.execute( + "SELECT name FROM sqlite_master " + "WHERE type='table' AND name='symbols'" + ).fetchone() + if table is None: + return [] + rows = conn.execute("SELECT * FROM symbols").fetchall() + return [dict(r) for r in rows] + finally: + conn.close() + except sqlite3.Error as e: + logger.debug(f"semantic_search_engine: failed to read symbols from {db_path}: {e}") + return [] + + +def build_index(db_path: str) -> SemanticIndex: + """Build a TF-IDF index over all symbols in ``db_path``. + + The result is cached per ``(db_path, mtime)`` — calling this function + twice with the same db file returns the same :class:`SemanticIndex` + instance. Re-scanning the workspace (which writes a new mtime on the + db file) automatically invalidates the cache. + + Args: + db_path: Absolute path to ``.codelens/codelens.db``. + + Returns: + A :class:`SemanticIndex`. If the db doesn't exist or has no + symbols, returns an empty index (``len(symbols) == 0``) whose + :meth:`SemanticIndex.query` always returns ``[]``. + """ + if not os.path.exists(db_path): + # Return an empty index — never raise. Callers should fall back + # to "no results" rather than crash. + return SemanticIndex(db_path, [], {}, [], [], []) + + try: + mtime = os.path.getmtime(db_path) + except OSError: + mtime = 0.0 + + cache_key = (db_path, mtime) + with _INDEX_CACHE_LOCK: + cached = _INDEX_CACHE.get(cache_key) + if cached is not None: + return cached + + symbols = _load_symbols_from_db(db_path) + + if not symbols: + index = SemanticIndex(db_path, [], {}, [], [], []) + with _INDEX_CACHE_LOCK: + _INDEX_CACHE[cache_key] = index + return index + + # ── Build document term-frequency vectors ── + doc_tf_vectors: List[Counter] = [] + doc_freq: Counter = Counter() # how many docs contain each term + for sym in symbols: + text = _build_symbol_text(sym) + toks = tokenize(text) + tf = Counter(toks) + doc_tf_vectors.append(tf) + # Document frequency: increment once per term per doc + for term in tf.keys(): + doc_freq[term] += 1 + + # ── Build vocabulary ── + # Sort terms for deterministic indexing (so the same db always produces + # the same vocabulary order, which makes debugging easier). + sorted_terms = sorted(doc_freq.keys()) + vocabulary: Dict[str, int] = {term: i for i, term in enumerate(sorted_terms)} + + # ── IDF (smoothed) ── + # idf(t) = ln((N + 1) / (df(t) + 1)) + 1 + # The +1 in numerator and denominator is the standard "add-one" + # smoothing that prevents division by zero and dampens the IDF of + # terms that appear in every document. The trailing +1 ensures + # IDF is always > 0 (so a term that appears in every doc still + # contributes a small amount, rather than being zeroed out). + n_docs = len(symbols) + idf: List[float] = [] + for term in sorted_terms: + df = doc_freq[term] + weight = math.log((n_docs + 1) / (df + 1)) + 1.0 + idf.append(weight) + + # ── TF-IDF document vectors + L2 norms ── + doc_vectors: List[Dict[int, float]] = [] + doc_norms: List[float] = [] + for tf in doc_tf_vectors: + total = sum(tf.values()) + if total == 0: + doc_vectors.append({}) + doc_norms.append(0.0) + continue + vec: Dict[int, float] = {} + for term, count in tf.items(): + idx = vocabulary[term] + tf_val = count / total + vec[idx] = tf_val * idf[idx] + norm = math.sqrt(sum(w * w for w in vec.values())) + doc_vectors.append(vec) + doc_norms.append(norm) + + index = SemanticIndex( + db_path=db_path, + symbols=symbols, + vocabulary=vocabulary, + idf=idf, + doc_vectors=doc_vectors, + doc_norms=doc_norms, + ) + with _INDEX_CACHE_LOCK: + _INDEX_CACHE[cache_key] = index + return index + + +# ─── Public API ─────────────────────────────────────────────── + +def semantic_query( + workspace: str, + query: str, + top_k: int = 10, + db_path: Optional[str] = None, +) -> Dict[str, Any]: + """Run a semantic search query against the workspace symbol registry. + + This is the main entry point used by the ``semantic-query`` CLI command + and the ``codelens_semantic_query`` MCP tool. It builds (or retrieves + a cached) TF-IDF index over the workspace's SQLite symbol registry + and returns the top-k symbols ranked by cosine similarity to the + query. + + Args: + workspace: Absolute or relative path to the workspace root. The + registry database is read from + ``/.codelens/codelens.db`` unless ``db_path`` is + given. + query: Natural-language or code-fragment query (e.g. + ``"user authentication flow"``, ``"parse jwt"``, + ``"error handler"``). + top_k: Maximum number of results to return. ``0`` returns all + matching symbols (still bounded by index size). Negative + values are treated as ``0``. Default: ``10``. + db_path: Override the database path. Mainly for tests; callers + should let the default resolve via :func:`utils.default_db_path`. + + Returns: + Dict with the following shape:: + + { + "status": "ok", + "query": "", + "workspace": "", + "top_k": , + "stats": { + "total_symbols": , # symbols in the registry + "returned": , # results in this response + "truncated": , # True if more than top_k matched + "index_size": , # vocabulary size + }, + "results": [ + { + "name": "", + "kind": "", + "file": "", + "line": , + "language": "", + "signature": "", + "score": , + "matched_terms": ["", ...], + }, + ... + ], + } + + On error (e.g. empty query), ``status`` is ``"error"`` with a + ``message`` field and empty ``results``. A workspace that has + never been scanned returns ``status="ok"`` with + ``total_symbols=0`` and empty ``results`` — this is not an error, + just a "nothing to search" state. + """ + workspace = os.path.abspath(workspace) + if db_path is None: + db_path = default_db_path(workspace) + + if not query or not query.strip(): + return { + "status": "error", + "message": "Query must be a non-empty string.", + "query": query, + "workspace": workspace, + "top_k": top_k, + "stats": { + "total_symbols": 0, + "returned": 0, + "truncated": False, + "index_size": 0, + }, + "results": [], + } + + try: + index = build_index(db_path) + except Exception as e: + # build_index is designed to never raise (it catches its own + # sqlite errors), but we add a defensive try here so the host + # command never crashes from a semantic-search failure. + logger.warning(f"semantic_search_engine: build_index failed: {e}", exc_info=True) + return { + "status": "error", + "message": f"Failed to build search index: {e}", + "query": query, + "workspace": workspace, + "top_k": top_k, + "stats": { + "total_symbols": 0, + "returned": 0, + "truncated": False, + "index_size": 0, + }, + "results": [], + } + + # Top-K clamp + if top_k < 0: + top_k = 0 + + ranked = index.query(query, top_k=top_k) + + # Build the response + results: List[Dict[str, Any]] = [] + for r in ranked: + sym = r["symbol"] + results.append({ + "name": sym.get("name", ""), + "kind": sym.get("kind", ""), + "file": sym.get("file_path", ""), + "line": sym.get("line_start") or 0, + "language": sym.get("language", ""), + "signature": sym.get("signature"), + "score": r["score"], + "matched_terms": r["matched_terms"], + }) + + # Truncation flag: True iff there were more than top_k matching + # symbols. We need to query with top_k=0 to count total matches. + total_matches = len(index.query(query, top_k=0)) if results else 0 + truncated = bool(top_k > 0 and total_matches > top_k) + + return { + "status": "ok", + "query": query, + "workspace": workspace, + "top_k": top_k, + "stats": { + "total_symbols": len(index.symbols), + "returned": len(results), + "truncated": truncated, + "index_size": len(index.vocabulary), + }, + "results": results, + } diff --git a/skill.json b/skill.json index d236d4ae..507d902f 100755 --- a/skill.json +++ b/skill.json @@ -1,7 +1,7 @@ { "name": "codelens", "version": "8.2.0", - "description": "Live Codebase Reference Intelligence. 71 commands for AI-powered code analysis, security auditing, quality scoring, and pre-write safety checks. Supports 28+ languages with regex+AST hybrid parsing. Must activate before writing/editing/deleting any class, id, or function.", + "description": "Live Codebase Reference Intelligence. 73 commands for AI-powered code analysis, security auditing, quality scoring, and pre-write safety checks. Supports 28+ languages with regex+AST hybrid parsing. Must activate before writing/editing/deleting any class, id, or function.", "author": "codelens", "command_categories": { "setup": [ diff --git a/tests/test_semantic_search_engine.py b/tests/test_semantic_search_engine.py new file mode 100644 index 00000000..08806963 --- /dev/null +++ b/tests/test_semantic_search_engine.py @@ -0,0 +1,538 @@ +""" +Tests for the semantic SEARCH engine (issue #11, Option A — TF-IDF). + +This file tests ``scripts/semantic_search_engine.py`` — the IR-style TF-IDF +symbol search. It is distinct from ``tests/test_semantic_engine.py``, which +tests the deprecated taint-analysis rules engine in +``scripts/semantic_engine.py``. The two modules share the "semantic" prefix +but have no other relationship. + +The tests build a tiny SQLite registry in a temp directory, then verify +that :func:`semantic_search_engine.semantic_query` returns ranked results +that match intuition: + +* A query for ``"auth"`` surfaces a symbol named ``verify_jwt_claims`` + even though the literal string ``"auth"`` never appears in the symbol + name — only in the file path. This is the core value proposition of + the issue (find by meaning, not just by literal name). +* Tokenization handles ``snake_case``, ``camelCase``, and path segments + uniformly. +* IDF weighting causes rare, discriminative terms (e.g. ``jwt``) to rank + higher than ubiquitous terms (e.g. ``function``). +* The engine degrades gracefully — empty query, missing db, missing + ``symbols`` table, and ``top_k=0`` all return well-formed responses + rather than raising. +* The cache invalidates on db mtime change, so a re-scan picks up new + symbols. +""" + +import json +import os +import shutil +import sqlite3 +import sys +import tempfile +import time + +import pytest + +# Make scripts/ importable +_SCRIPT_DIR = os.path.join( + os.path.dirname(os.path.dirname(os.path.abspath(__file__))), + "scripts", +) +if _SCRIPT_DIR not in sys.path: + sys.path.insert(0, _SCRIPT_DIR) + +import semantic_search_engine +from semantic_search_engine import ( + SemanticIndex, + build_index, + clear_cache, + semantic_query, + tokenize, +) + + +# ─── Fixtures ───────────────────────────────────────────────── + +def _make_db(workspace: str, symbols): + """Create a fake ``.codelens/codelens.db`` with the given symbol rows. + + ``symbols`` is a list of dicts with keys: name, kind, file_path, + line_start, language, signature, extra_json (dict, will be + JSON-encoded). + """ + codelens_dir = os.path.join(workspace, ".codelens") + os.makedirs(codelens_dir, exist_ok=True) + db_path = os.path.join(codelens_dir, "codelens.db") + conn = sqlite3.connect(db_path) + try: + conn.execute( + """ + CREATE TABLE symbols ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + name TEXT NOT NULL, + kind TEXT NOT NULL DEFAULT 'function', + file_path TEXT, + line_start INTEGER, + line_end INTEGER, + language TEXT, + signature TEXT, + hash TEXT, + extra_json TEXT + ) + """ + ) + for i, s in enumerate(symbols, start=1): + extra = s.get("extra_json") or {} + conn.execute( + """ + INSERT INTO symbols + (id, name, kind, file_path, line_start, line_end, + language, signature, hash, extra_json) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + i, + s["name"], + s.get("kind", "function"), + s.get("file_path", ""), + s.get("line_start"), + s.get("line_end"), + s.get("language", ""), + s.get("signature", ""), + s.get("hash", ""), + json.dumps(extra) if extra else None, + ), + ) + conn.commit() + finally: + conn.close() + return db_path + + +@pytest.fixture +def workspace(): + """Create a temp workspace with no db. Caller is responsible for + calling :func:`_make_db` to populate it.""" + ws = tempfile.mkdtemp(prefix="codelens_semantic_test_") + yield ws + # Drop the cache so the next test starts fresh + clear_cache() + shutil.rmtree(ws, ignore_errors=True) + + +def _auth_workspace(): + """A workspace with a mix of auth-related and unrelated symbols. + + Used by multiple tests to verify the core "find by meaning" behavior. + """ + ws = tempfile.mkdtemp(prefix="codelens_semantic_auth_") + _make_db(ws, [ + { + "name": "verify_jwt_claims", + "kind": "function", + "file_path": "auth/jwt.py", + "line_start": 42, + "language": "python", + "signature": "def verify_jwt_claims(token: str) -> bool", + "extra_json": {"status": "active", "ref_count": 7, "exported": True}, + }, + { + "name": "loginUser", + "kind": "function", + "file_path": "auth/login.py", + "line_start": 15, + "language": "python", + "signature": "def loginUser(email, password)", + "extra_json": {"status": "active", "ref_count": 3}, + }, + { + "name": "format_date", + "kind": "function", + "file_path": "utils/dates.py", + "line_start": 8, + "language": "python", + "signature": "def format_date(d: datetime) -> str", + "extra_json": {"status": "active", "ref_count": 12}, + }, + { + "name": "parse_csv_row", + "kind": "function", + "file_path": "io/csv_parser.py", + "line_start": 23, + "language": "python", + "signature": "def parse_csv_row(line: str) -> list", + "extra_json": {"status": "active", "ref_count": 4}, + }, + { + "name": "UserModel", + "kind": "class", + "file_path": "models/user.py", + "line_start": 5, + "language": "python", + "signature": "", + "extra_json": {"status": "active", "ref_count": 9, "exported": True}, + }, + ]) + return ws + + +@pytest.fixture +def auth_workspace(): + ws = _auth_workspace() + yield ws + clear_cache() + shutil.rmtree(ws, ignore_errors=True) + + +# ─── Tokenizer tests ────────────────────────────────────────── + +class TestTokenizer: + """Verify the tokenizer handles common code-identifier conventions.""" + + def test_snake_case_split(self): + assert tokenize("verify_jwt_claims") == ["verify", "jwt", "claims"] + + def test_camel_case_split(self): + # "loginUser" -> ["login", "user"] + assert tokenize("loginUser") == ["login", "user"] + + def test_pascal_case_split(self): + assert tokenize("UserModel") == ["user", "model"] + + def test_kebab_and_path_split(self): + # "auth/login-page" -> ["auth", "login", "page"] + assert tokenize("auth/login-page") == ["auth", "login", "page"] + + def test_mixed_conventions(self): + # verifyJwtClaims and verify_jwt_claims produce identical tokens + assert tokenize("verifyJwtClaims") == tokenize("verify_jwt_claims") + + def test_empty_input(self): + assert tokenize("") == [] + assert tokenize(None) == [] # type: ignore[arg-type] + + def test_single_char_filtered(self): + # Single chars (x, y) filtered out; signature punctuation is split + # so "def f(x, y)" tokenizes to ["def"] (f/x/y are all 1-char). + assert tokenize("def f(x, y)") == ["def"] + + def test_stopwords_filtered(self): + # "the", "is", "a" filtered out + tokens = tokenize("the user is a model") + assert "the" not in tokens + assert "is" not in tokens + assert "a" not in tokens + assert tokens == ["user", "model"] + + def test_lowercase(self): + # All tokens are lowercased + for t in tokenize("VerifyJWTClaims"): + assert t == t.lower() + + def test_signature_punctuation_split(self): + # Function signature punctuation (parens, comma, colon) is split + # so each identifier becomes its own token. + tokens = tokenize("def foo(x: int, y: str) -> bool") + assert "foo" in tokens + assert "int" in tokens + assert "str" in tokens + assert "bool" in tokens + # "def" is a single-char-word survivor? No, "def" is 3 chars — it + # stays. "x" and "y" are 1-char and filtered out. + assert "def" in tokens + assert "x" not in tokens + assert "y" not in tokens + + +# ─── Engine behavior tests ──────────────────────────────────── + +class TestSemanticQuery: + """End-to-end behavior of :func:`semantic_query`.""" + + def test_finds_auth_symbol_by_concept(self, auth_workspace): + """The core issue #11 use case: a query token that doesn't appear + in any symbol NAME still surfaces relevant symbols because file + paths, signatures, and kinds are all part of the TF-IDF document. + + Here, the query ``"auth"`` doesn't appear in any symbol name + (``verify_jwt_claims``, ``loginUser``, ``format_date``, etc.), but + it DOES appear in the file paths ``auth/jwt.py`` and + ``auth/login.py``. The engine must surface those symbols — that's + the "find by meaning, not just by literal name" value prop. + + Note: pure TF-IDF does NOT do stemming, so a query like + ``"authentication"`` would NOT match ``"auth"``. That's a known + limitation of Option A and is documented in the module docstring. + Option B (embedding models) would handle this; Option A trades + that capability for zero-dependency determinism. + """ + result = semantic_query(auth_workspace, "auth", top_k=10) + assert result["status"] == "ok" + # verify_jwt_claims should be in the top 3 — its file path + # contributes "auth" AND it has the rare "jwt" term. + top_names = [r["name"] for r in result["results"][:3]] + assert "verify_jwt_claims" in top_names, ( + f"Expected verify_jwt_claims in top 3, got: {top_names}" + ) + + def test_results_sorted_by_score_descending(self, auth_workspace): + result = semantic_query(auth_workspace, "jwt", top_k=10) + assert result["status"] == "ok" + scores = [r["score"] for r in result["results"]] + assert scores == sorted(scores, reverse=True), ( + f"Scores not in descending order: {scores}" + ) + # The single symbol with "jwt" in its name/file path must be #1 + assert result["results"][0]["name"] == "verify_jwt_claims" + + def test_top_k_limit(self, auth_workspace): + # The fixture has 5 symbols. top_k=2 should cap results at 2. + result = semantic_query(auth_workspace, "user", top_k=2) + assert result["status"] == "ok" + assert len(result["results"]) <= 2 + assert result["stats"]["returned"] == len(result["results"]) + # truncated flag should be True if we had more than 2 matches + # (UserModel + loginUser both contain "user"; that's 2 matches, + # so it may not be truncated — we just assert the field exists). + assert "truncated" in result["stats"] + + def test_top_k_zero_returns_all(self, auth_workspace): + result = semantic_query(auth_workspace, "user", top_k=0) + assert result["status"] == "ok" + # top_k=0 means "no limit"; we should get every symbol that + # shares at least one term with the query. + assert result["stats"]["returned"] == len(result["results"]) + assert result["stats"]["truncated"] is False + + def test_unknown_query_returns_empty(self, auth_workspace): + # A query whose tokens don't appear in any symbol returns empty. + result = semantic_query(auth_workspace, "nonexistent_thing_xyz", top_k=10) + assert result["status"] == "ok" + assert result["results"] == [] + assert result["stats"]["returned"] == 0 + + def test_empty_query_returns_error(self, auth_workspace): + result = semantic_query(auth_workspace, "", top_k=10) + assert result["status"] == "error" + assert "message" in result + assert result["results"] == [] + + def test_whitespace_query_returns_error(self, auth_workspace): + result = semantic_query(auth_workspace, " \t ", top_k=10) + assert result["status"] == "error" + + def test_result_shape(self, auth_workspace): + """Every result entry must have the documented fields.""" + result = semantic_query(auth_workspace, "user", top_k=10) + for r in result["results"]: + assert "name" in r + assert "kind" in r + assert "file" in r + assert "line" in r + assert "language" in r + assert "signature" in r + assert "score" in r + assert "matched_terms" in r + assert isinstance(r["matched_terms"], list) + assert 0.0 <= r["score"] <= 1.0 + + def test_matched_terms_populated(self, auth_workspace): + """``matched_terms`` lists the query tokens that appeared in the doc.""" + result = semantic_query(auth_workspace, "jwt verify", top_k=10) + assert result["status"] == "ok" + if result["results"]: + top = result["results"][0] + # The top result for "jwt verify" must match at least one + # of these terms. + assert any( + t in top["matched_terms"] for t in ("jwt", "verify") + ) + + def test_stats_total_symbols(self, auth_workspace): + """``stats.total_symbols`` reflects the registry size, not just matches.""" + result = semantic_query(auth_workspace, "jwt", top_k=10) + assert result["stats"]["total_symbols"] == 5 # the fixture has 5 symbols + + +# ─── Cache invalidation tests ───────────────────────────────── + +class TestCacheInvalidation: + """Verify the (db_path, mtime) cache key picks up re-scans.""" + + def test_cache_returns_same_instance_for_same_mtime(self, workspace): + _make_db(workspace, [ + {"name": "foo", "kind": "function", "file_path": "a.py", + "line_start": 1, "language": "python"}, + ]) + idx1 = build_index(os.path.join(workspace, ".codelens", "codelens.db")) + idx2 = build_index(os.path.join(workspace, ".codelens", "codelens.db")) + assert idx1 is idx2, "Same mtime should return the cached instance" + + def test_cache_invalidates_on_mtime_change(self, workspace): + db_path = os.path.join(workspace, ".codelens", "codelens.db") + _make_db(workspace, [ + {"name": "foo", "kind": "function", "file_path": "a.py", + "line_start": 1, "language": "python"}, + ]) + idx1 = build_index(db_path) + assert len(idx1.symbols) == 1 + + # Force mtime change by sleeping then writing new symbols + time.sleep(0.05) + conn = sqlite3.connect(db_path) + try: + conn.execute( + "INSERT INTO symbols (name, kind, file_path, line_start, language) " + "VALUES ('bar', 'function', 'b.py', 2, 'python')" + ) + conn.commit() + finally: + conn.close() + # Touch the file to ensure mtime updates (some filesystems have + # coarse mtime granularity). + os.utime(db_path, None) + + idx2 = build_index(db_path) + assert idx2 is not idx1, "Cache should have invalidated on mtime change" + assert len(idx2.symbols) == 2, "New index should reflect the new symbol" + + +# ─── Degradation tests ──────────────────────────────────────── + +class TestGracefulDegradation: + """The engine must never crash the host command.""" + + def test_missing_db_returns_empty_results(self, workspace): + # workspace fixture creates the dir but no .codelens/ + result = semantic_query(workspace, "anything", top_k=10) + assert result["status"] == "ok" + assert result["results"] == [] + assert result["stats"]["total_symbols"] == 0 + assert result["stats"]["returned"] == 0 + + def test_db_without_symbols_table_returns_empty(self, workspace): + # Create the db file but no `symbols` table + codelens_dir = os.path.join(workspace, ".codelens") + os.makedirs(codelens_dir, exist_ok=True) + db_path = os.path.join(codelens_dir, "codelens.db") + conn = sqlite3.connect(db_path) + conn.execute("CREATE TABLE unrelated (x INTEGER)") + conn.commit() + conn.close() + + result = semantic_query(workspace, "anything", top_k=10) + assert result["status"] == "ok" + assert result["results"] == [] + + def test_empty_symbols_table_returns_empty(self, workspace): + _make_db(workspace, []) + result = semantic_query(workspace, "anything", top_k=10) + assert result["status"] == "ok" + assert result["results"] == [] + assert result["stats"]["total_symbols"] == 0 + + def test_negative_top_k_treated_as_zero(self, auth_workspace): + # Should not raise; negative top_k is clamped to 0. + result = semantic_query(auth_workspace, "user", top_k=-5) + assert result["status"] == "ok" + # top_k=0 means "all matches" — just verify no exception. + + +# ─── IDF / ranking signal tests ─────────────────────────────── + +class TestRankingSignal: + """Verify TF-IDF math produces intuitive rankings.""" + + def test_rare_term_ranks_higher_than_common_term(self, workspace): + """A symbol that contains a rare, query-specific term should + outrank a symbol that only contains a common, ubiquitous term.""" + _make_db(workspace, [ + # 10 symbols that all contain "data" (common term) + *[{ + "name": f"process_data_{i}", + "kind": "function", + "file_path": f"mod{i}.py", + "line_start": i, + "language": "python", + } for i in range(10)], + # 1 symbol that contains the rare term "zebra" + { + "name": "find_zebra", + "kind": "function", + "file_path": "animals/zebra.py", + "line_start": 1, + "language": "python", + }, + ]) + # Query for the rare term — find_zebra should be the only result + result = semantic_query(workspace, "zebra", top_k=10) + assert result["status"] == "ok" + assert len(result["results"]) == 1 + assert result["results"][0]["name"] == "find_zebra" + assert result["results"][0]["score"] > 0.0 + + def test_multiple_query_terms_prefer_documents_with_all(self, workspace): + """A document containing BOTH query terms should outrank one + containing only one.""" + _make_db(workspace, [ + { + "name": "auth_token", + "kind": "function", + "file_path": "auth.py", + "line_start": 1, + "language": "python", + }, + { + "name": "auth_only", + "kind": "function", + "file_path": "x.py", + "line_start": 2, + "language": "python", + }, + { + "name": "token_only", + "kind": "function", + "file_path": "y.py", + "line_start": 3, + "language": "python", + }, + ]) + result = semantic_query(workspace, "auth token", top_k=10) + assert result["status"] == "ok" + # The symbol with both "auth" and "token" must be #1 + assert result["results"][0]["name"] == "auth_token" + # And it must have matched both terms + assert "auth" in result["results"][0]["matched_terms"] + assert "token" in result["results"][0]["matched_terms"] + + +# ─── CLI registration smoke test ────────────────────────────── + +class TestCommandRegistration: + """Verify the semantic-query command is registered and importable.""" + + def test_command_registered(self): + # Importing commands.semantic_query should register it + from commands import COMMAND_REGISTRY + assert "semantic-query" in COMMAND_REGISTRY, ( + "semantic-query must be in COMMAND_REGISTRY after import" + ) + + def test_command_has_help_text(self): + from commands import COMMAND_REGISTRY + info = COMMAND_REGISTRY["semantic-query"] + assert info["help"] + assert "semantic" in info["help"].lower() or "tf-idf" in info["help"].lower() + + def test_command_module_imports_cleanly(self): + # Re-import to make sure no import-time errors + import importlib + import commands.semantic_query as mod + importlib.reload(mod) + + +if __name__ == "__main__": + pytest.main([__file__, "-v", "--tb=short"])