diff --git a/README.md b/README.md index b62508a4..471d50e6 100644 --- a/README.md +++ b/README.md @@ -2,12 +2,12 @@ > **Before an AI writes a new class/id/function, CodeLens must be checked. This is not optional.** -CodeLens is an AI-native code intelligence platform that gives AI agents **full visibility** into a codebase before they write any code. It prevents collision, overwrite of existing logic, security vulnerabilities, and dead code through 66 CLI commands, an MCP server with 64 tools (50 static + 14 dynamic), AST-based taint analysis, live CVE/OSV scanning, a plugin system with OWASP Top 10 + Compliance rule packs, a true graph data model (nodes + edges) for structural code queries, and token-efficient `--format compact` output for high-volume agent workflows (issue #17). +CodeLens is an AI-native code intelligence platform that gives AI agents **full visibility** into a codebase before they write any code. It prevents collision, overwrite of existing logic, security vulnerabilities, and dead code through 67 CLI commands, an MCP server with 65 tools (50 static + 15 dynamic), AST-based taint analysis, live CVE/OSV scanning, a plugin system with OWASP Top 10 + Compliance rule packs, a true graph data model (nodes + edges) for structural code queries, and token-efficient `--format compact` output for high-volume agent workflows (issue #17). ## Features -- **66 CLI Commands** — From basic scan/query to AST taint analysis, CVE scanning, plugin management, auto-fix, dashboards, CI/CD quality gates, and `graph-schema` for cheap graph-shape introspection -- **MCP Server (64 Tools)** — Native AI agent integration via Model Context Protocol (JSON-RPC over stdio), 50 statically-defined tools + 14 dynamically discovered, every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`) +- **67 CLI Commands** — From basic scan/query to AST taint analysis, CVE scanning, plugin management, auto-fix, dashboards, CI/CD quality gates, and `graph-schema` for cheap graph-shape introspection +- **MCP Server (65 Tools)** — Native AI agent integration via Model Context Protocol (JSON-RPC over stdio), 50 statically-defined tools + 15 dynamically discovered, every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`) - **Token-Efficient Compact Output (v8.2, issue #17)** — `--format compact` produces single-char-key JSON with abbreviated types, omitted null fields, and relative paths — ~50% smaller than `json` on real trace output. Combined with `--limit`/`--offset` pagination, 5 structural queries now cost <5k tokens (down from 30-80k) - **AST Taint Engine** — Tree-sitter based taint analysis with return-value propagation, scope hierarchy, and branch condition refinement - **Live CVE/OSV Scanning** — Real-time vulnerability data from OSV.dev API with SQLite cache, 9 ecosystems (PyPI, npm, crates.io, Go, Maven, NuGet, RubyGems, Pub, Hex) @@ -225,8 +225,8 @@ codelens/ │ ├── changelog.md # Older changelog (per-version highlights) │ └── agent-integration.md # AI agent integration guide ├── scripts/ -│ ├── codelens.py # CLI entry point (66 commands registered) -│ ├── mcp_server.py # MCP JSON-RPC server (64 tools) +│ ├── codelens.py # CLI entry point (67 commands registered) +│ ├── mcp_server.py # MCP JSON-RPC server (65 tools) │ ├── registry.py # Registry read/write/build │ ├── persistent_registry.py # SQLite persistent storage (WAL mode) │ ├── base_parser.py # Base tree-sitter parser diff --git a/SKILL-QUICK.md b/SKILL-QUICK.md index cf29d54e..a3afe485 100755 --- a/SKILL-QUICK.md +++ b/SKILL-QUICK.md @@ -114,7 +114,7 @@ $CLI list --limit 5 --offset 10 --format compact # → paginated + co | "Cross-file taint" | `dataflow` | `taint` (taint is single-file, AST-deep) | | "Auto-fix issues" | `fix` | `check` (check just gates, doesn't fix) | -## All 66 Commands +## All 67 Commands ### Setup & Lifecycle (8+) `init` · `scan [--incremental] [--max-files N] [--full]` · `registry-validate` · `detect` · `watch [--debounce SECS] [--git-mode] [--interval SECS]` · `git-status` · `migrate` · `serve` · `lsp-status` (issue #33: `codelens --lsp-status` top-level flag is an alias of `codelens lsp-status` — both delegate to `hybrid_engine.get_lsp_status()` and return the identical payload) @@ -146,9 +146,9 @@ $CLI list --limit 5 --offset 10 --format compact # → paginated + co ### Tooling (1) `plugin ` -**Total: 66 commands** (auto-registered via `commands/__init__.py`; rerun `python3 scripts/sync_command_count.py --apply` after adding/removing a command) +**Total: 67 commands** (auto-registered via `commands/__init__.py`; rerun `python3 scripts/sync_command_count.py --apply` after adding/removing a command) -## MCP Server (64 Tools) +## MCP Server (65 Tools) Start the MCP server for AI agent integration: @@ -156,9 +156,9 @@ Start the MCP server for AI agent integration: python3 scripts/codelens.py serve ``` -Exposes 64 tools as `codelens_` (e.g., `codelens_query`, `codelens_taint`, `codelens_graph_schema`, `codelens_architecture`, `codelens_resolve_types`, `codelens_git_status`): +Exposes 65 tools as `codelens_` (e.g., `codelens_query`, `codelens_taint`, `codelens_graph_schema`, `codelens_architecture`, `codelens_resolve_types`, `codelens_git_status`): - 50 statically-defined tools (full JSON schemas in `mcp_server.py`) -- 14 dynamically-discovered tools (auto-discovered from `COMMAND_REGISTRY`; long-running `watch` and `serve` are excluded) +- 15 dynamically-discovered tools (auto-discovered from `COMMAND_REGISTRY`; long-running `watch` and `serve` are excluded) - Every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`). Use `format: "compact"` for token-efficient responses (~50% smaller than `json`). - `watch` and `serve` itself are excluded (long-running) diff --git a/SKILL.md b/SKILL.md index e3bd5eb2..e914b31c 100755 --- a/SKILL.md +++ b/SKILL.md @@ -1,10 +1,10 @@ --- name: codelens description: > - CodeLens — AI-Native Code Intelligence. 66 commands for AI-powered code analysis, + CodeLens — AI-Native Code Intelligence. 67 commands for AI-powered code analysis, security auditing, quality scoring, AST-based taint analysis, live CVE scanning, and pre-write safety checks. Supports 28+ languages with tree-sitter + regex - fallback parsing. MCP server exposes 64 tools for AI agent integration. + fallback parsing. MCP server exposes 65 tools for AI agent integration. For quick command reference with validated output schemas, see SKILL-QUICK.md. For version history, see CHANGELOG.md. --- diff --git a/pyproject.toml b/pyproject.toml index beed2de6..0a879294 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta" [project] name = "codelens" version = "8.2.0" -description = "Live Codebase Reference Intelligence — 66 commands for AI-powered code analysis, security auditing, and quality scoring" +description = "Live Codebase Reference Intelligence — 67 commands for AI-powered code analysis, security auditing, and quality scoring" readme = "README.md" license = {text = "MIT"} requires-python = ">=3.8" diff --git a/scripts/commands/memory.py b/scripts/commands/memory.py new file mode 100644 index 00000000..4278ff16 --- /dev/null +++ b/scripts/commands/memory.py @@ -0,0 +1,128 @@ +"""Memory command — Serena-style markdown memory system (issue #60). + +Provides cross-session memory for AI agents using CodeLens. Memory files are +plain markdown stored under ``.codelens/memories/`` in the workspace (project +scope) or ``~/.codelens/memories/global/`` (global scope, read-only via CLI). + +Usage:: + + codelens memory write # create/update project memory + codelens memory read # read memory (project or global) + codelens memory list # list all memories + codelens memory delete # delete (project memory only) + +The storage layout, file header rules, and ``mem:NAME`` reference validation +live in :mod:`memories.memory_manager` — this module is a thin CLI wrapper. +""" + +from __future__ import annotations + +from typing import Any, Dict + +from commands import register_command + + +def add_args(parser): + """Add memory subcommand arguments to the parser.""" + sub = parser.add_subparsers(dest="memory_action", help="Memory action") + + # memory write + write = sub.add_parser( + "write", + help="Create or update a project memory file", + description=( + "Create or update a project memory file at " + ".codelens/memories/.md. The file is given a canonical " + "'# Memory: ' header. mem:NAME references in are " + "validated and warnings are emitted for any references that do " + "not exist in project or global scope — the write itself always " + "succeeds (issue #60: warn, don't block)." + ), + ) + write.add_argument("name", help="Memory topic name (e.g. 'auth-flow')") + write.add_argument( + "content", + help="Memory content (markdown). May include 'mem:NAME' references.", + ) + + # memory read + read = sub.add_parser( + "read", + help="Read a memory file (project or global)", + description=( + "Read a memory file. Looks in the project scope first, then " + "falls back to the global scope. Returns 'not_found' if the " + "memory doesn't exist in either scope." + ), + ) + read.add_argument("name", help="Memory topic name") + + # memory list + sub.add_parser( + "list", + help="List all memory files (project + global)", + description=( + "List all memory files in project and global scope. Project " + "memories shadow global memories of the same name." + ), + ) + + # memory delete + delete = sub.add_parser( + "delete", + help="Delete a project memory file", + description=( + "Delete a project memory file. Global memories are read-only " + "via CLI and cannot be deleted here — remove them manually from " + "~/.codelens/memories/global/ if needed." + ), + ) + delete.add_argument("name", help="Memory topic name") + + +def execute(args, workspace): + """Dispatch the memory subcommand.""" + action = getattr(args, "memory_action", None) + if not action: + return { + "status": "error", + "error": "No memory action specified. Use: write, read, list, delete", + "usage": "codelens memory [args]", + "examples": [ + "codelens memory write auth-flow 'Uses JWT, see mem:tokens'", + "codelens memory read auth-flow", + "codelens memory list", + "codelens memory delete auth-flow", + ], + } + + # Lazy import so a broken memory_manager never breaks command discovery. + from memories.memory_manager import ( + write_memory, + read_memory, + list_memories, + delete_memory, + ) + + if action == "write": + return write_memory(workspace, args.name, args.content) + if action == "read": + return read_memory(workspace, args.name) + if action == "list": + return list_memories(workspace) + if action == "delete": + return delete_memory(workspace, args.name) + + return { + "status": "error", + "error": f"Unknown memory action: {action}", + "available_actions": ["write", "read", "list", "delete"], + } + + +register_command( + "memory", + "Serena-style markdown memory system (write/read/list/delete)", + add_args, + execute, +) diff --git a/scripts/graph_model.py b/scripts/graph_model.py index 22244927..7d224de7 100644 --- a/scripts/graph_model.py +++ b/scripts/graph_model.py @@ -13,7 +13,7 @@ - New tables `graph_nodes` and `graph_edges` are additive (prefixed `graph_` to avoid colliding with any existing table name). - The flat registry tables and JSON files are untouched. -- All 66 existing CLI commands continue to work unchanged. +- All 67 existing CLI commands continue to work unchanged. Schema: graph_nodes( diff --git a/scripts/memories/__init__.py b/scripts/memories/__init__.py new file mode 100644 index 00000000..3ec13168 --- /dev/null +++ b/scripts/memories/__init__.py @@ -0,0 +1,13 @@ +"""CodeLens memory package — Serena-style markdown memory system (issue #60). + +Exposes the :mod:`memory_manager` module which implements CRUD for project +and global memory files plus ``mem:NAME`` reference validation. + +Storage layout: +- Project memory: ``/.codelens/memories/.md`` +- Global memory: ``~/.codelens/memories/global/.md`` (read-only via CLI) +""" + +from . import memory_manager # noqa: F401 (re-export for convenience) + +__all__ = ["memory_manager"] diff --git a/scripts/memories/memory_manager.py b/scripts/memories/memory_manager.py new file mode 100644 index 00000000..4cc27265 --- /dev/null +++ b/scripts/memories/memory_manager.py @@ -0,0 +1,461 @@ +"""Serena-style markdown memory manager for CodeLens (issue #60). + +Provides cross-session memory for AI agents using CodeLens. Memory files are +plain markdown so they can be read, edited, and version-controlled by humans +or tools without specialised software. + +Storage layout +-------------- +- **Project memory:** ``/.codelens/memories/.md`` + Scoped to a single workspace; read/write via CLI. +- **Global memory:** ``~/.codelens/memories/global/.md`` + Scoped to the current user; **read-only via CLI**. Edit manually on the + filesystem to update. + +File format +----------- +Every memory file MUST start with a header line:: + + # Memory: + +The body is free-form markdown. A ``mem:NAME`` token anywhere in the body is +treated as a reference to another memory file named ``NAME``. References are +validated (non-blocking — see :func:`validate_references`) on write so authors +get a heads-up if they typo a topic name, but a missing reference never blocks +a write. +""" + +from __future__ import annotations + +import os +import re +import sys +from datetime import datetime, timezone +from typing import Any, Dict, List, Optional + +# ─── Public constants ────────────────────────────────────────────────────── + +# Memory file header prefix. Every memory file must start with this. +HEADER_PREFIX = "# Memory:" + +# Canonical header regex. Allows leading whitespace, requires '# Memory:', +# then captures the topic name (rest of line, trimmed). +_HEADER_RE = re.compile(r"^\s*#\s+Memory:\s*(.+?)\s*$", re.MULTILINE) + +# `mem:NAME` reference pattern. Names start with a letter and may contain +# letters, digits, underscores, dots, or hyphens — same charset enforced by +# :func:`_validate_name`. The leading word boundary prevents matching inside +# longer tokens like ``notmem:foo``. +_MEM_REF_RE = re.compile(r"(? str: + """Return the project memory directory for a workspace.""" + return os.path.join(workspace, ".codelens", "memories") + + +def global_memory_dir() -> str: + """Return the global memory directory (under the user's home).""" + return os.path.join(os.path.expanduser("~"), ".codelens", "memories", "global") + + +def project_memory_path(workspace: str, name: str) -> str: + """Return the absolute path to a project memory file.""" + _validate_name(name) + return os.path.join(project_memory_dir(workspace), f"{name}.md") + + +def global_memory_path(name: str) -> str: + """Return the absolute path to a global memory file.""" + _validate_name(name) + return os.path.join(global_memory_dir(), f"{name}.md") + + +# ─── Name validation ─────────────────────────────────────────────────────── + + +def _validate_name(name: str) -> None: + """Raise ``ValueError`` if ``name`` is not a valid memory topic name. + + Names must start with a letter and contain only letters, digits, + underscores, dots, or hyphens. This is the same charset that + :data:`_MEM_REF_RE` accepts inside ``mem:NAME`` references, so a writer + can always reference any validly-named memory. + """ + if not name: + raise ValueError("Memory name cannot be empty") + if not _NAME_RE.fullmatch(name): + raise ValueError( + f"Invalid memory name: {name!r}. " + "Names must start with a letter and contain only letters, digits, " + "underscores, dots, or hyphens." + ) + + +# ─── File content helpers ────────────────────────────────────────────────── + + +def build_file_content(name: str, content: str) -> str: + """Return canonical file content with the required header. + + If ``content`` already starts with a valid ``# Memory:`` header (with any + topic name), the existing header is replaced with the canonical one for + ``name``. Otherwise the header is prepended. + + The result is always terminated by a single newline so files are + diff-friendly and idempotent across writes. + """ + header = format_header(name) + stripped = content.lstrip() + # Detect and replace a pre-existing header (any topic). + if stripped.startswith(HEADER_PREFIX): + # Remove the first header line (and any blank line right after it). + without_header = _HEADER_RE.sub("", content, count=1).lstrip("\n") + body = without_header.rstrip() + else: + body = content.rstrip() + + if body: + return f"{header}\n\n{body}\n" + return f"{header}\n" + + +def format_header(name: str) -> str: + """Return the canonical header line for a memory named ``name``.""" + return f"{HEADER_PREFIX} {name}" + + +def parse_header_topic(first_line: str) -> Optional[str]: + """Extract the topic name from a header line. + + Returns ``None`` if the line is not a valid memory header. Used by + :func:`list_memories` to surface whether each file has a valid header. + """ + if not first_line: + return None + m = _HEADER_RE.match(first_line) + return m.group(1) if m else None + + +def has_valid_header(content: str) -> bool: + """Return True if ``content`` starts with a valid memory header line.""" + if not content: + return False + first_line = content.lstrip().splitlines()[0] if content.strip() else "" + return parse_header_topic(first_line) is not None + + +# ─── Reference handling ──────────────────────────────────────────────────── + + +def extract_references(content: str) -> List[str]: + """Return the unique ``mem:NAME`` references found in ``content``. + + Order is preserved (first occurrence wins). Self-references (``mem:NAME`` + inside the memory file named ``NAME``) are included — callers decide + whether to ignore them via the ``exclude`` parameter of + :func:`validate_references`. + """ + refs = _MEM_REF_RE.findall(content) + seen: set[str] = set() + unique: List[str] = [] + for r in refs: + if r not in seen: + seen.add(r) + unique.append(r) + return unique + + +def memory_exists(workspace: str, name: str) -> bool: + """Return True if a memory named ``name`` exists in project or global scope.""" + _validate_name(name) + return ( + os.path.isfile(project_memory_path(workspace, name)) + or os.path.isfile(global_memory_path(name)) + ) + + +def validate_references( + workspace: str, + content: str, + exclude: Optional[str] = None, +) -> Dict[str, Any]: + """Validate ``mem:NAME`` references in ``content`` (non-blocking). + + Returns a dict with: + + - ``references`` — all unique ``mem:NAME`` references found. + - ``missing`` — references whose target memory does not exist in + project or global scope. + - ``warnings`` — human-readable warning strings, one per missing + reference. Suitable for surfacing to the CLI user / AI agent. + + This function never raises on a missing reference — the contract is + "warn, don't block" (see issue #60). Invalid reference *syntax* is + silently ignored (the regex simply won't match). + """ + references = extract_references(content) + missing: List[str] = [] + for ref in references: + if exclude is not None and ref == exclude: + # Self-reference: the memory being written will exist after the + # write succeeds, so don't flag it. + continue + try: + if not memory_exists(workspace, ref): + missing.append(ref) + except ValueError: + # Shouldn't happen because the regex already validated the name, + # but be defensive — never block on validation. + continue + + warnings = [ + f"Reference 'mem:{ref}' does not exist (project or global)" + for ref in missing + ] + return { + "references": references, + "missing": missing, + "warnings": warnings, + } + + +# ─── CRUD operations ─────────────────────────────────────────────────────── + + +def write_memory(workspace: str, name: str, content: str) -> Dict[str, Any]: + """Create or update a project memory file. + + The file is written to ``/.codelens/memories/.md`` with a + canonical ``# Memory: `` header. ``mem:NAME`` references in + ``content`` are validated and any missing references are reported in the + result's ``warnings`` field — but the write always succeeds (issue #60: + warn, don't block). + + Global memories are not writable via this function — they are read-only + through the CLI by design. + """ + workspace = os.path.abspath(workspace) + _validate_name(name) + + file_content = build_file_content(name, content) + validation = validate_references(workspace, file_content, exclude=name) + + path = project_memory_path(workspace, name) + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "w", encoding="utf-8") as f: + f.write(file_content) + + result: Dict[str, Any] = { + "status": "ok", + "action": "written", + "scope": "project", + "name": name, + "path": path, + "references": validation["references"], + "size_bytes": len(file_content.encode("utf-8")), + } + if validation["missing"]: + result["missing_references"] = validation["missing"] + result["warnings"] = validation["warnings"] + return result + + +def read_memory(workspace: str, name: str) -> Dict[str, Any]: + """Read a memory file. Looks in project first, then global. + + Returns a ``not_found`` result (not an error) when the memory doesn't + exist in either scope, so callers can distinguish "missing" from "broken". + """ + workspace = os.path.abspath(workspace) + _validate_name(name) + + proj_path = project_memory_path(workspace, name) + if os.path.isfile(proj_path): + return _read_file(proj_path, name, "project") + + glob_path = global_memory_path(name) + if os.path.isfile(glob_path): + return _read_file(glob_path, name, "global") + + return { + "status": "not_found", + "name": name, + "message": ( + f"Memory '{name}' not found in project or global scope. " + "Run 'codelens memory list' to see available memories." + ), + } + + +def _read_file(path: str, name: str, scope: str) -> Dict[str, Any]: + """Read a memory file from disk and return a structured result.""" + try: + with open(path, "r", encoding="utf-8") as f: + content = f.read() + except OSError as e: + return { + "status": "error", + "error": f"Failed to read memory file: {e}", + "path": path, + "name": name, + "scope": scope, + } + + references = extract_references(content) + first_line = content.splitlines()[0] if content.strip() else "" + header_topic = parse_header_topic(first_line) + + return { + "status": "ok", + "scope": scope, + "name": name, + "path": path, + "content": content, + "references": references, + "header_topic": header_topic, + "has_valid_header": header_topic is not None, + "size_bytes": len(content.encode("utf-8")), + } + + +def list_memories(workspace: str) -> Dict[str, Any]: + """List all memories in project and global scope. + + Returns a dict with ``project`` and ``global`` lists plus a combined + ``memories`` list (deduplicated by name; project wins on collision). + """ + workspace = os.path.abspath(workspace) + + project = _list_dir(project_memory_dir(workspace), "project") + glob = _list_dir(global_memory_dir(), "global") + + # Deduplicate by name (project takes precedence — it shadows a global + # memory of the same name when reading via :func:`read_memory`). + seen: set[str] = set() + combined: List[Dict[str, Any]] = [] + for entry in project + glob: + if entry["name"] in seen: + continue + seen.add(entry["name"]) + combined.append(entry) + + return { + "status": "ok", + "total": len(combined), + "project_count": len(project), + "global_count": len(glob), + "project": project, + "global": glob, + "memories": combined, + } + + +def _list_dir(directory: str, scope: str) -> List[Dict[str, Any]]: + """List memory files in ``directory``. Returns ``[]`` if dir is missing.""" + if not os.path.isdir(directory): + return [] + + results: List[Dict[str, Any]] = [] + for fname in sorted(os.listdir(directory)): + if not fname.endswith(".md"): + continue + name = fname[:-3] + # Skip names that don't match our validation rules (e.g. manually + # created files with weird names). They still exist on disk but we + # don't surface them through the CLI. + if not _NAME_RE.fullmatch(name): + continue + path = os.path.join(directory, fname) + if not os.path.isfile(path): + continue + try: + stat = os.stat(path) + with open(path, "r", encoding="utf-8") as f: + first_line = f.readline().rstrip("\n") + except OSError: + continue + header_topic = parse_header_topic(first_line) + results.append({ + "name": name, + "scope": scope, + "path": path, + "size_bytes": stat.st_size, + "modified_at": datetime.fromtimestamp( + stat.st_mtime, tz=timezone.utc + ).isoformat(), + "header_topic": header_topic, + "has_valid_header": header_topic is not None, + }) + return results + + +def delete_memory(workspace: str, name: str) -> Dict[str, Any]: + """Delete a project memory file. + + Global memories are **read-only via CLI** and cannot be deleted here — + they must be removed manually from the filesystem. Returns ``not_found`` + when the project memory doesn't exist (rather than an error) so callers + can distinguish "missing" from "broken". + """ + workspace = os.path.abspath(workspace) + _validate_name(name) + + path = project_memory_path(workspace, name) + if not os.path.isfile(path): + return { + "status": "not_found", + "scope": "project", + "name": name, + "message": ( + f"Project memory '{name}' not found. " + "Global memories are read-only via CLI and cannot be deleted; " + "remove them manually from " + f"{global_memory_dir()!r}." + ), + } + + try: + os.remove(path) + except OSError as e: + return { + "status": "error", + "error": f"Failed to delete memory file: {e}", + "path": path, + "name": name, + "scope": "project", + } + + return { + "status": "ok", + "action": "deleted", + "scope": "project", + "name": name, + "path": path, + } + + +# ─── Module entry point for ad-hoc CLI inspection ────────────────────────── + + +def _main(argv: Optional[List[str]] = None) -> int: + """Tiny REPL-style entry point for manual debugging via ``python -m``. + + Not part of the public API — the real CLI lives in + ``scripts/commands/memory.py``. This exists so a developer can run + ``python3 -m memories.memory_manager`` to sanity-check imports. + """ + print("memory_manager module loaded successfully.") + print(f" Global memory dir: {global_memory_dir()}") + print(" Use 'codelens memory ' for real access.") + return 0 + + +if __name__ == "__main__": # pragma: no cover + sys.exit(_main()) diff --git a/skill.json b/skill.json index e2b2ad3d..04f55282 100755 --- a/skill.json +++ b/skill.json @@ -1,7 +1,7 @@ { "name": "codelens", "version": "8.2.0", - "description": "Live Codebase Reference Intelligence. 66 commands for AI-powered code analysis, security auditing, quality scoring, and pre-write safety checks. Supports 28+ languages with regex+AST hybrid parsing. Must activate before writing/editing/deleting any class, id, or function.", + "description": "Live Codebase Reference Intelligence. 67 commands for AI-powered code analysis, security auditing, quality scoring, and pre-write safety checks. Supports 28+ languages with regex+AST hybrid parsing. Must activate before writing/editing/deleting any class, id, or function.", "author": "codelens", "command_categories": { "setup": [ diff --git a/tests/test_integration.py b/tests/test_integration.py index 6aad33de..9b5af484 100644 --- a/tests/test_integration.py +++ b/tests/test_integration.py @@ -1,5 +1,5 @@ """ -Integration smoke tests for all 66 CodeLens commands. +Integration smoke tests for all 67 CodeLens commands. Tests that every command: 1. Runs without crash (valid JSON output) @@ -300,7 +300,7 @@ def test_command_registry_has_all_commands(self): sys.path.insert(0, SCRIPT_DIR) from commands import COMMAND_REGISTRY # Regression sentinel — see docstring above for update procedure. - EXPECTED_COMMAND_COUNT = 65 + EXPECTED_COMMAND_COUNT = 67 actual = len(COMMAND_REGISTRY) assert actual == EXPECTED_COMMAND_COUNT, ( f"Command count drift detected: expected {EXPECTED_COMMAND_COUNT}, " diff --git a/tests/test_memory.py b/tests/test_memory.py new file mode 100644 index 00000000..6f4b03d8 --- /dev/null +++ b/tests/test_memory.py @@ -0,0 +1,605 @@ +"""Tests for the Serena-style markdown memory system (issue #60). + +Covers: +- CRUD operations on project memory files (write/read/list/delete) +- Global memory fallback on read +- Global memory is read-only via CLI (write/delete reject it) +- File header is always present and canonical +- ``mem:NAME`` reference extraction + non-blocking validation (warn, not block) +- Name validation rejects invalid topic names +- The CLI ``memory`` command auto-registers and dispatches subcommands +""" + +from __future__ import annotations + +import os +import sys +import tempfile +from pathlib import Path + +import pytest + +# Make scripts/ importable. +SCRIPT_DIR = os.path.join( + os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "scripts" +) +if SCRIPT_DIR not in sys.path: + sys.path.insert(0, SCRIPT_DIR) + +from memories import memory_manager as mm # noqa: E402 +from commands import COMMAND_REGISTRY # noqa: E402 + + +# ─── Fixtures ────────────────────────────────────────────────────────────── + + +@pytest.fixture +def workspace(): + """Yield a temporary workspace directory.""" + d = tempfile.mkdtemp(prefix="codelens_memory_test_") + yield d + import shutil + shutil.rmtree(d, ignore_errors=True) + + +@pytest.fixture +def fake_home(monkeypatch, tmp_path): + """Redirect ``~`` to a temp dir so global memories don't touch the real home. + + Returns the path to the fake home directory. Tests that need a global + memory file should write to ``fake_home / ".codelens" / "memories" / "global"``. + """ + home = tmp_path / "fake_home" + home.mkdir() + monkeypatch.setenv("HOME", str(home)) + # os.path.expanduser caches nothing, but be explicit so any code reading + # HOME directly also gets the override. + monkeypatch.setattr(os.path, "expanduser", lambda p: str(home) if p == "~" else os.path.expanduser.__wrapped__(p) if hasattr(os.path.expanduser, "__wrapped__") else _expanduser(p, home)) + return home + + +def _expanduser(path: str, home: Path) -> str: + """Helper for monkeypatching os.path.expanduser without infinite recursion.""" + if path == "~": + return str(home) + if path.startswith("~/"): + return str(home / path[2:]) + return path + + +# ─── Name validation ─────────────────────────────────────────────────────── + + +class TestNameValidation: + """Names must match [A-Za-z][A-Za-z0-9_.-]* — same charset as mem:NAME.""" + + @pytest.mark.parametrize( + "name", + ["auth", "auth-flow", "auth_flow", "auth.flow", "auth2", "A", "a.b-c_d"], + ) + def test_valid_names_accepted(self, name): + mm._validate_name(name) # should not raise + # Path helpers should also accept these. + assert mm.project_memory_path("/ws", name).endswith(f"{name}.md") + assert mm.global_memory_path(name).endswith(f"{name}.md") + + @pytest.mark.parametrize( + "name", + [ + "", # empty + "1auth", # starts with digit + "-auth", # starts with hyphen + ".auth", # starts with dot + "_auth", # starts with underscore + "auth flow", # contains space + "auth/flow", # contains slash + "auth:flow", # contains colon + "auth$flow", # contains special char + ], + ) + def test_invalid_names_rejected(self, name): + with pytest.raises(ValueError): + mm._validate_name(name) + + def test_write_memory_rejects_invalid_name(self, workspace): + with pytest.raises(ValueError): + mm.write_memory(workspace, "1invalid", "content") + + def test_read_memory_rejects_invalid_name(self, workspace): + with pytest.raises(ValueError): + mm.read_memory(workspace, "1invalid") + + def test_delete_memory_rejects_invalid_name(self, workspace): + with pytest.raises(ValueError): + mm.delete_memory(workspace, "1invalid") + + +# ─── Header handling ─────────────────────────────────────────────────────── + + +class TestHeaderHandling: + """Every memory file must start with '# Memory: '.""" + + def test_write_adds_header_when_missing(self, workspace): + result = mm.write_memory(workspace, "topic", "Just body text.") + assert result["status"] == "ok" + with open(result["path"], "r", encoding="utf-8") as f: + content = f.read() + assert content.startswith("# Memory: topic\n") + assert "Just body text." in content + + def test_write_replaces_existing_header_with_canonical(self, workspace): + # Even if the user passes content with a different topic in the + # header, we overwrite with the canonical name. + result = mm.write_memory( + workspace, "real-name", "# Memory: wrong-name\n\nbody" + ) + with open(result["path"], "r", encoding="utf-8") as f: + content = f.read() + assert content.startswith("# Memory: real-name\n") + assert "wrong-name" not in content + assert "body" in content + + def test_write_preserves_body_when_only_header_given(self, workspace): + result = mm.write_memory(workspace, "t", "# Memory: t\n\nbody line 1\nbody line 2") + with open(result["path"], "r", encoding="utf-8") as f: + content = f.read() + assert "body line 1" in content + assert "body line 2" in content + + def test_write_idempotent(self, workspace): + """Writing the same content twice produces the same file.""" + mm.write_memory(workspace, "t", "body") + r1 = mm.read_memory(workspace, "t") + mm.write_memory(workspace, "t", "body") + r2 = mm.read_memory(workspace, "t") + assert r1["content"] == r2["content"] + assert r1["size_bytes"] == r2["size_bytes"] + + def test_has_valid_header(self): + assert mm.has_valid_header("# Memory: foo\n\nbody") + assert mm.has_valid_header(" # Memory: foo\nbody") + assert not mm.has_valid_header("No header here") + assert not mm.has_valid_header("") + assert not mm.has_valid_header("# Memory:\n") # missing topic + + def test_parse_header_topic(self): + assert mm.parse_header_topic("# Memory: auth-flow") == "auth-flow" + assert mm.parse_header_topic(" # Memory: spaced ") == "spaced" + assert mm.parse_header_topic("not a header") is None + assert mm.parse_header_topic("") is None + + +# ─── Write / Read / List / Delete ───────────────────────────────────────── + + +class TestWriteReadListDelete: + """Core CRUD lifecycle on project memory files.""" + + def test_write_creates_file_in_project_scope(self, workspace): + result = mm.write_memory(workspace, "auth", "Uses JWT.") + assert result["status"] == "ok" + assert result["action"] == "written" + assert result["scope"] == "project" + assert result["name"] == "auth" + assert result["path"] == os.path.join( + workspace, ".codelens", "memories", "auth.md" + ) + assert os.path.isfile(result["path"]) + assert result["size_bytes"] > 0 + + def test_write_creates_memories_dir(self, workspace): + memories_dir = os.path.join(workspace, ".codelens", "memories") + assert not os.path.exists(memories_dir) + mm.write_memory(workspace, "first", "content") + assert os.path.isdir(memories_dir) + + def test_write_updates_existing_file(self, workspace): + mm.write_memory(workspace, "topic", "v1") + r1 = mm.read_memory(workspace, "topic") + mm.write_memory(workspace, "topic", "v2 different content") + r2 = mm.read_memory(workspace, "topic") + assert r1["content"] != r2["content"] + assert "v2" in r2["content"] + + def test_read_returns_project_memory(self, workspace): + mm.write_memory(workspace, "topic", "Hello world") + result = mm.read_memory(workspace, "topic") + assert result["status"] == "ok" + assert result["scope"] == "project" + assert result["name"] == "topic" + assert "Hello world" in result["content"] + assert result["has_valid_header"] is True + assert result["header_topic"] == "topic" + + def test_read_returns_not_found_when_missing(self, workspace, fake_home): + result = mm.read_memory(workspace, "nonexistent") + assert result["status"] == "not_found" + assert "nonexistent" in result["message"] + + def test_read_falls_back_to_global(self, workspace, fake_home): + # Drop a global memory file directly (the only way to create one — + # write_memory only writes to project scope). + global_dir = fake_home / ".codelens" / "memories" / "global" + global_dir.mkdir(parents=True) + (global_dir / "global-topic.md").write_text( + "# Memory: global-topic\n\nGlobal content.\n", encoding="utf-8" + ) + + result = mm.read_memory(workspace, "global-topic") + assert result["status"] == "ok" + assert result["scope"] == "global" + assert "Global content." in result["content"] + + def test_read_prefers_project_over_global(self, workspace, fake_home): + # Both scopes have a memory with the same name. + global_dir = fake_home / ".codelens" / "memories" / "global" + global_dir.mkdir(parents=True) + (global_dir / "shared.md").write_text( + "# Memory: shared\n\nGlobal version.\n", encoding="utf-8" + ) + mm.write_memory(workspace, "shared", "Project version.") + + result = mm.read_memory(workspace, "shared") + assert result["status"] == "ok" + assert result["scope"] == "project" + assert "Project version." in result["content"] + + def test_list_empty_when_no_memories(self, workspace, fake_home): + result = mm.list_memories(workspace) + assert result["status"] == "ok" + assert result["total"] == 0 + assert result["project_count"] == 0 + assert result["global_count"] == 0 + assert result["memories"] == [] + + def test_list_returns_project_and_global(self, workspace, fake_home): + mm.write_memory(workspace, "p1", "project 1") + mm.write_memory(workspace, "p2", "project 2") + + global_dir = fake_home / ".codelens" / "memories" / "global" + global_dir.mkdir(parents=True) + (global_dir / "g1.md").write_text("# Memory: g1\n\nglobal 1\n", encoding="utf-8") + + result = mm.list_memories(workspace) + assert result["status"] == "ok" + assert result["project_count"] == 2 + assert result["global_count"] == 1 + assert result["total"] == 3 + + names = {m["name"] for m in result["memories"]} + assert names == {"p1", "p2", "g1"} + + # Each entry has the expected metadata. + for m in result["memories"]: + assert m["scope"] in ("project", "global") + assert m["path"] + assert m["size_bytes"] > 0 + assert m["modified_at"] + assert m["has_valid_header"] is True + assert m["header_topic"] == m["name"] + + def test_list_dedupes_with_project_taking_precedence(self, workspace, fake_home): + """A project memory shadows a global memory of the same name.""" + global_dir = fake_home / ".codelens" / "memories" / "global" + global_dir.mkdir(parents=True) + (global_dir / "shared.md").write_text( + "# Memory: shared\n\nglobal\n", encoding="utf-8" + ) + mm.write_memory(workspace, "shared", "project") + + result = mm.list_memories(workspace) + assert result["total"] == 1 + assert len(result["memories"]) == 1 + assert result["memories"][0]["scope"] == "project" + # The global list still shows the global entry. + assert result["global_count"] == 1 + assert result["project_count"] == 1 + + def test_delete_removes_project_memory(self, workspace): + mm.write_memory(workspace, "topic", "content") + path = os.path.join(workspace, ".codelens", "memories", "topic.md") + assert os.path.isfile(path) + + result = mm.delete_memory(workspace, "topic") + assert result["status"] == "ok" + assert result["action"] == "deleted" + assert result["scope"] == "project" + assert not os.path.exists(path) + + def test_delete_returns_not_found_for_missing_project(self, workspace): + result = mm.delete_memory(workspace, "never-existed") + assert result["status"] == "not_found" + assert "Global memories are read-only" in result["message"] + + def test_delete_cannot_delete_global(self, workspace, fake_home): + """Global memories are read-only via CLI — delete must refuse.""" + global_dir = fake_home / ".codelens" / "memories" / "global" + global_dir.mkdir(parents=True) + global_path = global_dir / "global-only.md" + global_path.write_text("# Memory: global-only\n\nbody\n", encoding="utf-8") + + # No project memory of the same name → not_found, file untouched. + result = mm.delete_memory(workspace, "global-only") + assert result["status"] == "not_found" + assert global_path.exists() # global file is untouched + + +# ─── Reference validation (warn, don't block) ───────────────────────────── + + +class TestReferenceValidation: + """``mem:NAME`` references are validated on write; missing refs warn.""" + + def test_extract_references_finds_all(self): + content = "See mem:auth and mem:tokens. Also mem:auth again." + refs = mm.extract_references(content) + # Deduplicated, order preserved. + assert refs == ["auth", "tokens"] + + def test_extract_references_ignores_non_matches(self): + # 'notmem:foo' should not match (leading word char before 'mem:'). + # 'mem:' alone has no name. 'mem:1foo' starts with digit, not allowed. + content = "notmem:foo mem: mem:1foo mem:valid" + refs = mm.extract_references(content) + assert refs == ["valid"] + + def test_extract_references_handles_special_chars_in_name(self): + content = "See mem:auth-flow, mem:tokens_v2, and mem:cache.hit" + refs = mm.extract_references(content) + assert "auth-flow" in refs + assert "tokens_v2" in refs + assert "cache.hit" in refs + + def test_extract_references_empty_content(self): + assert mm.extract_references("") == [] + assert mm.extract_references("no refs here") == [] + + def test_validate_references_no_missing(self, workspace): + mm.write_memory(workspace, "target", "target body") + result = mm.validate_references(workspace, "see mem:target") + assert result["references"] == ["target"] + assert result["missing"] == [] + assert result["warnings"] == [] + + def test_validate_references_reports_missing(self, workspace): + result = mm.validate_references(workspace, "see mem:nonexistent") + assert result["missing"] == ["nonexistent"] + assert "Reference 'mem:nonexistent' does not exist" in result["warnings"][0] + + def test_validate_references_exclude_self(self, workspace): + """When writing memory 'foo', a mem:foo self-reference is not flagged.""" + result = mm.validate_references( + workspace, "self-ref mem:foo and missing mem:bar", exclude="foo" + ) + assert "foo" in result["references"] + assert result["missing"] == ["bar"] # foo excluded + + def test_write_with_missing_reference_succeeds_with_warning(self, workspace): + """Issue #60: warn, don't block. The write must succeed.""" + result = mm.write_memory( + workspace, "login", "Login uses mem:auth-flow and mem:missing-topic." + ) + assert result["status"] == "ok" + assert result["action"] == "written" + # auth-flow doesn't exist either → both missing + assert "auth-flow" in result["missing_references"] + assert "missing-topic" in result["missing_references"] + assert any("auth-flow" in w for w in result["warnings"]) + assert any("missing-topic" in w for w in result["warnings"]) + # File was still written. + assert os.path.isfile(result["path"]) + + def test_write_with_existing_reference_no_warning(self, workspace): + """When all references exist, no warnings are emitted.""" + mm.write_memory(workspace, "auth", "auth body") + mm.write_memory(workspace, "tokens", "tokens body") + result = mm.write_memory( + workspace, "login", "Login uses mem:auth and mem:tokens." + ) + assert result["status"] == "ok" + assert "missing_references" not in result + assert "warnings" not in result + assert set(result["references"]) == {"auth", "tokens"} + + def test_write_with_self_reference_no_warning(self, workspace): + """A memory that references itself doesn't warn.""" + result = mm.write_memory( + workspace, "recursive", "see mem:recursive for details" + ) + assert result["status"] == "ok" + assert "recursive" in result["references"] + assert "missing_references" not in result + + def test_read_includes_references(self, workspace): + mm.write_memory(workspace, "topic", "see mem:other") + result = mm.read_memory(workspace, "topic") + assert result["references"] == ["other"] + + def test_validate_references_falls_back_to_global(self, workspace, fake_home): + """A reference to a global memory should not be flagged missing.""" + global_dir = fake_home / ".codelens" / "memories" / "global" + global_dir.mkdir(parents=True) + (global_dir / "global-topic.md").write_text( + "# Memory: global-topic\n\nbody\n", encoding="utf-8" + ) + result = mm.validate_references(workspace, "see mem:global-topic") + assert result["missing"] == [] + assert result["warnings"] == [] + + +# ─── Path helpers ───────────────────────────────────────────────────────── + + +class TestPathHelpers: + """Path helpers return deterministic, validated paths.""" + + def test_project_memory_dir(self, workspace): + assert mm.project_memory_dir(workspace) == os.path.join( + workspace, ".codelens", "memories" + ) + + def test_global_memory_dir_uses_home(self, fake_home): + assert mm.global_memory_dir() == str( + fake_home / ".codelens" / "memories" / "global" + ) + + def test_project_memory_path_validates_name(self): + with pytest.raises(ValueError): + mm.project_memory_path("/ws", "1invalid") + + def test_global_memory_path_validates_name(self): + with pytest.raises(ValueError): + mm.global_memory_path("1invalid") + + +# ─── CLI command registration & dispatch ────────────────────────────────── + + +class TestCommandRegistration: + """The ``memory`` command must auto-register via register_command().""" + + def test_memory_command_registered(self): + assert "memory" in COMMAND_REGISTRY + info = COMMAND_REGISTRY["memory"] + assert "Serena-style" in info["help"] + assert callable(info["add_args"]) + assert callable(info["execute"]) + + def test_command_registered_with_canonical_name(self): + """The execute function must belong to commands.memory module. + + This guards against the test_every_command_module_registers test in + test_command_registry.py — each commands/*.py must register at least + one command, and the registration must come from that module. + """ + info = COMMAND_REGISTRY["memory"] + assert info["execute"].__module__ == "commands.memory" + + +class TestCommandDispatch: + """End-to-end dispatch through the registered ``execute`` callback.""" + + def _parse_and_run(self, subcommand_args, workspace): + """Build an argparse namespace the way codelens.py does and dispatch.""" + import argparse + from commands.memory import add_args, execute + + parser = argparse.ArgumentParser(prog="codelens memory") + add_args(parser) + # The framework adds --format etc. to subparsers; we don't need them + # for these tests since execute() doesn't read them. + args = parser.parse_args(subcommand_args) + return execute(args, workspace) + + def test_no_action_returns_error(self, workspace): + result = self._parse_and_run([], workspace) + assert result["status"] == "error" + assert "No memory action" in result["error"] + + def test_write_via_dispatch(self, workspace): + result = self._parse_and_run( + ["write", "topic", "body content"], workspace + ) + assert result["status"] == "ok" + assert result["scope"] == "project" + + def test_read_via_dispatch(self, workspace): + self._parse_and_run(["write", "topic", "body content"], workspace) + result = self._parse_and_run(["read", "topic"], workspace) + assert result["status"] == "ok" + assert "body content" in result["content"] + + def test_list_via_dispatch(self, workspace): + self._parse_and_run(["write", "a", "alpha"], workspace) + self._parse_and_run(["write", "b", "beta"], workspace) + result = self._parse_and_run(["list"], workspace) + assert result["status"] == "ok" + assert result["total"] == 2 + + def test_delete_via_dispatch(self, workspace): + self._parse_and_run(["write", "topic", "body"], workspace) + result = self._parse_and_run(["delete", "topic"], workspace) + assert result["status"] == "ok" + assert result["action"] == "deleted" + + def test_unknown_action_returns_error(self, workspace): + # Argparse will reject unknown subcommands before execute() is called, + # but if memory_action is somehow None or unknown we should still + # return a structured error rather than crashing. + result = self._parse_and_run([], workspace) + assert result["status"] == "error" + + +# ─── CLI subprocess smoke test ──────────────────────────────────────────── + + +class TestCLISubprocess: + """Smoke-test the memory command end-to-end through codelens.py.""" + + def test_memory_command_runs_via_cli(self, workspace): + """`codelens memory write ...` end-to-end through codelens.py.""" + import subprocess + import json + + # Drop a project marker so the auto-detector finds this temp dir + # rather than walking up to the real CodeLens checkout. We also + # redirect HOME so the last-workspace cache (~/.codelens/...) doesn't + # leak across tests / pollute the developer's real home dir. + (Path(workspace) / "pyproject.toml").write_text( + "[project]\nname = 'test-ws'\nversion = '0'\n", encoding="utf-8" + ) + fake_home = Path(workspace) / "fake_home" + fake_home.mkdir() + + env = { + **os.environ, + "PYTHONPATH": SCRIPT_DIR, + "PYTHONUTF8": "1", + "HOME": str(fake_home), + } + codelens = os.path.join(SCRIPT_DIR, "codelens.py") + + # write + r = subprocess.run( + [sys.executable, codelens, "memory", "write", "topic", "body text"], + capture_output=True, text=True, env=env, cwd=workspace, timeout=30, + ) + assert r.returncode == 0, f"write failed: {r.stderr[:300]}" + + # The output should be JSON (after any [CodeLens] stderr lines). + stdout_lines = [ + line for line in r.stdout.splitlines() + if not line.startswith("[CodeLens]") + ] + data = json.loads("\n".join(stdout_lines)) + assert data["status"] == "ok" + # File must have landed inside the temp workspace, not somewhere else. + assert data["path"].startswith(workspace), ( + f"memory file written outside temp workspace: {data['path']}" + ) + assert os.path.isfile(data["path"]) + + def test_memory_no_action_via_cli(self): + """`codelens memory` with no action returns a structured error.""" + import subprocess + import json + + env = { + **os.environ, + "PYTHONPATH": SCRIPT_DIR, + "PYTHONUTF8": "1", + } + codelens = os.path.join(SCRIPT_DIR, "codelens.py") + r = subprocess.run( + [sys.executable, codelens, "memory"], + capture_output=True, text=True, env=env, timeout=30, + ) + assert r.returncode == 0 + stdout_lines = [ + line for line in r.stdout.splitlines() + if not line.startswith("[CodeLens]") + ] + data = json.loads("\n".join(stdout_lines)) + assert data["status"] == "error" + assert "No memory action" in data["error"]