diff --git a/.github/workflows/publish-pypi.yml b/.github/workflows/publish-pypi.yml new file mode 100644 index 00000000..d87b0a74 --- /dev/null +++ b/.github/workflows/publish-pypi.yml @@ -0,0 +1,92 @@ +name: Publish to PyPI + +# Issue #54 Phase 1: publish codelens to PyPI on tag push. +# Triggers on tags matching v*.*.* (e.g. v8.2.0, v8.3.0-rc1). +# Uses trusted publishing (OIDC) — no PyPI API token needed. + +on: + push: + tags: + - "v*.*.*" + workflow_dispatch: # Allow manual trigger for testing + +permissions: + contents: read + id-token: write # Required for PyPI trusted publishing (OIDC) + +jobs: + build: + name: Build sdist + wheel + runs-on: ubuntu-latest + + steps: + - name: Checkout repository + uses: actions/checkout@v4 + with: + fetch-depth: 0 # Full history for version detection + + - name: Set up Python 3.11 + uses: actions/setup-python@v5 + with: + python-version: "3.11" + + - name: Install build dependencies + run: | + python -m pip install --upgrade pip + pip install build twine + + - name: Build sdist and wheel + run: | + python -m build + + - name: Verify distributions + run: | + twine check dist/* + + - name: List built artifacts + run: | + ls -la dist/ + + - name: Upload build artifacts + uses: actions/upload-artifact@v4 + with: + name: dist + path: dist/ + + publish-pypi: + name: Publish to PyPI + runs-on: ubuntu-latest + needs: build + environment: pypi # Requires manual approval for first-time publishes + + steps: + - name: Download build artifacts + uses: actions/download-artifact@v4 + with: + name: dist + path: dist/ + + - name: Publish to PyPI (trusted publishing via OIDC) + uses: pypa/gh-action-pypi-publish@release/v1 + # No password/token — uses GitHub OIDC trusted publishing. + # Configure at https://pypi.org/manage/account/publishing/ + # with environment "pypi" and this workflow's name. + + publish-testpypi: + name: Publish to TestPyPI (on manual trigger) + runs-on: ubuntu-latest + needs: build + if: github.event_name == 'workflow_dispatch' + environment: testpypi + + steps: + - name: Download build artifacts + uses: actions/download-artifact@v4 + with: + name: dist + path: dist/ + + - name: Publish to TestPyPI (trusted publishing via OIDC) + uses: pypa/gh-action-pypi-publish@release/v1 + with: + repository-url: https://test.pypi.org/legacy/ diff --git a/README.md b/README.md index b90f8c93..b61b3a04 100644 --- a/README.md +++ b/README.md @@ -2,12 +2,12 @@ > **Before an AI writes a new class/id/function, CodeLens must be checked. This is not optional.** -CodeLens is an AI-native code intelligence platform that gives AI agents **full visibility** into a codebase before they write any code. It prevents collision, overwrite of existing logic, security vulnerabilities, and dead code through 68 CLI commands, an MCP server with 66 tools (54 static + 12 dynamic), AST-based taint analysis, live CVE/OSV scanning, a plugin system with OWASP Top 10 + Compliance rule packs, a true graph data model (nodes + edges) for structural code queries, and token-efficient `--format compact` output for high-volume agent workflows (issue #17). +CodeLens is an AI-native code intelligence platform that gives AI agents **full visibility** into a codebase before they write any code. It prevents collision, overwrite of existing logic, security vulnerabilities, and dead code through 69 CLI commands, an MCP server with 67 tools (54 static + 13 dynamic), AST-based taint analysis, live CVE/OSV scanning, a plugin system with OWASP Top 10 + Compliance rule packs, a true graph data model (nodes + edges) for structural code queries, and token-efficient `--format compact` output for high-volume agent workflows (issue #17). ## Features -- **68 CLI Commands** — From basic scan/query to AST taint analysis, CVE scanning, plugin management, auto-fix, dashboards, CI/CD quality gates, and `graph-schema` for cheap graph-shape introspection -- **MCP Server (66 Tools)** — Native AI agent integration via Model Context Protocol (JSON-RPC over stdio), 54 statically-defined tools + 12 dynamically discovered, every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`) +- **69 CLI Commands** — From basic scan/query to AST taint analysis, CVE scanning, plugin management, auto-fix, dashboards, CI/CD quality gates, and `graph-schema` for cheap graph-shape introspection +- **MCP Server (67 Tools)** — Native AI agent integration via Model Context Protocol (JSON-RPC over stdio), 54 statically-defined tools + 13 dynamically discovered, every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`) - **Token-Efficient Compact Output (v8.2, issue #17)** — `--format compact` produces single-char-key JSON with abbreviated types, omitted null fields, and relative paths — ~50% smaller than `json` on real trace output. Combined with `--limit`/`--offset` pagination, 5 structural queries now cost <5k tokens (down from 30-80k) - **AST Taint Engine** — Tree-sitter based taint analysis with return-value propagation, scope hierarchy, and branch condition refinement - **Live CVE/OSV Scanning** — Real-time vulnerability data from OSV.dev API with SQLite cache, 9 ecosystems (PyPI, npm, crates.io, Go, Maven, NuGet, RubyGems, Pub, Hex) @@ -225,8 +225,8 @@ codelens/ │ ├── changelog.md # Older changelog (per-version highlights) │ └── agent-integration.md # AI agent integration guide ├── scripts/ -│ ├── codelens.py # CLI entry point (68 commands registered) -│ ├── mcp_server.py # MCP JSON-RPC server (66 tools) +│ ├── codelens.py # CLI entry point (69 commands registered) +│ ├── mcp_server.py # MCP JSON-RPC server (67 tools) │ ├── registry.py # Registry read/write/build │ ├── persistent_registry.py # SQLite persistent storage (WAL mode) │ ├── base_parser.py # Base tree-sitter parser diff --git a/SKILL-QUICK.md b/SKILL-QUICK.md index 5764ffd7..4afc3bc1 100755 --- a/SKILL-QUICK.md +++ b/SKILL-QUICK.md @@ -114,7 +114,7 @@ $CLI list --limit 5 --offset 10 --format compact # → paginated + co | "Cross-file taint" | `dataflow` | `taint` (taint is single-file, AST-deep) | | "Auto-fix issues" | `fix` | `check` (check just gates, doesn't fix) | -## All 68 Commands +## All 69 Commands ### Setup & Lifecycle (8+) `init` · `scan [--incremental] [--max-files N] [--full]` · `registry-validate` · `detect` · `watch [--debounce SECS] [--git-mode] [--interval SECS]` · `git-status` · `migrate` · `serve` · `lsp-status` (issue #33: `codelens --lsp-status` top-level flag is an alias of `codelens lsp-status` — both delegate to `hybrid_engine.get_lsp_status()` and return the identical payload) @@ -146,9 +146,9 @@ $CLI list --limit 5 --offset 10 --format compact # → paginated + co ### Tooling (1) `plugin ` -**Total: 68 commands** (auto-registered via `commands/__init__.py`; rerun `python3 scripts/sync_command_count.py --apply` after adding/removing a command) +**Total: 69 commands** (auto-registered via `commands/__init__.py`; rerun `python3 scripts/sync_command_count.py --apply` after adding/removing a command) -## MCP Server (66 Tools) +## MCP Server (67 Tools) Start the MCP server for AI agent integration: @@ -156,9 +156,9 @@ Start the MCP server for AI agent integration: python3 scripts/codelens.py serve ``` -Exposes 66 tools as `codelens_` (e.g., `codelens_query`, `codelens_taint`, `codelens_graph_schema`, `codelens_architecture`, `codelens_resolve_types`, `codelens_git_status`): +Exposes 67 tools as `codelens_` (e.g., `codelens_query`, `codelens_taint`, `codelens_graph_schema`, `codelens_architecture`, `codelens_resolve_types`, `codelens_git_status`): - 50 statically-defined tools (full JSON schemas in `mcp_server.py`) -- 12 dynamically-discovered tools (auto-discovered from `COMMAND_REGISTRY`; long-running `watch` and `serve` are excluded) +- 13 dynamically-discovered tools (auto-discovered from `COMMAND_REGISTRY`; long-running `watch` and `serve` are excluded) - Every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`). Use `format: "compact"` for token-efficient responses (~50% smaller than `json`). - `watch` and `serve` itself are excluded (long-running) diff --git a/SKILL.md b/SKILL.md index bd91f902..379405c3 100755 --- a/SKILL.md +++ b/SKILL.md @@ -1,10 +1,10 @@ --- name: codelens description: > - CodeLens — AI-Native Code Intelligence. 68 commands for AI-powered code analysis, + CodeLens — AI-Native Code Intelligence. 69 commands for AI-powered code analysis, security auditing, quality scoring, AST-based taint analysis, live CVE scanning, and pre-write safety checks. Supports 28+ languages with tree-sitter + regex - fallback parsing. MCP server exposes 66 tools for AI agent integration. + fallback parsing. MCP server exposes 67 tools for AI agent integration. For quick command reference with validated output schemas, see SKILL-QUICK.md. For version history, see CHANGELOG.md. --- diff --git a/codelens/__init__.py b/codelens/__init__.py new file mode 100644 index 00000000..109a1345 --- /dev/null +++ b/codelens/__init__.py @@ -0,0 +1,140 @@ +"""CodeLens — AI-native code intelligence CLI + MCP server. + +This is the top-level package for the ``codelens`` distribution. It +provides a thin entry-point wrapper that delegates to the legacy +``scripts/codelens.py`` implementation so that ``pip install codelens`` +exposes a ``codelens`` console script. + +The actual implementation lives under ``scripts/`` (which is also the +``codelens`` import root via ``pyproject.toml``'s ``packages.find``). +This ``codelens/`` package only contains the ``__main__`` shim and +re-exports the ``main`` function for programmatic use. + +Issue #54 Phase 1 — Python package + PyPI distribution. +""" + +from __future__ import annotations + +import os +import sys +import warnings +from typing import Optional, List + + +__version__ = "8.2.0" + +# Path to the bundled scripts/ directory (legacy implementation). +# When installed as a package, scripts/ is shipped inside the codelens +# distribution and we add it to sys.path so the legacy imports keep +# working. When running from a source checkout, scripts/ is already +# accessible relative to the repo root. +_SCRIPTS_DIR: Optional[str] = None + + +def _resolve_scripts_dir() -> str: + """Locate the bundled ``scripts/`` directory. + + Search order: + 1. ``/../scripts`` — source checkout (repo root) + 2. ``/scripts`` — installed package (scripts bundled inside) + 3. Fall back to the legacy ``python3 scripts/codelens.py`` path + + Returns the absolute path to the scripts directory. + """ + global _SCRIPTS_DIR + if _SCRIPTS_DIR is not None: + return _SCRIPTS_DIR + + here = os.path.dirname(os.path.abspath(__file__)) + + # 1. Source checkout: /scripts/ + candidate = os.path.join(here, "..", "scripts") + if os.path.isfile(os.path.join(candidate, "codelens.py")): + _SCRIPTS_DIR = os.path.abspath(candidate) + return _SCRIPTS_DIR + + # 2. Installed package: /codelens/scripts/ + candidate = os.path.join(here, "scripts") + if os.path.isfile(os.path.join(candidate, "codelens.py")): + _SCRIPTS_DIR = os.path.abspath(candidate) + return _SCRIPTS_DIR + + # 3. Fall back: try importlib.resources or just leave it to the + # caller's sys.path (legacy ``python3 scripts/codelens.py`` mode). + _SCRIPTS_DIR = "" + return _SCRIPTS_DIR + + +def _ensure_scripts_on_path() -> str: + """Add the bundled scripts/ directory to sys.path if not present. + + Returns the scripts directory path (empty string if not found). + """ + scripts_dir = _resolve_scripts_dir() + if scripts_dir and scripts_dir not in sys.path: + sys.path.insert(0, scripts_dir) + return scripts_dir + + +def main(argv: Optional[List[str]] = None) -> int: + """Run the CodeLens CLI. + + Args: + argv: Optional argument list (defaults to ``sys.argv[1:]``). + + Returns: + Process exit code (0 on success, 1 on error). + """ + _ensure_scripts_on_path() + + # Import the legacy entry point. The scripts/codelens.py module uses + # sys.path-based imports relative to scripts/, so it must be on the + # path (handled above). + try: + # ``import codelens`` would be ambiguous here (this package is + # also named codelens), so import the module by file path. + import importlib.util + scripts_dir = _resolve_scripts_dir() + if not scripts_dir: + print( + "[codelens] ERROR: could not locate the scripts/ directory. " + "If you installed from source, run from the repo root or " + "install with 'pip install .'.", + file=sys.stderr, + ) + return 1 + + codelens_py = os.path.join(scripts_dir, "codelens.py") + spec = importlib.util.spec_from_file_location( + "_codelens_cli", codelens_py + ) + if spec is None or spec.loader is None: + print( + f"[codelens] ERROR: could not load {codelens_py}", + file=sys.stderr, + ) + return 1 + module = importlib.util.module_from_spec(spec) + # The module needs to be in sys.modules for its relative imports + # (e.g. ``from commands import ...``) to resolve correctly when + # scripts/ is on sys.path. + sys.modules["_codelens_cli"] = module + spec.loader.exec_module(module) + + # Temporarily replace sys.argv so the legacy main() picks up the + # caller's arguments. + old_argv = sys.argv + try: + sys.argv = ["codelens"] + (list(argv) if argv is not None else sys.argv[1:]) + module.main() + return 0 + except SystemExit as e: + return int(e.code) if e.code is not None else 0 + finally: + sys.argv = old_argv + except Exception as e: + print(f"[codelens] ERROR: {e}", file=sys.stderr) + return 1 + + +__all__ = ["main", "__version__"] diff --git a/codelens/__main__.py b/codelens/__main__.py new file mode 100644 index 00000000..7e29fa27 --- /dev/null +++ b/codelens/__main__.py @@ -0,0 +1,9 @@ +"""Allow ``python -m codelens`` to run the CLI.""" + +import sys + +from . import main + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/install.md b/docs/install.md new file mode 100644 index 00000000..3235e53b --- /dev/null +++ b/docs/install.md @@ -0,0 +1,149 @@ +# Installation + +CodeLens can be installed in several ways. Pick the one that fits your workflow. + +## Option 1 — pip install (recommended) + +```bash +pip install codelens +``` + +This installs the `codelens` console command and all core dependencies. +After install, run: + +```bash +codelens init /path/to/your/project +codelens scan /path/to/your/project +codelens query MyFunction /path/to/your/project +``` + +### With optional features + +```bash +# Tree-sitter grammars for AST-based analysis (recommended) +pip install "codelens[grammars]" + +# YAML rule engine (Semgrep-compat rules) +pip install "codelens[rules]" + +# File watcher (for `codelens watch`) +pip install "codelens[watch]" + +# Native LSP server (for `codelens lsp`) +pip install "codelens[lsp]" + +# Everything (for development) +pip install "codelens[all]" +``` + +## Option 2 — pipx (isolated environment) + +```bash +pipx install codelens +``` + +[pipx](https://pypa.github.io/pipx/) installs CodeLens in an isolated +environment so it doesn't conflict with your system Python packages. +The `codelens` command is available on your PATH. + +## Option 3 — python -m codelens + +If you prefer not to install a console script: + +```bash +pip install codelens +python -m codelens scan /path/to/your/project +``` + +## Option 4 — from source (legacy mode) + +```bash +git clone https://github.com/Wolfvin/CodeLens.git +cd CodeLens +pip install -e . +``` + +After install, both `codelens` (console command) and +`python3 scripts/codelens.py` (legacy mode) work. The legacy mode +prints a one-time deprecation warning pointing you to the console +command. + +### Running without install + +You can also run CodeLens directly from a source checkout without +installing: + +```bash +git clone https://github.com/Wolfvin/CodeLens.git +cd CodeLens +PYTHONUTF8=1 python3 scripts/codelens.py scan . +``` + +This is the original invocation method and remains fully supported. + +## Requirements + +- Python 3.8+ +- `tree-sitter>=0.21.0` (installed automatically by pip) + +### Optional dependencies + +| Extra | What it provides | Install | +|-------|------------------|---------| +| `grammars` | tree-sitter grammars for Python, JS, TS, Rust, HTML, CSS | `pip install "codelens[grammars]"` | +| `rules` | PyYAML for YAML rule engine | `pip install "codelens[rules]"` | +| `watch` | watchdog for `codelens watch` file monitoring | `pip install "codelens[watch]"` | +| `lsp` | pygls + lsprotocol for native LSP server | `pip install "codelens[lsp]"` | +| `dev` | pytest + pytest-cov for development | `pip install "codelens[dev]"` | +| `all` | All of the above | `pip install "codelens[all]"` | + +## Verifying the install + +```bash +codelens --command-count # Should print a number (67+) +codelens init /tmp/test # Initialize a test workspace +codelens scan /tmp/test # Scan it +``` + +## Troubleshooting + +### `codelens: command not found` after pip install + +Your pip bin directory may not be on PATH. Either: + +- Add it to PATH: `export PATH="$HOME/.local/bin:$PATH"` +- Or use `python -m codelens` instead +- Or use `pipx install codelens` which handles PATH automatically + +### UnicodeEncodeError on Windows + +Set `PYTHONUTF8=1` before running codelens: + +```cmd +set PYTHONUTF8=1 +codelens scan C:\path\to\project +``` + +### Tree-sitter grammar not found + +Install the grammars extra: + +```bash +pip install "codelens[grammars]" +``` + +Without grammars, CodeLens falls back to regex-based parsing which has +lower accuracy but works everywhere. + +## Upcoming distribution channels (issue #54) + +Phase 1 (this release) covers pip / pipx / source. Future phases will add: + +- **Phase 2**: Docker image (`docker run ghcr.io/wolfvin/codelens`) +- **Phase 3**: Self-contained binary (no Python needed) +- **Phase 4**: Homebrew (`brew install wolfvin/tap/codelens`), Scoop, Nix +- **Phase 5**: Release signing (minisign + Cosign) +- **Phase 6**: Auto-update command (`codelens upgrade`) + +See [issue #54](https://github.com/Wolfvin/CodeLens/issues/54) for the +full roadmap. diff --git a/pyproject.toml b/pyproject.toml index 51b37456..e766222b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta" [project] name = "codelens" version = "8.2.0" -description = "Live Codebase Reference Intelligence — 68 commands for AI-powered code analysis, security auditing, and quality scoring" +description = "Live Codebase Reference Intelligence — 69 commands for AI-powered code analysis, security auditing, and quality scoring" readme = "README.md" license = {text = "MIT"} requires-python = ">=3.8" @@ -74,13 +74,22 @@ all = [ "codelens[grammars,watch,rules,lsp,dev]", ] -# Entry point removed — codelens is run directly via python3 codelens.py -# The scripts/ directory uses sys.path-based imports, not a proper Python package. -# To install as a package, add __init__.py and convert to absolute imports. +# Issue #54 Phase 1: entry point for ``pip install codelens`` / ``pipx install codelens``. +# The codelens package (codelens/__init__.py) is a thin wrapper that delegates +# to scripts/codelens.py — the legacy implementation. +[project.scripts] +codelens = "codelens:main" +# Install both the new codelens/ package AND the legacy scripts/ directory +# (which contains the actual implementation). The scripts/ dir has no +# __init__.py so its subdirs (commands/, formatters/, etc.) are found as +# packages and its top-level .py files become importable modules. [tool.setuptools.packages.find] -where = ["scripts"] -include = ["*"] +where = [".", "scripts"] +include = ["codelens", "codelens.*", "commands", "commands.*", "formatters", "formatters.*", "parsers", "parsers.*", "plugins", "plugins.*"] +# Note: scripts/codelens.py is auto-discovered as a top-level py-module +# named "codelens" which would shadow the codelens/ package. We handle +# this in setup.py by explicitly excluding it — see setup.py for details. [tool.pytest.ini_options] testpaths = ["tests"] diff --git a/scripts/codelens.py b/scripts/codelens.py index 652aaddf..9c8592d1 100755 --- a/scripts/codelens.py +++ b/scripts/codelens.py @@ -1332,4 +1332,17 @@ def main(): if __name__ == "__main__": + # Issue #54 Phase 1: ``python3 scripts/codelens.py`` is the legacy + # invocation. After ``pip install codelens`` the ``codelens`` console + # script (or ``python -m codelens``) is preferred. This legacy entry + # point remains fully supported — the warning is informational only. + import warnings as _w + _w.warn( + "Running codelens via 'python3 scripts/codelens.py' is the legacy " + "mode. For a better experience, 'pip install codelens' and use the " + "'codelens' command directly (issue #54 Phase 1). This legacy mode " + "will continue to work.", + DeprecationWarning, + stacklevel=1, + ) main() diff --git a/scripts/graph_model.py b/scripts/graph_model.py index 1790fb7a..727635bf 100644 --- a/scripts/graph_model.py +++ b/scripts/graph_model.py @@ -13,7 +13,7 @@ - New tables `graph_nodes` and `graph_edges` are additive (prefixed `graph_` to avoid colliding with any existing table name). - The flat registry tables and JSON files are untouched. -- All 68 existing CLI commands continue to work unchanged. +- All 69 existing CLI commands continue to work unchanged. Schema: graph_nodes( diff --git a/setup.py b/setup.py new file mode 100644 index 00000000..9feb641c --- /dev/null +++ b/setup.py @@ -0,0 +1,26 @@ +"""Setup script for CodeLens (issue #54 Phase 1). + +pyproject.toml handles most of the build config, but we need a setup.py +to exclude ``scripts/codelens.py`` from top-level py-module discovery. +Without this exclusion, setuptools auto-discovers ``scripts/codelens.py`` +as a module named ``codelens`` which shadows the ``codelens/`` package +(both have the same name). The package loads scripts/codelens.py +explicitly via importlib in ``codelens/__init__.py``. +""" + +from setuptools import setup + +# Collect all top-level .py files in scripts/ EXCEPT codelens.py. +# These become importable top-level modules after pip install. +import os +_SCRIPTS_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "scripts") +_py_modules = [] +if os.path.isdir(_SCRIPTS_DIR): + for fname in os.listdir(_SCRIPTS_DIR): + if fname.endswith(".py") and fname != "codelens.py" and fname != "__init__.py": + # Strip .py extension to get module name + _py_modules.append(fname[:-3]) + +setup( + py_modules=_py_modules, +) diff --git a/skill.json b/skill.json index 20d1a64d..77df8998 100755 --- a/skill.json +++ b/skill.json @@ -1,7 +1,7 @@ { "name": "codelens", "version": "8.2.0", - "description": "Live Codebase Reference Intelligence. 68 commands for AI-powered code analysis, security auditing, quality scoring, and pre-write safety checks. Supports 28+ languages with regex+AST hybrid parsing. Must activate before writing/editing/deleting any class, id, or function.", + "description": "Live Codebase Reference Intelligence. 69 commands for AI-powered code analysis, security auditing, quality scoring, and pre-write safety checks. Supports 28+ languages with regex+AST hybrid parsing. Must activate before writing/editing/deleting any class, id, or function.", "author": "codelens", "command_categories": { "setup": [