Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 97 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,103 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [8.2.0] — Unreleased

### Token-Efficient Output + Pagination (issue #17)

Adds a 5th output format (`compact`) and pagination to all list-type commands
so AI agents pay fewer tokens for the same information. A single `trace` call
that previously returned 5-10KB of verbose JSON now returns ~2.5-5KB of
compact single-char-key JSON. Target: 5 structural queries cost <5k tokens
total (down from 30-80k).

### Added (issue #17)

- **`--format compact`** — New 5th output format alongside `json`/`markdown`/
`ai`/`sarif`. Implemented in `scripts/formatters/compact.py`:
- Omits null/empty fields (saves ~15% on average).
- Abbreviates node types: `function→fn`, `class→cls`, `file→f`, `module→m`,
`route→r`, `type→t`, `interface→i`.
- Abbreviates edge types: `CALLS→C`, `IMPORTS→I`, `DEFINES→D`, `INHERITS→H`,
`IMPLEMENTS→M`, `USES_TYPE→U`.
- Uses single-char keys: `name→n`, `file→f`, `line→l`, `type→t`, `status→s`,
`confidence→c`, etc. (full map in `formatters/compact.py:FIELD_KEY_ABBR`).
- Strips the workspace prefix from absolute paths.
- Output is still valid JSON — MCP clients parse it directly.

- **`graph-schema` command + `codelens_graph_schema` MCP tool** — Returns
node + edge counts, node-type distribution, edge-type distribution, and
index count in one cheap call. Example compact output:
`{"s":"ok","n":31,"e":97,"nts":{"function":30,"class":1},"ets":{"CALLS":97},"ix":6}`.
The cheapest way for an agent to understand the graph shape before issuing
structural queries.

- **`--limit N` / `--offset N` pagination** on `list`, `search`, `trace`,
`symbols`, `outline`. Default `--limit 20`. All paginated commands now
return `total_count`, `count`, `offset`, `limit`, `has_more` fields. The
existing `--top N` flag is preserved as an alias for `--limit N --offset 0`.

- **`format` parameter on every MCP tool** — All MCP tools now accept a
`format` parameter with the enum `[json, markdown, ai, sarif, compact]`.
Default remains `ai` (normalized schema). Pass `format: "compact"` for
token-efficient responses.

- **`tests/test_compact_format.py`** — 28 test cases covering compact
formatter rules, pagination behavior, graph-schema command, MCP tool
advertisement, and token-savings assertions.

### Changed (issue #17)

- **`scripts/codelens.py`** — Global `--format` flag (and per-subparser flag)
now accept `compact` as a 5th choice. Pre-parse loop updated to recognize
`compact` before subcommand dispatch.
- **`scripts/formatters/__init__.py`** — `format_output()` now dispatches to
`formatters.compact.format_compact` when `format_type == "compact"`.
- **`scripts/commands/search.py`** — Adds `--limit`/`--offset`, paginates
the `matches` list, adds `total_count`/`count`/`offset`/`limit`/`has_more`
fields.
- **`scripts/commands/list.py`** — Default `--limit` lowered from 200 to 20
(per issue #17 spec); adds `total_count` field alongside the existing
`total`. `--limit 0` means unlimited (preserves backward compat).
- **`scripts/commands/trace.py`** — Adds `--limit`/`--offset`, paginates
`chains.up` and `chains.down`, adds `total_count` field.
- **`scripts/commands/symbols.py`** — Adds `--limit`/`--offset`, paginates
`results`, adds `total_count`/`has_more` fields.
- **`scripts/commands/outline.py`** — Adds `--limit`/`--offset`, paginates
`outlines`, adds `total_count`/`has_more` fields.
- **`scripts/mcp_server.py`** — Adds `graph-schema` to `_TOOL_DEFINITIONS`.
Adds `_inject_format_enum()` helper that injects the shared `format`
property into every tool's inputSchema. `_execute_command` now respects
`arguments["format"]` — when set to `"compact"`, returns the compacted
dict via `formatters.compact.compact_dict` instead of the AI-normalized
schema.

### Non-Breaking (issue #17)

- Existing `--format json/ai/markdown/sarif` outputs are unchanged.
- Existing `--top N` flag still works (alias for `--limit N --offset 0`).
- Existing `list --limit 200` (the old default) still works — only the
default value changed from 200 to 20. Pass `--limit 200` explicitly to
restore the old behavior, or `--limit 0` for unlimited.
- All 56 existing CLI commands continue to work unchanged.
- 28 new tests pass; 4 pre-existing `test_hybrid_engine.py` failures
(confidence-field assertions) are unchanged — NOT caused by this change.

### Migration Notes for Engine Authors

The compact formatter is purely a presentation-layer concern. Engines do
not need to know about it — the formatter reads the engine's existing
output dict and produces a compacted JSON string. To verify your engine's
output compacts well, run:

```bash
$CLI <your-command> --format compact | python3 -m json.tool
```

If a field you depend on disappears in compact output, it's because the
value was null/empty (the formatter drops these). Either populate the
field with a meaningful default, or accept that null fields are noise.

---

### Graph Data Model (issue #8)

Replaces the ad-hoc flat-registry graph traversal with a true node + edge graph
Expand Down
40 changes: 28 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,13 @@

> **Before an AI writes a new class/id/function, CodeLens must be checked. This is not optional.**

CodeLens is an AI-native code intelligence platform that gives AI agents **full visibility** into a codebase before they write any code. It prevents collision, overwrite of existing logic, security vulnerabilities, and dead code through 56 CLI commands, an MCP server with 54 tools (49 static + 5 dynamic), AST-based taint analysis, live CVE/OSV scanning, a plugin system with OWASP Top 10 + Compliance rule packs, and a true graph data model (nodes + edges) for structural code queries.
CodeLens is an AI-native code intelligence platform that gives AI agents **full visibility** into a codebase before they write any code. It prevents collision, overwrite of existing logic, security vulnerabilities, and dead code through 57 CLI commands, an MCP server with 55 tools (50 static + 5 dynamic), AST-based taint analysis, live CVE/OSV scanning, a plugin system with OWASP Top 10 + Compliance rule packs, a true graph data model (nodes + edges) for structural code queries, and token-efficient `--format compact` output for high-volume agent workflows (issue #17).

## Features

- **56 CLI Commands** — From basic scan/query to AST taint analysis, CVE scanning, plugin management, auto-fix, dashboards, and CI/CD quality gates
- **MCP Server (54 Tools)** — Native AI agent integration via Model Context Protocol (JSON-RPC over stdio), 49 statically-defined tools + 5 dynamically discovered
- **57 CLI Commands** — From basic scan/query to AST taint analysis, CVE scanning, plugin management, auto-fix, dashboards, CI/CD quality gates, and `graph-schema` for cheap graph-shape introspection
- **MCP Server (55 Tools)** — Native AI agent integration via Model Context Protocol (JSON-RPC over stdio), 50 statically-defined tools + 5 dynamically discovered, every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`)
- **Token-Efficient Compact Output (v8.2, issue #17)** — `--format compact` produces single-char-key JSON with abbreviated types, omitted null fields, and relative paths — ~50% smaller than `json` on real trace output. Combined with `--limit`/`--offset` pagination, 5 structural queries now cost <5k tokens (down from 30-80k)
- **AST Taint Engine** — Tree-sitter based taint analysis with return-value propagation, scope hierarchy, and branch condition refinement
- **Live CVE/OSV Scanning** — Real-time vulnerability data from OSV.dev API with SQLite cache, 9 ecosystems (PyPI, npm, crates.io, Go, Maven, NuGet, RubyGems, Pub, Hex)
- **Cross-File Call Graph** — Workspace-wide call graph with import resolution and bidirectional taint propagation
Expand All @@ -21,7 +22,7 @@ CodeLens is an AI-native code intelligence platform that gives AI agents **full
- **Framework Auto-Detection** — React/Next.js, Vue, Svelte, Tailwind CSS, Express, Fastify, Koa, Hono, Django, Flask, FastAPI, Tauri, and more
- **Incremental Scanning** — Only re-parse changed files for speed, with SQLite persistent registry storage
- **Workspace Auto-Detect** — No need to specify workspace path if you're already in the project
- **AI-Optimized Output** — `--format ai` and `--lite` flags for token-efficient AI agent consumption
- **AI-Optimized Output** — `--format ai` (normalized schema) and `--format compact` (token-efficient single-char keys) flags for AI agent consumption
- **Auto-Fix Engine** — Confidence-scored auto-fixes with dry-run-by-default safety
- **HTML Dashboard** — Generate visual dashboards with historical trend tracking
- **Hybrid LSP Engine** — Optional LSP-enhanced deep analysis (`--deep` flag) when language servers are available
Expand Down Expand Up @@ -101,14 +102,15 @@ python3 scripts/codelens.py query "myFunction" --lite

| Command | Description |
|---------|-------------|
| `search "pattern" [workspace] [--type] [--context]` | Regex search across workspace |
| `symbols "name" [workspace] [--fuzzy]` | Search symbol in registry |
| `trace "name" [workspace] [--direction up\|down\|both] [--depth N]` | Deep call chain tracing |
| `search "pattern" [workspace] [--type] [--context] [--limit N] [--offset N]` | Regex search across workspace (paginated, default limit=20) |
| `symbols "name" [workspace] [--fuzzy] [--limit N] [--offset N]` | Search symbol in registry (paginated) |
| `trace "name" [workspace] [--direction up\|down\|both] [--depth N] [--limit N] [--offset N]` | Deep call chain tracing (paginated) |
| `context "name" [workspace]` | Rich symbol context (definition, callers, callees) |
| `outline [workspace] [--file] [--all]` | File structure outline |
| `outline [workspace] [--file] [--all] [--limit N] [--offset N]` | File structure outline (paginated) |
| `missing-refs [workspace]` | Detect CSS/HTML mismatches |
| `dependents "file" [workspace]` | Module-level import tracking |
| `list [workspace] [--domain] [--filter]` | List entries with filter (dead, collision, duplicate, etc.) |
| `list [workspace] [--domain] [--filter] [--limit N] [--offset N]` | List entries with filter (paginated, default limit=20) |
| `graph-schema [workspace]` | Return graph shape: node/edge counts, type distribution, indexes (issue #17) |
| `ask "question"` | Ask a question in natural language (auto-dispatches to relevant commands) |
| `summary [workspace] [--focus ...] [--detail ...]` | Auto-summary with prioritized findings (anti-overload) |

Expand Down Expand Up @@ -136,6 +138,7 @@ python3 scripts/codelens.py query "myFunction" --lite
| `state-map [workspace]` | Track global state management |
| `diff [workspace]` | Compare registry snapshots |
| `circular [workspace]` | Detect circular dependencies |
| `graph-schema [workspace]` | Cheap graph-shape introspection: node/edge counts, type distribution, indexes (issue #17) |
| `handbook [workspace]` | Generate project handbook for AI agents |
| `dashboard [workspace]` | Generate HTML visualization dashboard |
| `history [workspace]` | Show historical trend data and charts |
Expand Down Expand Up @@ -275,8 +278,8 @@ codelens/
│ ├── plugin_system.py # Plugin system & marketplace
│ ├── pre_commit_hook.py # Git pre-commit hook integration
│ ├── utils.py # Shared utilities (version, helpers)
│ ├── commands/ # One file per CLI command (auto-registered)
│ ├── formatters/ # Output formatters (markdown, sarif)
│ ├── commands/ # One file per CLI command (auto-registered, 57 commands incl. graph-schema)
│ ├── formatters/ # Output formatters (markdown, sarif, compact)
│ ├── parsers/ # Tree-sitter + fallback parsers
│ │ ├── html_parser.py, css_parser.py, js_frontend_parser.py, js_backend_parser.py
│ │ ├── rust_parser.py, python_parser.py, tsx_parser.py, ts_backend_parser.py
Expand Down Expand Up @@ -323,13 +326,26 @@ CodeLens is designed to be used by AI coding agents. The full integration guide

### MCP Server Integration

CodeLens ships with a native MCP server (54 tools) for direct AI agent integration:
CodeLens ships with a native MCP server (55 tools) for direct AI agent integration:

```bash
# Start MCP server (JSON-RPC over stdio)
python3 scripts/codelens.py serve
```

Every MCP tool accepts a `format` parameter with the enum `[json, markdown, ai, sarif, compact]`.
For high-volume agent workflows, pass `format: "compact"` to cut token usage ~50%. Example:

```json
// tools/call with format=compact
{"name": "codelens_graph_schema", "arguments": {"workspace": "/path/to/proj", "format": "compact"}}
// → {"s":"ok","n":1234,"e":5678,"nts":{"function":1000,"class":234},"ets":{"CALLS":5678},"ix":6}
```

The new `codelens_graph_schema` tool (issue #17) returns the graph shape in one cheap call —
use it first to decide whether structural queries (callers/callees/blast-radius) will return
meaningful results before paying tokens for them.

See `mcp_config.json` for Claude Desktop, Cursor, VS Code Copilot, Continue.dev, and Cline configuration templates.

### Guard Hooks for AI Agents
Expand Down
32 changes: 19 additions & 13 deletions SKILL-QUICK.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,20 @@
# CodeLens v8.1.0 — Quick Reference
# CodeLens v8.2 — Quick Reference

**MUST activate before writing/editing/deleting any class, id, or function.**

> Read THIS FILE FIRST. All commands: auto-detect workspace, auto-setup, smart `--top 20`, `--lite`, `--max-tokens N`, `--format ai`.
> Read THIS FILE FIRST. All commands: auto-detect workspace, auto-setup, smart `--top 20`, `--lite`, `--max-tokens N`, `--format ai`, `--format compact` (issue #17).

## Zero-Config Usage

```bash
CLI="python3 /path/to/codelens/scripts/codelens.py"
export CODELENS_AI_MODE=1 # Optional: --format ai becomes default
$CLI query "myFunction" --lite # → {found, action}. Auto-init+scan if needed.
$CLI smell # → Auto --top 20, sorted by severity
$CLI complexity --top 5 --lite # → Top 5 most complex, minimal output
$CLI query "myFunction" --lite # → {found, action}
$CLI smell # → Auto --top 20, sorted by severity
$CLI complexity --top 5 --lite # → Top 5 most complex, minimal output
$CLI trace "main" --direction down --format compact # → token-efficient single-char keys (issue #17)
$CLI graph-schema # → graph shape: nodes/edges/types in ~50 bytes (issue #17)
$CLI list --limit 5 --offset 10 --format compact # → paginated + compact
```

**Auto-setup** caps at 3000 files to prevent timeout. For full analysis: `$CLI scan` manually.
Expand All @@ -24,8 +27,10 @@ $CLI complexity --top 5 --lite # → Top 5 most complex, minimal output
| `--lite` | Command-specific minimal output (see table below) |
| `--max-tokens N` | Auto-truncate to fit ~N tokens |
| `--format ai` | Normalized: `{stats, items[], truncated, recommendations}` |
| `--deep` | Enable LSP-enhanced deep analysis (requires language server; check with `lsp-status`) |
| `--format compact` | Token-efficient: single-char keys + abbreviated types (issue #17). ~50% smaller than `json`. Best for high-volume MCP tool calls |
| `--format sarif` | SARIF v2.1.0 output for GitHub Advanced Security / VS Code |
| `--limit N` / `--offset N` | Pagination on list-type commands (`list`, `search`, `trace`, `symbols`, `outline`). Default limit=20 (issue #17). `--top N` is an alias for `--limit N --offset 0` |
| `--deep` | Enable LSP-enhanced deep analysis (requires language server; check with `lsp-status`) |
| `--db-path PATH` | Custom SQLite database path (default: `.codelens/codelens.db`) |

### Lite Mode Per Command
Expand Down Expand Up @@ -113,10 +118,10 @@ $CLI complexity --top 5 --lite # → Top 5 most complex, minimal output
`query "name" [--domain ...] [--fuzzy]` · `impact "name" [--action modify|delete]` · `refactor-safe "name" [--action rename|move]` · `guard (--pre|--post) --file PATH` · `check [--severity ...] [--max-findings N]`

### Navigation (10)
`summary [--focus security|quality|architecture|all] [--detail minimal|standard|full]` · `context "name"` · `trace "name" [--direction up|down|both]` · `search "pattern"` · `symbols "name" [--fuzzy]` · `outline [--file path]` · `dependents "file"` · `list [--filter ...]` · `ask "question"` · `diff`
`summary [--focus security|quality|architecture|all] [--detail minimal|standard|full]` · `context "name"` · `trace "name" [--direction up|down|both] [--limit N] [--offset N]` · `search "pattern" [--limit N] [--offset N]` · `symbols "name" [--fuzzy] [--limit N] [--offset N]` · `outline [--file path] [--limit N] [--offset N]` · `dependents "file"` · `list [--filter ...] [--limit N] [--offset N]` · `ask "question"` · `diff`

### Architecture (8)
`entrypoints` · `api-map` · `state-map` · `detect` · `handbook` · `diff` · `dashboard` · `history`
### Architecture (9)
`entrypoints` · `api-map` · `state-map` · `detect` · `handbook` · `diff` · `dashboard` · `history` · `graph-schema`

### Security (5)
`secrets [--severity ...]` · `taint` (AST-based) · `dataflow [--source ...] [--sink ...]` (cross-file) · `vuln-scan` (OSV.dev + native audit) · `env-check [--var NAME]`
Expand All @@ -136,19 +141,20 @@ $CLI complexity --top 5 --lite # → Top 5 most complex, minimal output
### Tooling (1)
`plugin <install|list|search|update|info|validate>`

**Total: 56 commands** (verified via `commands/__init__.py` auto-registration)
**Total: 57 commands** (56 + `graph-schema` added in v8.2 issue #17; verified via `commands/__init__.py` auto-registration)

## MCP Server (54 Tools)
## MCP Server (55 Tools)

Start the MCP server for AI agent integration:

```bash
python3 scripts/codelens.py serve
```

Exposes 54 tools as `codelens_<command>` (e.g., `codelens_query`, `codelens_taint`):
- 49 statically-defined tools (full JSON schemas in `mcp_server.py`)
Exposes 55 tools as `codelens_<command>` (e.g., `codelens_query`, `codelens_taint`, `codelens_graph_schema`):
- 50 statically-defined tools (full JSON schemas in `mcp_server.py`) including the new `codelens_graph_schema` (issue #17)
- 5 dynamically-discovered tools (`benchmark`, `dashboard`, `history`, `lsp-status`, `migrate`)
- Every tool accepts a `format` parameter (`json`/`markdown`/`ai`/`sarif`/`compact`). Use `format: "compact"` for token-efficient responses (~50% smaller than `json`).
- `watch` and `serve` itself are excluded (long-running)

See `mcp_config.json` for Claude Desktop, Cursor, VS Code Copilot, Continue.dev, and Cline configuration templates.
Expand Down
Loading
Loading