Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 63 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,15 +23,31 @@ permissions:

jobs:
test:
name: test (node ${{ matrix.node-version }})
name: test (${{ matrix.os }}, node ${{ matrix.node-version }})
# The schedule trigger exists only to re-run `audit` against a moving
# advisory DB; the code is unchanged between crons, so skip the rest.
if: github.event_name != 'schedule'
runs-on: ubuntu-latest
runs-on: ${{ matrix.os }}
# Node versions fan out on Linux only; Windows and macOS get one pinned
# version each via `include`. A full 3x3 cross-product would be nine jobs to
# re-answer a question the Linux column already answers — what these two add
# is the PLATFORM, not another Node.
#
# They are here because the code has real per-platform branches (win32 paths
# in skills/install.mjs, path separators through the resolver, git behaviour
# differences) and the docs target Windows, yet every one of the 477 tests
# had only ever run on Linux. macOS is the maintainer's own platform and is
# exercised locally; Windows genuinely was not covered anywhere.
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest]
node-version: [20, 22, 24]
include:
- os: windows-latest
node-version: 22
- os: macos-latest
node-version: 22
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
Expand All @@ -45,6 +61,14 @@ jobs:
- name: Install dependencies
run: npm ci

# checkJs over ~250KB of hand-written JS — the only static analysis that
# reads this codebase, since none of it is TypeScript. Deliberately runs on
# every OS: the two errors it caught when introduced were a stale JSDoc type
# and an unfollowable dynamic import, and path-shaped mistakes are exactly
# the kind that differ per platform.
- name: Typecheck
run: npm run typecheck

# Black-box suite — drives the real CLI as a subprocess against temp repos.
# Runs via npm so CI uses the same glob set as `npm test` locally
# (test/*.test.mjs alone silently skipped the test/vue-sfc/ suite).
Expand Down Expand Up @@ -73,6 +97,14 @@ jobs:
# So: pack, install from the tarball, and drive the real binary. Exit status is
# NOT sufficient evidence here — exit 0 with empty stdout was the bug's exact
# signature, so every step below asserts on OUTPUT.
#
# Deliberately Linux-only, unlike the `test` job above. Every step here is a
# bash script with `set -euo pipefail`, absolute /tmp paths and $GITHUB_ENV
# export syntax; on windows-latest the default shell is PowerShell, so porting
# this means `shell: bash` plus rewriting the paths, and a half-ported version
# that silently skips a step is worse than an honest gap. The Windows install
# path is therefore NOT covered — recorded here rather than left to be inferred
# from the matrix.
install-smoke:
name: install smoke (node ${{ matrix.node-version }})
if: github.event_name != 'schedule'
Expand Down Expand Up @@ -165,6 +197,35 @@ jobs:
echo "generatedSha $BEFORE -> $AFTER"
test "$AFTER" != "$BEFORE" || { echo "::error::post-commit hook did not refresh the map — auto-refresh is dead"; exit 1; }

# Coverage floor, so a shipped file that nothing executes stays visible.
#
# Uses node --test's own coverage rather than c8: the thresholds land natively
# from Node 22, and the near-zero-deps rule is easier to keep than to argue with.
# That is also why this is its own job pinned to one version — the flags do not
# exist on Node 20, which the test matrix still supports.
#
# Floors sit BELOW the measured 93.77% lines / 75.92% branches on purpose. They
# are a regression alarm, not a target: a gate set at the current number reddens
# on ordinary work and gets raised until someone stops reading it.
coverage:
name: Coverage floor
if: github.event_name != 'schedule'
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Setup Node
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 24

- name: Install dependencies
run: npm ci

- name: Run tests under coverage
run: npm run coverage

# Audit for high-severity vulnerabilities in the dependency tree. Deliberately
# its own job rather than a step in the matrix above: it depends only on the
# lockfile, so running it per Node version was three identical checks, and a
Expand Down
41 changes: 41 additions & 0 deletions CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Code of Conduct

## The standard

This project adopts the [Contributor Covenant v2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/).
Read it there — it is the authoritative text, and restating it here only creates a
copy to drift.

In short: be respectful, assume good faith, and keep discussion about the work.
Harassment, personal attacks, and demeaning comments are not welcome, in issues,
pull requests, commit messages, or anywhere else this project is discussed.

## Scope

Applies in all project spaces — issues, pull requests, discussions, commits — and
when representing the project publicly.

## Technical disagreement is not a violation

This repository argues with itself in writing. `ROADMAP.md` records items closed as
**refuted** with the measurements that refuted them, several of which contradict
what a maintainer previously believed. `CONTRIBUTING.md` says maintainers may decline
in-scope changes. Being told your patch is wrong, or that a number does not
reproduce, is the process working.

What is not on that list: making it personal, or about the person.

## Reporting

Report anything that crosses the line to **raymondchin.s@gmail.com**, the address
already on every commit in this repository. Reports are handled privately, and the
reporter's identity is not shared with the person reported.

If the report concerns the maintainer, GitHub's own
[reporting channels](https://docs.github.com/en/communities/maintaining-your-safety-on-github/reporting-abuse-or-spam)
exist for exactly that reason and are the right escalation.

## Enforcement

Responses are proportionate: a private correction, a warning, removal of a comment,
or a block. The maintainer decides, and will say which is being applied and why.
17 changes: 15 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,9 +120,22 @@ When you touch caching, building, or the schema:
## Submitting a PR

1. For anything non-trivial, open (or link) an issue describing the change first.
2. Branch, make the change, run `npm test` — all green on Node 20+.
2. Branch, make the change, then run all three gates — CI runs the same ones:

```bash
npm test # black-box suite, all green on Node 20+
npm run typecheck # checkJs over the .mjs sources (see jsconfig.json)
npm run coverage # suite again with the line/branch floor enforced
```

`typecheck` is intentionally non-strict — `jsconfig.json` explains what that
buys and what it gives up. `coverage` needs Node 22+ for the threshold flags;
`npm test` alone is fine on Node 20.
3. Tests are dependency-free black-box drivers over throwaway git repos (see
`test/helpers.mjs`). New behavior needs a test in that style.
`test/helpers.mjs`). New behavior needs a test in that style. Assert what the
output *should be*, not merely that two runs agree — `test/determinism.test.mjs`
passes unchanged with the hub comparator inverted, which is why
`test/ranking-quality.test.mjs` exists.
4. Keep the diff minimal and the output byte-identical for existing commands —
unless the change *is* the output (then call it out).
5. Fill in the PR checklist. Maintainers may decline in-scope-but-bloating
Expand Down
8 changes: 8 additions & 0 deletions EVAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,14 @@ where naive grep returns a noisy superset (high recall, low precision) — and `
actually costs **more** tokens than `grep -l` because it returns the full blast radius
(exports + imports + dependents + related), not just the file list.

> **Reconciling this with `RESULTS.md`'s 99.2% blast-radius row.** Both numbers are real;
> they price different baselines. `RESULTS.md` scenario D compares against an agent that
> `cat`s every dependent file — agentmap wins by ~99%. This eval compares against `grep -l`,
> which returns a file list and nothing else — agentmap loses on tokens. The list is cheaper
> because it is *less* correct: at **59.9%** precision, roughly 4 of every 10 paths on it are
> not dependents, and the agent pays for them on the next turn when it opens them. Neither
> file is the whole picture on its own; quote them together.

### Per fixture

| Repo | commit | def n | agentmap top1/top3 | grep top1/top3 | deps n | agentmap recall/prec | grep recall/prec |
Expand Down
38 changes: 30 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,12 +133,17 @@ follows the chain and names the file that actually declares it.
- **The win scales with the work.** The 63% and 11% rows are the floor. A *trivial
single-file* lookup can cost **more** than `cat` + `grep` — taxonomy's file-import task hit
**−313%**, and it stays in the table.
- **The 98.3% combined figure is skewed** by the whole-repo row (150 K vs 1 K). Excluding it,
the per-task average is ~32× rather than 58×. Both are real; the headline captures the
common worst case (repo dump on session start).
- **The 98.3% headline is carried by its two biggest rows** — repo dump (150,281 → 1,127) and
blast radius (81,038 → 616). Drop the repo dump and it's **96.9%**; drop both and it's
**89.8%** here, **93.7%** pooled across all three repos, and **73.1%** on the smallest one.
All of those are real — they answer different questions. The headline is the common worst
case: an agent dumping the repo at session start.
- **`--relates` returns the full blast radius**, so it costs *more* than a bare `grep -l` file
list. Complete-and-correct over short-and-wrong — but it is a trade, stated in
[EVAL.md](./EVAL.md).
list. That's why the same command reads as 99.2% *saved* in the benchmark and *more
expensive* in the eval: the benchmark's baseline is an agent that `cat`s all 65 dependent
files, the eval's is a file list nobody reads. Against the list, agentmap trades tokens for
precision — 100% vs 59.9%, so ~4 in 10 files on the grep list don't belong.
Complete-and-correct over short-and-wrong, but it is a trade → [EVAL.md](./EVAL.md).
- **Numbers are context-token volume**, not answer quality or wall-clock.
- **Token counts are estimates** (`chars / 4`), applied identically to both sides.
- **TypeScript/JavaScript only** (+ Vue SFC) — see [Scope & limitations](#scope--limitations).
Expand All @@ -164,17 +169,20 @@ OS-event file watcher (FSEvents/inotify) with debounced auto-sync and an install
auto-configures eight agent CLIs. agentmap's honest edge over the multi-language graph tools is
narrower and sharper: **TS/JS resolution the others approximate, with a published accuracy eval.**

| | **agentmap** | Aider repo map | RepoMapper | Repomix | code2prompt |
| | **agentmap** | [Aider repo map](https://github.com/Aider-AI/aider) | [RepoMapper](https://github.com/nuptcode/repomapper) | [Repomix](https://github.com/yamadashy/repomix) | [code2prompt](https://github.com/mufeedvh/code2prompt) |
| --- | --- | --- | --- | --- | --- |
| **Ranking algorithm** | Personalized PageRank (file + symbol graphs) | PageRank (graph ranking) | Importance heuristics | None (file order) | None (file order) |
| **Languages** | TS/JS + Vue SFC (via ts-morph) | Many (tree-sitter) | Many (tree-sitter) | Language-agnostic (text) | Language-agnostic (text) |
| **Token-budget output** | Yes — `--map [--tokens N]` ranked digest | Yes (built into Aider's context) | Partial | Yes (size caps) | Yes (templates/caps) |
| **TS/JS resolution depth** | **Compiler-grade — `tsconfig` paths + `vite`/`webpack` alias + `#imports` + workspaces (ts-morph)** | Basename/regex heuristics | Basename/regex heuristics | N/A (text) | N/A (text) |
| **Retrieval-accuracy eval** | **Yes — published [`EVAL.md`](./EVAL.md) vs live ground truth** | No | No | No | No |
| **Agent-loop wiring** | Yes — post-commit auto-refresh + PreToolUse hook | In-process (Aider only) | No | No | No |
| **Agent-loop wiring** | Yes — post-commit auto-refresh + PreToolUse hook | In-process (Aider only) | No | MCP server (no auto-refresh, no nudge) | No |
| **Dependencies** | `ts-morph` only | Python + tree-sitter stack | Python + tree-sitter | Node | Rust binary |
| **Install** | `npx @raymondchins/agentmap` | `pip install aider-chat` | `pip install` | `npx`/global | `cargo`/binary |

<sub>Comparison as of <b>2026-07-27</b>, from each project's own docs. These are moving targets — if a
cell is out of date, that's a bug: <a href="https://github.com/raymondchins/agentmap/issues">open an issue</a>.</sub>

What that table is **not** claiming: agentmap is TS/JS-only (the others are multi-language),
and it's a **file-level import graph**, not a full call-site/reference resolver (see
[Scope & limitations](#scope--limitations)). The differentiators are narrow and honest:
Expand Down Expand Up @@ -353,7 +361,7 @@ skill/rule the agent may or may not consult). Honest matrix:
| Platform | Install | Enforcement | Known gaps |
|----------|---------|-------------|------------|
| **Claude Code** | `/plugin install agentmap@agentmap` (or `--install-hooks`) | **live hook** — `PreToolUse` nudge on `Grep` + Bash searchers | non-blocking (never denies grep); bare-symbol `Grep` nudge requires the #3 hook fix |
| **Gemini CLI** | `--install-skill --platform gemini` | **live hook** — `.gemini/settings.json` nudge | fires on the `AfterTool`/`systemMessage` path (the earlier `BeforeTool` + `additionalContext` combo was silently dropped — fixed in #4) |
| **Gemini CLI** | `--install-skill --platform gemini` | **live hook** — `.gemini/settings.json` nudge | fires on `BeforeTool` and emits a top-level `systemMessage`; Gemini parses and then **drops** `hookSpecificOutput.additionalContext` on `BeforeTool`, which is why the nudge used to vanish silently |
| **OpenCode** | `--install-skill --platform opencode` | **log-only** — `.opencode/plugins/agentmap-nudge.js` writes to the log, does not inject context | plugin can't steer the model; relies on the `AGENTS.md` block being read |
| **Cursor** | `--install-skill --platform cursor` + `.cursor/mcp.json` (below) | **MCP + docs** — `alwaysApply` rule + the MCP server | Cursor's own hooks aren't wired; the rule is advisory |
| **Codex CLI** | `--install-skill --platform codex` | **live gate** — `.codex/config.toml` PreToolUse hook | denies only high-confidence structural greps; allow-fallback for logs/pipes/non-TS-JS; `AGENTMAP_CODEX_GATE=0` bypasses; needs a trusted dir + Codex hooks-GA |
Expand Down Expand Up @@ -403,6 +411,10 @@ leaves the rest of your `AGENTS.md` / `GEMINI.md` intact.
| Codex/Gemini nudge never fires | Codex's gate is opt-in — set `[features] hooks = true` in `.codex/config.toml` (`AGENTMAP_CODEX_GATE=0` disables it). Gemini needs the `BeforeTool` hook that `--install-skill` writes. |
| Installed the wrong `agentmap` | This is **`@raymondchins/agentmap`** (npm scope) — not the unrelated unscoped `agentmap` packages. |
| Cursor MCP tools missing | `--mcp` doesn't auto-wire Cursor; add the copy-paste `.cursor/mcp.json` from the matrix above and restart Cursor. |
| Hook works in your shell, not in the agent | Almost always **nvm**. Your interactive shell sources `~/.nvm/nvm.sh`; the git hook and the agent's tool runner do not, so `node` isn't on their `PATH`. Point the hook at an absolute node (`which node`) or install a system-wide node. |
| `JavaScript heap out of memory` | Raise the ceiling — the parse peaks and there is no in-process warning that can fire in time (the process dies inside a single call, with heap use still at ~40% one sample earlier). Re-run as `NODE_OPTIONS=--max-old-space-size=8192 npx @raymondchins/agentmap`. Repo **size is not the axis**: measured, a 252-file Next.js app peaks at 683 MB while 4,000 dependency-free files peak at 756 MB, because the dependency `.d.ts` closure (~300 MB, ~1,800 extra program files on a 393-file app) dominates. A small repo with heavy `@types` can need more than a large plain one. |
| Skill file looks out of date | Each installed skill dir carries a `.agentmap_version`. `agentmap --doctor` compares it against the running version and flags the drift; `--install-skill` again overwrites it. |
| `0 files mapped` | agentmap indexes `git ls-files --cached --others --exclude-standard`, so uncommitted files *are* included but **`.gitignore`d ones are not** — a source tree matched by an ignore rule maps to nothing, as does a directory that is not a git repo at all. Confirm with `git ls-files --others --exclude-standard \| head`. |

---

Expand Down Expand Up @@ -752,6 +764,16 @@ top 10 ranked symbols (Aider-style):
0.015034 lib/errors.ts → ErrorCode (TypeAliasDeclaration)
```

`map.json` persists the top 80. Asking for more re-ranks from the cached map rather than
truncating, so `--symbols 200` really does return 200 where the repo has them. When a repo
has fewer ranked symbols than you asked for, the header says so and `--json` carries
`requested` / `shown` / `truncated`:

```
$ node agentmap.mjs --symbols 200
top 62 ranked symbols (Aider-style) — asked for 200, this repo only ranks 62:
```

### `--map [--tokens N] [--focus <path>]` — token-budgeted ranked digest

The token-budgeted digest (Aider's killer feature): a ranked, files-and-symbols summary
Expand Down
Loading