Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,31 @@ Notable changes, newest first. The format follows
[`docs/roadmap.md`](docs/roadmap.md): integrations marked 🟡 there are first
drafts even when they appear in a release.

## [Unreleased]

### Added

- **`agentsec scan` says what the repository is before what is wrong with it.**
The runtime framework fingerprint is composed into the `project` plane as
`project.fingerprint`, so the CLI, the dashboard page and
`agentsec://project/risks` read one classification: `confirmed`, `likely`,
`configuration_only`, `not_detected` or `unsupported`, with the framework and
its entrypoints named. Runtime agents and coding-agent configuration are
carried as separate lists at every hop — a `.mcp.json` cannot become a claimed
runtime agent by way of a template.
- The classification is reported **before** `agentsec init`, since whether there
is an agent in a checkout does not depend on whether anyone wrote a manifest.

### Changed

- `PUBLISH_SCHEMA_VERSION` is `1.4.0`. `project.fingerprint` is a new optional
key inside a plane that already existed; no plane was added or merged, and no
published shape changed.
- The `project` plane is now projected field by field like every other published
document rather than passed through whole. Its new content is derived from
reading arbitrary repository files, so "the producer promises no source text"
stopped being a guarantee that could live in one place.

## [0.2.0] — 2026-08-06

The release that gives AgentSec a first step. In 0.1.0 the entry point was
Expand Down
18 changes: 17 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,23 @@ agentsec init # write .agentsec/project.yaml, then read it and commit it
agentsec scan # find the attack surface, and rank what it finds
```

`scan` reads what this repository gives an AI agent — project instructions,
`scan` answers two questions in order. First, whether this repository implements
an AI agent at all — read from dependencies, imports and builder calls, without
importing or running any of it:

```
AI agent confirmed
runtime agent code in this repository
langgraph (python) src/agent/graph.py
coding-agent config: claude_code, mcp
```

A repository holding only a `CLAUDE.md` and a `.mcp.json` reports
`configuration only`: a coding agent works *on* this checkout, which is not the
same as this checkout *being* an agent. An ordinary repository reports
`not detected` — an absence of evidence, never a pass.

Then it reads what this repository gives an AI agent — project instructions,
subagent definitions, skills, hooks, pre-approved tool grants, MCP servers and
memory stores — and applies the deterministic rules in
[`inspect/`](src/agentsec/inspect/). Each risk says whether anything here can
Expand Down
24 changes: 16 additions & 8 deletions docs/feature-matrix.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ Changes here need an ADR. These are load-bearing.
| Verdict precedence | `evaluation/axes.py` | `error > detection_gap > prevention_gap > evidence_gap > response_gap > secure`. Frozen. |
| Selected-project resolution + manifest | `project/` | Which repository. A process-boundary decision, never a tool argument ([ADR 0003](adr/0003-constrained-mcp-tools.md)). |
| Surface discovery | `project/discovery.py` | Agents, skills, hooks, settings, instructions, MCP servers, tool grants, memory. Inventory only. |
| Runtime framework fingerprint | `project/fingerprint.py` | Distinguishes application runtime agents from Claude Code, Codex, Gemini CLI, Cursor and MCP development configuration without importing repository code. |
| Runtime framework fingerprint | `project/fingerprint.py` | Distinguishes application runtime agents from Claude Code, Codex, Gemini CLI, Cursor and MCP development configuration without importing repository code. Composed into the `project` plane, so `scan`, the dashboard and the MCP resource read one classification. |
| Repository risk plane | `inspect/` | Turns the inventory into ranked risks, and each risk into `verified` / `verifiable` / `not_verifiable` ([ADR 0009](adr/0009-repository-first-golden-path.md)). |
| `config-surface:` correlation | `scenario/surface_tags.py` | The one bridge from a static surface to a runnable scenario. Shared by the risk and posture planes so they cannot disagree. |
| Replay executor + fixture corpus | `execution/replay.py` | Deterministic. What CI relies on. |
Expand All @@ -83,7 +83,7 @@ well-meaning change:
| `repo_risk` | What in this repository is worth testing? | `inspected` / severity / `verified`–`verifiable`–`not_verifiable` |
| `skill_assurance` | Do this repository's skills behave? | `pass` / `fail` / `not_tested` |
| `static_posture` | What did a third-party scanner flag? | `ingested` / `covered`–`not_tested`–`n/a` |
| `project` | Which repository is this? | `declared` / `not_initialised` / `invalid` |
| `project` | Which repository is this, and is it an agent? | `declared` / `not_initialised` / `invalid`, and `confirmed`–`likely`–`configuration_only`–`not_detected`–`unsupported` |

Each status enum is spelled differently on purpose. A single number averaging
them would answer none of the four questions, and the fastest way to build one
Expand Down Expand Up @@ -158,11 +158,12 @@ is refusing to widen the surface while the middle of it is unproven.
Stated here rather than left for a reader to discover, because a matrix that
only lists what works is marketing.

1. **The framework fingerprint is not yet composed into `agentsec scan`.** The
deterministic detector exists and distinguishes `confirmed`, `likely`,
`configuration_only`, `not_detected` and `unsupported`, but the next DTO/CLI PR must make
that classification visible on the golden path without merging it into a
Purple verdict.
1. **The fingerprint classifies six framework families, not every framework.**
LangChain/LangGraph, OpenAI Agents SDK, AutoGen, Semantic Kernel, CrewAI and
framework-neutral Python/Node tool calling. Anything else in a repository that
is plainly an agent reports `not_detected`, which is why that word means
"no evidence" and never "no agent" — and why an unparsed file is reported as
`unsupported` rather than counted as absence.
2. **No scenario covers the tool-grant or settings surface.** `ASI-TOOL-BROAD-GRANT`
and `ASI-TOOL-PERMISSION-BYPASS` fire — the second at `critical` — and both
report `not_verifiable`, because no `AGT-CONFIG-*` scenario is tagged at
Expand All @@ -176,7 +177,14 @@ only lists what works is marketing.
`demo-agent-fixture` (environment `local`) correctly refuses with exit 2
rather than selecting nothing and reporting success. Until fixtures exist,
`--verify` needs a real staging target.
5. **No end-to-end run against a live agent has happened.** This remains the
5. **The catalogue does not travel with the CLI.** `Settings.scenarios_dir` is
`<workspace>/scenarios`, so in a repository that is not a checkout of
AgentSec there are no scenarios to triage against and *every* risk resolves
to `not_verifiable` — correct, and misleading at a glance now that the screen
above it can say `confirmed langgraph`. Bundling the reviewed catalogue as
package data is the fix; which catalogue version an installed CLI should
trust is the question that makes it more than a one-line change.
6. **No end-to-end run against a live agent has happened.** This remains the
single most valuable open item, exactly as #32 says.

---
Expand Down
5 changes: 1 addition & 4 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ agentsec init → agentsec scan → agentsec scan --verify -t <id> → dashboard
| CLI with meaningful exit codes | ✅ | `0` clean, `1` blocking, `2` could not tell |
| Selected-project manifest and discovery | ✅ | `.agentsec/project.yaml`; relative locations only, traversal and symlink escape refused |
| Runtime framework fingerprint engine | ✅ | deterministic, read-only detection for LangGraph/LangChain, OpenAI Agents SDK, AutoGen, Semantic Kernel, CrewAI and framework-neutral tool calling; development-agent config stays separate |
| Fingerprint composed into `scan`, dashboard and MCP resource | ✅ | `project.fingerprint`; reported even before `agentsec init`, and `not_detected` never renders as a pass ([#32](https://github.com/trionnemesis/AgentSec/issues/32)) |
| Tool-grant and memory surfaces | ✅ | one entry per permission rule; `.claude/memory` declared like any other surface ([#32](https://github.com/trionnemesis/AgentSec/issues/32)) |
| **Repository risk plane** (`agentsec scan`) | ✅ | 10 deterministic rules across agents, skills, hooks, tool grants, MCP and memory ([ADR 0009](adr/0009-repository-first-golden-path.md)) |
| **Risk → scenario triage** | ✅ | `verified` / `verifiable` / `not_verifiable`; `scan --verify` drains the queue |
Expand All @@ -44,10 +45,6 @@ agentsec init → agentsec scan → agentsec scan --verify -t <id> → dashboard

### Core — open

- [ ] **Compose the framework fingerprint into the inspection DTO and CLI.**
The detector is built and tested, but `agentsec scan` does not expose it
yet; that integration must not turn coding-agent configuration into a
runtime-agent claim or widen `PurpleVerdict`.
- [ ] **Run against one real staging agent end to end**, and fix what that
reveals. Still the single most valuable open item.
- [ ] **A scenario covering the tool-grant / settings surface.**
Expand Down
72 changes: 70 additions & 2 deletions schemas/project-dashboard.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@
"$defs": {
"project": {
"type": "object",
"description": "Which repository this is. `status` is not decoration: a dashboard for a repository that was never initialised must say so rather than render as an anonymous but healthy project.",
"description": "Which repository this is, what agent surfaces it declares, and whether it implements an AI agent at all (`fingerprint`). `status` is not decoration: a dashboard for a repository that was never initialised must say so rather than render as an anonymous but healthy project. The fingerprint is reported even then — whether there is an agent here does not depend on whether someone ran `agentsec init`.",
"required": ["status"],
"additionalProperties": false,
"properties": {
Expand All @@ -56,11 +56,79 @@
"type": "object",
"description": "Counts only. The inventory itself is not part of this document.",
"additionalProperties": { "type": "integer", "minimum": 0 }
}
},
"fingerprint": { "$ref": "#/$defs/fingerprint" }
},
"if": { "properties": { "status": { "const": "declared" } } },
"then": { "required": ["status", "project_id", "name"] }
},
"fingerprint": {
"type": "object",
"description": "Whether this checkout implements an AI agent, and in what (issue #32). Read statically: no repository code is imported or executed, and every path is relative to the selected root. Inside `project` rather than beside it because it qualifies the identity that plane already reports, and because a sixth top-level key would have read as a sixth kind of conclusion. It is not one — nothing here has run, so `not_detected` is an absence of evidence and never a pass.",
"required": ["agent_presence", "confidence"],
"additionalProperties": false,
"properties": {
"agent_presence": {
"enum": ["confirmed", "likely", "configuration_only", "not_detected", "unsupported"],
"description": "confirmed: framework code with a builder or entrypoint. likely: dependencies or tool-calling shape without a runtime entrypoint. configuration_only: CLAUDE.md, skills, hooks or .mcp.json and no runtime agent — a coding agent works *on* this repository, which is not the same as this repository *being* one. not_detected: no evidence either way. unsupported: something was found and could not be classified, which is stated rather than rounded down to absence."
},
"confidence": { "enum": ["high", "medium", "none"] },
"schema_version": { "type": "string", "pattern": "^[0-9]+\\.[0-9]+\\.[0-9]+$" },
"runtime_agents": {
"type": "array",
"description": "Application agents implemented in this repository. Never concatenated with development_agent_config: merging the two is exactly how a .mcp.json becomes a claimed runtime agent.",
"items": {
"type": "object",
"required": ["framework", "language", "confidence"],
"properties": {
"framework": { "type": "string", "maxLength": 60 },
"language": { "enum": ["python", "javascript", "typescript", "mixed", "unknown"] },
"confidence": { "enum": ["high", "medium"] },
"entrypoints": { "type": "array", "items": { "type": "string" } },
"evidence": {
"type": "array",
"description": "Bounded structural facts: a dependency name, an import path, a builder symbol, a relative file. Never source text — see project/fingerprint.py.",
"items": {
"type": "object",
"required": ["kind", "file", "value"],
"properties": {
"kind": {
"enum": ["dependency", "import", "builder_call", "runtime_config", "tool_calling"]
},
"file": { "type": "string" },
"value": { "type": "string", "maxLength": 500 }
}
}
}
}
}
},
"development_agent_config": {
"type": "array",
"description": "Coding-agent configuration found in the checkout. Present on its own, this is what makes a repository configuration_only.",
"items": {
"type": "object",
"required": ["platform"],
"properties": {
"platform": { "enum": ["claude_code", "codex", "gemini_cli", "cursor", "mcp"] },
"paths": { "type": "array", "items": { "type": "string" } }
}
}
},
"problems": {
"type": "array",
"description": "Candidates the detector could not read completely. An unparsed file is stated, never counted as absence of an agent.",
"items": {
"type": "object",
"properties": {
"path": { "type": "string" },
"kind": { "type": "string" },
"detail": { "type": "string" }
}
}
}
}
},
"repoRisk": {
"type": "object",
"description": "The repository risk plane (issue #32): this repository's own agent configuration — skills, agents, hooks, tool grants, MCP servers and memory stores — read statically by first-party rules and triaged against the scenario catalogue. Deliberately not verdict-shaped. A risk is a reason to run a scenario, not a result of having run one; nothing here has executed anything or given a detection control the chance to fire. The `verification` block is the whole point: it says whether a deterministic conclusion is available (`verified`), obtainable (`verifiable`), or out of reach (`not_verifiable`). The last is the common case and must never render as green.",
Expand Down
68 changes: 62 additions & 6 deletions src/agentsec/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -491,6 +491,23 @@ def init(
"verifiable": "runnable now",
"not_verifiable": "no scenario covers this",
}
#: What each classification means to someone who did not read the schema. Every
#: line says what was found; none of them says the repository is safe, because
#: this classifier has executed nothing.
_PRESENCE_LABEL = {
"confirmed": ("confirmed", typer.colors.CYAN),
"likely": ("likely", typer.colors.CYAN),
"configuration_only": ("configuration only", typer.colors.WHITE),
"not_detected": ("not detected", typer.colors.WHITE),
"unsupported": ("could not classify", typer.colors.YELLOW),
}
_PRESENCE_DETAIL = {
"confirmed": "runtime agent code in this repository",
"likely": "agent dependencies or tool calling, but no runtime entrypoint",
"configuration_only": "coding-agent configuration only — no runtime agent code",
"not_detected": "no agent framework, tool calling or agent configuration found",
"unsupported": "something was found here that these rules cannot classify",
}


@app.command()
Expand All @@ -512,10 +529,16 @@ def scan(
) -> None:
"""Inspect the selected repository for agent attack surface, and rank what it finds.

The engineer's entry point. Reads this repository's skills, agents, hooks,
tool grants, MCP servers and memory stores, applies the static rules in
`agentsec.inspect`, and reports each risk alongside whether anything here
can turn it into a deterministic conclusion.
The engineer's entry point. Answers two questions in order: whether this
repository implements an AI agent — and in what framework — then what in its
skills, agents, hooks, tool grants, MCP servers and memory stores is worth
testing, with whether anything here can turn each risk into a deterministic
conclusion.

The first answer never asserts the second. A repository holding only a
`CLAUDE.md` and a `.mcp.json` reports `configuration only`: a coding agent
works *on* this checkout, which is not the same as this checkout *being* an
agent. `not detected` is likewise an absence of evidence, not a pass.

Static only, by design. A risk is a reason to run a scenario, never the
result of having run one — so `--verify` is the second half: it selects the
Expand Down Expand Up @@ -567,18 +590,51 @@ def scan(
_fail(exc)


def _print_agent(project: dict) -> None:
"""What this repository *is*, before what is wrong with it.

First, because it is the question an engineer opening an unfamiliar
repository actually has, and because every risk below it means something
different depending on the answer. Printed even when the risk plane could
not run: whether there is an agent here does not depend on whether anyone
has run `agentsec init`.
"""
fingerprint = project.get("fingerprint") or {}
presence = fingerprint.get("agent_presence", "unsupported")
label, colour = _PRESENCE_LABEL.get(presence, (presence, typer.colors.YELLOW))

typer.secho(f"\n AI agent {label}", fg=colour, bold=True)
typer.echo(f" {_PRESENCE_DETAIL.get(presence, '')}")

for agent in fingerprint.get("runtime_agents") or []:
where = ", ".join(agent.get("entrypoints") or []) or "no entrypoint found"
typer.echo(f" {agent['framework']} ({agent['language']}) {where}")
platforms = ", ".join(
config["platform"] for config in fingerprint.get("development_agent_config") or []
)
if platforms:
typer.echo(f" coding-agent config: {platforms}")
if fingerprint.get("problems"):
typer.secho(
f" {len(fingerprint['problems'])} file(s) could not be parsed; "
"absence of a framework here is not proof there is none",
fg=typer.colors.YELLOW,
)


def _print_scan(document: dict) -> None:
project, plane = document["project"], document["repo_risk"]
_print_agent(project)

if plane.get("status") != "inspected":
typer.secho(
f"not inspected [{plane.get('reason', 'unknown')}]: {plane.get('detail', '')}",
f"\nnot inspected [{plane.get('reason', 'unknown')}]: {plane.get('detail', '')}",
fg=typer.colors.YELLOW,
)
return

typer.echo(
f"\nproject {project.get('project_id', '?')} "
f"\n project {project.get('project_id', '?')} "
f"surfaces {sum((project.get('surfaces') or {}).values())} "
f"risks {plane['counts']['total']}\n"
)
Expand Down
Loading
Loading