diff --git a/CHANGELOG.md b/CHANGELOG.md index 795f52e..aa064b2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,31 @@ Notable changes, newest first. The format follows [`docs/roadmap.md`](docs/roadmap.md): integrations marked ๐ŸŸก there are first drafts even when they appear in a release. +## [Unreleased] + +### Added + +- **`agentsec scan` says what the repository is before what is wrong with it.** + The runtime framework fingerprint is composed into the `project` plane as + `project.fingerprint`, so the CLI, the dashboard page and + `agentsec://project/risks` read one classification: `confirmed`, `likely`, + `configuration_only`, `not_detected` or `unsupported`, with the framework and + its entrypoints named. Runtime agents and coding-agent configuration are + carried as separate lists at every hop โ€” a `.mcp.json` cannot become a claimed + runtime agent by way of a template. +- The classification is reported **before** `agentsec init`, since whether there + is an agent in a checkout does not depend on whether anyone wrote a manifest. + +### Changed + +- `PUBLISH_SCHEMA_VERSION` is `1.4.0`. `project.fingerprint` is a new optional + key inside a plane that already existed; no plane was added or merged, and no + published shape changed. +- The `project` plane is now projected field by field like every other published + document rather than passed through whole. Its new content is derived from + reading arbitrary repository files, so "the producer promises no source text" + stopped being a guarantee that could live in one place. + ## [0.2.0] โ€” 2026-08-06 The release that gives AgentSec a first step. In 0.1.0 the entry point was diff --git a/README.md b/README.md index d5ffb0b..7bf1a29 100644 --- a/README.md +++ b/README.md @@ -116,7 +116,23 @@ agentsec init # write .agentsec/project.yaml, then read it and commit it agentsec scan # find the attack surface, and rank what it finds ``` -`scan` reads what this repository gives an AI agent โ€” project instructions, +`scan` answers two questions in order. First, whether this repository implements +an AI agent at all โ€” read from dependencies, imports and builder calls, without +importing or running any of it: + +``` + AI agent confirmed + runtime agent code in this repository + langgraph (python) src/agent/graph.py + coding-agent config: claude_code, mcp +``` + +A repository holding only a `CLAUDE.md` and a `.mcp.json` reports +`configuration only`: a coding agent works *on* this checkout, which is not the +same as this checkout *being* an agent. An ordinary repository reports +`not detected` โ€” an absence of evidence, never a pass. + +Then it reads what this repository gives an AI agent โ€” project instructions, subagent definitions, skills, hooks, pre-approved tool grants, MCP servers and memory stores โ€” and applies the deterministic rules in [`inspect/`](src/agentsec/inspect/). Each risk says whether anything here can diff --git a/docs/feature-matrix.md b/docs/feature-matrix.md index c8058e4..1c970b3 100644 --- a/docs/feature-matrix.md +++ b/docs/feature-matrix.md @@ -62,7 +62,7 @@ Changes here need an ADR. These are load-bearing. | Verdict precedence | `evaluation/axes.py` | `error > detection_gap > prevention_gap > evidence_gap > response_gap > secure`. Frozen. | | Selected-project resolution + manifest | `project/` | Which repository. A process-boundary decision, never a tool argument ([ADR 0003](adr/0003-constrained-mcp-tools.md)). | | Surface discovery | `project/discovery.py` | Agents, skills, hooks, settings, instructions, MCP servers, tool grants, memory. Inventory only. | -| Runtime framework fingerprint | `project/fingerprint.py` | Distinguishes application runtime agents from Claude Code, Codex, Gemini CLI, Cursor and MCP development configuration without importing repository code. | +| Runtime framework fingerprint | `project/fingerprint.py` | Distinguishes application runtime agents from Claude Code, Codex, Gemini CLI, Cursor and MCP development configuration without importing repository code. Composed into the `project` plane, so `scan`, the dashboard and the MCP resource read one classification. | | Repository risk plane | `inspect/` | Turns the inventory into ranked risks, and each risk into `verified` / `verifiable` / `not_verifiable` ([ADR 0009](adr/0009-repository-first-golden-path.md)). | | `config-surface:` correlation | `scenario/surface_tags.py` | The one bridge from a static surface to a runnable scenario. Shared by the risk and posture planes so they cannot disagree. | | Replay executor + fixture corpus | `execution/replay.py` | Deterministic. What CI relies on. | @@ -83,7 +83,7 @@ well-meaning change: | `repo_risk` | What in this repository is worth testing? | `inspected` / severity / `verified`โ€“`verifiable`โ€“`not_verifiable` | | `skill_assurance` | Do this repository's skills behave? | `pass` / `fail` / `not_tested` | | `static_posture` | What did a third-party scanner flag? | `ingested` / `covered`โ€“`not_tested`โ€“`n/a` | -| `project` | Which repository is this? | `declared` / `not_initialised` / `invalid` | +| `project` | Which repository is this, and is it an agent? | `declared` / `not_initialised` / `invalid`, and `confirmed`โ€“`likely`โ€“`configuration_only`โ€“`not_detected`โ€“`unsupported` | Each status enum is spelled differently on purpose. A single number averaging them would answer none of the four questions, and the fastest way to build one @@ -158,11 +158,12 @@ is refusing to widen the surface while the middle of it is unproven. Stated here rather than left for a reader to discover, because a matrix that only lists what works is marketing. -1. **The framework fingerprint is not yet composed into `agentsec scan`.** The - deterministic detector exists and distinguishes `confirmed`, `likely`, - `configuration_only`, `not_detected` and `unsupported`, but the next DTO/CLI PR must make - that classification visible on the golden path without merging it into a - Purple verdict. +1. **The fingerprint classifies six framework families, not every framework.** + LangChain/LangGraph, OpenAI Agents SDK, AutoGen, Semantic Kernel, CrewAI and + framework-neutral Python/Node tool calling. Anything else in a repository that + is plainly an agent reports `not_detected`, which is why that word means + "no evidence" and never "no agent" โ€” and why an unparsed file is reported as + `unsupported` rather than counted as absence. 2. **No scenario covers the tool-grant or settings surface.** `ASI-TOOL-BROAD-GRANT` and `ASI-TOOL-PERMISSION-BYPASS` fire โ€” the second at `critical` โ€” and both report `not_verifiable`, because no `AGT-CONFIG-*` scenario is tagged at @@ -176,7 +177,14 @@ only lists what works is marketing. `demo-agent-fixture` (environment `local`) correctly refuses with exit 2 rather than selecting nothing and reporting success. Until fixtures exist, `--verify` needs a real staging target. -5. **No end-to-end run against a live agent has happened.** This remains the +5. **The catalogue does not travel with the CLI.** `Settings.scenarios_dir` is + `/scenarios`, so in a repository that is not a checkout of + AgentSec there are no scenarios to triage against and *every* risk resolves + to `not_verifiable` โ€” correct, and misleading at a glance now that the screen + above it can say `confirmed langgraph`. Bundling the reviewed catalogue as + package data is the fix; which catalogue version an installed CLI should + trust is the question that makes it more than a one-line change. +6. **No end-to-end run against a live agent has happened.** This remains the single most valuable open item, exactly as #32 says. --- diff --git a/docs/roadmap.md b/docs/roadmap.md index 521867c..4d54792 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -34,6 +34,7 @@ agentsec init โ†’ agentsec scan โ†’ agentsec scan --verify -t โ†’ dashboard | CLI with meaningful exit codes | โœ… | `0` clean, `1` blocking, `2` could not tell | | Selected-project manifest and discovery | โœ… | `.agentsec/project.yaml`; relative locations only, traversal and symlink escape refused | | Runtime framework fingerprint engine | โœ… | deterministic, read-only detection for LangGraph/LangChain, OpenAI Agents SDK, AutoGen, Semantic Kernel, CrewAI and framework-neutral tool calling; development-agent config stays separate | +| Fingerprint composed into `scan`, dashboard and MCP resource | โœ… | `project.fingerprint`; reported even before `agentsec init`, and `not_detected` never renders as a pass ([#32](https://github.com/trionnemesis/AgentSec/issues/32)) | | Tool-grant and memory surfaces | โœ… | one entry per permission rule; `.claude/memory` declared like any other surface ([#32](https://github.com/trionnemesis/AgentSec/issues/32)) | | **Repository risk plane** (`agentsec scan`) | โœ… | 10 deterministic rules across agents, skills, hooks, tool grants, MCP and memory ([ADR 0009](adr/0009-repository-first-golden-path.md)) | | **Risk โ†’ scenario triage** | โœ… | `verified` / `verifiable` / `not_verifiable`; `scan --verify` drains the queue | @@ -44,10 +45,6 @@ agentsec init โ†’ agentsec scan โ†’ agentsec scan --verify -t โ†’ dashboard ### Core โ€” open -- [ ] **Compose the framework fingerprint into the inspection DTO and CLI.** - The detector is built and tested, but `agentsec scan` does not expose it - yet; that integration must not turn coding-agent configuration into a - runtime-agent claim or widen `PurpleVerdict`. - [ ] **Run against one real staging agent end to end**, and fix what that reveals. Still the single most valuable open item. - [ ] **A scenario covering the tool-grant / settings surface.** diff --git a/schemas/project-dashboard.schema.json b/schemas/project-dashboard.schema.json index db29f02..d12c022 100644 --- a/schemas/project-dashboard.schema.json +++ b/schemas/project-dashboard.schema.json @@ -41,7 +41,7 @@ "$defs": { "project": { "type": "object", - "description": "Which repository this is. `status` is not decoration: a dashboard for a repository that was never initialised must say so rather than render as an anonymous but healthy project.", + "description": "Which repository this is, what agent surfaces it declares, and whether it implements an AI agent at all (`fingerprint`). `status` is not decoration: a dashboard for a repository that was never initialised must say so rather than render as an anonymous but healthy project. The fingerprint is reported even then โ€” whether there is an agent here does not depend on whether someone ran `agentsec init`.", "required": ["status"], "additionalProperties": false, "properties": { @@ -56,11 +56,79 @@ "type": "object", "description": "Counts only. The inventory itself is not part of this document.", "additionalProperties": { "type": "integer", "minimum": 0 } - } + }, + "fingerprint": { "$ref": "#/$defs/fingerprint" } }, "if": { "properties": { "status": { "const": "declared" } } }, "then": { "required": ["status", "project_id", "name"] } }, + "fingerprint": { + "type": "object", + "description": "Whether this checkout implements an AI agent, and in what (issue #32). Read statically: no repository code is imported or executed, and every path is relative to the selected root. Inside `project` rather than beside it because it qualifies the identity that plane already reports, and because a sixth top-level key would have read as a sixth kind of conclusion. It is not one โ€” nothing here has run, so `not_detected` is an absence of evidence and never a pass.", + "required": ["agent_presence", "confidence"], + "additionalProperties": false, + "properties": { + "agent_presence": { + "enum": ["confirmed", "likely", "configuration_only", "not_detected", "unsupported"], + "description": "confirmed: framework code with a builder or entrypoint. likely: dependencies or tool-calling shape without a runtime entrypoint. configuration_only: CLAUDE.md, skills, hooks or .mcp.json and no runtime agent โ€” a coding agent works *on* this repository, which is not the same as this repository *being* one. not_detected: no evidence either way. unsupported: something was found and could not be classified, which is stated rather than rounded down to absence." + }, + "confidence": { "enum": ["high", "medium", "none"] }, + "schema_version": { "type": "string", "pattern": "^[0-9]+\\.[0-9]+\\.[0-9]+$" }, + "runtime_agents": { + "type": "array", + "description": "Application agents implemented in this repository. Never concatenated with development_agent_config: merging the two is exactly how a .mcp.json becomes a claimed runtime agent.", + "items": { + "type": "object", + "required": ["framework", "language", "confidence"], + "properties": { + "framework": { "type": "string", "maxLength": 60 }, + "language": { "enum": ["python", "javascript", "typescript", "mixed", "unknown"] }, + "confidence": { "enum": ["high", "medium"] }, + "entrypoints": { "type": "array", "items": { "type": "string" } }, + "evidence": { + "type": "array", + "description": "Bounded structural facts: a dependency name, an import path, a builder symbol, a relative file. Never source text โ€” see project/fingerprint.py.", + "items": { + "type": "object", + "required": ["kind", "file", "value"], + "properties": { + "kind": { + "enum": ["dependency", "import", "builder_call", "runtime_config", "tool_calling"] + }, + "file": { "type": "string" }, + "value": { "type": "string", "maxLength": 500 } + } + } + } + } + } + }, + "development_agent_config": { + "type": "array", + "description": "Coding-agent configuration found in the checkout. Present on its own, this is what makes a repository configuration_only.", + "items": { + "type": "object", + "required": ["platform"], + "properties": { + "platform": { "enum": ["claude_code", "codex", "gemini_cli", "cursor", "mcp"] }, + "paths": { "type": "array", "items": { "type": "string" } } + } + } + }, + "problems": { + "type": "array", + "description": "Candidates the detector could not read completely. An unparsed file is stated, never counted as absence of an agent.", + "items": { + "type": "object", + "properties": { + "path": { "type": "string" }, + "kind": { "type": "string" }, + "detail": { "type": "string" } + } + } + } + } + }, "repoRisk": { "type": "object", "description": "The repository risk plane (issue #32): this repository's own agent configuration โ€” skills, agents, hooks, tool grants, MCP servers and memory stores โ€” read statically by first-party rules and triaged against the scenario catalogue. Deliberately not verdict-shaped. A risk is a reason to run a scenario, not a result of having run one; nothing here has executed anything or given a detection control the chance to fire. The `verification` block is the whole point: it says whether a deterministic conclusion is available (`verified`), obtainable (`verifiable`), or out of reach (`not_verifiable`). The last is the common case and must never render as green.", diff --git a/src/agentsec/cli.py b/src/agentsec/cli.py index ccab551..8fbf55b 100644 --- a/src/agentsec/cli.py +++ b/src/agentsec/cli.py @@ -491,6 +491,23 @@ def init( "verifiable": "runnable now", "not_verifiable": "no scenario covers this", } +#: What each classification means to someone who did not read the schema. Every +#: line says what was found; none of them says the repository is safe, because +#: this classifier has executed nothing. +_PRESENCE_LABEL = { + "confirmed": ("confirmed", typer.colors.CYAN), + "likely": ("likely", typer.colors.CYAN), + "configuration_only": ("configuration only", typer.colors.WHITE), + "not_detected": ("not detected", typer.colors.WHITE), + "unsupported": ("could not classify", typer.colors.YELLOW), +} +_PRESENCE_DETAIL = { + "confirmed": "runtime agent code in this repository", + "likely": "agent dependencies or tool calling, but no runtime entrypoint", + "configuration_only": "coding-agent configuration only โ€” no runtime agent code", + "not_detected": "no agent framework, tool calling or agent configuration found", + "unsupported": "something was found here that these rules cannot classify", +} @app.command() @@ -512,10 +529,16 @@ def scan( ) -> None: """Inspect the selected repository for agent attack surface, and rank what it finds. - The engineer's entry point. Reads this repository's skills, agents, hooks, - tool grants, MCP servers and memory stores, applies the static rules in - `agentsec.inspect`, and reports each risk alongside whether anything here - can turn it into a deterministic conclusion. + The engineer's entry point. Answers two questions in order: whether this + repository implements an AI agent โ€” and in what framework โ€” then what in its + skills, agents, hooks, tool grants, MCP servers and memory stores is worth + testing, with whether anything here can turn each risk into a deterministic + conclusion. + + The first answer never asserts the second. A repository holding only a + `CLAUDE.md` and a `.mcp.json` reports `configuration only`: a coding agent + works *on* this checkout, which is not the same as this checkout *being* an + agent. `not detected` is likewise an absence of evidence, not a pass. Static only, by design. A risk is a reason to run a scenario, never the result of having run one โ€” so `--verify` is the second half: it selects the @@ -567,18 +590,51 @@ def scan( _fail(exc) +def _print_agent(project: dict) -> None: + """What this repository *is*, before what is wrong with it. + + First, because it is the question an engineer opening an unfamiliar + repository actually has, and because every risk below it means something + different depending on the answer. Printed even when the risk plane could + not run: whether there is an agent here does not depend on whether anyone + has run `agentsec init`. + """ + fingerprint = project.get("fingerprint") or {} + presence = fingerprint.get("agent_presence", "unsupported") + label, colour = _PRESENCE_LABEL.get(presence, (presence, typer.colors.YELLOW)) + + typer.secho(f"\n AI agent {label}", fg=colour, bold=True) + typer.echo(f" {_PRESENCE_DETAIL.get(presence, '')}") + + for agent in fingerprint.get("runtime_agents") or []: + where = ", ".join(agent.get("entrypoints") or []) or "no entrypoint found" + typer.echo(f" {agent['framework']} ({agent['language']}) {where}") + platforms = ", ".join( + config["platform"] for config in fingerprint.get("development_agent_config") or [] + ) + if platforms: + typer.echo(f" coding-agent config: {platforms}") + if fingerprint.get("problems"): + typer.secho( + f" {len(fingerprint['problems'])} file(s) could not be parsed; " + "absence of a framework here is not proof there is none", + fg=typer.colors.YELLOW, + ) + + def _print_scan(document: dict) -> None: project, plane = document["project"], document["repo_risk"] + _print_agent(project) if plane.get("status") != "inspected": typer.secho( - f"not inspected [{plane.get('reason', 'unknown')}]: {plane.get('detail', '')}", + f"\nnot inspected [{plane.get('reason', 'unknown')}]: {plane.get('detail', '')}", fg=typer.colors.YELLOW, ) return typer.echo( - f"\nproject {project.get('project_id', '?')} " + f"\n project {project.get('project_id', '?')} " f"surfaces {sum((project.get('surfaces') or {}).values())} " f"risks {plane['counts']['total']}\n" ) diff --git a/src/agentsec/mcp/contract.py b/src/agentsec/mcp/contract.py index d1b1447..2a66633 100644 --- a/src/agentsec/mcp/contract.py +++ b/src/agentsec/mcp/contract.py @@ -385,12 +385,15 @@ def _obj( uri_template="agentsec://project/risks", title="Repository risk plane", description=( - "This repository's own agent attack surface โ€” skills, agents, hooks, tool " - "grants, MCP servers, memory stores โ€” read by the static rules in " - "`agentsec.inspect` and triaged against the scenario catalogue. Each risk " - "says whether a deterministic conclusion is available, obtainable, or out " - "of reach. Not a verdict: nothing here has executed anything. Takes no " - "arguments โ€” which repository is a process-boundary decision (ADR 0003)." + "Whether this repository implements an AI agent and in what framework " + "(`project.fingerprint`), and its own agent attack surface โ€” skills, " + "agents, hooks, tool grants, MCP servers, memory stores โ€” read by the " + "static rules in `agentsec.inspect` and triaged against the scenario " + "catalogue. Each risk says whether a deterministic conclusion is " + "available, obtainable, or out of reach. Not a verdict: nothing here has " + "executed anything, so a repository with no runtime agent reads as " + "`not_detected` rather than as a pass. Takes no arguments โ€” which " + "repository is a process-boundary decision (ADR 0003)." ), handler="inspect_repository", publish="repo_risk_document", @@ -406,10 +409,12 @@ def _obj( uri_template="agentsec://dashboard/latest", title="Project dashboard", description=( - "The latest state of this project as one document: identity, the " - "four-axis purple rollup, and the Skill Assurance summary, each in " - "its own property. Computed in memory โ€” reading it starts no run and " - "writes no file. Schema: schemas/project-dashboard.schema.json." + "The latest state of this project as one document: identity and " + "runtime-agent classification, the repository risk plane, the " + "four-axis purple rollup, the Skill Assurance summary and ingested " + "static posture, each in its own property and never merged. Computed " + "in memory โ€” reading it starts no run and writes no file. Schema: " + "schemas/project-dashboard.schema.json." ), handler="dashboard", publish="dashboard", diff --git a/src/agentsec/reporting/publish.py b/src/agentsec/reporting/publish.py index 3ac769e..d6660e2 100644 --- a/src/agentsec/reporting/publish.py +++ b/src/agentsec/reporting/publish.py @@ -56,7 +56,12 @@ #: composed `dashboard`, inert everywhere else. Minor rather than major because #: a consumer that reads the planes it knows keeps working โ€” the shapes it was #: already reading are untouched. -PUBLISH_SCHEMA_VERSION = "1.3.0" +#: 1.4.0 added `project.fingerprint` (#32) โ€” an *optional* key inside a plane +#: that already existed, rather than a sixth plane, because "does this +#: repository implement an agent" qualifies the identity `project` already +#: reports. No plane was added, none was merged, and nothing already published +#: changed shape. +PUBLISH_SCHEMA_VERSION = "1.4.0" #: Names the ruleset, so a stored export says which policy produced it. PUBLISH_POLICY = "observed-data-v1" @@ -469,6 +474,82 @@ def publish_posture(document: dict[str, Any] | None) -> dict[str, Any]: return body +def publish_project(document: dict[str, Any] | None) -> dict[str, Any]: + """Project the identity plane: which repository, what is in it, is it an agent. + + Named field by field like every other publisher here, and newly so: this + plane used to be passed through whole because everything in it was a short + service-authored string. The fingerprint changes that. It is derived from + reading arbitrary repository files, and while the detector is built never to + carry source text โ€” evidence is a dependency name, an import path, a builder + symbol, a relative file path โ€” "the producer promises" is the wrong place + for that guarantee to live alone. Listing the fields here means a detector + that starts carrying a snippet tomorrow has to be argued for in this file. + """ + document = document or {"status": "not_initialised"} + body: dict[str, Any] = {"status": document.get("status", "not_initialised")} + for key in ("project_id", "name"): + if document.get(key) is not None: + body[key] = document[key] + if document.get("detail") is not None: + body["detail"] = scrub(document["detail"]) + if "surfaces" in document: + body["surfaces"] = dict(document["surfaces"]) + if document.get("fingerprint") is not None: + body["fingerprint"] = _publish_fingerprint(document["fingerprint"]) + return body + + +def _publish_fingerprint(document: dict[str, Any]) -> dict[str, Any]: + """The runtime-agent classification, with the two lists kept apart. + + ``runtime_agents`` and ``development_agent_config`` are projected + separately and never concatenated. A consumer that merged them would be one + template change away from announcing a runtime AI agent in a repository + whose only evidence is a ``CLAUDE.md``, which is the specific overclaim #32 + asked this feature not to make. + """ + body: dict[str, Any] = { + "agent_presence": document.get("agent_presence", "unsupported"), + "confidence": document.get("confidence", "none"), + "runtime_agents": [ + { + "framework": agent.get("framework"), + "language": agent.get("language"), + "confidence": agent.get("confidence"), + "entrypoints": list(agent.get("entrypoints") or []), + "evidence": [ + { + "kind": item.get("kind"), + "file": item.get("file"), + # A package, module or builder symbol the detector + # matched โ€” bounded vocabulary, never matched source. + "value": scrub(item.get("value")), + } + for item in agent.get("evidence") or [] + ], + } + for agent in document.get("runtime_agents") or [] + ], + "development_agent_config": [ + {"platform": config.get("platform"), "paths": list(config.get("paths") or [])} + for config in document.get("development_agent_config") or [] + ], + } + if document.get("schema_version") is not None: + body["schema_version"] = document["schema_version"] + if document.get("problems"): + body["problems"] = [ + { + "path": problem.get("path"), + "kind": problem.get("kind"), + "detail": scrub(problem.get("detail")), + } + for problem in document["problems"] + ] + return body + + def publish_repo_risk(document: dict[str, Any] | None) -> dict[str, Any]: """Project the repository risk plane (`inspect/`). @@ -537,11 +618,10 @@ def publish_repo_risk_document(document: dict[str, Any] | None) -> dict[str, Any checkout. """ document = document or {} - project = document.get("project") or {"status": "not_initialised"} return _envelope( "repo_risk", [], - project=dict(project), + project=publish_project(document.get("project")), repo_risk=publish_repo_risk(document.get("repo_risk")), ) @@ -686,7 +766,9 @@ def publish_dashboard(document: dict[str, Any]) -> dict[str, Any]: "schema_version": PUBLISH_SCHEMA_VERSION, "kind": "dashboard", "generated_at": document.get("generated_at"), - "project": document.get("project"), + # Carries the runtime-agent fingerprint (#32), so this one is projected + # rather than passed through; see publish_project. + "project": publish_project(document.get("project")), # Already the one shape every output renders from, and it stamps its own # version. Republishing it is a no-op by design; see publish_report. "purple": document.get("purple"), @@ -726,6 +808,7 @@ def publish_dashboard(document: dict[str, Any]) -> dict[str, Any]: "report": publish_report, "dashboard": publish_dashboard, "posture": publish_posture, + "project": publish_project, "repo_risk": publish_repo_risk, "repo_risk_document": publish_repo_risk_document, } diff --git a/src/agentsec/reporting/templates/dashboard.html.j2 b/src/agentsec/reporting/templates/dashboard.html.j2 index 2648258..a871341 100644 --- a/src/agentsec/reporting/templates/dashboard.html.j2 +++ b/src/agentsec/reporting/templates/dashboard.html.j2 @@ -36,6 +36,16 @@ {% else %} project {{ d.project.status.replace('_', ' ') }} {% endif %} + {% if d.project.fingerprint %} + {# The first thing on the page, because every count below it means + something different depending on the answer. A classification, not + a conclusion: nothing has been executed to produce it. #} + ยท AI agent + {{ d.project.fingerprint.agent_presence.replace('_', ' ') }} + {% for agent in d.project.fingerprint.runtime_agents %} + {{ agent.framework }}{{ '' if loop.last else ',' }} + {% endfor %} + {% endif %} ยท target {{ d.purple.target_id }} ยท profile {{ d.purple.profile }} @@ -74,6 +84,50 @@ PurpleVerdict and does not belong in a verdict count; see ADR 0008. A repo risk is not one either โ€” it is a reason to run a scenario, not a result. #}
+ {% if d.project.fingerprint %} + {# Runtime agents and coding-agent configuration are rendered as two + separate rows and never summed. A single "agent surfaces" number would + let a .mcp.json read as a runtime agent, which is the one overclaim + this classification exists to prevent (#32). #} +
+

+ AI agent + {{ d.project.fingerprint.agent_presence.replace('_', ' ') }} +

+
+
confidence
{{ d.project.fingerprint.confidence }}
+ {% for agent in d.project.fingerprint.runtime_agents %} +
{{ agent.framework }}
+
+ {{ agent.language }}{% if agent.entrypoints %} ยท + {% for path in agent.entrypoints %}{{ path }}{{ '' if loop.last else ' ' }}{% endfor %} + {% else %} ยท no entrypoint found{% endif %} +
+ {% endfor %} + {% if d.project.fingerprint.development_agent_config %} +
coding-agent config
+
{{ d.project.fingerprint.development_agent_config | map(attribute='platform') | join(', ') }}
+ {% endif %} + {% if d.project.fingerprint.problems %} +
unparsed
{{ d.project.fingerprint.problems | length }} file(s)
+ {% endif %} +
+

+ {% if d.project.fingerprint.agent_presence == 'configuration_only' %} + A coding agent works on this repository; the repository does not implement one. + {% elif d.project.fingerprint.agent_presence == 'not_detected' %} + No agent framework, tool calling or agent configuration was found. That is an + absence of evidence, not a pass โ€” nothing was executed to establish it. + {% elif d.project.fingerprint.agent_presence == 'unsupported' %} + Something here could not be classified. Stated rather than rounded down to + "no agent found". + {% else %} + Read statically from dependencies, imports and builder calls. No repository + code was imported or run. + {% endif %} +

+
+ {% endif %}

Repository risk diff --git a/src/agentsec/service/harness.py b/src/agentsec/service/harness.py index b590c98..dc78df8 100644 --- a/src/agentsec/service/harness.py +++ b/src/agentsec/service/harness.py @@ -47,7 +47,13 @@ from agentsec.policy.profiles import Profile, load_profiles from agentsec.posture.adapter import load_posture_report, resolve_report_path from agentsec.posture.coverage import compute_posture_coverage, coverage_counts -from agentsec.project import MANIFEST_PATH, Discovery, discover +from agentsec.project import ( + FINGERPRINT_SCHEMA_VERSION, + MANIFEST_PATH, + Discovery, + discover, + fingerprint_repository, +) from agentsec.reporting.html import write_html_report from agentsec.reporting.junit import render_junit from agentsec.reporting.normalizer import ( @@ -784,7 +790,14 @@ def _project_planes( Skill plane says `not_tested`, because "we do not know which repository this is" must never render as a clean result. Static posture follows the same rule (#25): no manifest means nothing was ingested, not a clean scan. + + The fingerprint is attached to every branch, including the ones where + discovery failed, because it answers a question that does not depend on + a manifest: whether there is an agent in this checkout at all. A reader + told only "not initialised" learns nothing about whether it was worth + initialising. """ + fingerprint = self._fingerprint() try: discovery = discover(self.settings.workspace) except ProjectNotInitialised: @@ -792,6 +805,7 @@ def _project_planes( { "status": "not_initialised", "detail": f"no {MANIFEST_PATH}; run `agentsec init` in this repository", + "fingerprint": fingerprint, }, { "status": "not_tested", @@ -811,7 +825,7 @@ def _project_planes( ) except ConfigError as exc: return ( - {"status": "invalid", "detail": exc.message[:500]}, + {"status": "invalid", "detail": exc.message[:500], "fingerprint": fingerprint}, { "status": "not_tested", "reason": "project_invalid", @@ -836,12 +850,46 @@ def _project_planes( "project_id": discovery.project_id, "name": discovery.name, "surfaces": counts, + "fingerprint": fingerprint, }, {**discovery.skill_assurance(), "counts": counts}, self._static_posture(discovery, scenarios_with_a_verdict or set()), self._repo_risk(discovery, scenarios_with_a_verdict or set()), ) + def _fingerprint(self) -> dict[str, Any]: + """Whether this checkout implements an AI agent, and in what. + + Sits inside the ``project`` plane rather than beside it: it answers + "which repository is this", which is the question that plane already + exists for, and it is read on the same screen as the surface counts it + qualifies. A sixth plane would have implied a sixth kind of conclusion. + + The distinction the whole thing turns on is preserved by the model: + ``runtime_agents`` is application code, ``development_agent_config`` is + Claude Code, Codex, Cursor or an MCP config. A repository holding only + the latter is ``configuration_only``, never a runtime agent, and + ``not_detected`` is an absence of evidence rather than a pass โ€” nothing + here has executed anything. + """ + try: + return fingerprint_repository(self.settings.workspace).to_dict() + except AgentSecError as exc: + # The classifier could not read the checkout. Reporting + # `not_detected` here would turn "we could not look" into "there is + # nothing to find", which is the one answer this plane must never + # invent; `unsupported` is the model's word for an incomplete read. + return { + "schema_version": FINGERPRINT_SCHEMA_VERSION, + "agent_presence": "unsupported", + "confidence": "none", + "runtime_agents": [], + "development_agent_config": [], + "problems": [ + {"path": ".", "kind": "unreadable_root", "detail": exc.message[:500]} + ], + } + def _repo_risk( self, discovery: Discovery, scenarios_with_a_verdict: set[str] ) -> dict[str, Any]: diff --git a/tests/test_dashboard.py b/tests/test_dashboard.py index 5123f8b..e91fcc5 100644 --- a/tests/test_dashboard.py +++ b/tests/test_dashboard.py @@ -156,6 +156,15 @@ def test_a_workspace_with_a_manifest_is_named(service: HarnessService) -> None: "skills": 0, "supported_skills": 0, "agents": 0, "hooks": 0, "mcp_servers": 0, "tool_grants": 0, "memory": 0, "problems": 0, }, + # A workspace of scenarios and fixtures is not an agent, and says so + # rather than reporting nothing. + "fingerprint": { + "agent_presence": "not_detected", + "confidence": "none", + "runtime_agents": [], + "development_agent_config": [], + "schema_version": "1.0.0", + }, } diff --git a/tests/test_fingerprint.py b/tests/test_fingerprint.py index 7428a40..2e4342c 100644 --- a/tests/test_fingerprint.py +++ b/tests/test_fingerprint.py @@ -1,4 +1,9 @@ -"""Runtime-agent fingerprinting stays distinct from coding-agent config.""" +"""Runtime-agent fingerprinting stays distinct from coding-agent config. + +The second half of this file is the composition (#32): the same classification +read through the inspection DTO, the `agentsec scan` output and the published +dashboard, where the distinction has to survive three more hops. +""" from __future__ import annotations @@ -8,8 +13,11 @@ import pytest +from agentsec.config import Settings from agentsec.models.fingerprint import FingerprintReport, RuntimeAgentFingerprint from agentsec.project import fingerprint_repository +from agentsec.reporting.publish import publish +from agentsec.service.harness import HarnessService def write(path: Path, text: str) -> Path: @@ -285,3 +293,179 @@ def test_evidence_never_contains_source_text(tmp_path: Path) -> None: report = fingerprint_repository(tmp_path) assert secret not in json.dumps(report.to_dict()) + + +# -- composition into the inspection DTO, the CLI and the dashboard ----------- + +MANIFEST = """\ +apiVersion: agentsec.dev/v1alpha1 +kind: Project +project_id: demo-project +name: Demo +""" + + +@pytest.fixture +def langgraph_repo(workspace: Path) -> Path: + """A checkout that really does implement an agent.""" + write(workspace / ".agentsec" / "project.yaml", MANIFEST) + write( + workspace / "pyproject.toml", + '[project]\nname = "order-agent"\nversion = "0.1.0"\ndependencies = ["langgraph"]\n', + ) + write( + workspace / "src" / "agent" / "graph.py", + "from langgraph.graph import StateGraph\n\n" + "builder = StateGraph(dict)\n" + "app = builder.compile()\n", + ) + return workspace + + +def project_plane(settings: Settings) -> dict: + return publish("dashboard", HarnessService(settings, actor="pytest").dashboard())["project"] + + +def test_the_project_plane_names_the_framework_and_where_it_lives( + langgraph_repo: Path, settings: Settings +) -> None: + fingerprint = project_plane(settings)["fingerprint"] + + assert fingerprint["agent_presence"] == "confirmed" + assert fingerprint["confidence"] == "high" + [agent] = fingerprint["runtime_agents"] + assert agent["framework"] == "langgraph" + assert agent["entrypoints"] == ["src/agent/graph.py"] + + +def test_coding_agent_configuration_is_never_published_as_a_runtime_agent( + workspace: Path, settings: Settings +) -> None: + """The overclaim this composition exists to prevent (#32).""" + write(workspace / ".agentsec" / "project.yaml", MANIFEST) + write(workspace / ".claude" / "skills" / "greet" / "SKILL.md", "---\nname: greet\n---\nHi.\n") + write(workspace / ".mcp.json", json.dumps({"mcpServers": {"local": {"command": "x"}}})) + + fingerprint = project_plane(settings)["fingerprint"] + + assert fingerprint["agent_presence"] == "configuration_only" + assert fingerprint["runtime_agents"] == [] + assert {c["platform"] for c in fingerprint["development_agent_config"]} == { + "claude_code", "mcp", + } + + +def test_an_ordinary_repository_is_not_detected_and_never_reads_as_a_pass( + workspace: Path, settings: Settings +) -> None: + write(workspace / ".agentsec" / "project.yaml", MANIFEST) + write(workspace / "README.md", "# A normal repository\n") + + project = project_plane(settings) + + assert project["fingerprint"]["agent_presence"] == "not_detected" + assert project["fingerprint"]["confidence"] == "none" + serialised = json.dumps(project) + for verdict in ("secure", "pass", "prevention_gap", "detection_gap"): + assert verdict not in serialised + + +def test_the_fingerprint_is_reported_before_the_repository_is_initialised( + workspace: Path, settings: Settings +) -> None: + """Whether there is an agent here does not depend on running `agentsec init`.""" + write( + workspace / "src" / "graph.py", + "from langgraph.graph import StateGraph\ngraph = StateGraph(dict)\n", + ) + + project = project_plane(settings) + + assert project["status"] == "not_initialised" + assert project["fingerprint"]["agent_presence"] == "confirmed" + + +def test_the_fingerprint_never_reaches_the_purple_plane( + langgraph_repo: Path, settings: Settings +) -> None: + """Composition, not merging: no plane may borrow another's vocabulary.""" + dashboard = publish("dashboard", HarnessService(settings, actor="pytest").dashboard()) + + purple = json.dumps(dashboard["purple"]) + assert "langgraph" not in purple + assert "agent_presence" not in purple + assert set(dashboard["purple"]["verdict_counts"]) <= { + "secure", "prevention_gap", "detection_gap", "evidence_gap", "response_gap", "error", + } + + +def test_the_standalone_risk_resource_carries_the_fingerprint( + langgraph_repo: Path, settings: Settings +) -> None: + service = HarnessService(settings, actor="pytest") + + document = publish("repo_risk_document", service.inspect_repository()) + + assert document["project"]["fingerprint"]["agent_presence"] == "confirmed" + assert document["repo_risk"]["status"] == "inspected" + + +def test_the_project_projection_names_its_fields(langgraph_repo: Path, settings: Settings) -> None: + """A key the service grows tomorrow is absent from published output until argued for.""" + raw = HarnessService(settings, actor="pytest").dashboard()["project"] + raw["operator_notes"] = "an internal field nobody decided was publishable" + raw["fingerprint"]["runtime_agents"][0]["source"] = "a snippet a future detector kept" + + published = publish("project", raw) + + assert "operator_notes" not in published + assert "source" not in published["fingerprint"]["runtime_agents"][0] + + +def test_published_evidence_never_carries_source_text( + workspace: Path, settings: Settings +) -> None: + secret = "customer-secret-should-stay-in-source" + write(workspace / ".agentsec" / "project.yaml", MANIFEST) + write( + workspace / "agent.py", + "from langgraph.graph import StateGraph\n" + f"SECRET = {secret!r}\n" + "graph = StateGraph(dict)\n", + ) + + assert secret not in json.dumps(project_plane(settings)) + + +def test_scan_leads_with_what_the_repository_is(langgraph_repo: Path) -> None: + from typer.testing import CliRunner + + from agentsec.cli import app + + result = CliRunner().invoke(app, ["scan", "--workspace", str(langgraph_repo)]) + + assert result.exit_code == 0, result.output + agent_line = result.output.index("AI agent") + assert agent_line < result.output.index("project demo-project") + assert "langgraph" in result.output + assert "src/agent/graph.py" in result.output + + +def test_scan_states_the_classification_when_the_repository_is_uninitialised( + workspace: Path, +) -> None: + """`not inspected` on its own tells an engineer nothing about what is here.""" + from typer.testing import CliRunner + + from agentsec.cli import app + + write( + workspace / "src" / "graph.py", + "from langgraph.graph import StateGraph\ngraph = StateGraph(dict)\n", + ) + + result = CliRunner().invoke(app, ["scan", "--workspace", str(workspace)]) + + assert result.exit_code == 0, result.output + assert "confirmed" in result.output + assert "not inspected [project_not_initialised]" in result.output