feat(inspect): compose the framework fingerprint into scan and the dashboard - #39
Conversation
…shboard `agentsec scan` now answers what the repository *is* before what is wrong with it, and the dashboard leads with the same answer. The detector landed in #37 with nothing reading it; this is the DTO, CLI and page it was built for. The classification lives at `project.fingerprint` rather than in a sixth plane. "Does this repository implement an agent" qualifies the identity the `project` plane already reports, and it is read on the same screen as the surface counts it explains — a sixth top-level key would have implied a sixth kind of conclusion, which this is not: nothing here executes anything. Runtime agents and coding-agent configuration stay in separate lists at every hop — service, publisher, schema, template, terminal. Merging them is the one overclaim #32 named, and it is a template change away unless each layer keeps them apart. Reported before `agentsec init`, since whether there is an agent in a checkout does not depend on whether anyone wrote a manifest. The `project` plane is now projected field by field like every other published document instead of being passed through whole. Its content is newly derived from reading arbitrary repository files; "the producer promises no source text" should not be a guarantee that lives in only one place. - `PUBLISH_SCHEMA_VERSION` 1.3.0 → 1.4.0: an optional key inside an existing plane. No plane added, none merged, no published shape changed. - 10 composition tests: framework and entrypoint named, configuration-only never published as a runtime agent, `not_detected` never rendering as a pass, the fingerprint absent from the purple plane, a stray service key refused by the projection, source text absent from published evidence, and `scan` leading with the classification in both the initialised and uninitialised cases. 397 passed, ruff and mypy clean. `docs/feature-matrix.md` records a gap this made visible: outside a checkout of AgentSec the catalogue is empty, so every risk resolves to `not_verifiable` under a screen that can now say `confirmed`. Refs #32
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cce17fffac
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| told only "not initialised" learns nothing about whether it was worth | ||
| initialising. | ||
| """ | ||
| fingerprint = self._fingerprint() |
There was a problem hiding this comment.
When agentsec://dashboard/latest or agentsec://project/risks is polled, this unconditional call runs synchronously on every read. fingerprint_repository recursively enumerates and reads up to 2,000 source files of up to 512 KiB each, so a large monorepo can make every refresh reread and process roughly 1 GiB while delaying the resource response. Cache the fingerprint with suitable invalidation, or move the full scan out of the per-read path.
Useful? React with 👍 / 👎.
2 participants
What changed
agentsec scannow says what the repository is before what is wrong with it, and the dashboard leads with the same answer. #37 landed the detector with nothing reading it; this is the DTO, CLI and page it was built for.Same classification through
agentsec dashboard, the rendered page andagentsec://project/risks. Closes the first three of #32's acceptance conditions end to end.Type
Purple coverage
tests/test_fingerprint.py::test_the_fingerprint_never_reaches_the_purple_planeasserts neither the framework name noragent_presenceappears anywhere in the purple plane, and thatverdict_countskeeps its five-verdict vocabulary.Why
project.fingerprintand not a sixth plane"Does this repository implement an agent" qualifies the identity the
projectplane already reports, and it is read on the same screen as the surface counts it explains —skills 3 · hooks 2 · mcp 2means something different depending on whether there is a runtime agent underneath them. A sixth top-level key would also have implied a sixth kind of conclusion. It is not one: nothing here executes anything,not_detectedis an absence of evidence, and #32 asked specifically that new planes stay frozen while the first screen gets finished.The distinction is enforced at every hop
Runtime agents and coding-agent configuration are separate lists in the service, the publisher, the schema, the template and the terminal output, and are never concatenated. That is the one overclaim #32 named — a
.mcp.jsonbecoming a claimed runtime agent — and it is a single template change away unless each layer keeps them apart.configuration_onlyis what AgentSec's own checkout reports.The
projectplane is now projected rather than passed throughIt used to be forwarded whole, which was defensible while everything in it was a short service-authored string. The fingerprint is derived from reading arbitrary repository files. The detector is built never to carry source text — evidence is a dependency name, an import path, a builder symbol, a relative path — but that guarantee should not live in only one place, so
publish_projectnames its fields like every other publisher here. A stray key on the service dict, and asourcefield on a hypothetical future detector, are both asserted absent from published output.Schema
PUBLISH_SCHEMA_VERSION1.3.0 → 1.4.0.project.fingerprintis an optional key inside a plane that already existed; every shape a consumer was already reading is untouched.$defs/fingerprintclosesadditionalPropertiesand enumerates both status vocabularies.Gap this exposed by being run
Recorded in
docs/feature-matrix.mdrather than fixed here:Settings.scenarios_diris<workspace>/scenarios, so in a repository that is not a checkout of AgentSec the catalogue is empty and every risk resolves tonot_verifiable. Correct, and newly misleading — it now sits under a screen that can sayconfirmed langgraph. Bundling the reviewed catalogue as package data is the fix; which catalogue version an installed CLI should trust is what makes it more than a one-liner.Checks
make check— ruff, mypy and pytest green on 3.11, 3.12 and 3.13 in CI. 397 passed, 2 skipped locally (387 before, 10 new).agentsec validate --strictpasses on all 8 scenariosagentsec run --target demo-agent-fixture --profile nightlystill exits 1 with exactlyAGT-TENANT-001andAGT-MEMPOIS-001blocking — asserted by the harness runs against its own fixtures CI job, and bytests/test_pipeline.py::test_exit_code_reflects_blocking_findings. Not runnable from this session's shell: this repository's guard hook refusesagentsec runfrom Bash, which is the hook working as intended.If this touches the MCP surface
HarnessServicefirst (_fingerprint, composed by_project_planes), so CLI and CI reach it tootests/test_mcp_contract.pypasses; both resource descriptions updated to state what they now carryNotes for the reviewer
Three worth challenging:
project.fingerprintrather than a top-level plane. Argued above; the counter-argument is that a reader scanning top-level keys will not find it.unsupportedon a classifier failure._fingerprintcatchesAgentSecErrorand reportsunsupportedwith a problem, nevernot_detected— "we could not look" must not become "there is nothing here". The invented problem kind isunreadable_root, which the schema accepts as free-formkindlike the other problem lists do.warnstyling forconfirmed/likely. Finding an agent is not a warning, but the palette has no neutral-but-notable colour andskipgrey read as "nothing here". Open to a better answer.