feat(project): add runtime agent framework fingerprinting - #37
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 31d6e11242
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| resolved = _resolve_call_name(node.func, aliases) | ||
| short = resolved.rsplit(".", 1)[-1] | ||
| for framework, _module in imported_frameworks: | ||
| if short in _FRAMEWORKS[framework].builders: |
There was a problem hiding this comment.
When a file imports a recognized framework but also calls an unrelated local function whose short name matches one of that framework's builders, this check records a high-confidence builder call. For example, import langgraph followed by a locally defined StateGraph() is reported as a confirmed LangGraph runtime because the resolved callee is reduced to its final component and never checked against the imported module. Associate the resolved call with the corresponding framework import before raising confidence.
Useful? React with 👍 / 👎.
| if re.search(rf"\b(?:new\s+)?{re.escape(builder)}\s*\(", text): | ||
| self.observations[framework].builders.add((relative, builder, language)) |
There was a problem hiding this comment.
When a JavaScript or TypeScript file imports a recognized package but mentions a builder only in a comment or string, this raw-source regex still records a builder call and upgrades the repository to confirmed. A commented-out new Agent(...) beside an active @openai/agents import is therefore reported as a live runtime entrypoint. Tokenize or otherwise remove comments and literals before using builder matches as high-confidence evidence.
Useful? React with 👍 / 👎.
| elif isinstance(node, ast.ImportFrom) and node.module: | ||
| framework = _framework_for_module(node.module) | ||
| if framework: | ||
| frameworks.add((framework, node.module)) |
There was a problem hiding this comment.
When application code uses a relative module whose name matches a framework, ast.ImportFrom exposes the module name without the leading dots and records the relative import as third-party framework evidence. For example, from .langgraph import StateGraph is classified as a LangGraph import and its call produces a confirmed runtime even though the module is entirely local. Check node.level before attributing an ImportFrom node to an external framework.
Useful? React with 👍 / 👎.
1 participant
What changed
Adds the first bounded PR from #32's remaining repository-inspection work: a deterministic, read-only runtime framework fingerprint.
confirmed,likely,configuration_only,not_detected, orunsupported.crew.jsoncprojects), and framework-neutral Python/Node tool calling.unsupportedrather than absence.PurpleVerdict.Why
The current discovery layer can inventory coding-agent surfaces, but it cannot answer whether the repository itself implements a runtime AI agent. Treating
.mcp.jsonorCLAUDE.mdas runtime proof would overstate what AgentSec knows. This change creates the evidence-backed classifier without coupling it to the dashboard or Purple evaluator yet.The package/import/builder vocabulary was checked against current primary documentation:
Safety and semantics
not_detected, neversecure.configuration_only, not as a runtime agent.Checks
make checkpytest -q --cov=agentsec --cov-report=term-missing— 80.82% total coverage (72% floor)Refs #32