AgentSleuth is a project-level framework for evaluating and supervising the behavior safety of an Agent inside a system-provided MCP test environment.
The only tested object is the Agent. MCP Server, Tool, Resource, Prompt, Tool Response templates, risk rules, and test cases are internal test fixtures.
Important for A-line PyRIT runtime: before running model-backed PyRIT checks, read docs/A/p3-a-pyrit-runtime-usage.md. Use OPENAI_CHAT_MODEL=deepseek-v4-pro, provide OPENAI_CHAT_KEY or a provider key such as DEEPSEEK_API_KEY, and point OPENAI_CHAT_ENDPOINT to the AgentSleuth PyRIT/OpenClaw shim: http://127.0.0.1:3100/api/v1/pyrit/openclaw/v1. Do not use the AgentSleuth realtime MCP endpoint as OPENAI_CHAT_ENDPOINT.
backend/ Backend runtime, API boundary, Agent execution, MCP sandbox, monitoring, risk, and report modules
frontend/ Web console workspace for dashboard, traces, risk reports, configs, and system views
packages/ Shared packages; contracts is the single front/back interface source
configs/ Internal tools, resources, prompts, tool responses, rules, test cases, oracles, scenarios, and attack-library indexes
outputs/ Generated runs, traces, reports, and exported artifacts
docs/ Architecture, directory, interface, ownership, and development specifications
scripts/ Engineering scripts for validation, build, and local startup
tests/ Cross-system end-to-end tests
third_party/ Vendored reference sources used by line-specific adapters, such as the A-line PyRIT adapted attack libraryLine-specific implementation notes are grouped under docs/A/, docs/B/, and docs/C/.
See docs/architecture.md for the system boundary, directory baseline, dependency rules, and development workflow.
See docs/ownership.md for strict A/B/C workspace ownership, shared controlled files, and cross-workspace change rules.
See docs/README.md for the documentation index and source-of-truth rules.
packages/contracts is the only shared interface package between backend and frontend. Frontend code must not import backend/src/**, and backend runtime modules must not place business logic in the contracts package.
AgentUnderTest
-> AgentAdapterConfig
-> TestContext
-> TestRun
-> InteractionTrace
-> RiskEvaluationResult
-> RiskReport
-> ReportArtifact[]npm run demo
npm run demo:p2
npm run frontend
npm run typecheck
npm run verify:all
npm run verify:e2enpm run demo starts the isolated display demo from frontend/demo. It is for product-flow demonstration only and is not the formal frontend implementation baseline.
npm run demo:p2 starts the P2 demo services and the formal Vite frontend. The frontend URL is http://127.0.0.1:5173 by default.
npm run verify:all runs the standard typecheck and module verification suite. npm run verify:e2e runs the three-stage end-to-end pipeline: pre-supervision detection, supervised rerun, and defense report export.
A-line P3 PyRIT corpus/runtime checks:
npm run verify:a-pyrit-library
npm run verify:a-corpus
npm run pyrit:bridge-smoke
npm run verify:a-pyrit-runtimeFor model-backed PyRIT runs in PowerShell:
. .\scripts\setup-pyrit-openclaw-env.ps1
npm run a:pyrit-runtime