diff --git a/.devcontainer/README.md b/.devcontainer/README.md index 36d5af9..0067ef8 100644 --- a/.devcontainer/README.md +++ b/.devcontainer/README.md @@ -18,8 +18,9 @@ or build directly with `docker build -f .devcontainer/Dockerfile .`. Every component is MIT, Apache-2.0, or BSD — passes CLAUDE.md §3.8. -The 17 skills under `.claude/skills/` and the 6 MCPs in `.mcp.json` come -along for free (they're just files in the repo + `npx`/`uvx` commands the +The 18 skills under `.claude/skills/` and the MCPs across `.mcp*.json` +(1 in the safe-default `.mcp.json`, 6 in `.mcp.cloud.json` / `.mcp.dual.json`) +come along for free (they're just files in the repo + `npx`/`uvx` commands the container can run). ## What's NOT inside (by design) @@ -42,7 +43,7 @@ The image carries **no secrets**. `devcontainer.json` uses `${localEnv:VAR}` ref |---|---|---| | `ANTHROPIC_API_KEY` | host shell or `.env` | preferred cloud credential; container env only — never written to disk inside the image | | `OPENROUTER_API_KEY` | host shell or `.env` | optional host-side AI-agent fallback; never passed into microsandboxes | -| `GITHUB_TOKEN` | host shell | passed to the `github` MCP via `.mcp.json` (already env-var-ref only) | +| `GITHUB_TOKEN` | host shell | passed to the `github` MCP via `.mcp.cloud.json` / `.mcp.dual.json` (env-var-ref only) | | `LANGFUSE_*` | host shell or `.env` | pointed at `host.docker.internal:3000` | | `SGLANG_*` | hard-coded to `host.docker.internal:{30000,30001}` | overrides the `.env` defaults of `localhost:*` so the container reaches the host | @@ -113,7 +114,7 @@ openssl rand -base64 32 | gpg --symmetric --output ~/.verdict/key.gpg chmod 600 ~/.verdict/key.gpg ``` -After this, `verdict doctor` (when the CLI lands per `BUILD_PLAN.md` W6) will report green on whatever the host can actually provide. +After this, `verdict doctor` will report green on whatever the host can actually provide. ## Rebuilding diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json index bf23304..1e383bb 100644 --- a/.devcontainer/devcontainer.json +++ b/.devcontainer/devcontainer.json @@ -1,6 +1,6 @@ // VERDICT dev container — opens any host with Docker into a reproducible -// toolchain (uv/Python 3.11, Node 20, pnpm, Rust 1.88) plus the 6 MCPs -// from .mcp.json and the 19 skills under .claude/skills/. See README.md +// toolchain (uv/Python 3.11, Node 20, pnpm, Rust 1.88) plus the MCPs across +// .mcp*.json and the 18 skills under .claude/skills/. See README.md // in this folder for what is NOT in here (Microsandbox, SGLang) and why. { "name": "VERDICT dev", diff --git a/CLAUDE.md b/CLAUDE.md index e2ebdf6..a901b7b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -35,6 +35,9 @@ VERDICT extends — but does not vendor — the upstream `protocol-sift/` Claude | `docs/ARCHITECTURE.md` | **Current authoritative architecture.** Supersedes everything in `docs/spec/`. Single source of truth for components, data flow, schemas, verifier strategies, threat model. | Default reference for any code or design question. | | `docs/BUILD_PLAN.md` | **Execution sequencing.** 6-week / 75-teammate-day TDD plan with task IDs (W1.A.3.a, W1.B.7, …), ownership, hours, acceptance gates. | Pick your next task; use task IDs in commits; use weekly gates as the definition of done. | | `docs/DEVPOST_COMPLIANCE.md` | **Submission rule-to-artifact mapping.** Every Devpost requirement traced to the file/commit that satisfies it. | Before any submission packaging. | +| `docs/FAILURE_MODES.md` | Runtime failure matrix: sandbox spawn, tool errors, fanout timeout, TSI failure, ledger write failures, and UNVERIFIABLE semantics. | Before implementing error paths or explaining graceful degradation to judges. | +| `docs/CASE_ISOLATION.md` | Case, chain, checkpoint, reverify, export, approval, and mode-lock boundaries. | Before implementing `verdict reverify`, `resume`, `export`, `approve`, or `validate`. | +| `docs/DFIR_MEMORY.md` | VERDICT's self-evolving memory model — evidence-first mutation rules, memory layers (case / technique / pattern / meta), and governance constraints. | Before implementing anything under `src/verdict/memory/`. | ### Audits (cross-doc consistency) @@ -78,18 +81,18 @@ These are non-negotiable. Each ties back to a schema validator, a wrapper, or a ### 3.1 Evidence integrity - **Never write to `/evidence/`.** It is a read-only microsandbox mount with `noexec` on data partitions; the host also `chattr +i`s evidence files. Any tool wrapper that writes to evidence is a bug, not a feature. -- **Hash on entry, re-hash periodically.** Every evidence file gets a SHA-256 at `case_init` recorded in the `EvidenceManifest`. The runtime re-hashes every 10 super-steps (`verdict/runtime/evidence_recheck.py`). Mismatch raises `HashMismatchError` and halts the case. +- **Hash on entry, re-hash periodically.** Every evidence file gets a SHA-256 at `case_init` recorded in the `EvidenceManifest`. The runtime re-hashes every 10 super-steps (`src/verdict/runtime/evidence_recheck.py`). Mismatch raises `HashMismatchError` and halts the case. - **Per-invocation hash.** Every tool call records `invocation_hash = blake3(tool_name + tool_version + args + evidence_hash)` in its `ToolOutput` and ledger entry. - **Per-output-file hash.** `LedgerEntry.output_files_sha256: dict[str, str]` records SHA-256 of every file the tool emits. NIST SP 800-86 §5.1.2 / §5.1.4 compliance. ### 3.2 Multi-artifact corroboration - `Finding.artifact_paths` and `Finding.artifact_classes` both have `min_length=2`. Single-artifact execution claims are forensically unsound and the validator rejects them. -- **Execution-class MITRE techniques** — T1059, T1106, T1204, T1218, T1543, T1547 — require **≥2 distinct `ArtifactClass` values** (not just two paths in the same class). Validator: `Finding._execution_requires_two_classes`. +- **Execution-class MITRE techniques** — T1059, T1106, T1204, T1218, T1543, T1547 — require **≥2 distinct `ArtifactClass` values** (not just two paths in the same class). Validator: `Finding._forensic_corroboration`. ### 3.3 Tier-1 caveat acknowledgment -`Finding.caveats_acknowledged: list[CaveatID]` is enforced at the schema layer. Cite the artifact, acknowledge the caveat. Caveat triggers are keyed by `Finding.artifact_classes` membership unless otherwise noted; `LOGON_TYPE_3_VS_10` is the named exception (triggered by `EVTX_4624` artifact_class AND the `EvtxRecord.LogonType` field equaling 3 or 10). The seven Tier-1 caveats (encoded in `verdict/schemas/caveat_id.py` and `verdict/prompts/examiner_caveats.md`): +`Finding.caveats_acknowledged: list[CaveatID]` is enforced at the schema layer. Cite the artifact, acknowledge the caveat. Caveat triggers are keyed by `Finding.artifact_classes` membership unless otherwise noted; `LOGON_TYPE_3_VS_10` is the named exception (triggered by `EVTX_4624` artifact_class AND `Finding.evtx_4624_logon_types` empty or containing 3 or 10). The seven Tier-1 caveats (encoded in `src/verdict/schemas/caveat_id.py` and `src/verdict/planning/prompts/examiner_caveats.md`): | CaveatID | Trigger | |----------|---------| @@ -153,7 +156,7 @@ Every new dependency must be **MIT or Apache-2.0** unless explicitly approved in - API keys, OAuth tokens, and bearer tokens **never enter a microVM**. This includes `ANTHROPIC_API_KEY`, `OPENROUTER_API_KEY`, GitHub tokens, Langfuse keys, and any bearer token used by host-side AI agents. They are injected via TSI on host egress only; tcpdump-verifiable. - HMAC ledger key is TPM-backed (`/dev/tpmrm0`) when available, else gpg-encrypted at `~/.verdict/key.gpg` with passphrase prompted at gateway init. -- Ledger redaction strips `authorization`, `auth_user`, `api_key` **before** the entry is hashed and HMAC-signed (`verdict/ledger/redaction.py`). +- Ledger redaction strips `authorization`, `auth_user`, `api_key` **before** the entry is hashed and HMAC-signed (`src/verdict/ledger/redaction.py`). - Anthropic OAuth tokens (Claude Code interactive auth) are not redistributable per Anthropic's commercial terms — do not commit, do not bake into images. ### 3.10 No mocks, no stubs, no placeholders — full-stack real @@ -183,7 +186,7 @@ If you find yourself reaching for a mock to make a test fast or hermetic, you ar ## 4. Architecture at a glance -9-node LangGraph state machine: **planner → planner_critique (CoVe) → comprehension_gate → executor_fanout (n=4; each branch composed of DenyRuleWrapper / ToolExecutor / LedgerEmitter) → pivot (≤15) → quorum → replan (≤3) → unverifiable_finalize → finalize**. +8-node LangGraph state machine: **planner → planner_critique (CoVe) → comprehension_gate → executor_fanout (n=4; each branch composed of DenyRuleWrapper / ToolExecutor / LedgerEmitter) → pivot (≤15) → quorum → replan (≤3) → finalize**. (`unverifiable_finalize_node` is a helper called from `replan_node` after budget exhaustion; it is not a registered graph node.) Three operational modes, auto-detected at `case_init` and **locked**: @@ -197,39 +200,43 @@ Three-layer immutability defense: (1) Claude `PreToolUse` hook (best-effort, per ## 5. Tech stack (one line each) -Python 3.11 (`uv` / `pytest` / `ruff`); Rust 1.88 for FastMCP 3.x; Node 20 (pnpm, deferred v2). Inference: SGLang primary, vLLM fallback; models Qwen3-30B-A3B-Thinking-2507 (Apache-2.0) + GLM-4.5-Air (MIT, verifier only). Orchestration: LangGraph + SqliteSaver (WAL+fsync). Schemas: Pydantic v2 + Pydantic-AI. Sandbox: Microsandbox (libkrun, ~200 ms cold). Hashing: blake3. Observability: Langfuse v2 self-host + OpenLLMetry. Eval: Inspect AI. CI: GitHub Actions. +Python 3.11 (`uv` / `pytest` / `ruff`); Rust 1.88 for FastMCP 3.x; Node 20 (pnpm, deferred v2). Inference: SGLang primary, vLLM fallback; models Qwen3-30B-A3B-Thinking-2507 (Apache-2.0) + GLM-4.5-Air (MIT, verifier only). Orchestration: LangGraph + SqliteSaver (WAL+fsync). Schemas: Pydantic v2. Sandbox: Microsandbox (libkrun, ~200 ms cold). Hashing: blake3. Observability: Langfuse v2 self-host + OpenLLMetry. Eval: Inspect AI. CI: GitHub Actions. → Full version pins, license notes, hard-NO list: **`docs/ARCHITECTURE.md` §7** (and §3.8 above for forbidden deps). ## 6. Target repo layout -Today the workspace holds docs only — code scaffolding is **W1.A** of `docs/BUILD_PLAN.md`. Active layout: +Active layout (W1 scaffolding is live under `src/verdict/`): ``` Verdict/ ├── CLAUDE.md README.md CONTRIBUTING.md SECURITY.md LICENSE .env.example ├── docs/ │ ├── ARCHITECTURE.md BUILD_PLAN.md DEVPOST_COMPLIANCE.md DOCS_ACCURACY_REPORT.md -│ └── spec/ ← frozen audit archive (01..05 + README) +│ ├── FAILURE_MODES.md CASE_ISOLATION.md DFIR_MEMORY.md RELEASE.md TLDR.md +│ ├── AGENT_SWARM.md MCP_FRAMEWORK.md SKILLS_FRAMEWORK.md SKILLS_LICENSE_AUDIT.md +│ ├── AGENTIC_WORKFLOW_REVIEW.md README.md (+ ARCHITECTURE_DIAGRAM.svg/.mmd) +│ └── spec/ ← frozen audit archive (01..04 + README) ├── downloads/ ← SIFT OVA, evidence samples (gitignored) └── protocol-sift/ ← upstream submodule ``` -Code surface to land in W1+ (target ~140 files; full tree in `docs/BUILD_PLAN.md` §file-layout): +Code surface (target ~140 files; full tree in `docs/BUILD_PLAN.md` §file-layout): ``` -verdict/ +src/verdict/ ├── runtime/ schemas/ verification/ planning/ ├── playbooks/ knowledge/ graph/wrappers/ tools/vol3/ ├── sandboxes/ ledger/ observability/ cli/ adapters/ -verdict-skills/ tests/{schemas,graph,tools,chaos,smoke,e2e,…} +├── memory/ proof/ +tests/{schemas,graph,tools,chaos,smoke,e2e,…} inspect_ai/{tasks,scorers,ground_truth/case_00{1..3}_*} scripts/ .github/workflows/ packer/ ``` ## 7. Forensic doctrine (one-paragraph summaries) -The SANS-canonical knowledge an agent must internalise. Encoded in `verdict/playbooks/`, `verdict/knowledge/`, `verdict/prompts/` — never duplicated in narrative code comments. Full discipline (with rationale, validators, schema field references): **`docs/ARCHITECTURE.md` §4**. +The SANS-canonical knowledge an agent must internalise. Encoded in `src/verdict/playbooks/`, `src/verdict/knowledge/`, `src/verdict/planning/prompts/` — never duplicated in narrative code comments. Full discipline (with rationale, validators, schema field references): **`docs/ARCHITECTURE.md` §4**. - **Canonical first moves.** Memory → `windows.info`. Disk → `image_hash_verify` → `mmls` → `fsstat`. Triage zip → registry hives first. - **DKOM / T1014.** `set(psscan_pids) - set(pslist_pids)` non-empty → emit T1014 hypothesis. First-class playbook rule (v4.6 F4), not a prompt suggestion. @@ -241,7 +248,7 @@ The SANS-canonical knowledge an agent must internalise. Encoded in `verdict/play ## 8. Verifier strategies (one-line each) -`verdict/verification/strategy.py` — `VerifierStrategy` Protocol; quorum dispatches per locked mode. +`src/verdict/verification/strategy.py` — `VerifierStrategy` Protocol; quorum dispatches per locked mode. - `CloudSelfConsistency` — n=3 with three blake3-keyed seeds at `temp=0.7` (v4.6 F1; **never** temp=0, that collapses to n=1). ≥ 2-of-3 → `VETTED_CLOUD`. - `AirGapCrossEngine` — Qwen3 + GLM-4.5-Air both execute. Jaccard ≥ 0.80 on `artifact_paths` AND identical `mitre_technique` → `VETTED_AIRGAP`. @@ -254,7 +261,7 @@ The SANS-canonical knowledge an agent must internalise. Encoded in `verdict/play ## 9. Ledger discipline -`verdict/ledger/writer.py` + `chain.py`. The ledger is the chain-of-custody artifact a SANS judge will scrutinise. +`src/verdict/ledger/writer.py`. The ledger is the chain-of-custody artifact a SANS judge will scrutinise. - JSONL append-only at `cases//ledger.jsonl`. `prev_entry_hash` chains entries; `verdict validate ` walks them. - Three-tier IDs on every entry: `case_id` → `langfuse_trace_id` → `langgraph_checkpoint_id`. @@ -265,7 +272,7 @@ The SANS-canonical knowledge an agent must internalise. Encoded in `verdict/play - `write() + fsync() + verify-readback` in `LedgerEmitter`. No buffered writes. - Bidirectional cross-link with Langfuse trace tree via `trace_id`. -Event types: `case_init`, `tool_call`, `finding`, `approval`, `rejection`, `mode_lock`, `comprehension_check`, `critique_verdict`, `pivot`, `exhausted_replan`, `evidence_hash_recheck`, `sandbox_failure`, `planner_cot`. +Event types: `case_init`, `tool_call`, `finding`, `approval`, `rejection`, `mode_lock`, `comprehension_check`, `critique_verdict`, `pivot`, `exhausted_replan`, `evidence_hash_recheck`, `sandbox_failure`, `planner_cot`, `case_conclusion`. → Schema fields, validation flow, key-management decisions: **`docs/ARCHITECTURE.md` §5**. @@ -275,7 +282,7 @@ Event types: `case_init`, `tool_call`, `finding`, `approval`, `rejection`, `mode ```bash # Microsandbox -curl -sSL https://get.microsandbox.dev | sh +curl -fsSL https://install.microsandbox.dev | sh # Bootstrap (three-credential-path detection) bash scripts/install.sh @@ -290,25 +297,28 @@ sglang_server_v1 --model-path /path/to/qwen3 --tool-call-parser qwen --port 3000 sglang_server_v1 --model-path /path/to/glm-4.5-air --tool-call-parser glm --port 30001 # Langfuse v2 self-host -docker-compose up -d +docker-compose -f infra/langfuse/docker-compose.yml up -d curl http://localhost:3000/api/public/health # expect 200 ``` ### 10.2 CLI surface ```bash -verdict doctor # pre-flight: API, SGLang, microsandbox, Langfuse, HMAC key +verdict doctor # pre-flight: API, SGLang, microsandbox, HMAC key verdict mode # show detected + locked mode verdict init [--mode {cloud,airgap,dual}] +verdict run-tool # run a single registered SIFT tool +verdict run-case # run the canonical triage sequence verdict resume verdict reverify --mode dual # parallel re-run; non-mutating -verdict status +verdict status verdict ls verdict show -verdict export [--format {json,csv,sigtools_triage}] +verdict export [--format {jsonl,execution-logs,html}] verdict validate # ledger chain + HMAC integrity -verdict approve # HMAC-signed approval w/ timestamp -verdict gc # garbage-collect old cases +verdict approve --approver # HMAC-signed approval w/ timestamp +verdict gc # list cases eligible for cleanup +verdict package-check [--output ] # validate + optionally zip Devpost artifacts verdict health ``` @@ -316,23 +326,23 @@ verdict health ```bash # Schema/playbook/knowledge gates (W1) -uv run --directory services/agent pytest tests/schemas/ -v +uv run pytest tests/schemas/ -v uv run pytest tests/playbooks/ -v uv run pytest tests/knowledge/ -v -# Tool wrapper tests +# Tool wrapper tests (lands at W1.E.1.a) uv run pytest tests/tools/test_vol_psscan.py -v # Smoke tests (xfail markers expected) pytest tests/smoke/ -# Chaos (kill-9 resume — must be 100/100 zero-loss) +# Chaos (kill-9 resume — must be 100/100 zero-loss; lands at W3.E.6.a) pytest tests/chaos/test_kill_9_resume.py -v -# Inference smoke (tool-call parse rate ≥ 98%) +# Inference smoke (tool-call parse rate ≥ 98%; lands at W3.A) python scripts/inference-smoke.py -# Full per-service run +# Full per-service run (lands with scripts/run-all-tests.sh) bash scripts/run-all-tests.sh # Per-mode end-to-end evals (THIS IS THE TEST SURFACE — see §3.10). @@ -348,8 +358,8 @@ CI hard gate: hallucination rate ≤10% in every mode by end of week 4, else fre ### 10.4 Package + submit ```bash -bash scripts/package-devpost.sh # → dist/verdict-devpost-v1.zip -git tag v-submit && git push origin v-submit # fires .github/workflows/devpost-submit.yml +bash scripts/package-devpost.sh --output dist/verdict-devpost-v1.zip +git tag v-submit && git push origin v-submit ``` ## 11. Submission deliverables @@ -382,14 +392,14 @@ Encoded in `docs/RELEASE.md`. Every item must demonstrably pass in the demo reco - **3:00 – 4:00** dual mode (new case, mode-locked): three-way verification. - **4:00 – 5:00** architecture recap + accuracy tables. -### 11.3 Eight submission docs +### 11.3 Six submission docs 1. `README.md` — problem statement, architecture, demo link, install, mode reference, license, contributing. 2. `docs/ARCHITECTURE.md` 3. `docs/RELEASE.md` — exact build steps from a fresh SIFT VM, verified on a second VM, plus scope, CLI, demo, judge checklist, dataset, accuracy, production-audit, and novelty sections. 4. `CONTRIBUTING.md` + `LICENSE` (MIT) 5. `docs/DEVPOST_COMPLIANCE.md` — rule-to-artifact mapping and submission checklist. -6. `docs/RELEASE.md` — v4 triage, judge checklist, per-mode hallucination rate, executor agreement, findings precision/recall, sub-technique precision, negative-hypothesis quality, step efficiency, contested-resolution rate, Qwen3-vs-GLM disagreement-correlation across 50 findings, and 5-min sequence with timing per beat. +6. `docs/FAILURE_MODES.md` + `docs/CASE_ISOLATION.md` — runtime failure matrix and case/chain/checkpoint isolation boundaries (both required by `verdict package-check`). ## 12. Pointers (read directly, do not summarise from memory) @@ -408,6 +418,9 @@ Encoded in `docs/RELEASE.md`. Every item must demonstrably pass in the demo reco | MCP server allowlist + credential isolation | `docs/MCP_FRAMEWORK.md` (allowlist in `.mcp.json`) | | Vendored skill stack composition + house-rules overlay | `docs/SKILLS_FRAMEWORK.md` (stack in `.claude/skills/`) | | License audit for vendored skills/hooks/MCPs | `docs/SKILLS_LICENSE_AUDIT.md` | +| Runtime failure modes, graceful degradation, UNVERIFIABLE semantics | `docs/FAILURE_MODES.md` | +| Case/chain/checkpoint isolation, reverify, export, approve contracts | `docs/CASE_ISOLATION.md` | +| Evidence-first memory model, mutation rules, memory layers | `docs/DFIR_MEMORY.md` | | Audit-history rationale ("why was X decided?") | `docs/spec/` (`01..04` + `README.md`) | | Official hackathon rules (scraped) | `docs/hackathon/RULES.md` | | Hackathon overview + resource links | `docs/hackathon/OVERVIEW.md` + https://findevil.devpost.com/ + `downloads/README.md` | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e65835d..5e1a89c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -6,7 +6,7 @@ New-contributor setup guide. If you've already done the SANS Find Evil! 2026 tea **Default branch:** `main` **License:** MIT **Deadline gate:** 2026-06-15 22:45 CDT (team-internal target: 2026-06-14 EOD = ~28 h buffer) -**Recommended platform:** SANS **SIFT Workstation VM** (canonical — all forensic tools, Microsandbox, SGLang, evidence mounts work out-of-box) **or** any modern **Linux box** (Ubuntu 22.04+ / Debian 12+ / Fedora 39+ / Arch). **macOS host is acceptable** for schema/planner/MCP/unit-test work that doesn't shell out to forensic tools — but the moment your task touches `verdict/sandboxes/`, `verdict/tools/vol3/`, or anything that runs in Microsandbox, switch into the SIFT VM. **Windows host is not supported** for development; use WSL2 + Ubuntu or pull the SIFT VM (it runs under Hyper-V / VMware / VirtualBox). +**Recommended platform:** SANS **SIFT Workstation VM** (canonical — all forensic tools, Microsandbox, SGLang, evidence mounts work out-of-box) **or** any modern **Linux box** (Ubuntu 22.04+ / Debian 12+ / Fedora 39+ / Arch). **macOS host is acceptable** for schema/planner/MCP/unit-test work that doesn't shell out to forensic tools — but the moment your task touches `src/verdict/sandboxes/`, `src/verdict/tools/`, or anything that runs in Microsandbox, switch into the SIFT VM. **Windows host is not supported** for development; use WSL2 + Ubuntu or pull the SIFT VM (it runs under Hyper-V / VMware / VirtualBox). Authority chain when docs disagree: Devpost rules → `DEVPOST_COMPLIANCE.md` → `ARCHITECTURE.md` → `BUILD_PLAN.md` → this file. Code + lockfiles win over docs (per `CLAUDE.md`); update the doc, don't roll back the code, unless the code is wrong. @@ -16,7 +16,7 @@ Authority chain when docs disagree: Devpost rules → `DEVPOST_COMPLIANCE.md` > > The minimum sweep: > -> - Did you add/rename/remove a file under `verdict/`? Grep `docs/` + `CLAUDE.md` + `README.md` for the old path; update every reference. +> - Did you add/rename/remove a file under `src/verdict/`? Grep `docs/` + `CLAUDE.md` + `README.md` for the old path; update every reference. > - Did you add/change/remove a CLI command, flag, or env var? Update `CLAUDE.md` §10 and `README.md` "CLI surface". > - Did you add/change/remove a schema field, validator, enum member, or playbook rule? Update `docs/ARCHITECTURE.md` §4 and any `CLAUDE.md` §3.x rule that cites it. > - Did you add/remove a dependency, change a version pin, or vendor a skill/MCP? Update `CONTRIBUTING.md` §2 toolchain table + `docs/RELEASE.md` (deps) or `docs/SKILLS_FRAMEWORK.md` / `docs/MCP_FRAMEWORK.md` + license audit (vendored). @@ -118,13 +118,13 @@ Idempotent. Installs uv + Python 3.11, Rust 1.88, Node 20 + pnpm, and Microsandb **Where to develop.** Three supported configurations, in order of preference: -1. **Inside the SIFT VM** (canonical, recommended for everyone). All forensic tools, Microsandbox, SGLang, and the evidence mounts resolve here without setup. **Required** for any work touching the executor branches, the Microsandbox layer, evidence I/O, or anything under `verdict/sandboxes/`, `verdict/tools/vol3/`, or `services/mcp/src/tools/`. Pull the OVA from `downloads/README.md`; snapshot it as `clean-install` before installing anything. +1. **Inside the SIFT VM** (canonical, recommended for everyone). All forensic tools, Microsandbox, SGLang, and the evidence mounts resolve here without setup. **Required** for any work touching the executor branches, the Microsandbox layer, evidence I/O, or anything under `src/verdict/sandboxes/` or `src/verdict/tools/`. Pull the OVA from `downloads/README.md`; snapshot it as `clean-install` before installing anything. 2. **A modern Linux box** (Ubuntu 22.04+ / Debian 12+ / Fedora 39+ / Arch). Acceptable for the full stack as long as you can install Microsandbox (Linux-only) and run the SIFT toolchain (`apt install sleuthkit volatility ...`). Faster I/O than the VM. Take a `pre-verdict` snapshot of your home before installing forensic tools — they leave state. 3. **macOS host with the SIFT VM as a runtime target** (faster edit loop, smaller surface). Use VS Code Remote-SSH or `Develop on a Container` into the VM. Acceptable for schema, planner, MCP gateway, and unit-test work that doesn't shell out to forensic tools or Microsandbox. The moment your task touches a Microsandbox path or a forensic CLI, switch into the VM. **Windows host is not supported** for development. Either use WSL2 + Ubuntu (treat as configuration #2) or run the SIFT VM under Hyper-V / VMware / VirtualBox (configuration #1). Native Windows breaks Microsandbox (libkrun is Linux-only), pre-commit hooks (path/exec semantics), and several SIFT tools. -If your work touches `services/mcp/src/tools/`, `services/agent_mcp/`, or anything under a Microsandbox path, you **must** run integration tests inside the SIFT VM (or your Linux box with Microsandbox installed) before opening a PR. macOS-only test runs do not satisfy the gate. +If your work touches `src/verdict/tools/`, `src/verdict/sandboxes/`, or anything under a Microsandbox path, you **must** run integration tests inside the SIFT VM (or your Linux box with Microsandbox installed) before opening a PR. macOS-only test runs do not satisfy the gate. --- @@ -173,8 +173,8 @@ test -f pnpm-lock.yaml && pnpm install --frozen-lockfile Sanity check: ```bash uv run pytest -q # must pass on a clean clone — if it doesn't, that's a P0 -cargo test --workspace -q -pnpm test +test -f Cargo.toml && cargo test --workspace -q +test -f pnpm-lock.yaml && pnpm test ``` If the workspace files don't exist yet (we're early in Week 1), clone is enough — the scaffold lands per `BUILD_PLAN.md` Phase W1.A. @@ -188,7 +188,7 @@ If the workspace files don't exist yet (we're early in Week 1), clone is enough This section is contributor-specific workflow on top of those rules. ### Branches -Format: `/-` — e.g. `feat/W1-B-1-artifact-class-enum`. Branch from `main`, rebase before PR, squash on merge only if the branch was a single logical task. Otherwise preserve the TDD red→green commits — they're the audit trail. +Format: `/-` — e.g. `feat/W1.B.1-artifact-class-enum`. Branch from `main`, rebase before PR, squash on merge only if the branch was a single logical task. Otherwise preserve the TDD red→green commits — they're the audit trail. ### Commit + push: use `/qc` @@ -204,7 +204,7 @@ gh pr create \ --draft \ --title "feat(scope): summary [W1.B.1]" \ --body "$(cat <<'EOF' -**Task:** [docs/BUILD_PLAN.md W1.B.1](../blob/main/docs/BUILD_PLAN.md) +**Task:** [docs/BUILD_PLAN.md W1.B.1](docs/BUILD_PLAN.md) **Mode(s):** all **Test evidence:** \`\`\` @@ -234,7 +234,7 @@ Concretely, when a verified fact lands: | Dependency added, removed, or version pinned | this `CONTRIBUTING.md` §2 + `docs/RELEASE.md` | | LangGraph node added / removed / renamed | `docs/ARCHITECTURE.md` §2 + `docs/BUILD_PLAN.md` task body | | Verifier strategy semantics changed | `docs/ARCHITECTURE.md` §1 + `CLAUDE.md` §8 | -| Caveat list changed | `CLAUDE.md` §3.3 + `docs/TLDR.md` (if cited) + `verdict/prompts/examiner_caveats.md` | +| Caveat list changed | `CLAUDE.md` §3.3 + `docs/TLDR.md` (if cited) + `src/verdict/planning/prompts/examiner_caveats.md` | | New skill or MCP vendored | `docs/SKILLS_FRAMEWORK.md` (skills) or `docs/MCP_FRAMEWORK.md` (MCPs) + audit log | | Audit-history correction (research contradicts a frozen `spec/` claim) | log in `docs/DOCS_ACCURACY_REPORT.md`. Do **not** edit `spec/`. | @@ -260,7 +260,7 @@ claude # interactive Claude Code session if you're on cloud/dual mode Air-gap operators: ask PUG for the bridged + Tesla-mode entry point. -Expected output: structured Findings citing tool-call IDs, a quorum verdict, an HMAC-signed ledger entry, and a Langfuse trace (if Langfuse is up — see `ARCHITECTURE.md` §Observability). +Expected output: structured Findings citing tool-call IDs, a quorum verdict, an HMAC-signed ledger entry, and a Langfuse trace (if Langfuse is up — see `ARCHITECTURE.md` §5). If the run fails before producing a Finding, you have a P0 environment problem. Post the trace ID + `verdict doctor` output in team chat before opening a code PR. @@ -271,7 +271,7 @@ If the run fails before producing a Finding, you have a P0 environment problem. Authority order (escalate left-to-right): 1. **NotebookLM Q&A:** https://notebooklm.google.com/notebook/f0957a60-6fb2-452b-93d4-ecd73ba47779?authuser=1 — chief location for "how does X work?" -2. **`docs/` and the audit history in `archive/`** — most "why did we choose X?" questions are answered in `archive/03-audit-v4.5.md`. +2. **`docs/` and the audit history in `docs/spec/`** — most "why did we choose X?" questions are answered in `docs/spec/03-audit-v4.5.md`. 3. **Team chat** — PUG / Beaver / Haley / KP. Use the appropriate thread; don't DM PUG for things the team should see. 4. **Devpost platform issues only:** https://help.devpost.com/ diff --git a/README.md b/README.md index ba8526d..96b78e0 100644 --- a/README.md +++ b/README.md @@ -41,7 +41,7 @@ Source code lives under `src/verdict/`. The outer `Verdict/` directory is the re │ 1. Plans the investigation │ "Where would evil hide?" │ 2. Runs forensic tools │ vol3, hayabusa, plaso, MFTECmd... │ 3. Two models cross-check │ Qwen3 vs GLM-4.5-Air - │ 4. Quorum decides verdict │ VERIFIED / CONTESTED / UNVERIFIABLE + │ 4. Quorum decides verdict │ VETTED_* / CONTESTED / UNVERIFIABLE │ 5. HMAC-signs the audit log │ Tamper-evident chain └───────────────────────────────┘ │ @@ -103,7 +103,7 @@ Roles: Tim (infra, ledger, ops), Beaver (orchestration, verifiers), Haley (infer │ ⓶ Hunt Evil masquerade (scvhost.exe parent=cmd.exe) │ ⓷ Amcache caveat acknowledged in rationale │ ⓸ Pivot in action (1 pivot, 0 replans) - │ ⓹ Disagreement → CONTESTED → replan → VERIFIED ★ (Devpost-required self-correction) + │ ⓹ Disagreement → CONTESTED → replan → VETTED_AIRGAP ★ (Devpost-required self-correction) │ ⓺ TSI tcpdump proof (key never enters VM) │ ⓻ Kill -9 + verdict resume 3:00 ─┤ DUAL MODE (60s) — three-way verification → VETTED_DUAL @@ -116,7 +116,7 @@ Roles: Tim (infra, ledger, ops), Beaver (orchestration, verifiers), Haley (infer ## Mapped to the 6 Devpost judging criteria 1. **Autonomous Execution Quality** *(tiebreaker)* — Mode-aware verifier strategy. Plan-then-Execute with planner_critique CoVe + comprehension_gate + pivot vs replan + unverifiable_finalize. Self-correction via cross-engine quorum. -2. **IR Accuracy** — Artifact-pair rule, Tier-1 caveats, MITRE sub-techniques, VETTED_CLOUD vs VERIFIED honesty, Hunt Evil masquerade, DKOM auto-detection. Five Inspect AI scorers per mode. +2. **IR Accuracy** — Artifact-pair rule, Tier-1 caveats, MITRE sub-techniques, honest VETTED_CLOUD framing (same-model self-consistency ≠ true cross-family verification), Hunt Evil masquerade, DKOM auto-detection. Five Inspect AI scorers per mode. 3. **Breadth and Depth of Analysis** — "Depth on fewer types beats shallow coverage of many" — Devpost rubric line we lean on directly. Windows-DFIR-depth-first; v2 extension points named (5th `net_executor` + `live_executor` branches). 4. **Constraint Implementation** — Three-layer immutability (PreToolUse + DenyRule + microsandbox kernel mount). HMAC ledger. TSI. Mode lock at case_init. **Architectural, not prompt.** 5. **Audit Trail Quality** — HMAC ledger ↔ Langfuse bidirectional. Three-tier ID hierarchy. Per-output-file SHA-256. Examination-environment metadata (NIST SP 800-86). @@ -130,7 +130,7 @@ Roles: Tim (infra, ledger, ops), Beaver (orchestration, verifiers), Haley (infer ``` RISK #1 — Microsandbox hits a blocker week 4+ - Likelihood: MEDIUM (pre-1.0; latest 0.1.x). Impact: HIGH (kills TSI hero shot). + Likelihood: MEDIUM (pre-1.0; pinned 0.4.x line). Impact: HIGH (kills TSI hero shot). Mitigation: Test it HARD in week 2, not week 4. RISK #2 — Case 001 doesn't disagree by end of week 4 @@ -213,7 +213,7 @@ Follow [`CONTRIBUTING.md`](CONTRIBUTING.md) Step 0–7 for the full human-at-key ### CLI surface (one-liner) ``` -verdict {doctor, mode, init, resume, reverify, status, ls, show, export, validate, approve, gc, health} +verdict {doctor, mode, init, run-tool, run-case, resume, reverify, status, ls, show, export, validate, approve, gc, package-check, health} ``` `verdict reverify --mode ` re-runs verification under a new mode and writes a *parallel* verdict chain — the original ledger is never mutated. Full CLI reference in `CLAUDE.md` §10.2; flag spec in `docs/BUILD_PLAN.md` W3.C.2. diff --git a/SECURITY.md b/SECURITY.md index efbd98c..c7de21d 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -8,12 +8,12 @@ Include: - Affected version / commit SHA - Repro steps (minimal evidence + commands) -- Threat surface (see `docs/spec/VERDICT_AUDIT_v4.4.md` — insider, prompt-injection-from-evidence, malicious-tool-output, external-attacker) +- Threat surface (see `docs/spec/02-audit-v4.4.md` — insider, prompt-injection-from-evidence, malicious-tool-output, external-attacker) - Suggested mitigation if you have one ## Scope -VERDICT is a forensic agent that touches **evidence** — disk images, memory captures, packet captures, registry hives, event logs. Even though all tool execution happens inside read-only microsandboxes (`CLAUDE.md` §4.2), evidence integrity is the highest-value asset; we treat **any** path that lets a writer reach `/evidence/` as critical. +VERDICT is a forensic agent that touches **evidence** — disk images, memory captures, packet captures, registry hives, event logs. Even though all tool execution happens inside read-only microsandboxes (`CLAUDE.md` §4), evidence integrity is the highest-value asset; we treat **any** path that lets a writer reach `/evidence/` as critical. In scope: @@ -47,23 +47,23 @@ Issues discovered by internal review and disclosed here so contributors and judg ### VERDICT-2026-001 — Layer-2 deny rule bypassable by `..`-traversal and `//`-prefix * **Severity:** High -* **Affected:** `verdict/graph/wrappers/deny_rule.py:221-248` (`_to_path_str`, `_is_under_evidence`) +* **Affected:** `src/verdict/graph/wrappers/deny_rule.py:41-43` (`_deny_evidence_output`) * **Discovered:** 2026-05-02 (internal security review of `feat/W2.C.4-compose-executor-work`) * **Status:** Open — fix tracked under W2.C.1.b (deny-rule normalization hardening) * **Scope mapping:** "Bypass of the three-layer immutability defense" (in-scope §) -`_to_path_str` uses `pathlib.PurePosixPath(value)`, which deliberately does **not** resolve `..` segments or collapse leading `//`. The deny check then compares the un-normalized string against `"/evidence/"` via prefix match. Inputs like `/work/../evidence/out.txt`, `/tmp/../evidence/out.txt`, and `//evidence/out.txt` slip past Layer 2 even though the kernel resolves them to `/evidence/out.txt` at syscall time. Layer 3 (read-only mount + `noexec` + host `chattr +i`) is the actual write blocker in correctly-configured deployments, but `CLAUDE.md` §3.1 designates Layer 2 as the architectural guarantee that fires in all three modes — defense-in-depth must hold even if Layer 3 is degraded. +`_deny_evidence_output` checks `path == "/evidence" or path.startswith("/evidence/")` using a plain string prefix match without normalising `..` segments or collapsing leading `//`. Inputs like `/work/../evidence/out.txt`, `/tmp/../evidence/out.txt`, and `//evidence/out.txt` slip past Layer 2 even though the kernel resolves them to `/evidence/out.txt` at syscall time. Layer 3 (read-only mount + `noexec` + host `chattr +i`) is the actual write blocker in correctly-configured deployments, but `CLAUDE.md` §3.1 designates Layer 2 as the architectural guarantee that fires in all three modes — defense-in-depth must hold even if Layer 3 is degraded. -**Remediation:** replace `PurePosixPath` with `os.path.normpath` (lexical `..` collapse) plus an explicit double-slash strip, then compare via `pathlib.PurePosixPath` parents rather than string prefix. Add RED tests for `..` traversal, `//`-prefix, NUL injection, and symlink-style siblings of `/evidence`. +**Remediation:** replace the plain `startswith` check with `os.path.normpath` (lexical `..` collapse) plus an explicit double-slash strip, then compare via `pathlib.PurePosixPath` parents rather than string prefix. Add RED tests for `..` traversal, `//`-prefix, NUL injection, and symlink-style siblings of `/evidence`. ### VERDICT-2026-002 — TPM HMAC silently truncates ledger message to 1024 bytes * **Severity:** High -* **Affected:** `verdict/ledger/hmac_key.py:134` (`_TPMHMACProvider.sign` and the symmetric `verify`) +* **Affected:** `src/verdict/ledger/hmac_key.py:9-11` (TPM guard in `load_or_create_hmac_key`) * **Discovered:** 2026-05-02 (internal security review of `feat/W2.C.4-compose-executor-work`) -* **Status:** Open — fix tracked under W2.C.3.b (TPM HMAC sequencing) +* **Status:** Open — fix tracked under W2.C.3.b (TPM HMAC sequencing); `_TPMHMACProvider` not yet implemented * **Scope mapping:** "HMAC ledger forgery / chain-of-custody breakage" (in-scope §) -`_TPMHMACProvider.sign()` truncates its message argument with `TPM2B_MAX_BUFFER(message[:1024])` and returns the digest as if it covered the full input. There is no length guard, no error on overflow, no chunked path via `TPM2_HMAC_Start` / `TPM2_SequenceUpdate` / `TPM2_SequenceComplete`. `LedgerWriter._compute_payload_hash` (writer.py:73-81) appends `prev_entry_hash` and `entry_id` **after** the JSON payload — for any tool-call entry whose serialized payload exceeds ~960 bytes (the steady state once `langfuse_trace_id`, `output_files_sha256`, `parse_warnings`, and the NIST SP 800-86 metadata are populated), the chain-linkage bytes fall past the truncation window and are not authenticated at all. An attacker with write access to `cases//ledger.jsonl` can rewrite `prev_entry_hash` / `entry_id` (or splice forged entries) while `verdict validate` still reports the chain as intact in the TPM configuration. Software (`hmac.HMAC`) and gpg-derived paths are unaffected. +`_TPMHMACProvider` was not implemented in the current codebase. The TPM path in `load_or_create_hmac_key` raises `RuntimeError("TPM-backed HMAC key path is not implemented on this host yet")`, so the specific truncation vulnerability described in the original report does not currently exist. However, when TPM HMAC IS implemented, the same risk applies: signing code must not truncate the ledger entry message silently. `LedgerWriter._compute_payload_hash` appends `prev_entry_hash` and `entry_id` **after** the JSON payload — any TPM `TPM2B_MAX_BUFFER` truncation that falls short of those fields leaves chain-linkage bytes unauthenticated. Software (`hmac.HMAC`) and gpg-derived paths are unaffected. -**Remediation:** raise an explicit `HMACMessageTooLargeError` for inputs > `TPM2B_MAX_BUFFER` until sequenced-HMAC is implemented; then implement `TPM2_HMAC_Start` / `SequenceUpdate` / `SequenceComplete` chunking. Mirror the change in `verify`. Add a unit test (the §3.10 single-system-boundary mock exception applies at the `tpm2_pytss` boundary) that signs a ≥ 4 KB message and asserts that two messages differing only in bytes after position 1024 produce different signatures. +**Remediation:** when implementing `_TPMHMACProvider`, raise an explicit `HMACMessageTooLargeError` for inputs > `TPM2B_MAX_BUFFER` until sequenced-HMAC is implemented; then implement `TPM2_HMAC_Start` / `SequenceUpdate` / `SequenceComplete` chunking. Mirror in `verify`. Add a unit test (the §3.10 single-system-boundary mock exception applies at the `tpm2_pytss` boundary) that signs a ≥ 4 KB message and asserts that two messages differing only in bytes after position 1024 produce different signatures. diff --git a/docs/AGENTIC_WORKFLOW_REVIEW.md b/docs/AGENTIC_WORKFLOW_REVIEW.md index dab0705..da30d06 100644 --- a/docs/AGENTIC_WORKFLOW_REVIEW.md +++ b/docs/AGENTIC_WORKFLOW_REVIEW.md @@ -87,6 +87,8 @@ Finding IDs are prefixed `R*` (runtime workflow) or `D*` (development workflow) **Fix:** Add task to `BUILD_PLAN.md` `W1.A.7` (pre-commit setup): write a ~40-LOC custom AST hook (`scripts/check_no_mocks.py`) that walks all `.py` files under `verdict/` and `tests/` and rejects any of: `import unittest.mock`, `from unittest import mock`, `import responses`, `import vcr`, `import betamax`, `import httpx_mock`, regex `if .*(MOCK|TEST_MODE).*:`, regex `os\.environ\.get\(['"]VERDICT_TEST`. Wire as a `pre-commit` local hook. +**Fix status:** `scripts/check_no_mocks.py` exists and is wired as the `check-no-mocks` hook in `.pre-commit-config.yaml`, scanning `src/verdict tests scripts swarm`. + --- ## HIGH — must fix before W2.B (LangGraph compile) @@ -133,7 +135,7 @@ Finding IDs are prefixed `R*` (runtime workflow) or `D*` (development workflow) **Claim:** "PIVOT (cheap, `pivot_max=15`): single Hypothesis added on basis of an executor's output. Re-enters `executor_work` only." -**Reality:** When `pivot_node` adds one hypothesis to `InvestigationPlan.hypotheses` and re-enters `executor_fanout`, does the fanout run on (a) all hypotheses including the previous round's, or (b) only the newly added one? Reducer dedup behavior on `case.findings` isn't defined either — same hypothesis run twice could yield two near-identical Finding rows. +**Reality:** When `pivot_node` adds one hypothesis to `InvestigationPlan.positive_hypotheses` and re-enters `executor_fanout`, does the fanout run on (a) all hypotheses including the previous round's, or (b) only the newly added one? Reducer dedup behavior on `case.findings` isn't defined either — same hypothesis run twice could yield two near-identical Finding rows. **Fix:** Spec in ARCH §2 Pivot subsection: pivot re-runs the 4 executor branches against the **single new hypothesis only** (not the full hypothesis list). The fanout reducer **appends** results to `case.findings` without deduplication; downstream `quorum_node` does the per-hypothesis grouping. State invariant: after N pivots, `len(case.findings) ≈ 4 × (initial_hypotheses + N)`, modulo branch timeouts. @@ -185,6 +187,8 @@ Finding IDs are prefixed `R*` (runtime workflow) or `D*` (development workflow) **Fix:** Pull `.pre-commit-config.yaml` creation into the W1.A.7 acceptance criteria (today it's implicit). Bare-minimum config: (a) `commitizen check` against a regex requiring `^(feat|fix|test|chore|docs|refactor)\(\w+\): .* \[W\d+\.[A-Z]\.\d+(\.[a-z])?\]$`; (b) `ruff check --select ALL`; (c) the `scripts/check_no_mocks.py` AST hook from D1; (d) `cargo fmt --check`. Once the file lands, drop the `test -f` short-circuit in `CONTRIBUTING.md` line 140. +**Fix status:** `.pre-commit-config.yaml` now exists at the repo root with `verdict-commit-msg`, `check-no-mocks`, and `ruff-check` local hooks. `CONTRIBUTING.md` no longer guards the `pre-commit install` call with `test -f`. + --- ## MEDIUM — clarify before judge demo @@ -247,6 +251,8 @@ Finding IDs are prefixed `R*` (runtime workflow) or `D*` (development workflow) **Fix:** Stage a real workflow `.github/workflows/eval-hallucination-gate.yml` in W1.A.9 that runs `inspect eval inspect_ai/tasks/verdict_eval_cloud.py --score hallucination_rate` once `verdict doctor --mode cloud` succeeds. Until the real scorer exists in W4.D.1, the job must fail closed with `scorer_not_implemented` rather than returning a passing score. W1-W3 may publish an advisory ≤10% trend report; W4+ release gates are hard ≤5%. +**Fix status:** `.github/workflows/eval-hallucination-gate.yml` now exists (landed in W1.A.9). The file runs `inspect eval` against the three per-mode eval tasks and fails closed when the scorer is not yet implemented. + --- ## LOW — verified accurate (no action) @@ -269,16 +275,16 @@ The following workflow claims were checked and hold up: ## Summary punchlist (priority order) 1. **R1 — RESOLVED** — `VerdictStatus` enum cascade across 4 files now uses CLAUDE.md's six-value list as canonical; engine-quorum-vs-case-verdict distinction lives in CLAUDE.md §3.6. -2. **R2 + R3** — Add a quorum dispatch table to ARCH §1 covering all three strategies; add empty-set-is-DISAGREEMENT rule. -3. **R4** — Spec `ModeLockedError` exit-code + stderr in CLAUDE.md §3.4. -4. **R5** — Add `max_clarify_iterations=2` to ARCH §2 comprehension_gate. -5. **R7** — Spec pivot state-merge: 4 branches × 1 new hypothesis, append to `case.findings`, no dedup. -6. **R9** — Add `_unverifiable_relaxes_corroboration` validator branch + `failure_reason` field; cite from ARCH §6. -7. **D1 + D3** — Pull `.pre-commit-config.yaml` into W1.A.7 acceptance; add `scripts/check_no_mocks.py` AST hook task. -8. **D2** — BUILD_PLAN sweep: every `*.a` "Failing test" subtask names the literal RED assertion; add RED-line policy to BUILD_PLAN intro. -9. **R12** — Caveat-trigger-keying note at top of CLAUDE.md §3.3 table. -10. **R13** — Sub-technique-applies-to-negatives sentence appended to CLAUDE.md §3.5. -11. **D4** — Stage `.github/workflows/eval-hallucination-gate.yml` fail-closed task in W1.A.9. +2. **R2 + R3 — RESOLVED** — Quorum dispatch table added to `ARCHITECTURE.md` §1; empty-set-is-DISAGREEMENT rule added at the bottom of the dispatch table. +3. **R4 — RESOLVED** — `ModeLockedError` exit-code + stderr message specced in `CLAUDE.md` §3.4. +4. **R5 — RESOLVED** — `max_clarify_iterations=2` and exhaustion-to-CONTESTED path added to `ARCHITECTURE.md` §2 comprehension-gate clarify budget subsection. +5. **R7 — RESOLVED** — Pivot state-merge contract (4 branches × 1 new hypothesis, append to `case.findings`, no dedup, state invariant) added to `ARCHITECTURE.md` §2. +6. **R9 — RESOLVED** — `_unverifiable_relaxes_corroboration` validator branch + `failure_reason` field added to `ARCHITECTURE.md` §6 tool-call argument validation subsection. +7. **D1 + D3 — RESOLVED** — `scripts/check_no_mocks.py` exists and is wired as `check-no-mocks` hook in `.pre-commit-config.yaml`; `CONTRIBUTING.md` no longer guards `pre-commit install` with `test -f`. +8. **D2 — RESOLVED** — RED-line policy added to `BUILD_PLAN.md` intro; every `*.a` subtask now names a test path and the literal failing assertion. +9. **R12 — RESOLVED** — Caveat-trigger-keying note (artifact_classes membership; `LOGON_TYPE_3_VS_10` named exception) added at top of `CLAUDE.md` §3.3 table. +10. **R13 — RESOLVED** — Sub-technique-precision-applies-to-negatives sentence added to `CLAUDE.md` §3.5. +11. **D4 — RESOLVED** — `.github/workflows/eval-hallucination-gate.yml` now exists (W1.A.9); see Fix status note above. 12. **R6, R10, R11, R8 — RESOLVED** — `docs/FAILURE_MODES.md` covers branch timeout, sandbox spawn failure, and TSI proxy failure; `docs/CASE_ISOLATION.md` covers reverify chain semantics. -**Estimated fix effort:** ~75 minutes for items 1–11 (in-place edits to 5 existing docs); items 12 are next-turn doc creation already scoped. +All punchlist items are resolved. No open items remain from this audit pass. diff --git a/docs/AGENT_SWARM.md b/docs/AGENT_SWARM.md index cd6057a..aa32dd0 100644 --- a/docs/AGENT_SWARM.md +++ b/docs/AGENT_SWARM.md @@ -38,7 +38,7 @@ A subagent-driven fallback path lives in `scripts/run-swarm.sh` for cases where | Model — `schema/sandbox/tool-wrapper/eval-engineer` | `claude-sonnet-4-6` | Bulk implementation work; mechanical with judgment. ~5× cheaper than Opus, fits §14 budget. | | Model — reviewer (TaskCompleted hook) | `claude-sonnet-4-6` | Mechanical pass/fail on lint/test output. Runs as a hook script, not a teammate. | | Model — auditor (TaskCompleted hook) | `claude-haiku-4-5` | Pattern-match scan over a diff. Cheapest tier. Runs as a hook script, not a teammate. | -| Storage (cross-launch state) | SQLite WAL + fsync (`swarm/state.db`) | Same discipline as the runtime ledger (CLAUDE.md §9). Agent Teams' shared task list lives at `~/.claude/tasks//` and does NOT survive cleanup; persistent state belongs in `swarm/state.db`. | +| Storage (cross-launch state) | SQLite WAL + fsync (`swarm/swarm.db`) | Same discipline as the runtime ledger (CLAUDE.md §9). Agent Teams' shared task list lives at `~/.claude/tasks//` and does NOT survive cleanup; persistent state belongs in `swarm/swarm.db`. | | Auth | `CLAUDE_CODE_OAUTH_TOKEN` (Pro/Max subscription) | Each teammate is a full Claude Code session inheriting the lead's auth. `ANTHROPIC_API_KEY` is the direct API fallback; `OPENROUTER_API_KEY` is an optional host-side fallback for build-side AI agents if direct Anthropic quota is constrained. | | Per-task token ceiling | $20 USD | Tracked in `swarm/state.py:tasks.token_spend_usd`. Worker exceeds → exits `turn_budget_exceeded`. | @@ -263,7 +263,7 @@ Body: ```markdown ## Task -[W1.B.7](BUILD_PLAN.md#L371) +[W1.B.7](BUILD_PLAN.md#L396) ## Mode(s) affected all diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 58786a8..137d9a1 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -21,7 +21,7 @@ VERDICT detects available infrastructure at startup and selects one of three mod | Mode | Trigger | Engines | Verifier strategy | Use case | |---|---|---|---|---| | **cloud-only** | Internet ✓ + GPU ✗ | Claude Code (Agent SDK) | n=3 self-consistency at temperature=0.7 with three case_id-derived blake3 seeds. ≥2-of-3 → `VETTED_CLOUD`; below → `CONTESTED` (escalates to `replan_node`). **Best-effort vetting, not true verification** — same model shares failure modes. | SOC analyst on corporate laptop | -| **air-gap-only** | Internet ✗ + GPU ✓ | SGLang serving Qwen3-30B-A3B-Thinking + GLM-4.5-Air | Cross-family quorum: both engines must independently agree on artifact set (Jaccard ≥0.80) and identical MITRE technique. Independence is **partial-not-absolute** (overlapping web pretraining); empirical disagreement-correlation measured in W4.G.1. | DCO operator on classified network | +| **air-gap-only** | Internet ✗ + GPU ✓ | SGLang serving Qwen3-30B-A3B-Thinking-2507 + GLM-4.5-Air | Cross-family quorum: both engines must independently agree on artifact set (Jaccard ≥0.80) and identical MITRE technique. Independence is **partial-not-absolute** (overlapping web pretraining); empirical disagreement-correlation measured in W4.G.1. | DCO operator on classified network | | **dual** | Internet ✓ + GPU ✓ | Claude + Qwen3 + GLM-4.5-Air | Three-way: cloud agrees with at least one local + locals agree with each other. Strongest verification. | Forensic lab | ### Why diverse seeds matter (cloud-only) @@ -31,18 +31,12 @@ Same seed + same temperature + same prompt = three identical outputs. Wang et al ```python from blake3 import blake3 +_SEED_DERIVATION_KEY = b"VERDICT self-consistency seeds\0\0" + def derive_seeds(case_id: str) -> tuple[int, int, int]: - """Three reproducible-but-diverse seeds per case via blake3 derive_key contexts.""" - return tuple( - int.from_bytes( - blake3( - case_id.encode(), - derive_key_context=f"verdict.seeds.v1.{label}", - ).digest(length=4), - "big", - ) - for label in ("a", "b", "c") - ) + """Three reproducible-but-diverse seeds per case via keyed blake3.""" + digest = blake3(case_id.encode(), key=_SEED_DERIVATION_KEY).digest(length=12) + return tuple(int.from_bytes(digest[i : i + 4], "big") for i in range(0, 12, 4)) ``` Reproducibility-with-diversity: re-running the case yields the same three samples (audit-friendly), but the three samples differ from each other (verifier-friendly). @@ -62,7 +56,7 @@ Reproducibility-with-diversity: re-running the case yields the same three sample | `DualLaneCrossEngine` | cloud disagrees with both locals | `CONTESTED` | `replan_node` | | `DualLaneCrossEngine` | cloud agrees with 1 local, locals disagree with each other | `CONTESTED` | `replan_node` | | any | After `replan_max=3` exhaustion | `EXHAUSTED_REPLAN` | `unverifiable_finalize_node` | -| any | Tool / sandbox / args exhaustion (see §6 + `FAILURE_MODES.md`) | `UNVERIFIABLE` | `finalize_node` (with `failure_reason` set) | +| any | Tool / sandbox / args exhaustion (see §6 + `FAILURE_MODES.md`) | `UNVERIFIABLE` | `finalize_node` (`failure_reason` field planned — see §6) | **Empty-set rule:** if any quorum participant returns `parsed_artifacts=[]` (zero findings — e.g., GLM crashed silently, executor branch timed out per R6), it is treated as DISAGREEMENT for Jaccard / pair-agreement purposes. Empty-set is **never** a null vote that lets the non-empty engine win by default. Otherwise an executor that crashes silently becomes a free pass for the other lane and destroys the cross-engine guarantee. @@ -96,7 +90,7 @@ START │ │ parsed_success_criteria_hash. Mismatch │ │ → clarify sub-state (re-prompts within │ │ the same node, not a separate top-level -│ │ node — total node count stays 9). +│ │ node — total node count stays 8). └────────┬────────┘ ▼ (fanout — 4 parallel branches) ┌─────────────────┐ @@ -142,7 +136,7 @@ Real DFIR pivots 8–15 times per investigation; v4.4 research showed that bound ### Pivot state-merge contract -When `pivot_node` adds a hypothesis it appends one entry to `InvestigationPlan.hypotheses` and re-enters `executor_fanout`. The fanout runs the 4 branches against the **single new hypothesis only** (not the full hypothesis list — re-running prior hypotheses would inflate the ledger and double-count for quorum). The fanout reducer **appends** the new findings to `case.findings` with no deduplication; downstream `quorum_node` does the per-hypothesis grouping. State invariant after N pivots: `len(case.findings) ≈ 4 × (initial_hypotheses + N)`, modulo branch timeouts (see `FAILURE_MODES.md`). +When `pivot_node` adds a hypothesis it appends one entry to `InvestigationPlan.positive_hypotheses` and re-enters `executor_fanout`. The fanout runs the 4 branches against the **single new hypothesis only** (not the full hypothesis list — re-running prior hypotheses would inflate the ledger and double-count for quorum). The fanout reducer **appends** the new findings to `case.findings` with no deduplication; downstream `quorum_node` does the per-hypothesis grouping. State invariant after N pivots: `len(case.findings) ≈ 4 × (initial_hypotheses + N)`, modulo branch timeouts (see `FAILURE_MODES.md`). ### Interrupt idempotency contract @@ -189,52 +183,71 @@ The differentiator vs. competitors. Schema validators reject sloppy findings bef ### Artifact-pair corroboration ```python +AVAILABLE_CAVEAT_TRIGGERS = { + ArtifactClass.AMCACHE: CaveatID.AMCACHE_LASTMODIFIED_NOT_EXEC, + ArtifactClass.SHIMCACHE: CaveatID.SHIMCACHE_ORDER_CHANGED_WIN81, + ArtifactClass.PREFETCH: CaveatID.PREFETCH_SSD_DISABLED, + ArtifactClass.MFT: CaveatID.MFT_SI_STOMPABLE, + ArtifactClass.USNJRNL: CaveatID.USNJRNL_WRAPS, + ArtifactClass.SYSMON_1: CaveatID.SYSMON_PROCESSGUID_OVER_PID, +} + class Finding(BaseModel): artifact_paths: list[Path] = Field(min_length=2) artifact_classes: list[ArtifactClass] = Field(min_length=2) caveats_acknowledged: list[CaveatID] = [] mitre_technique: str | None # validated against ^T\d{4}(\.\d{3})?$ + evtx_4624_logon_types: list[int] = [] @model_validator(mode="after") - def _execution_claims_need_two_classes(self): - is_exec = any( - self.mitre_technique and self.mitre_technique.startswith(p) - for p in ("T1059", "T1106", "T1204", "T1218", "T1543", "T1547") + def _forensic_corroboration(self): + # Execution-class techniques need ≥2 distinct artifact classes + is_execution = self.mitre_technique and self.mitre_technique.startswith( + ("T1059", "T1106", "T1204", "T1218", "T1543", "T1547") ) - if is_exec and len(set(self.artifact_classes)) < 2: - raise ValueError(f"execution claim needs ≥2 distinct artifact classes") - return self - - @model_validator(mode="after") - def _amcache_caveat_required(self): - if ArtifactClass.AMCACHE in self.artifact_classes: - if CaveatID.AMCACHE_LASTMODIFIED_NOT_EXEC not in self.caveats_acknowledged: - raise ValueError("Finding cites Amcache without LastModified caveat") + if is_execution and len(set(self.artifact_classes)) < 2: + raise ValueError("execution claims require two distinct artifact classes") + + # Each cited artifact class requires its Tier-1 caveat to be acknowledged + acknowledged = set(self.caveats_acknowledged) + for artifact_class, required_caveat in AVAILABLE_CAVEAT_TRIGGERS.items(): + if artifact_class in self.artifact_classes and required_caveat not in acknowledged: + raise ValueError(f"{required_caveat.value} must be acknowledged") + + # EVTX_4624 logon-type caveat (named exception — not keyed by artifact class alone) + evtx_4624_cited = ArtifactClass.EVTX_4624 in self.artifact_classes + evtx_type_requires_caveat = not self.evtx_4624_logon_types or any( + t in {3, 10} for t in self.evtx_4624_logon_types + ) + if evtx_4624_cited and evtx_type_requires_caveat: + if CaveatID.LOGON_TYPE_3_VS_10 not in acknowledged: + raise ValueError(f"{CaveatID.LOGON_TYPE_3_VS_10.value} must be acknowledged") return self ``` ### CaveatID — Tier-1 examiner caveats from `CLAUDE.md` §3.3 ```python -class CaveatID(str, Enum): - AMCACHE_LASTMODIFIED_NOT_EXEC = "amcache_lastmodified_neq_execution" - SHIMCACHE_ORDER_CHANGED_WIN81 = "shimcache_order_lru_pre81_insertion_post81" - PREFETCH_SSD_DISABLED = "prefetch_disabled_on_ssd_or_gpo" - MFT_SI_STOMPABLE = "mft_si_timestomp_use_fn" - USNJRNL_WRAPS = "usnjrnl_wraps_treat_gaps_carefully" - LOGON_TYPE_3_VS_10 = "evtx_4624_type3_network_neq_type10_rdp" - SYSMON_PROCESSGUID_OVER_PID = "sysmon_processguid_correlation_key_not_pid" +class CaveatID(StrEnum): + AMCACHE_LASTMODIFIED_NOT_EXEC = "AMCACHE_LASTMODIFIED_NOT_EXEC" + SHIMCACHE_ORDER_CHANGED_WIN81 = "SHIMCACHE_ORDER_CHANGED_WIN81" + PREFETCH_SSD_DISABLED = "PREFETCH_SSD_DISABLED" + MFT_SI_STOMPABLE = "MFT_SI_STOMPABLE" + USNJRNL_WRAPS = "USNJRNL_WRAPS" + LOGON_TYPE_3_VS_10 = "LOGON_TYPE_3_VS_10" + SYSMON_PROCESSGUID_OVER_PID = "SYSMON_PROCESSGUID_OVER_PID" ``` -Loaded into every executor system prompt via `verdict/planning/prompts/examiner_caveats.md`. +Loaded into every executor system prompt via `src/verdict/planning/prompts/examiner_caveats.md`. ### ArtifactClass — multi-source corroboration vocabulary ```python -class ArtifactClass(str, Enum): +class ArtifactClass(StrEnum): PREFETCH = "prefetch" AMCACHE = "amcache" SHIMCACHE = "shimcache" + EVTX_4624 = "evtx_4624" EVTX_4688 = "evtx_4688" SYSMON_1 = "sysmon_1" NETWORK = "network" @@ -242,6 +255,7 @@ class ArtifactClass(str, Enum): TASK_SCHEDULER = "task_scheduler" WMI_SUBSCRIPTION = "wmi_subscription" MFT = "mft" + USNJRNL = "usnjrnl" PROCESS_MEMORY = "process_memory" YARA_HIT = "yara_hit" SIGMA_HIT = "sigma_hit" @@ -249,7 +263,7 @@ class ArtifactClass(str, Enum): ### Playbooks — SANS canonical tool sequencing -Three YAMLs in `verdict/playbooks/` (memory.yml / disk.yml / triage.yml) encode the SANS-canonical sequencing summarized in `CLAUDE.md` §7 and this document's forensic doctrine. Loaded into planner system prompt at case_init based on detected evidence type. +Three YAMLs in `src/verdict/playbooks/` (memory.yml / disk.yml / triage.yml) encode the SANS-canonical sequencing summarized in `CLAUDE.md` §7 and this document's forensic doctrine. Loaded into planner system prompt at case_init based on detected evidence type. `memory.yml` example rule (DKOM detection): ```yaml @@ -263,7 +277,7 @@ This is one of the architecture's clearest moats — DKOM/T1014 detection auto-f ### Hunt Evil baseline -`verdict/knowledge/hunt_evil.yml` keyed by process name with expected parent / path / signing / instance count for 8 canonical Windows processes (svchost, lsass, csrss, winlogon, services, wininit, explorer, smss). `ProcessBaselineAnomaly` Hypothesis subtype maps to `T1036.005` (Match Legitimate Name or Location). Catches `scvhost.exe` with parent `cmd.exe` automatically. +`src/verdict/knowledge/hunt_evil.yml` keyed by process name with `expected_parent_names`, `expected_path_prefixes`, and `expected_user_names` for 8 canonical Windows processes (svchost, lsass, csrss, winlogon, services, wininit, explorer, smss). `ProcessBaselineAnomaly` maps to `T1036.005` (Match Legitimate Name or Location). Catches `scvhost.exe` with parent `cmd.exe` automatically. --- @@ -273,19 +287,19 @@ This is one of the architecture's clearest moats — DKOM/T1014 detection auto-f ```python class LedgerEntry(BaseModel): - entry_id: str # ULID + entry_id: str # "{case_id}:{event_type}:{timestamp_utc}" case_id: str # ROOT — eternal finding_id: str | None event_type: Literal[ "case_init", "tool_call", "finding", "approval", "rejection", "mode_lock", "comprehension_check", "critique_verdict", "pivot", "exhausted_replan", "evidence_hash_recheck", - "sandbox_failure", "planner_cot", + "sandbox_failure", "planner_cot", "case_conclusion", ] timestamp_utc: datetime # Mode lock - mode_at_case_init: Mode + mode_at_case_init: str verifier_strategy_used: str # Langfuse cross-references @@ -299,10 +313,10 @@ class LedgerEntry(BaseModel): langgraph_checkpoint_id: str # Examination-environment metadata (NIST SP 800-86 §5.1.4) - microsandbox_version: str | None = None - rootfs_sha256: str | None = None - tool_version: str | None = None - kernel_version: str | None = None + microsandbox_version: str + rootfs_sha256: str + tool_version: str + kernel_version: str # Per-output-file hashes (NIST SP 800-86 §5.1.2) output_files_sha256: dict[str, str] = {} @@ -310,7 +324,7 @@ class LedgerEntry(BaseModel): # Ledger chain integrity payload: dict payload_redactions: list[str] = [] - prev_entry_hash: str + prev_entry_hash: str | None # None for the first entry in a case hmac_sig: str schema_version: int = 1 ``` @@ -329,11 +343,11 @@ Every 10 super-steps, re-hash all `EvidenceItem` files against the manifest. Mis --- -## 6. Tool surface (12 SIFT tools, 23 wrappers) +## 6. Tool surface (12 SIFT tools, 24 wrappers) | Tool family | Wrappers | |---|---| -| Volatility 3 | `windows.{pslist,psscan,pstree,cmdline,dlllist,malfind,netscan,svcscan,handles,callbacks}` (10 typed plugin wrappers; `windows.info` is invoked through the generic vol3 allow-list — see §6 *Tool-call argument validation*) | +| Volatility 3 | `windows.{info,pslist,psscan,pstree,cmdline,dlllist,malfind,netscan,svcscan,handles,callbacks}` (11 typed plugin wrappers; `vol3.info` is registered as a typed `ExternalToolSpec` in `src/verdict/tools/registry.py`) | | Hayabusa | Split: `hayabusa_csv_timeline` (extract) + `hayabusa_filter` (analyst-driven filter by sigma_level + time_range) | | Plaso | Split: `plaso_extract` (log2timeline.py → .plaso) + `psort_filter` (psort.py + filter expression) | | Sleuth Kit | `mmls`, `fls`, `fsstat` | @@ -384,14 +398,14 @@ Pattern 2 intentionally allows TSI-mediated egress to a single allowlisted origi ### Tool-call argument validation -Pydantic-AI `args_validator` runs *before* `microsandbox.spawn`: +`ArgsValidator` (Pydantic v2, `src/verdict/tools/args_validators.py`) runs *before* `microsandbox.spawn`: - vol3: validate plugin against allow-list (parse `vol3 --help` once at startup, hash-pin); `--pid` is positive int; reject unknown flags. - plaso: pre-validate filter expression with `psteal --validate-filter` in ephemeral sandbox. - Hayabusa: validate timeline-flag combinations against playbook matrix. On validation failure: raise `ModelRetry`, bounded by `tool_arg_retry_max=2`, then UNVERIFIABLE. -When `tool_arg_retry_max` exhausts, the executor emits `Finding(status=UNVERIFIABLE, artifact_paths=[], caveats_acknowledged=[], failure_reason="tool_args_failed_validation_after_2_retries")`. This would normally fail the `Finding._artifact_paths_min_length=2` and execution-class corroboration validators; the schema exempts UNVERIFIABLE findings via the `_unverifiable_relaxes_corroboration` validator branch — when `Finding.status == UNVERIFIABLE` AND `Finding.failure_reason` is set, `artifact_paths` and `caveats_acknowledged` may be empty. The same exemption covers `failure_reason ∈ {sandbox_spawn_failed, tsi_proxy_unreachable, branch_timeout}` (see `FAILURE_MODES.md`). +When `tool_arg_retry_max` exhausts, the executor routes the hypothesis to `UNVERIFIABLE`. The current `Finding` schema enforces `min_length=2` on `artifact_paths` and `artifact_classes` for **all** findings; there is no current exemption for UNVERIFIABLE status. **Planned:** add `failure_reason: str | None` field to `Finding` and a `_unverifiable_relaxes_corroboration` model validator that allows empty `artifact_paths`/`caveats_acknowledged` when `status == UNVERIFIABLE` and `failure_reason` is set (values: `tool_args_failed_validation_after_2_retries`, `sandbox_spawn_failed`, `tool_execution_failed`, `tsi_proxy_unreachable`, `branch_timeout`). Until that field lands, UNVERIFIABLE findings from tool/sandbox failures must populate `artifact_paths` with whatever paths were available (see `FAILURE_MODES.md`). ### No-evil case conclusion @@ -399,7 +413,6 @@ Benign or red-herring cases do not produce a `Finding` with empty artifacts. The ```python class CaseConclusion(BaseModel): - case_id: str status: Literal["NO_EVIL_FOUND", "EVIL_FOUND", "UNVERIFIABLE"] playbook_steps_executed: list[str] = Field(min_length=1) evidence_hashes: dict[Path, str] @@ -410,7 +423,7 @@ class CaseConclusion(BaseModel): ### Sanitization for prompt injection -`verdict/tools/sanitization.py` scans tool stdout for prompt-injection patterns (`IGNORE PREVIOUS`, `SYSTEM:`, ``, `[INST]`, `### Instruction`, common jailbreak suffixes). Detected → `ToolOutput.sanitization_flags` populated; surfaced to planner. Defense against malicious memory images where attacker-controlled strings end up in `vol3.cmdline` output. +`src/verdict/tools/sanitization.py` scans tool stdout for prompt-injection patterns (`IGNORE PREVIOUS`, `SYSTEM:`, ``, `[INST]`, `### Instruction`, common jailbreak suffixes). Detected → `ToolOutput.sanitization_flags` populated; surfaced to planner. Defense against malicious memory images where attacker-controlled strings end up in `vol3.cmdline` output. --- @@ -424,7 +437,7 @@ class CaseConclusion(BaseModel): | Local Model A | Qwen3-30B-A3B-Thinking-2507 | Apache-2.0 | | Local Model B (verifier) | GLM-4.5-Air | MIT | | Orchestration | LangGraph | MIT | -| Schema layer | Pydantic v2 + Pydantic-AI | MIT | +| Schema layer | Pydantic v2 | MIT | | MCP gateway | FastMCP 3.x | Apache-2.0 | | Sandbox primary | Microsandbox (libkrun microVM; beta, verified per release) | Apache-2.0 | | Sandbox secondary | bubblewrap | LGPL-2.0 (linking-clean) | @@ -432,7 +445,6 @@ class CaseConclusion(BaseModel): | Eval harness | Inspect AI | MIT | | Tracing | Langfuse self-hosted (core) + OpenLLMetry | MIT + Apache-2.0 | | Durable execution | LangGraph SqliteSaver | MIT | -| Rails | NeMo Guardrails | Apache-2.0 | | Skills | agentskills.io standard | open standard | **Hard nos** (license-incompatible or architecturally rejected): Daytona (AGPL-3.0), REMnux MCP for vendoring (GPL-3.0; network-call only allowed), Llama 4 / Gemma 3 (community licenses, not OSI), Modal (closed), LangSmith / Braintrust (closed), Arize Phoenix (ELv2), AutoGen v0.4 migration (maintenance mode Oct 2025), Microsoft Agent Framework (Azure-coupled, late). AGPL clean-room rewrites do not strip copyright. @@ -510,7 +522,7 @@ These were raised during the v4.4 research and v4.5 system-design review. They'r 2. **Schema strictness vs recall tradeoff.** `Finding` validators reject sloppy findings — good for credibility, but each invariant is a place a *legitimate* finding gets rejected for the wrong reason. Mitigation: have `executor_work` *infer* `artifact_classes` deterministically from `artifact_paths` (e.g. paths matching `\Prefetch\*.pf` → `PREFETCH`) so the LLM never has to know the enum exists. Validator-as-projection rather than validator-as-gate. -3. **"No evil found" verdict is unmodeled.** `Finding.artifact_paths min_length=2` rejects null findings. Some engineered cases will be benign-with-red-herrings. Need a `NegativeFinding` schema variant citing playbook *steps executed* rather than artifact paths. Currently a v1 gap. +3. **"No evil found" verdict — resolved by `CaseConclusion` (see §6).** `Finding.artifact_paths min_length=2` still rejects empty-artifact findings; benign/red-herring cases instead produce `CaseConclusion(status="NO_EVIL_FOUND")` citing completed playbook steps and evidence hashes. No `NegativeFinding` variant is needed. 4. **Plan-then-Execute is fundamentally batch; DFIR is fundamentally adaptive.** `pivot_max=15` recreates ReAct in a more constrained form. If demo runs show 90% sequential pivots, the topology pays overhead for a parallelism that doesn't materialize. Lean the demo narrative on the *audit-trail review story* (sequential is fine; structure is what reviewability needs) rather than runtime parallelism. @@ -530,5 +542,5 @@ These were raised during the v4.4 research and v4.5 system-design review. They'r | v4.4 agentic + DFIR research findings (raw) | `docs/spec/02-audit-v4.4.md` | | v4.6 schema patches (raw spec) | `docs/spec/04-spec-plan-v4.6.md` | | Project-wide build conventions | `../CLAUDE.md` | -| Tier-1 examiner caveat source | `../CLAUDE.md` §3.3 and planned `../verdict/planning/prompts/examiner_caveats.md` | -| Tool sequencing playbook source | `../verdict/playbooks/*.yml` and `docs/ARCHITECTURE.md` §4 | +| Tier-1 examiner caveat source | `../CLAUDE.md` §3.3 and `../src/verdict/planning/prompts/examiner_caveats.md` | +| Tool sequencing playbook source | `../src/verdict/playbooks/*.yml` and `docs/ARCHITECTURE.md` §4 | diff --git a/docs/BUILD_PLAN.md b/docs/BUILD_PLAN.md index 367e4bb..227385d 100644 --- a/docs/BUILD_PLAN.md +++ b/docs/BUILD_PLAN.md @@ -35,8 +35,8 @@ Use the ID in commit messages: `feat(schema): add ArtifactClass enum [W1.B.1]`. VERDICT is a mode-aware verifier-gateway for forensic LLM agents. By June 14: 1. **Three operational modes** (cloud-only / air-gap-only / dual) auto-selected by infrastructure detection; operator overrides via `--mode={cloud,airgap,dual}`. Mode locked at `case_init`. -2. **Plan-then-Execute LangGraph topology** (9 nodes) with named nodes: `planner` → `planner_critique` (CoVe) → `comprehension_gate` → `executor_fanout` (per-branch composition: `DenyRuleWrapper → ToolExecutor → LedgerEmitter`; the composition is referred to internally as `executor_work` and is a sub-state of fanout, not a separate top-level node) → `pivot_node` → `quorum` → `replan` → `unverifiable_finalize` → `finalize`. -3. **12 SIFT tool wrappers** as MCP tools running in per-call ephemeral microsandbox VMs: `mmls`, `fls`, `fsstat`, `vol3` (10 plugins), `hayabusa` (split: csv-timeline + filter), `plaso` (split: extract + filter), `MFTECmd`, `RECmd`, `PECmd`, `bulk_extractor`, `exiftool`, `capa`. +2. **Plan-then-Execute LangGraph topology** (8 nodes) with named nodes: `planner` → `planner_critique` (CoVe) → `comprehension_gate` → `executor_fanout` (per-branch composition: `DenyRuleWrapper → ToolExecutor → LedgerEmitter`; the composition is referred to internally as `executor_work` and is a sub-state of fanout, not a separate top-level node) → `pivot_node` → `quorum` → `replan` → `finalize`. (`unverifiable_finalize_node` is a helper called from `replan_node`, not a registered graph node.) +3. **12 SIFT tool wrappers** as MCP tools running in per-call ephemeral microsandbox VMs: `mmls`, `fls`, `fsstat`, `vol3` (11 plugins), `hayabusa` (split: csv-timeline + filter), `plaso` (split: extract + filter), `MFTECmd`, `RECmd`, `PECmd`, `bulk_extractor`, `exiftool`, `capa`. 4. **Three-layer immutability defense**: Layer 1 = Claude PreToolUse hook (best-effort, version-dependent caveat per #33106/#37210); Layer 2 = LangGraph `DenyRuleWrapper` (architectural guarantee, all modes); Layer 3 = Microsandbox read-only mount (kernel-enforced). 5. **Cryptographic chain-of-custody**: HMAC-signed append-only JSONL ledger with `prev_entry_hash`, three-tier ID hierarchy (`case_id` / `langfuse_trace_id` / `langgraph_checkpoint_id`), per-output-file SHA-256, examination-environment metadata (`microsandbox_version`, `rootfs_sha256`, `tool_version`, `kernel_version`). 6. **Forensic discipline encoded in code**: `Finding.artifact_paths min_length=2`, `Finding.artifact_classes min_length=2`, `Finding.caveats_acknowledged` field with model_validators, three playbook YAMLs (memory/disk/triage), `examiner_caveats.md` system-prompt include, `hunt_evil.yml` with 8 process baselines, DKOM/T1014 detection via pslist+psscan divergence, sub-technique-aware MITRE field validation. @@ -54,11 +54,10 @@ VERDICT is a mode-aware verifier-gateway for forensic LLM agents. By June 14: | Architecture rationale | `docs/ARCHITECTURE.md` | | Schema patches + DFIR rule encoding | `docs/ARCHITECTURE.md` §4 | | Project-level conventions | `CLAUDE.md` (this repo) | -| Tier-1 examiner caveats | `CLAUDE.md` §3.3 and planned `verdict/planning/prompts/examiner_caveats.md` | -| Per-evidence-type tool sequencing | `docs/ARCHITECTURE.md` §4 and planned `verdict/playbooks/*.yml` | -| Tool surface (Rust MCP) | `services/mcp/src/tools/` | -| Tool surface (Python MCP) | `services/agent_mcp/` | -| Decision history | `CHANGELOG.md` + `git log --oneline` | +| Tier-1 examiner caveats | `CLAUDE.md` §3.3 and `src/verdict/planning/prompts/examiner_caveats.md` | +| Per-evidence-type tool sequencing | `docs/ARCHITECTURE.md` §4 and `src/verdict/playbooks/*.yml` | +| Tool surface | `src/verdict/tools/` | +| Decision history | `git log --oneline` | | Why we picked X over Y | v4.5 §"Lock-In Decisions" + v4.5 §"Per-Tool Deep Dives" | If two authorities conflict: **code + lockfiles win** over docs (per `CLAUDE.md` §"Spec/code divergences"). Update the doc rather than rolling back the code, unless the code is wrong. @@ -73,7 +72,7 @@ If two authorities conflict: **code + lockfiles win** over docs (per `CLAUDE.md` - **Local Model A:** Qwen3-30B-A3B-Thinking-2507 (Apache-2.0). Air-gap planner/executor; dual-mode local executor/verifier lane. Cloud-only remains Claude Code/Agent SDK only because the mode trigger is GPU absent. - **Local Model B (verifier):** GLM-4.5-Air (MIT). Verifier only; never planner. Cross-family verification partner for air-gap and dual modes. - **Orchestration:** LangGraph (MIT) state machine. Five core nodes per Plan-then-Execute, plus comprehension_gate (v4.3), planner_critique (v4.4 SHOULD-FIX), pivot_node (v4.4 SHOULD-FIX), unverifiable_finalize (v4.4 SHOULD-FIX). -- **Schema layer:** Pydantic v2 + Pydantic-AI (MIT) for typed tool args + `ModelRetry` flow. +- **Schema layer:** Pydantic v2 (MIT) for typed tool args + `ModelRetry` flow via `ArgsValidator`. - **MCP gateway:** FastMCP 3.x (Apache-2.0). - **Sandbox primary:** Microsandbox (Apache-2.0, beta). libkrun microVM, sub-200ms cold start, built-in MCP server, TSI for credential injection. - **Sandbox secondary:** bubblewrap (LGPL-2.0, linking-clean) for non-microVM tools. @@ -81,7 +80,6 @@ If two authorities conflict: **code + lockfiles win** over docs (per `CLAUDE.md` - **Eval harness:** Inspect AI (MIT) — UKGovernmentBEIS/inspect_ai. - **Tracing:** Langfuse self-hosted (core MIT) + OpenLLMetry (Apache-2.0). Cross-linked to JSONL ledger. - **Durable execution:** LangGraph SqliteSaver (MIT). Single-writer; reducer pattern handles fanout. -- **Rails:** NeMo Guardrails (Apache-2.0) for input/output rails. - **Skills:** agentskills.io standard (open standard) — portable across Claude Code, Hermes, Cursor, Codex. **Hard nos:** Daytona (AGPL-3.0), REMnux MCP for vendoring (GPL-3.0; network-call only allowed), Llama 4 / Gemma 3 (community licenses, not OSI), Modal (closed), LangSmith / Braintrust (closed), Arize Phoenix (ELv2). AutoGen v0.4 migration (maintenance mode Oct 2025; succeeded by Microsoft Agent Framework which is Azure-coupled and late). Microsoft Agent Framework. AGPL clean-room rewrites. @@ -97,13 +95,13 @@ By June 14 the repo will have this shape: ├── LICENSE # MIT ├── README.md ├── CONTRIBUTING.md -├── CHANGELOG.md ├── CLAUDE.md # Project conventions ├── pyproject.toml # Workspace root (uv) ├── uv.lock ├── package.json # Root for pnpm workspaces (mcp-widgets deferred V2) -├── docker-compose.yml # Langfuse v2 self-host (default) -├── docker-compose.langfuse-v3.yml # ClickHouse-backed alt for >=16GB RAM hosts +├── infra/ +│ └── langfuse/ +│ └── docker-compose.yml # Langfuse v2 self-host ├── docs/ │ ├── ARCHITECTURE.md │ ├── RELEASE.md # build, scope, CLI, demo, accuracy, dataset, novelty @@ -111,11 +109,12 @@ By June 14 the repo will have this shape: ├── src/verdict/ # Python application source package │ ├── __init__.py │ ├── runtime/ -│ │ ├── mode_detect.py # detect_mode() (W5.A.1) -│ │ └── gateway.py # FastMCP gateway init +│ │ ├── mode_detect.py # detect_mode() + Mode enum (W5.A.1) +│ │ ├── gateway.py # FastMCP gateway init +│ │ ├── evidence_recheck.py # periodic evidence re-hash (W1.G.7) +│ │ └── mode_lock.py # mode-lock enforcement on resume (W3.C.1) │ ├── schemas/ │ │ ├── __init__.py -│ │ ├── mode.py # Mode enum │ │ ├── artifact_class.py # ArtifactClass enum (W1.B.1) │ │ ├── caveat_id.py # CaveatID enum (W1.B.2) │ │ ├── evidence.py # EvidenceItem + EvidenceManifest (W1.B.3) @@ -125,6 +124,9 @@ By June 14 the repo will have this shape: │ │ ├── ledger.py # LedgerEntry (W1.B.11) │ │ ├── playbook.py # Playbook + Step (W1.F.1) │ │ ├── hunt_evil.py # HuntEvilBaseline + ProcessBaselineAnomaly (W1.F.8) +│ │ ├── verdict_status.py # VerdictStatus enum (W1.B.13) +│ │ ├── case_conclusion.py # CaseConclusion for no-evil terminal cases (W1.B.14) +│ │ ├── memory.py # MemoryEntry + MemoryUpdateProposal + MemoryType │ │ └── version.py # SCHEMA_VERSION constant + migration helpers │ ├── verification/ │ │ ├── strategy.py # VerifierStrategy Protocol @@ -155,11 +157,18 @@ By June 14 the repo will have this shape: │ │ ├── topology.py # build_graph(mode) → CompiledGraph │ │ ├── reducers.py # State reducers for fanout merge │ │ ├── checkpoint.py # SqliteSaver setup + WAL pragmas (W3.E.1) -│ │ └── interrupt.py # interrupt() helpers + unverifiable_finalize wiring (W3.D.4) +│ │ ├── comprehension_gate.py # Comprehension gate node (W2.B.2) +│ │ ├── interrupt.py # interrupt() helpers + unverifiable_finalize wiring (W3.D.4) +│ │ └── wrappers/ +│ │ ├── deny_rule.py # DenyRuleWrapper Layer 2 immutability (W2.C.1) +│ │ └── ledger_emitter.py # LedgerEmitter wrapper (W2.C.3) │ ├── tools/ │ │ ├── base.py # ToolWrapper abstract -│ │ ├── args_validators.py # Pydantic-AI args_validator framework (W2.E.1) -│ │ ├── vol3/ # 10 Volatility plugin wrappers +│ │ ├── external.py # ExternalToolSpec dataclass +│ │ ├── parsers.py # Tool stdout → ToolOutput parser +│ │ ├── registry.py # TOOL_SPECS registry + helpers +│ │ ├── args_validators.py # ArgsValidator (Pydantic v2; W2.E.1) +│ │ ├── vol3/ # 10 of 11 vol3 plugin wrapper files (vol3.info registered in registry.py) │ │ │ ├── pslist.py │ │ │ ├── psscan.py # NEW (W1.E.1) │ │ │ ├── pstree.py @@ -179,14 +188,13 @@ By June 14 the repo will have this shape: │ │ ├── bulk_extractor.py │ │ ├── exiftool.py │ │ ├── capa.py -│ │ └── sanitization.py # Prompt-injection scanner (W2.E.4) +│ │ └── sanitization.py # Prompt-injection scanner (W2.E.5) │ ├── sandboxes/ │ │ ├── microsandbox_provider.py # Pattern 1 ephemeral VM (W1.A.6) │ │ ├── tsi_provider.py # Pattern 2 TSI enrichment (W3.B.1) │ │ └── rootfs_pin.py # SHA-256 pin verification │ ├── ledger/ -│ │ ├── writer.py # write + fsync + verify-readback (W2.G.1) -│ │ ├── chain.py # prev_entry_hash chain verification +│ │ ├── writer.py # write + fsync + verify-readback + verify_ledger_chain (W2.G.1) │ │ ├── hmac_key.py # TPM-backed or gpg-encrypted (W1.G.6) │ │ └── redaction.py # Auth-field stripping (W3.B.3) │ ├── observability/ @@ -195,10 +203,17 @@ By June 14 the repo will have this shape: │ │ └── trace_link.py # trace_id ↔ ledger cross-link (W3.E.5) │ ├── cli/ │ │ ├── __main__.py +│ │ ├── credentials.py # Cloud credential detection helper (W1.A.1) │ │ ├── init.py / resume.py / reverify.py / status.py │ │ ├── ls.py / show.py / export.py / validate.py │ │ ├── mode.py / gc.py / health.py │ │ └── doctor.py # Pre-flight checks (W5.A.4) +│ ├── memory/ +│ │ ├── __init__.py +│ │ └── store.py # SQLite-backed append-only MemoryStore +│ ├── proof/ +│ │ ├── __init__.py +│ │ └── cloud.py # Cloud-only Claude proof harness │ └── adapters/ │ ├── opencti_mcp.py # OpenCTI enrichment (W5.B.1) │ ├── velociraptor_mcp.py # Live-endpoint mode (W5.B.2) @@ -225,7 +240,11 @@ By June 14 the repo will have this shape: │ ├── tools/ # Each tool wrapper integration test │ ├── sandboxes/ # Microsandbox + TSI tests │ ├── ledger/ # Chain integrity + HMAC tests +│ ├── memory/ # MemoryStore tests │ ├── observability/ # Langfuse + OTel tests +│ ├── policy/ # No-mocks hook, source-layout, eval-fail-closed gates +│ ├── proof/ # Cloud proof harness tests +│ ├── runtime/ # Gateway, mode-detect, mode-lock, evidence-recheck tests │ ├── cli/ # CLI tests │ ├── chaos/ # kill-9 chaos tests (W3.E.6) │ ├── smoke/ # Cross-cutting smoke tests @@ -287,19 +306,19 @@ The plan below is exhaustive. Every task has owner, hours, and TDD substeps. Tas ## Phase W1.A — Infrastructure stand-up (Tim, ~2 days) ### W1.A.1 — `scripts/install.sh` with cloud credential detection -- [ ] **W1.A.1.a** — Write failing test `tests/cli/test_install_credentials.py::test_detects_oauth_token_first`. Launch the credentials helper in a subprocess with only `CLAUDE_CODE_OAUTH_TOKEN` set in that subprocess environment; assert install reports `mode=oauth`. Run → RED. -- [ ] **W1.A.1.b** — Write credential detection logic (`CLAUDE_CODE_OAUTH_TOKEN` env → interactive `~/.claude/` → `ANTHROPIC_API_KEY` → optional `OPENROUTER_API_KEY`) in `scripts/install.sh` + Python helper `verdict/cli/credentials.py`. Run → GREEN. `OPENROUTER_API_KEY` is host-side only for AI-agent fallback; `verdict doctor` must fail if any cloud credential would be passed into a microsandbox env. -- [ ] **W1.A.1.c** — Commit: `feat(cli): three-credential-path install per A1 [W1.A.1]` +- [x] **W1.A.1.a** — Write failing test `tests/cli/test_install_credentials.py::test_detects_oauth_token_first`. Launch the credentials helper in a subprocess with only `CLAUDE_CODE_OAUTH_TOKEN` set in that subprocess environment; assert install reports `mode=oauth`. Run → RED. +- [x] **W1.A.1.b** — Write credential detection logic (`CLAUDE_CODE_OAUTH_TOKEN` env → interactive `~/.claude/` → `ANTHROPIC_API_KEY` → optional `OPENROUTER_API_KEY`) in `scripts/install.sh` + Python helper `src/verdict/cli/credentials.py`. Run → GREEN. `OPENROUTER_API_KEY` is host-side only for AI-agent fallback; `verdict doctor` must fail if any cloud credential would be passed into a microsandbox env. +- [x] **W1.A.1.c** — Commit: `feat(cli): three-credential-path install per A1 [W1.A.1]` ### W1.A.2 — SIFT VM provisioning (manual + scripted) -- [ ] **W1.A.2.a** — Document VM specs in `docs/RELEASE.md`: 32GB RAM, 8 vCPU, 200GB disk, KVM enabled. Convert `sift-2026.03.24.ova` to VMware Workstation per project's existing `scripts/sift-vm-bootstrap.sh`. +- [ ] **W1.A.2.a** — Document VM specs in `docs/RELEASE.md`: 32GB RAM, 8 vCPU, 200GB disk, KVM enabled. Convert `sift-2026.03.24.ova` to VMware Workstation manually (see SIFT install docs). - [ ] **W1.A.2.b** — Smoke test: `vol3 -h` runs inside VM. Verify Python 3.11 present. - [ ] **W1.A.2.c** — Commit: `docs(build): SIFT VM provisioning checklist [W1.A.2]` ### W1.A.3 — Microsandbox install -- [ ] **W1.A.3.a** — Run `curl -sSL https://get.microsandbox.dev | sh` inside SIFT VM. Verify `microsandbox --version` returns. Verify `microsandbox-mcp` binary present. +- [ ] **W1.A.3.a** — Run `curl -fsSL https://install.microsandbox.dev | sh` inside SIFT VM. Verify `microsandbox --version` returns. Verify `microsandbox-mcp` binary present. - [ ] **W1.A.3.b** — Smoke test: spawn an Ubuntu 22.04 microVM, run `vol3 -h`, destroy. Document spawn time in `docs/RELEASE.md` (target <500ms). -- [ ] **W1.A.3.c** — Build `verdict-sift-tools` rootfs Docker image with the 12 forensic tools pinned to versions: `vol3==2.10.0`, `hayabusa==2.18.0`, `plaso==20260427`, `MFTECmd==1.2.x`, etc. Push as `verdict-sift-tools:v0.1` and capture SHA-256. +- [ ] **W1.A.3.c** — Build `verdict-sift-tools` rootfs Docker image with the 12 forensic tools pinned to versions: `vol3==2.28.0`, `hayabusa==2.18.0`, `plaso==20260427`, `MFTECmd==1.2.x`, etc. Push as `verdict-sift-tools:v0.1` and capture SHA-256. - [ ] **W1.A.3.d** — Commit: `feat(sandbox): microsandbox install + verdict-sift-tools rootfs pinned [W1.A.3]` ### W1.A.4 — SGLang + Qwen3 + GLM-4.5-Air on dev rig (Haley, ~1 day) @@ -311,131 +330,135 @@ The plan below is exhaustive. Every task has owner, hours, and TDD substeps. Tas - [ ] **W1.A.4.e** — Commit: `feat(inference): SGLang + Qwen3 + GLM-4.5-Air serving with tool-call parsers [W1.A.4]` ### W1.A.5 — FastMCP gateway skeleton (Tim) -- [ ] **W1.A.5.a** — Write failing test `tests/runtime/test_gateway_case_init.py::test_case_init_returns_handle`. Start the real FastMCP gateway process and call the real `case_init` tool against a temporary case directory and read-only sample evidence path; assert it returns `{case_id, mode}` and writes a `case_init` ledger entry. Run → RED. -- [ ] **W1.A.5.b** — Implement `verdict/runtime/gateway.py` with FastMCP, single tool `case_init`. Wire the real `detect_mode()` contract from day one: cloud requires Anthropic reachability, air-gap requires SGLang reachability, dual requires both. If neither prerequisite is reachable, fail closed with a diagnostic rather than returning a default mode. -- [ ] **W1.A.5.c** — Commit: `feat(runtime): FastMCP gateway skeleton with case_init [W1.A.5]` +- [x] **W1.A.5.a** — Write failing test `tests/runtime/test_gateway.py::test_case_init_missing_evidence_fails_before_case_directory`. Start the real FastMCP gateway process and call the real `case_init` tool against a temporary case directory and read-only sample evidence path; assert it returns `{case_id, mode}` and writes a `case_init` ledger entry. Run → RED. +- [x] **W1.A.5.b** — Implement `src/verdict/runtime/gateway.py` with FastMCP, single tool `case_init`. Wire the real `detect_mode()` contract from day one: cloud requires Anthropic reachability, air-gap requires SGLang reachability, dual requires both. If neither prerequisite is reachable, fail closed with a diagnostic rather than returning a default mode. +- [x] **W1.A.5.c** — Commit: `feat(runtime): FastMCP gateway skeleton with case_init [W1.A.5]` ### W1.A.6 — Microsandbox provider Pattern 1 (per-tool ephemeral microVM) -- [ ] **W1.A.6.a** — Write failing test `tests/sandboxes/test_microsandbox_provider.py::test_per_call_ephemeral_microvm`. Spawn sandbox with read-only `/evidence` mount, run `cat /etc/os-release`, destroy. Assert `network=False` enforced. Run → RED. -- [ ] **W1.A.6.b** — Implement `verdict/sandboxes/microsandbox_provider.py` per v4.5 line 461 sketch. Network=False default; `mounts=[ReadOnly(...)]`; SHA-256 stdout. -- [ ] **W1.A.6.c** — Commit: `feat(sandbox): per-tool ephemeral microsandbox provider Pattern 1 [W1.A.6]` +- [x] **W1.A.6.a** — Write failing test `tests/sandboxes/test_microsandbox_provider.py::test_microsandbox_command_mounts_evidence_readonly`. Spawn sandbox with read-only `/evidence` mount, run `cat /etc/os-release`, destroy. Assert `network=False` enforced. Run → RED. +- [x] **W1.A.6.b** — Implement `src/verdict/sandboxes/microsandbox_provider.py` per v4.5 line 461 sketch. Network=False default; `mounts=[ReadOnly(...)]`; SHA-256 stdout. +- [x] **W1.A.6.c** — Commit: `feat(sandbox): per-tool ephemeral microsandbox provider Pattern 1 [W1.A.6]` ### W1.A.7 — Langfuse self-host (Tim) -- [ ] **W1.A.7.a** — Stand up `docker-compose.yml` with Langfuse v2 (Postgres-only, ~1.5GB RAM). Verify UI loads on `http://localhost:3000`. **Threshold:** if v2 deployment hits 4-hour blocker, fall back to OpenLLMetry → local Tempo viewer; document why in `docs/RELEASE.md`. +- [x] **W1.A.7.a** — Stand up `infra/langfuse/docker-compose.yml` with Langfuse v2 (Postgres-only, ~1.5GB RAM). Verify UI loads on `http://localhost:3000`. **Threshold:** if v2 deployment hits 4-hour blocker, fall back to OpenLLMetry → local Tempo viewer; document why in `docs/RELEASE.md`. - [ ] **W1.A.7.b** — Write smoke test `tests/observability/test_langfuse_smoke.py::test_one_trace_renders`. Send one synthetic trace via SDK; assert `/api/public/traces/{id}` returns 200. - [ ] **W1.A.7.c** — Commit: `feat(observability): Langfuse v2 self-host + smoke trace [W1.A.7]` -### W1.A.8 — Inspect AI hello-world -- [ ] **W1.A.8.a** — `pip install inspect-ai` per CLAUDE.md. Author `inspect_ai/tasks/hello_world.py` minimal task. Run `inspect eval inspect_ai/tasks/hello_world.py`. Assert pass. -- [ ] **W1.A.8.b** — Commit: `feat(eval): Inspect AI hello-world task [W1.A.8]` +### W1.A.8 — Inspect AI eval scaffolds +**Superseded by W4.D.1.** The original hello-world task was replaced by the three +per-mode eval scaffolds (`inspect_ai/tasks/verdict_eval_{cloud,airgap,dual}.py`) +already landed at main. No further action needed for W1.A.8. + +- [x] **W1.A.8.a** — Per-mode eval scaffolds present at `inspect_ai/tasks/verdict_eval_{cloud,airgap,dual}.py`. Gate: `ls inspect_ai/tasks/verdict_eval_{cloud,airgap,dual}.py` passes. *(Done via W4.D.1)* +- [x] **W1.A.8.b** — Commits: `feat(eval): add fail-closed per-mode scaffolds [W4.D.1]` ### W1.A.9 — Mechanical hard-rule enforcement (Tim, ~3 hours) Pulls forward what `CONTRIBUTING.md` already promises and what `CLAUDE.md` §3.7 + §3.10 require. Without this task, the hard rules are rules of prose only. -- [ ] **W1.A.9.a** — Failing test `tests/policy/test_no_mocks_hook.py::test_rejects_unittest_mock_import`. Assertion: `check_no_mocks.scan(["tests/policy/fixtures/has_mock_import.py"]).violations` is non-empty AND the offending line is reported. Plus `test_allows_third_party_boundary_patch` — patching `httpx` in a single targeted test passes. -- [ ] **W1.A.9.b** — Implement `scripts/check_no_mocks.py` (~40 LOC AST walker). Rejects: `import unittest.mock`, `from unittest import mock`, `import responses`, `import vcr`, `import betamax`, `import httpx_mock`, regex `^\s*if .*(MOCK|TEST_MODE).*:\s*$`, regex `os\.environ\.get\(['"]VERDICT_TEST`. Walks all `.py` under `verdict/` and `tests/`. -- [ ] **W1.A.9.c** — Author `.pre-commit-config.yaml` at repo root with hooks: (1) `commitizen check` enforcing `^(feat|fix|test|chore|docs|refactor)\(\w+\): .* \[W\d+\.[A-Z]\.\d+(\.[a-z])?\]$` on commit message; (2) `ruff check --select ALL`; (3) the local `check-no-mocks` hook from W1.A.9.b; (4) `cargo fmt --check`. Run `pre-commit install --install-hooks` in `scripts/install.sh`. -- [ ] **W1.A.9.d** — Add `.github/workflows/eval-hallucination-gate.yml`: on PR, runs `inspect eval inspect_ai/tasks/verdict_eval_cloud.py --score hallucination_rate` against the real evaluator once `verdict doctor --mode cloud` succeeds. Until the real scorer exists in W4.D.1, the workflow must fail with `scorer_not_implemented` rather than returning a passing score. -- [ ] **W1.A.9.e** — Drop the `test -f .pre-commit-config.yaml &&` short-circuit at `CONTRIBUTING.md` line 140 (file exists now; the guard is no longer needed and silently masks a missing config). -- [ ] **W1.A.9.f** — Commit: `feat(policy): mechanical enforcement of §3.7 + §3.10 (no-mocks AST hook + commit-msg regex + hallucination CI gate) [W1.A.9]` +- [x] **W1.A.9.a** — Failing test `tests/policy/test_no_mocks_hook.py::test_rejects_unittest_mock_import`. Assertion: `check_no_mocks.scan(["tests/policy/fixtures/has_mock_import.py"]).violations` is non-empty AND the offending line is reported. Plus `test_allows_third_party_boundary_patch` — patching `httpx` in a single targeted test passes. +- [x] **W1.A.9.b** — Implement `scripts/check_no_mocks.py` (~40 LOC AST walker). Rejects: `import unittest.mock`, `from unittest import mock`, `import responses`, `import vcr`, `import betamax`, `import httpx_mock`, regex `^\s*if .*(MOCK|TEST_MODE).*:\s*$`, regex `os\.environ\.get\(['"]VERDICT_TEST`. Walks all `.py` under `src/verdict/` and `tests/`. +- [x] **W1.A.9.c** — Author `.pre-commit-config.yaml` at repo root with hooks: (1) `commitizen check` enforcing `^(feat|fix|test|chore|docs|refactor)\(\w+\): .* \[W\d+\.[A-Z]\.\d+(\.[a-z])?\]$` on commit message; (2) `ruff check --select ALL`; (3) the local `check-no-mocks` hook from W1.A.9.b; (4) `cargo fmt --check`. Run `pre-commit install --install-hooks` in `scripts/install.sh`. +- [x] **W1.A.9.d** — Add `.github/workflows/eval-hallucination-gate.yml`: on PR, runs `inspect eval inspect_ai/tasks/verdict_eval_cloud.py --score hallucination_rate` against the real evaluator once `verdict doctor --mode cloud` succeeds. Until the real scorer exists in W4.D.1, the workflow must fail with `scorer_not_implemented` rather than returning a passing score. +- [x] **W1.A.9.e** — Drop the `test -f .pre-commit-config.yaml &&` short-circuit at `CONTRIBUTING.md` line 140 (file exists now; the guard is no longer needed and silently masks a missing config). +- [x] **W1.A.9.f** — Commit: `feat(policy): mechanical enforcement of §3.7 + §3.10 (no-mocks AST hook + commit-msg regex + hallucination CI gate) [W1.A.9]` ## Phase W1.B — Schema bundle (Tim, ~2 hours) This is the contract every teammate will code against. **Lock by Sunday May 4.** All schema work must reconcile against `docs/ARCHITECTURE.md` §4. ### W1.B.1 — `ArtifactClass` enum -- [ ] **W1.B.1.a** — Write failing test `tests/schemas/test_artifact_class.py::test_enum_has_13_required_members`. Run → RED. -- [ ] **W1.B.1.b** — Implement `verdict/schemas/artifact_class.py` per Appendix A.1. -- [ ] **W1.B.1.c** — Commit: `feat(schema): ArtifactClass enum (FOR500 corroboration) [W1.B.1]` +- [x] **W1.B.1.a** — Write failing test `tests/schemas/test_artifact_class.py::test_enum_has_required_members_for_tier_1_caveat_triggers`. Assert all 15 `ArtifactClass` members are present. Run → RED. +- [x] **W1.B.1.b** — Implement `src/verdict/schemas/artifact_class.py` per Appendix A.1. +- [x] **W1.B.1.c** — Commit: `feat(schema): ArtifactClass enum (FOR500 corroboration) [W1.B.1]` ### W1.B.2 — `CaveatID` enum -- [ ] **W1.B.2.a** — Write failing test `tests/schemas/test_caveat_id.py::test_enum_covers_tier1_examiner_caveats`. Assert all 7 from the current planned caveat source `verdict/prompts/examiner_caveats.md` and the root `CLAUDE.md` §3.3 table. Run → RED. -- [ ] **W1.B.2.b** — Implement `verdict/schemas/caveat_id.py` per Appendix A.2. -- [ ] **W1.B.2.c** — Commit: `feat(schema): CaveatID enum from project MEMORY.md Tier-1 [W1.B.2]` +- [x] **W1.B.2.a** — Write failing test `tests/schemas/test_caveat_id.py::test_enum_has_seven_tier_1_caveats`. Assert all 7 from the current planned caveat source `src/verdict/planning/prompts/examiner_caveats.md` and the root `CLAUDE.md` §3.3 table. Run → RED. +- [x] **W1.B.2.b** — Implement `src/verdict/schemas/caveat_id.py` per Appendix A.2. +- [x] **W1.B.2.c** — Commit: `feat(schema): CaveatID enum from CLAUDE.md §3.3 Tier-1 caveats [W1.B.2]` ### W1.B.3 — `EvidenceItem` + `EvidenceManifest` -- [ ] **W1.B.3.a** — Write failing test `tests/schemas/test_evidence.py::test_manifest_hash_is_blake3_of_sorted_pairs`. Run → RED. -- [ ] **W1.B.3.b** — Implement `verdict/schemas/evidence.py` per v4.5 lines 153–168 + Appendix A.3. -- [ ] **W1.B.3.c** — Commit: `feat(schema): EvidenceItem + EvidenceManifest schemas [W1.B.3]` +- [x] **W1.B.3.a** — Write failing test `tests/schemas/test_evidence.py::test_manifest_hash_is_blake3_of_sorted_pairs`. Run → RED. +- [x] **W1.B.3.b** — Implement `src/verdict/schemas/evidence.py` per v4.5 lines 153–168 + Appendix A.3. +- [x] **W1.B.3.c** — Commit: `feat(schema): EvidenceItem + EvidenceManifest schemas [W1.B.3]` ### W1.B.4 — `Artifact` + `ToolOutput` base -- [ ] **W1.B.4.a** — Write failing test `tests/schemas/test_tool_output.py::test_invocation_hash_combines_name_version_args_evidence`. Run → RED. -- [ ] **W1.B.4.b** — Implement `verdict/schemas/tool_output.py` per v4.5 lines 170–193 + Appendix A.4. -- [ ] **W1.B.4.c** — Commit: `feat(schema): Artifact + ToolOutput base for tool wrapper contract [W1.B.4]` +- [x] **W1.B.4.a** — Write failing test `tests/schemas/test_tool_output.py::test_invocation_hash_combines_name_version_args_evidence`. Run → RED. +- [x] **W1.B.4.b** — Implement `src/verdict/schemas/tool_output.py` per v4.5 lines 170–193 + Appendix A.4. +- [x] **W1.B.4.c** — Commit: `feat(schema): Artifact + ToolOutput base for tool wrapper contract [W1.B.4]` ### W1.B.5 — `Hypothesis` + `InvestigationPlan` + `PlanComprehensionEcho` + `PlannerCritiqueVerdict` -- [ ] **W1.B.5.a** — Write failing tests in `tests/schemas/test_plan.py`: `test_mitre_subtechnique_regex_validates_T1055_012` (passes) and `test_mitre_invalid_format_rejected` (raises). Plus `test_negative_hypothesis_quality_rejects_degenerate`. Run → RED. -- [ ] **W1.B.5.b** — Implement `verdict/schemas/plan.py` with all four classes + the `mitre_technique` regex validator (`^T\d{4}(\.\d{3})?$`) + `_negative_hypothesis_quality` validator (deny-list: cosmic/alien/nothing/not-relevant/n-a; require non-None mitre_technique; require non-empty artifact_families). -- [ ] **W1.B.5.c** — Commit: `feat(schema): Hypothesis + InvestigationPlan + comprehension/critique schemas [W1.B.5]` +- [x] **W1.B.5.a** — Write failing tests in `tests/schemas/test_plan.py`: `test_mitre_subtechnique_regex_validates_t1055_012` (passes) and `test_mitre_invalid_format_rejected` (raises). Plus `test_negative_hypothesis_quality_rejects_degenerate`. Run → RED. +- [x] **W1.B.5.b** — Implement `src/verdict/schemas/plan.py` with all four classes + the `mitre_technique` regex validator (`^T\d{4}(\.\d{3})?$`) + `_negative_hypothesis_quality` validator (deny-list: cosmic/alien/nothing/not-relevant/n-a; require non-None mitre_technique; require non-empty artifact_families). +- [x] **W1.B.5.c** — Commit: `feat(schema): Hypothesis + InvestigationPlan + comprehension/critique schemas [W1.B.5]` ### W1.B.6 — `Finding` skeleton -- [ ] **W1.B.6.a** — Write failing test `tests/schemas/test_finding.py::test_finding_round_trips_through_json`. Run → RED. -- [ ] **W1.B.6.b** — Implement `verdict/schemas/finding.py` skeleton: all v4.5 fields plus the new ones (`artifact_classes`, `caveats_acknowledged`). -- [ ] **W1.B.6.c** — Commit: `feat(schema): Finding skeleton [W1.B.6]` +- [x] **W1.B.6.a** — Write failing test `tests/schemas/test_finding.py::test_finding_round_trips_through_json`. Run → RED. +- [x] **W1.B.6.b** — Implement `src/verdict/schemas/finding.py` skeleton: all v4.5 fields plus the new ones (`artifact_classes`, `caveats_acknowledged`). +- [x] **W1.B.6.c** — Commit: `feat(schema): Finding skeleton [W1.B.6]` ### W1.B.7 — Patch `Finding.artifact_paths` to `Field(min_length=2)` -- [ ] **W1.B.7.a** — Failing test: `test_artifact_paths_min_length_2`. Run → RED. -- [ ] **W1.B.7.b** — Implement. -- [ ] **W1.B.7.c** — Commit: `feat(schema): require ≥2 artifact paths per Finding (FOR500) [W1.B.7]` +- [x] **W1.B.7.a** — Failing test: `test_artifact_paths_min_length_2`. Run → RED. +- [x] **W1.B.7.b** — Implement. +- [x] **W1.B.7.c** — Commit: `feat(schema): require ≥2 artifact paths per Finding (FOR500) [W1.B.7]` ### W1.B.8 — `Finding.artifact_classes` field -- [ ] **W1.B.8.a** — Failing test: `test_artifact_classes_min_length_2`. Run → RED. -- [ ] **W1.B.8.b** — Implement. -- [ ] **W1.B.8.c** — Commit: `feat(schema): Finding.artifact_classes min_length=2 [W1.B.8]` +- [x] **W1.B.8.a** — Failing test: `test_artifact_classes_min_length_2`. Run → RED. +- [x] **W1.B.8.b** — Implement. +- [x] **W1.B.8.c** — Commit: `feat(schema): Finding.artifact_classes min_length=2 [W1.B.8]` ### W1.B.9 — `Finding.caveats_acknowledged` field -- [ ] **W1.B.9.a** — Failing test: `test_caveats_acknowledged_default_empty`. Run → RED. -- [ ] **W1.B.9.b** — Implement. -- [ ] **W1.B.9.c** — Commit: `feat(schema): Finding.caveats_acknowledged field [W1.B.9]` +- [x] **W1.B.9.a** — Failing test: `test_caveats_acknowledged_default_empty`. Run → RED. +- [x] **W1.B.9.b** — Implement. +- [x] **W1.B.9.c** — Commit: `feat(schema): Finding.caveats_acknowledged field [W1.B.9]` -### W1.B.10 — Execution-claim validator + Amcache-caveat validator + 6 other caveat validators -- [ ] **W1.B.10.a** — Failing tests: `test_execution_claim_requires_two_classes` (T1059, T1106, T1204, T1218, T1543, T1547 prefixes), `test_amcache_requires_caveat`, plus one test per remaining CaveatID (`test_shimcache_caveat_required_when_shimcache_cited`, etc.). Run → RED. -- [ ] **W1.B.10.b** — Implement `_execution_claims_need_two_classes` + `_amcache_caveat_required` + 6 sibling validators (one per CaveatID where the artifact_class triggers the caveat). -- [ ] **W1.B.10.c** — Commit: `feat(schema): Finding validators enforce caveat acknowledgment [W1.B.10]` +### W1.B.10 — Unified forensic-corroboration validator +- [x] **W1.B.10.a** — Failing tests: `test_execution_claim_requires_two_classes` (T1059, T1106, T1204, T1218, T1543, T1547 prefixes), `test_available_caveat_required_when_artifact_class_cited` (parametrised over all ArtifactClass caveat triggers). Run → RED. +- [x] **W1.B.10.b** — Implement a single `_forensic_corroboration` `@model_validator(mode="after")` that covers both the execution-class two-class requirement and all Tier-1 caveat acknowledgments (including the EVTX_4624 logon-type named exception). +- [x] **W1.B.10.c** — Commit: `feat(schema): Finding validators enforce caveat acknowledgment [W1.B.10]` ### W1.B.11 — `LedgerEntry` schema -- [ ] **W1.B.11.a** — Failing test: `test_ledger_entry_three_id_hierarchy`. Assert `case_id`, `langfuse_trace_id`, `langgraph_checkpoint_id` are distinct fields. Plus `test_ledger_entry_records_examination_environment` for `microsandbox_version`/`rootfs_sha256`/`tool_version`/`kernel_version`. -- [ ] **W1.B.11.b** — Implement `verdict/schemas/ledger.py` per v4.5 lines 245–278 plus the v4.4 environment-metadata fields. Add `output_files_sha256: dict[str, str] = {}` field. -- [ ] **W1.B.11.c** — Commit: `feat(schema): LedgerEntry with three-ID hierarchy + exam-env metadata [W1.B.11]` +- [x] **W1.B.11.a** — Failing test: `test_ledger_entry_three_id_hierarchy`. Assert `case_id`, `langfuse_trace_id`, `langgraph_checkpoint_id` are distinct fields. Plus `test_ledger_entry_records_examination_environment` for `microsandbox_version`/`rootfs_sha256`/`tool_version`/`kernel_version`. +- [x] **W1.B.11.b** — Implement `src/verdict/schemas/ledger.py` per v4.5 lines 245–278 plus the v4.4 environment-metadata fields. Add `output_files_sha256: dict[str, str] = {}` field. +- [x] **W1.B.11.c** — Commit: `feat(schema): LedgerEntry with three-ID hierarchy + exam-env metadata [W1.B.11]` -### W1.B.12 — `schema_version` discipline + `verdict/schemas/version.py` -- [ ] **W1.B.12.a** — Failing test: `test_schema_version_is_1_on_all_top_level_models`. Loop through `[InvestigationPlan, Finding, LedgerEntry, EvidenceManifest, ToolOutput]`; assert `.schema_version == 1`. -- [ ] **W1.B.12.b** — Implement: add `schema_version: int = 1` to all five top-level schemas; centralize in `verdict/schemas/version.py`. -- [ ] **W1.B.12.c** — Commit: `feat(schema): schema_version discipline across top-level models [W1.B.12]` +### W1.B.12 — `schema_version` discipline + `src/verdict/schemas/version.py` +- [x] **W1.B.12.a** — Failing test: `test_schema_version_is_1_on_all_top_level_models`. Loop through `[InvestigationPlan, Finding, LedgerEntry, EvidenceManifest, ToolOutput]`; assert `.schema_version == 1`. +- [x] **W1.B.12.b** — Implement: add `schema_version: int = 1` to all five top-level schemas; centralize in `src/verdict/schemas/version.py`. +- [x] **W1.B.12.c** — Commit: `feat(schema): schema_version discipline across top-level models [W1.B.12]` ### W1.B.13 — `VerdictStatus` enum -- [ ] **W1.B.13.a** — Failing test: `test_verdict_status_has_canonical_states`. Assert exactly the six statuses from `CLAUDE.md` §3.6: `VETTED_CLOUD`, `VETTED_AIRGAP`, `VETTED_DUAL`, `CONTESTED`, `UNVERIFIABLE`, `EXHAUSTED_REPLAN`. Assert `DRAFT`, `APPROVED`, and `REJECTED` live only on the separate `Finding.review_state` enum. -- [ ] **W1.B.13.b** — Implement. -- [ ] **W1.B.13.c** — Commit: `feat(schema): VerdictStatus enum [W1.B.13]` +- [x] **W1.B.13.a** — Failing test: `test_verdict_status_has_canonical_states`. Assert exactly the six statuses from `CLAUDE.md` §3.6: `VETTED_CLOUD`, `VETTED_AIRGAP`, `VETTED_DUAL`, `CONTESTED`, `UNVERIFIABLE`, `EXHAUSTED_REPLAN`. Assert `DRAFT`, `APPROVED`, and `REJECTED` live only on the separate `Finding.review_state` enum. +- [x] **W1.B.13.b** — Implement. +- [x] **W1.B.13.c** — Commit: `feat(schema): VerdictStatus enum [W1.B.13]` ### W1.B.14 — `CaseConclusion` for no-evil terminal cases -- [ ] **W1.B.14.a** — Failing test `tests/schemas/test_case_conclusion.py::test_no_evil_found_requires_playbook_steps`. Assert `CaseConclusion(status="NO_EVIL_FOUND", playbook_steps_executed=[])` raises validation error, and a conclusion with at least one playbook step plus evidence hashes validates. -- [ ] **W1.B.14.b** — Implement `verdict/schemas/case_conclusion.py` with status values `NO_EVIL_FOUND`, `EVIL_FOUND`, `UNVERIFIABLE`; require `playbook_steps_executed: list[str] = Field(min_length=1)`, `evidence_hashes: dict[Path, str]`, and `rationale: str`. Do not add `NO_EVIL_FOUND` to `VerdictStatus`. -- [ ] **W1.B.14.c** — Commit: `feat(schema): CaseConclusion for no-evil terminal cases [W1.B.14]` +- [x] **W1.B.14.a** — Failing test `tests/schemas/test_case_conclusion.py::test_no_evil_found_requires_playbook_steps`. Assert `CaseConclusion(status="NO_EVIL_FOUND", playbook_steps_executed=[])` raises validation error, and a conclusion with at least one playbook step plus evidence hashes validates. +- [x] **W1.B.14.b** — Implement `src/verdict/schemas/case_conclusion.py` with status values `NO_EVIL_FOUND`, `EVIL_FOUND`, `UNVERIFIABLE`; require `playbook_steps_executed: list[str] = Field(min_length=1)`, `evidence_hashes: dict[Path, str]`, and `rationale: str`. Do not add `NO_EVIL_FOUND` to `VerdictStatus`. +- [x] **W1.B.14.c** — Commit: `feat(schema): CaseConclusion for no-evil terminal cases [W1.B.14]` ## Phase W1.C — Verifier strategy seed-derivation fix (Beaver, ~1 hour) ### W1.C.1 — `derive_seeds(case_id)` helper -- [ ] **W1.C.1.a** — Failing test `tests/verification/test_derive_seeds.py::test_three_distinct_deterministic_per_case`. Run → RED. -- [ ] **W1.C.1.b** — Implement `verdict/verification/derive_seeds.py` using blake3 keyed-hash pattern. -- [ ] **W1.C.1.c** — Commit: `feat(verification): derive_seeds(case_id) for n=3 self-consistency [W1.C.1]` +- [x] **W1.C.1.a** — Failing test `tests/verification/test_derive_seeds.py::test_three_distinct_deterministic_per_case`. Run → RED. +- [x] **W1.C.1.b** — Implement `src/verdict/verification/derive_seeds.py` using blake3 keyed-hash pattern. +- [x] **W1.C.1.c** — Commit: `feat(verification): derive_seeds(case_id) for n=3 self-consistency [W1.C.1]` ### W1.C.2 — `CloudSelfConsistency` impl - [ ] **W1.C.2.a** — Failing integration test `tests/verification/test_cloud_self_consistency.py::test_three_distinct_seeds_in_api_calls`. Require `verdict doctor --mode cloud` first; execute the real Anthropic/Claude client in a bounded smoke request and assert 3 calls, 3 distinct seeds, `temperature=0.7`, and non-empty verifier outputs. Run → RED. -- [ ] **W1.C.2.b** — Implement `verdict/verification/cloud_self_consistency.py` per Appendix A.5. +- [ ] **W1.C.2.b** — Implement `src/verdict/verification/cloud_self_consistency.py` per Appendix A.5. - [ ] **W1.C.2.c** — Commit: `fix(verification): CloudSelfConsistency samples 3 diverse paths (Wang 2022) [W1.C.2]` ### W1.C.3 — `VerifierStrategy` Protocol + Universal Self-Consistency baseline -- [ ] **W1.C.3.a** — Failing test `tests/verification/test_strategy_protocol.py::test_strategy_returns_verdict_result`. Define a tiny in-test concrete strategy that computes its result from supplied verifier outputs; assert it conforms to the `VerifierStrategy` Protocol without any hardcoded production verdict. Run → RED. -- [ ] **W1.C.3.b** — Implement `verdict/verification/strategy.py` (Protocol) + a real `universal_self_consistency.py` fallback that takes already-produced verifier candidates, groups by `(artifact_paths, mitre_technique)`, and returns `CONTESTED` or the matching `VETTED_*`/`UNVERIFIABLE` result according to the documented quorum rule. No placeholder implementation lands. -- [ ] **W1.C.3.c** — Commit: `feat(verification): VerifierStrategy Protocol + USC baseline [W1.C.3]` +- [x] **W1.C.3.a** — Failing test `tests/verification/test_strategy_protocol.py::test_strategy_returns_verdict_result`. Define a tiny in-test concrete strategy that computes its result from supplied verifier outputs; assert it conforms to the `VerifierStrategy` Protocol without any hardcoded production verdict. Run → RED. +- [x] **W1.C.3.b** — Implement `src/verdict/verification/strategy.py` (Protocol) + a real `universal_self_consistency.py` fallback that takes already-produced verifier candidates, groups by `(artifact_paths, mitre_technique)`, and returns `CONTESTED` or the matching `VETTED_*`/`UNVERIFIABLE` result according to the documented quorum rule. No placeholder implementation lands. +- [x] **W1.C.3.c** — Commit: `feat(verification): VerifierStrategy Protocol + USC baseline [W1.C.3]` ## Phase W1.D — PreToolUse caveat + smoke scaffold (Tim, ~30 min) ### W1.D.1 — CI smoke-test scaffold (xfail-marked) -- [ ] **W1.D.1.a** — Author `tests/smoke/test_pretooluse_deny.py` marked `pytest.mark.xfail(reason="anthropics/claude-code#33106 + #37210")`. Test invokes `claude` subprocess with a PreToolUse hook returning `permissionDecision: "deny"` for an MCP write; asserts the call is blocked. -- [ ] **W1.D.1.b** — Add `[smoke]` marker to `pyproject.toml` so `pytest -m smoke` finds it. -- [ ] **W1.D.1.c** — Commit: `test(smoke): PreToolUse deny scaffold (xfail per #33106 #37210) [W1.D.1]` +- [x] **W1.D.1.a** — Author `tests/smoke/test_pretooluse_deny.py` marked `pytest.mark.xfail(reason="anthropics/claude-code#33106 + #37210")`. Test invokes `claude` subprocess with a PreToolUse hook returning `permissionDecision: "deny"` for an MCP write; asserts the call is blocked. +- [x] **W1.D.1.b** — Add `[smoke]` marker to `pyproject.toml` so `pytest -m smoke` finds it. +- [x] **W1.D.1.c** — Commit: `test(smoke): PreToolUse deny scaffold (xfail per #33106 #37210) [W1.D.1]` ### W1.D.2 — Apply v4.6 P2 to v4.5 audit doc - [ ] **W1.D.2.a** — Append the Layer-1 caveat paragraph to v4.5 architecture caption (line 144). @@ -447,13 +470,13 @@ The 12 tool wrappers ship in W2.E. This phase ships the schema scaffolding + `ps ### W1.E.1 — `vol_psscan` MCP tool wrapper - [ ] **W1.E.1.a** — Failing integration test `tests/tools/test_vol_psscan.py::test_psscan_returns_pids`. Require `verdict doctor --mode airgap` and a real memory image from `inspect_ai/ground_truth/case_001_lolbins/`; invoke `vol3 windows.psscan` through the real microsandbox provider; assert returned `ToolOutput` contains process artifacts and a valid invocation hash. Run → RED. -- [ ] **W1.E.1.b** — Implement `verdict/tools/vol3/psscan.py` mirroring `vol_pslist` shape from project's `services/mcp/`. +- [ ] **W1.E.1.b** — Implement `src/verdict/tools/vol3/psscan.py` mirroring the `ToolWrapper` shape from `src/verdict/tools/base.py`. - [ ] **W1.E.1.c** — Commit: `feat(tools): vol_psscan wrapper for DKOM/T1014 cross-validation [W1.E.1]` ### W1.E.2 — Tool wrapper base class -- [ ] **W1.E.2.a** — Failing test `tests/tools/test_tool_base.py::test_base_records_invocation_hash`. Assert any wrapper extending `ToolWrapper` records `invocation_hash = blake3(tool_name + tool_version + args + evidence_hash)`. -- [ ] **W1.E.2.b** — Implement `verdict/tools/base.py` abstract `ToolWrapper` with `pre_run` (compute invocation hash) + `run` (subclass impl) + `post_run` (sandbox destroy + ledger write hooks). -- [ ] **W1.E.2.c** — Commit: `feat(tools): ToolWrapper abstract base with invocation hashing [W1.E.2]` +- [x] **W1.E.2.a** — Failing test `tests/tools/test_tool_base.py::test_base_records_invocation_hash`. Assert any wrapper extending `ToolWrapper` records `invocation_hash = blake3(tool_name + tool_version + args + evidence_hash)`. +- [x] **W1.E.2.b** — Implement `src/verdict/tools/base.py` abstract `ToolWrapper` with `pre_run` (compute invocation hash) + `run` (subclass impl) + `post_run` (sandbox destroy + ledger write hooks). +- [x] **W1.E.2.c** — Commit: `feat(tools): ToolWrapper abstract base with invocation hashing [W1.E.2]` ### W1.E.3 — Apply v4.6 P3 + P4 to v4.5 audit doc - [ ] **W1.E.3.a** — Update v4.5 line 796 tool list to include 10 vol3 plugins. @@ -463,52 +486,52 @@ The 12 tool wrappers ship in W2.E. This phase ships the schema scaffolding + `ps ## Phase W1.F — KP content authoring (KP, ~1.5 days) ### W1.F.1 — `Playbook` Pydantic schema -- [ ] **W1.F.1.a** — Failing test `tests/schemas/test_playbook.py::test_playbook_loads_yaml`. Run → RED. -- [ ] **W1.F.1.b** — Implement `verdict/schemas/playbook.py` with `Step` + `Playbook` classes per v4.6. -- [ ] **W1.F.1.c** — Commit: `feat(schema): Playbook + Step for planner methodology injection [W1.F.1]` +- [x] **W1.F.1.a** — Failing test `tests/schemas/test_playbook.py::test_playbook_loads_yaml`. Run → RED. +- [x] **W1.F.1.b** — Implement `src/verdict/schemas/playbook.py` with `Step` + `Playbook` classes per v4.6. +- [x] **W1.F.1.c** — Commit: `feat(schema): Playbook + Step for planner methodology injection [W1.F.1]` -### W1.F.2 — Author `verdict/playbooks/memory.yml` -- [ ] **W1.F.2.a** — Failing test `tests/playbooks/test_memory_yml.py::test_memory_playbook_has_dkom_rule`. Run → RED. -- [ ] **W1.F.2.b** — Author per Appendix C.1. -- [ ] **W1.F.2.c** — Commit: `feat(playbooks): memory.yml — Volatility 3 sequence + DKOM rule [W1.F.2]` +### W1.F.2 — Author `src/verdict/playbooks/memory.yml` +- [x] **W1.F.2.a** — Failing test `tests/playbooks/test_memory_yml.py::test_memory_playbook_has_dkom_rule`. Run → RED. +- [x] **W1.F.2.b** — Author per Appendix C.1. +- [x] **W1.F.2.c** — Commit: `feat(playbooks): memory.yml — Volatility 3 sequence + DKOM rule [W1.F.2]` -### W1.F.3 — Author `verdict/playbooks/disk.yml` -- [ ] **W1.F.3.a** — Failing test `tests/playbooks/test_disk_yml.py::test_plaso_after_lighter_tools`. Run → RED. -- [ ] **W1.F.3.b** — Author per Appendix C.2. -- [ ] **W1.F.3.c** — Commit: `feat(playbooks): disk.yml [W1.F.3]` +### W1.F.3 — Author `src/verdict/playbooks/disk.yml` +- [x] **W1.F.3.a** — Failing test `tests/playbooks/test_disk_yml.py::test_plaso_after_lighter_tools`. Run → RED. +- [x] **W1.F.3.b** — Author per Appendix C.2. +- [x] **W1.F.3.c** — Commit: `feat(playbooks): disk.yml [W1.F.3]` -### W1.F.4 — Author `verdict/playbooks/triage.yml` -- [ ] **W1.F.4.a** — Failing test `tests/playbooks/test_triage_yml.py::test_registry_first`. Run → RED. -- [ ] **W1.F.4.b** — Author per Appendix C.3. -- [ ] **W1.F.4.c** — Commit: `feat(playbooks): triage.yml [W1.F.4]` +### W1.F.4 — Author `src/verdict/playbooks/triage.yml` +- [x] **W1.F.4.a** — Failing test `tests/playbooks/test_triage_yml.py::test_registry_first`. Run → RED. +- [x] **W1.F.4.b** — Author per Appendix C.3. +- [x] **W1.F.4.c** — Commit: `feat(playbooks): triage.yml [W1.F.4]` ### W1.F.5 — Apply v4.6 P5 to v4.5 audit doc - [ ] **W1.F.5** — Append multi-artifact corroboration caveat. Commit: `docs(audit): v4.6 P5 — multi-artifact corroboration [W1.F.5]` ### W1.F.6 — `playbook_loader` injects into planner prompt -- [ ] **W1.F.6.a** — Failing test `tests/planning/test_playbook_loader.py::test_loader_picks_by_evidence_type`. Run → RED. -- [ ] **W1.F.6.b** — Implement `verdict/planning/playbook_loader.py::load_playbook_prompt(manifest: EvidenceManifest) -> str`. -- [ ] **W1.F.6.c** — Commit: `feat(planning): playbook_loader injects methodology by evidence type [W1.F.6]` +- [x] **W1.F.6.a** — Failing test `tests/planning/test_playbook_loader.py::test_loader_picks_by_evidence_type`. Run → RED. +- [x] **W1.F.6.b** — Implement `src/verdict/planning/playbook_loader.py::load_playbook_prompt(manifest: EvidenceManifest) -> str`. +- [x] **W1.F.6.c** — Commit: `feat(planning): playbook_loader injects methodology by evidence type [W1.F.6]` -### W1.F.7 — Author `verdict/planning/prompts/examiner_caveats.md` -- [ ] **W1.F.7.a** — Failing test `tests/prompts/test_examiner_caveats.py::test_all_seven_caveats_present`. Run → RED. -- [ ] **W1.F.7.b** — Author per Appendix B.1. -- [ ] **W1.F.7.c** — Commit: `feat(prompts): examiner_caveats.md — Tier-1 caveats include [W1.F.7]` +### W1.F.7 — Author `src/verdict/planning/prompts/examiner_caveats.md` +- [x] **W1.F.7.a** — Failing test `tests/prompts/test_examiner_caveats.py::test_all_seven_caveats_present`. Run → RED. +- [x] **W1.F.7.b** — Author per Appendix B.1. +- [x] **W1.F.7.c** — Commit: `feat(prompts): examiner_caveats.md — Tier-1 caveats include [W1.F.7]` ### W1.F.8 — `HuntEvilBaseline` schema + `ProcessBaselineAnomaly` Hypothesis subtype -- [ ] **W1.F.8.a** — Failing test `tests/schemas/test_hunt_evil.py::test_baseline_loads`. Plus `test_anomaly_maps_to_T1036_005`. -- [ ] **W1.F.8.b** — Implement `verdict/schemas/hunt_evil.py` with both classes. -- [ ] **W1.F.8.c** — Commit: `feat(schema): HuntEvilBaseline + ProcessBaselineAnomaly (T1036.005) [W1.F.8]` +- [x] **W1.F.8.a** — Failing test `tests/schemas/test_hunt_evil.py::test_baseline_loads`. Plus `test_anomaly_maps_to_T1036_005`. +- [x] **W1.F.8.b** — Implement `src/verdict/schemas/hunt_evil.py` with both classes. +- [x] **W1.F.8.c** — Commit: `feat(schema): HuntEvilBaseline + ProcessBaselineAnomaly (T1036.005) [W1.F.8]` -### W1.F.9 — Author `verdict/knowledge/hunt_evil.yml` -- [ ] **W1.F.9.a** — Failing test `tests/knowledge/test_hunt_evil_yml.py::test_eight_canonical_processes`. Run → RED. -- [ ] **W1.F.9.b** — Author per Appendix C.4 — 8 processes (svchost, lsass, csrss, winlogon, services, wininit, explorer, smss). -- [ ] **W1.F.9.c** — Commit: `feat(knowledge): hunt_evil.yml — 8 canonical Windows process baselines [W1.F.9]` +### W1.F.9 — Author `src/verdict/knowledge/hunt_evil.yml` +- [x] **W1.F.9.a** — Failing test `tests/knowledge/test_hunt_evil_yml.py::test_eight_canonical_processes`. Run → RED. +- [x] **W1.F.9.b** — Author per Appendix C.4 — 8 processes (svchost, lsass, csrss, winlogon, services, wininit, explorer, smss). +- [x] **W1.F.9.c** — Commit: `feat(knowledge): hunt_evil.yml — 8 canonical Windows process baselines [W1.F.9]` ### W1.F.10 — Executor system-prompt include -- [ ] **W1.F.10.a** — Failing test `tests/planning/test_executor_prompt.py::test_includes_caveats_and_hunt_evil`. Assert prompt contains `AMCACHE_LASTMODIFIED_NOT_EXEC` and `svchost.exe`. -- [ ] **W1.F.10.b** — Implement `verdict/planning/executor_prompt.py::render_executor_prompt(role: str) -> str` that composes examiner_caveats.md + relevant hunt_evil entries. -- [ ] **W1.F.10.c** — Commit: `feat(planning): executor system prompt with caveats + hunt evil [W1.F.10]` +- [x] **W1.F.10.a** — Failing test `tests/planning/test_executor_prompt.py::test_includes_caveats_and_hunt_evil`. Assert prompt contains `AMCACHE_LASTMODIFIED_NOT_EXEC` and `svchost.exe`. +- [x] **W1.F.10.b** — Implement `src/verdict/planning/executor_prompt.py::render_executor_prompt(role: str) -> str` that composes examiner_caveats.md + relevant hunt_evil entries. +- [x] **W1.F.10.c** — Commit: `feat(planning): executor system prompt with caveats + hunt evil [W1.F.10]` ### W1.F.11 — Apply v4.6 P6 to v4.5 audit doc - [ ] **W1.F.11** — Append Tier-1 caveat caveat. Commit: `docs(audit): v4.6 P6 — Tier-1 caveats encoded [W1.F.11]` @@ -531,19 +554,19 @@ The 12 tool wrappers ship in W2.E. This phase ships the schema scaffolding + `ps - [ ] **W1.G.4** — Author migration policy: `schema_version` field on all top-level schemas; breaking changes ship a `migrations/v{N}_to_v{N+1}.py` script. Commit: `docs(release): document schema migration policy [W1.G.4]` ### W1.G.5 — `Planner` Protocol + `CloudPlanner` + `LocalPlanner` (Beaver collaborates) -- [ ] **W1.G.5.a** — Failing test `tests/planning/test_planner_protocol.py::test_protocol_returns_investigation_plan`. Plus `test_planner_bound_at_gateway_init` — assert mode-switching code lives in `runtime/mode_detect.py`, not in `planner_node`. -- [ ] **W1.G.5.b** — Implement `verdict/planning/planner.py` with the Protocol + two impls. -- [ ] **W1.G.5.c** — Commit: `feat(planning): Planner Protocol + CloudPlanner + LocalPlanner [W1.G.5]` +- [x] **W1.G.5.a** — Failing test `tests/planning/test_planner_protocol.py::test_protocol_returns_investigation_plan`. Plus `test_planner_bound_at_gateway_init` — assert mode-switching code lives in `runtime/mode_detect.py`, not in `planner_node`. +- [x] **W1.G.5.b** — Implement `src/verdict/planning/planner.py` with the Protocol + two impls. +- [x] **W1.G.5.c** — Commit: `feat(planning): Planner Protocol + CloudPlanner + LocalPlanner [W1.G.5]` ### W1.G.6 — HMAC key handling (TPM-backed if present, else gpg-encrypted) -- [ ] **W1.G.6.a** — Failing integration tests: `tests/ledger/test_hmac_key.py::test_tpm_path_when_dev_tpmrm0_present` runs only on hosts with `/dev/tpmrm0` and verifies the TPM-backed path; `test_gpg_path_when_dev_tpmrm0_absent` runs in the CI environment that lacks `/dev/tpmrm0` and verifies the gpg-encrypted fallback. If the host does not match a test prerequisite, fail with a prerequisite diagnostic rather than simulating the device. -- [ ] **W1.G.6.b** — Implement `verdict/ledger/hmac_key.py` with both paths. Passphrase prompt at gateway init for the gpg path. -- [ ] **W1.G.6.c** — Commit: `feat(ledger): HMAC key TPM-backed or gpg-encrypted [W1.G.6]` +- [x] **W1.G.6.a** — Failing integration tests: `tests/ledger/test_hmac_key.py::test_tpm_path_when_dev_tpmrm0_present` runs only on hosts with `/dev/tpmrm0` and verifies the TPM-backed path; `test_gpg_path_when_dev_tpmrm0_absent` runs in the CI environment that lacks `/dev/tpmrm0` and verifies the gpg-encrypted fallback. If the host does not match a test prerequisite, fail with a prerequisite diagnostic rather than simulating the device. +- [x] **W1.G.6.b** — Implement `src/verdict/ledger/hmac_key.py` with both paths. Passphrase prompt at gateway init for the gpg path. +- [x] **W1.G.6.c** — Commit: `feat(ledger): HMAC key TPM-backed or gpg-encrypted [W1.G.6]` ### W1.G.7 — Evidence manifest with periodic re-hash check -- [ ] **W1.G.7.a** — Failing test `tests/runtime/test_evidence_recheck.py::test_recheck_every_10_super_steps`. Plus `test_mismatch_writes_ledger_entry_and_halts`. -- [ ] **W1.G.7.b** — Implement re-hash loop in `verdict/runtime/evidence_recheck.py`. Mismatch → `LedgerEntry(event_type="evidence_hash_recheck")` with both hashes + halt with `HashMismatchError`. -- [ ] **W1.G.7.c** — Commit: `feat(runtime): periodic evidence re-hash check (10 super-steps) [W1.G.7]` +- [x] **W1.G.7.a** — Failing test `tests/runtime/test_evidence_recheck.py::test_recheck_every_10_super_steps`. Plus `test_mismatch_writes_ledger_entry_and_halts`. +- [x] **W1.G.7.b** — Implement re-hash loop in `src/verdict/runtime/evidence_recheck.py`. Mismatch → `LedgerEntry(event_type="evidence_hash_recheck")` with both hashes + halt with `HashMismatchError`. +- [x] **W1.G.7.c** — Commit: `feat(runtime): periodic evidence re-hash check (10 super-steps) [W1.G.7]` ## Week 1 — acceptance gates @@ -551,17 +574,17 @@ By end of day Thursday May 8 ALL the following must be true. If any is FALSE on | Gate | Verification command | |---|---| -| All schema tests pass | `uv run --directory services/agent pytest tests/schemas/ -v` | +| All schema tests pass | `uv run pytest tests/schemas/ -v` | | All playbook tests pass | `uv run pytest tests/playbooks/ -v` | | All knowledge tests pass | `uv run pytest tests/knowledge/ -v` | -| Microsandbox spawns + runs vol3 -h on a sample image | `bash scripts/healthcheck.sh microsandbox` | +| Microsandbox spawns + runs vol3 -h on a sample image | `python -c "import microsandbox; print('ok')"` (full check lands at W3.F.1 `scripts/healthcheck.sh`) | | SGLang serves both Qwen3 + GLM with ≥98% tool-call parse rate | Output of `python scripts/inference-smoke.py` | | Langfuse UI loads + smoke trace renders | `curl http://localhost:3000/api/public/health` returns 200 | -| Inspect AI hello-world passes | `inspect eval inspect_ai/tasks/hello_world.py` | +| Inspect AI per-mode eval scaffolds present | `ls inspect_ai/tasks/verdict_eval_{cloud,airgap,dual}.py` | | `vol_psscan` wrapper integration test passes | `uv run pytest tests/tools/test_vol_psscan.py -v` | | Architecture-review docs present | `ls docs/{RELEASE,FAILURE_MODES}.md` | -| `examiner_caveats.md` includes all 7 CaveatID values | `grep -c "## " verdict/planning/prompts/examiner_caveats.md` returns 7 | -| `hunt_evil.yml` has 8 canonical processes | `python -c "import yaml; print(len(yaml.safe_load(open('verdict/knowledge/hunt_evil.yml'))))"` returns 8 | +| `examiner_caveats.md` includes all 7 CaveatID values | `grep -c "## " src/verdict/planning/prompts/examiner_caveats.md` returns 7 | +| `hunt_evil.yml` has 8 canonical processes | `python -c "import yaml; print(len(yaml.safe_load(open('src/verdict/knowledge/hunt_evil.yml'))['process_baselines']))"` returns 8 | | Schema and DFIR rule patches represented in current architecture | `docs/ARCHITECTURE.md` cites the current decisions | | Conventional Commits enforced (no `--no-verify`) | `git log --oneline -50 | grep -c '^[a-f0-9]\+ \(feat\|test\|fix\|docs\|chore\)' = 50` | @@ -572,7 +595,7 @@ If any gate is RED on May 8: **descope before slipping**. Drop in this priority # WEEK 2 (May 9 – May 15): Tool surface + Plan-then-Execute refactor **Theme:** Wrap all 12 SIFT tools as MCP tools. Refactor LangGraph topology to explicit Plan-then-Execute. Add `planner_critique_node`. Wire per-tool args validators. Split plaso/Hayabusa. -**Critical-path output:** All 12 tools callable through gateway. LangGraph compiles with 9 nodes: `planner` → `planner_critique` → `comprehension_gate` → `executor_fanout` (composes DenyRuleWrapper / ToolExecutor / LedgerEmitter per branch) → `pivot` → `quorum` → `replan` → `unverifiable_finalize` → `finalize`. +**Critical-path output:** All 12 tools callable through gateway. LangGraph compiles with 8 nodes: `planner` → `planner_critique` → `comprehension_gate` → `executor_fanout` (composes DenyRuleWrapper / ToolExecutor / LedgerEmitter per branch) → `pivot` → `quorum` → `replan` → `finalize`. (`unverifiable_finalize_node` is a helper called from `replan_node`, not a registered graph node.) **If this week slips:** week 3 verifier work pushes; cut pivot_node + unverifiable_finalize from v1; ship pure replan_max=3 → quietly stuck CONTESTED (v4.4 SHOULD-FIX leaks back in). **Cumulative team-days:** Tim ~5, Beaver ~5, Haley ~1, KP ~3. @@ -580,7 +603,7 @@ If any gate is RED on May 8: **descope before slipping**. Drop in this priority For each tool: - [ ] Failing integration test in `tests/tools/test_.py` against a fixture. -- [ ] Implement `verdict/tools/.py` extending `ToolWrapper`. +- [ ] Implement `src/verdict/tools/.py` extending `ToolWrapper`. - [ ] Add to gateway tool registry. - [ ] Commit: `feat(tools): wrapper [W2.A.]` @@ -609,25 +632,25 @@ For each tool: ## Phase W2.B — Plan-then-Execute LangGraph refactor (Beaver, ~2 days) -### W2.B.1 — Five core nodes -- [ ] **W2.B.1.a** — Failing test `tests/graph/test_topology_compiles.py::test_five_nodes_present`. Assert nodes `planner`, `executor_fanout`, `quorum`, `replan`, `finalize` exist on the compiled graph. -- [ ] **W2.B.1.b** — Implement `verdict/graph/topology.py::build_graph(mode: Mode) -> CompiledGraph` and `verdict/graph/nodes.py` with real minimal node bodies for all five. Each node must read/write typed state and raise explicit `NotImplementedError` only for dependencies that are scheduled in a later task and never on the production happy path. -- [ ] **W2.B.1.c** — Commit: `feat(graph): five-node Plan-then-Execute topology [W2.B.1]` +### W2.B.1 — Eight registered nodes +- [x] **W2.B.1.a** — Failing test `tests/graph/test_topology_compiles.py::test_planner_critique_is_wired_before_comprehension_gate`. Assert all 8 registered nodes (`planner`, `planner_critique`, `comprehension_gate`, `executor_fanout`, `pivot`, `quorum`, `replan`, `finalize`) and entrypoint/edge wiring exist on the compiled graph. +- [x] **W2.B.1.b** — Implement `src/verdict/graph/topology.py::build_graph(mode: Mode) -> CompiledGraph` and `src/verdict/graph/nodes.py` with real minimal node bodies for all eight. Each node must read/write typed state and raise explicit `NotImplementedError` only for dependencies that are scheduled in a later task and never on the production happy path. +- [x] **W2.B.1.c** — Commit: `feat(graph): eight-node Plan-then-Execute topology [W2.B.1]` ### W2.B.2 — `comprehension_gate` node (v4.3) -- [ ] **W2.B.2.a** — Failing test `tests/graph/test_comprehension_gate.py::test_consensus_advances_executor_work` and `test_mismatch_routes_to_clarify`. -- [ ] **W2.B.2.b** — Implement gate node. Collects `PlanComprehensionEcho`s; validates consensus on `parsed_positive_hypothesis_ids`, `parsed_negative_hypothesis_ids`, `parsed_success_criteria_hash`. -- [ ] **W2.B.2.c** — Commit: `feat(graph): comprehension_gate validates executor consensus [W2.B.2]` +- [x] **W2.B.2.a** — Failing test `tests/graph/test_comprehension_gate.py::test_consensus_advances_executor_work` and `test_mismatch_routes_to_clarify`. +- [x] **W2.B.2.b** — Implement gate node. Collects `PlanComprehensionEcho`s; validates consensus on `parsed_positive_hypothesis_ids`, `parsed_negative_hypothesis_ids`, `parsed_success_criteria_hash`. +- [x] **W2.B.2.c** — Commit: `feat(graph): comprehension_gate validates executor consensus [W2.B.2]` ### W2.B.3 — `ComprehensionMismatch` ledger entry on disagreement -- [ ] **W2.B.3.a** — Failing test: ledger contains structured per-executor diff on mismatch. -- [ ] **W2.B.3.b** — Implement event type + payload schema. -- [ ] **W2.B.3.c** — Commit: `feat(ledger): ComprehensionMismatch event with per-executor diff [W2.B.3]` +- [x] **W2.B.3.a** — Failing test: ledger contains structured per-executor diff on mismatch. +- [x] **W2.B.3.b** — Implement event type + payload schema. +- [x] **W2.B.3.c** — Commit: `feat(ledger): ComprehensionMismatch event with per-executor diff [W2.B.3]` ### W2.B.4 — Reducer pattern for fanout merge + race test -- [ ] **W2.B.4.a** — Failing test `tests/graph/test_fanout_race.py::test_4_executors_merge_deterministically`. 4 executors, randomized 0–500ms sleep each; assert final state contains all 4 outputs in deterministic order. -- [ ] **W2.B.4.b** — Implement `verdict/graph/reducers.py` with `Annotated[..., reducer]` for `executor_results` field. -- [ ] **W2.B.4.c** — Commit: `feat(graph): reducer pattern for parallel-executor merge [W2.B.4]` +- [x] **W2.B.4.a** — Failing test `tests/graph/test_fanout_race.py::test_4_executors_merge_deterministically`. 4 executors, randomized 0–500ms sleep each; assert final state contains all 4 outputs in deterministic order. +- [x] **W2.B.4.b** — Implement `src/verdict/graph/reducers.py` with `Annotated[..., reducer]` for `executor_results` field. +- [x] **W2.B.4.c** — Commit: `feat(graph): reducer pattern for parallel-executor merge [W2.B.4]` ### W2.B.5 — Pin LangGraph version - [ ] **W2.B.5** — Pin in `pyproject.toml`. Commit: `chore(deps): pin langgraph version that passes fanout-race test [W2.B.5]` @@ -635,36 +658,36 @@ For each tool: ## Phase W2.C — `executor_work` split into 3 wrappers (v4.5 fix from architecture review) ### W2.C.1 — `DenyRuleWrapper` (Layer 2 of three-layer immutability) -- [ ] **W2.C.1.a** — Failing test `tests/graph/test_deny_rule_wrapper.py::test_blocks_evidence_writes_in_all_modes`. Test args denied for cloud, airgap, dual. -- [ ] **W2.C.1.b** — Implement `verdict/graph/wrappers/deny_rule.py`. Layer 2 of three-layer defense — fires regardless of model. Owns deny-rule list (Tim). -- [ ] **W2.C.1.c** — Commit: `feat(graph): DenyRuleWrapper Layer 2 immutability [W2.C.1]` +- [x] **W2.C.1.a** — Failing test `tests/graph/test_deny_rule_wrapper.py::test_blocks_evidence_writes_in_all_modes`. Test args denied for cloud, airgap, dual. +- [x] **W2.C.1.b** — Implement `src/verdict/graph/wrappers/deny_rule.py`. Layer 2 of three-layer defense — fires regardless of model. Owns deny-rule list (Tim). +- [x] **W2.C.1.c** — Commit: `feat(graph): DenyRuleWrapper Layer 2 immutability [W2.C.1]` ### W2.C.2 — `ToolExecutor` (Beaver owns) -- [ ] **W2.C.2.a** — Failing test for typed dispatch + microsandbox spawn + result parsing into ToolOutput. -- [ ] **W2.C.2.b** — Implement. -- [ ] **W2.C.2.c** — Commit: `feat(graph): ToolExecutor wrapper [W2.C.2]` +- [x] **W2.C.2.a** — Failing test for typed dispatch + microsandbox spawn + result parsing into ToolOutput. +- [x] **W2.C.2.b** — Implement. +- [x] **W2.C.2.c** — Commit: `feat(graph): ToolExecutor wrapper [W2.C.2]` ### W2.C.3 — `LedgerEmitter` (Tim owns) -- [ ] **W2.C.3.a** — Failing test for write+fsync+verify-readback. Plus chain-integrity assertion. -- [ ] **W2.C.3.b** — Implement `verdict/graph/wrappers/ledger_emitter.py` + `verdict/ledger/writer.py` with the durability discipline. -- [ ] **W2.C.3.c** — Commit: `feat(ledger): LedgerEmitter wrapper with write+fsync+verify-readback [W2.C.3]` +- [x] **W2.C.3.a** — Failing test for write+fsync+verify-readback. Plus chain-integrity assertion. +- [x] **W2.C.3.b** — Implement `src/verdict/graph/wrappers/ledger_emitter.py` + `src/verdict/ledger/writer.py` with the durability discipline. +- [x] **W2.C.3.c** — Commit: `feat(ledger): LedgerEmitter wrapper with write+fsync+verify-readback [W2.C.3]` ### W2.C.4 — Compose three wrappers + replace executor_work -- [ ] **W2.C.4.a** — Failing test: end-to-end through composed `DenyRuleWrapper → ToolExecutor → LedgerEmitter`. -- [ ] **W2.C.4.b** — Wire composition in `verdict/graph/topology.py`. -- [ ] **W2.C.4.c** — Commit: `feat(graph): compose 3-wrapper executor_work [W2.C.4]` +- [x] **W2.C.4.a** — Failing test: end-to-end through composed `DenyRuleWrapper → ToolExecutor → LedgerEmitter`. +- [x] **W2.C.4.b** — Wire composition in `src/verdict/graph/topology.py`. +- [x] **W2.C.4.c** — Commit: `feat(graph): compose 3-wrapper executor_work [W2.C.4]` ## Phase W2.D — `planner_critique_node` (Beaver, ~1 day) ### W2.D.1 — CoVe (Chain-of-Verification, Dhuliawala 2023) -- [ ] **W2.D.1.a** — Failing test `tests/planning/test_planner_critique.py::test_failed_questions_route_back_to_planner`. Plus `test_all_pass_advances_to_comprehension_gate`. -- [ ] **W2.D.1.b** — Implement `verdict/planning/planner_critique.py`. Same model drafts CoVe questions ABOUT THE PLAN ITSELF (does plan cover most-likely attacker techniques given evidence type? does it have positive AND negative for each artifact family? are success criteria measurable?). Answers them against case_init evidence summary; failed questions route back to planner with hint. -- [ ] **W2.D.1.c** — Commit: `feat(planning): planner_critique_node CoVe [W2.D.1]` +- [x] **W2.D.1.a** — Failing test `tests/planning/test_planner_critique.py::test_failed_questions_route_back_to_planner`. Plus `test_all_pass_advances_to_comprehension_gate`. +- [x] **W2.D.1.b** — Implement `src/verdict/planning/planner_critique.py`. Same model drafts CoVe questions ABOUT THE PLAN ITSELF (does plan cover most-likely attacker techniques given evidence type? does it have positive AND negative for each artifact family? are success criteria measurable?). Answers them against case_init evidence summary; failed questions route back to planner with hint. +- [x] **W2.D.1.c** — Commit: `feat(planning): planner_critique_node CoVe [W2.D.1]` ### W2.D.2 — `PlannerCritiqueVerdict` schema + `critique_verdict` ledger event -- [ ] **W2.D.2.a** — Failing test `tests/planning/test_planner_critique_verdict.py::test_schema_rejects_missing_failed_questions_when_route_back`. Plus `test_ledger_emits_critique_verdict_event_with_route_decision`. Assertions: `PlannerCritiqueVerdict(route="planner", failed_questions=[]).model_validate()` raises `ValidationError`; `ledger.last_entry.event_type == "critique_verdict"` after `planner_critique_node` runs. -- [ ] **W2.D.2.b** — Wire into LangGraph + ledger. -- [ ] **W2.D.2.c** — Commit: `feat(graph): planner_critique_node wired between planner + comprehension_gate [W2.D.2]` +- [x] **W2.D.2.a** — Failing test `tests/planning/test_planner_critique_verdict.py::test_schema_rejects_missing_failed_questions_when_route_back`. Plus `test_ledger_emits_critique_verdict_event_with_route_decision`. Assertions: `PlannerCritiqueVerdict(route="planner", failed_questions=[]).model_validate()` raises `ValidationError`; `ledger.last_entry.event_type == "critique_verdict"` after `planner_critique_node` runs. +- [x] **W2.D.2.b** — Wire into LangGraph + ledger. +- [x] **W2.D.2.c** — Commit: `feat(graph): planner_critique_node wired between planner + comprehension_gate [W2.D.2]` ### W2.D.3 — Planner CoT capture - [ ] **W2.D.3.a** — Failing test `tests/planning/test_cot_capture.py::test_gzipped_cot_in_ledger`. Plus `test_8kb_attached_to_langfuse_span`. @@ -680,7 +703,7 @@ For each tool: ### W2.E.1 — `args_validators` framework - [ ] **W2.E.1.a** — Failing test `tests/tools/test_args_validator.py::test_unknown_flag_raises_modelretry`. Plus `test_invalid_pid_type_raises`. -- [ ] **W2.E.1.b** — Implement `verdict/tools/args_validators.py` with Pydantic-AI `args_validator` framework. `tool_arg_retry_max=2`, then UNVERIFIABLE. +- [ ] **W2.E.1.b** — Implement `src/verdict/tools/args_validators.py` with Pydantic v2 `ArgsValidator`. `tool_arg_retry_max=2`, then UNVERIFIABLE. - [ ] **W2.E.1.c** — Commit: `feat(tools): args_validator framework with retry budget 2 [W2.E.1]` ### W2.E.2 — `vol3` validators (parse `vol3 --help` once at startup; allow-list of 26 plugins) @@ -695,12 +718,12 @@ For each tool: ### W2.E.4 — `Hayabusa` flag-matrix validator - [ ] **W2.E.4.a** — Failing test: invalid timeline-flag combinations rejected. -- [ ] **W2.E.4.b** — Implement against the matrix in `verdict/playbooks/memory.yml` rules section. +- [ ] **W2.E.4.b** — Implement against the matrix in `src/verdict/playbooks/memory.yml` rules section. - [ ] **W2.E.4.c** — Commit: `feat(tools): hayabusa flag-matrix validator [W2.E.4]` ### W2.E.5 — Sanitization scanner for prompt injection in tool stdout - [ ] **W2.E.5.a** — Failing test `tests/tools/test_sanitization.py::test_detects_ignore_previous_instructions`. Plus standard jailbreak suffixes. -- [ ] **W2.E.5.b** — Implement `verdict/tools/sanitization.py`. Patterns include `IGNORE PREVIOUS`, `SYSTEM:`, ``, `[INST]`, `### Instruction`. Detected → `ToolOutput.sanitization_flags` populated; surface to planner. +- [ ] **W2.E.5.b** — Implement `src/verdict/tools/sanitization.py`. Patterns include `IGNORE PREVIOUS`, `SYSTEM:`, ``, `[INST]`, `### Instruction`. Detected → `ToolOutput.sanitization_flags` populated; surface to planner. - [ ] **W2.E.5.c** — Commit: `feat(tools): sanitization scanner for prompt-injection patterns [W2.E.5]` ## Phase W2.F — Plaso/Hayabusa split (Beaver, ~0.5 day) @@ -746,11 +769,11 @@ For each tool: | Gate | Verification | |---|---| -| All 23 tool wrappers callable via gateway | `pytest tests/tools/ -v` green | +| All 24 tool wrappers callable via gateway | `pytest tests/tools/ -v` green | | LangGraph compiles in all three modes | `pytest tests/graph/test_topology_compiles.py` green for cloud/airgap/dual | | `comprehension_gate` + `planner_critique_node` integrated | Inspect AI smoke run shows both nodes in trace | -| `executor_work` is composition of 3 wrappers, three owners | `git blame` shows distinct authors on `deny_rule.py`, `tool_executor.py`, `ledger_emitter.py` | -| Plaso + Hayabusa split into extract+filter | `grep -c "extract" verdict/tools/plaso_*.py` returns ≥1 each | +| `executor_work` is composition of wrappers with distinct owners | `git blame` shows distinct authors on `deny_rule.py`, `ledger_emitter.py` | +| Plaso + Hayabusa split into extract+filter | `grep -c "extract" src/verdict/tools/plaso_*.py` returns ≥1 each | | Args validators reject unknown flags | `pytest tests/tools/test_args_validator.py` green | | Sanitization flags detected on injection patterns | `pytest tests/tools/test_sanitization.py` green | | Langfuse spans show real prompt_tokens > 0 | Manual UI check + integration test | @@ -787,7 +810,7 @@ If RED: drop W2.D.3 (planner CoT capture, push to W3) → drop W2.E.3-4 (plaso/H ### W3.B.1 — TSI provider Pattern 2 - [ ] **W3.B.1.a** — Failing test `tests/sandboxes/test_tsi_provider.py::test_credentials_never_enter_microvm`. Use tcpdump capture comparison: bearer header on egress to `opencti.local:8080`, NOT inside microvm. -- [ ] **W3.B.1.b** — Implement `verdict/sandboxes/tsi_provider.py` per v4.5 lines 482–489. +- [ ] **W3.B.1.b** — Implement `src/verdict/sandboxes/tsi_provider.py` per v4.5 lines 482–489. - [ ] **W3.B.1.c** — Commit: `feat(sandbox): TSI Pattern 2 with credential injection [W3.B.1]` ### W3.B.2 — TSI demo prep (Tim's W4.3 carryover) @@ -796,50 +819,50 @@ If RED: drop W2.D.3 (planner CoT capture, push to W3) → drop W2.E.3-4 (plaso/H - [ ] **W3.B.2.c** — Commit: `chore(demo): TSI tcpdump demonstration assets [W3.B.2]` ### W3.B.3 — Ledger redaction pass -- [ ] **W3.B.3.a** — Failing test: `test_redacts_authorization_header_before_hash`. Plus `auth_user`, `api_key`. -- [ ] **W3.B.3.b** — Implement `verdict/ledger/redaction.py`. Strip + record in `payload_redactions` field. -- [ ] **W3.B.3.c** — Commit: `feat(ledger): redact auth fields before hash + write [W3.B.3]` +- [x] **W3.B.3.a** — Failing test: `test_redacts_authorization_header_before_hash`. Plus `auth_user`, `api_key`. +- [x] **W3.B.3.b** — Implement `src/verdict/ledger/redaction.py`. Strip + record in `payload_redactions` field. +- [x] **W3.B.3.c** — Commit: `feat(ledger): redact auth fields before hash + write [W3.B.3]` ## Phase W3.C — Mode lock (Beaver, ~0.5 day) ### W3.C.1 — Mode-lock enforcement at `case_init` -- [ ] **W3.C.1.a** — Failing test `tests/runtime/test_mode_lock.py::test_resume_with_different_mode_refuses`. Plus `test_mode_at_case_init_immutable`. -- [ ] **W3.C.1.b** — Implement: write `mode_at_case_init` to ledger; refuse to advance if resume detects mode mismatch with current autodetect. -- [ ] **W3.C.1.c** — Commit: `feat(runtime): mode lock at case_init enforced on resume [W3.C.1]` +- [x] **W3.C.1.a** — Failing test `tests/runtime/test_mode_lock.py::test_resume_with_different_mode_refuses`. Plus `test_mode_at_case_init_immutable`. +- [x] **W3.C.1.b** — Implement: write `mode_at_case_init` to ledger; refuse to advance if resume detects mode mismatch with current autodetect. +- [x] **W3.C.1.c** — Commit: `feat(runtime): mode lock at case_init enforced on resume [W3.C.1]` ### W3.C.2 — `verdict reverify` command - [ ] **W3.C.2.a** — Failing test: `verdict reverify --mode dual` produces parallel verdict chain without mutating original. -- [ ] **W3.C.2.b** — Implement in `verdict/cli/reverify.py`. Fork a new parallel chain with the original `EvidenceManifest`, a new `chain_id`, the requested `mode_at_case_init`, and fresh mode-appropriate planner/executor/quorum ledger entries. Never mutate the original chain. +- [ ] **W3.C.2.b** — Implement in `src/verdict/cli/reverify.py`. Fork a new parallel chain with the original `EvidenceManifest`, a new `chain_id`, the requested `mode_at_case_init`, and fresh mode-appropriate planner/executor/quorum ledger entries. Never mutate the original chain. - [ ] **W3.C.2.c** — Commit: `feat(cli): verdict reverify produces parallel verdict chain [W3.C.2]` ## Phase W3.D — Pivot + replan + unverifiable_finalize (Beaver, ~1 day) ### W3.D.1 — `pivot_node` (cheap follow-up) -- [ ] **W3.D.1.a** — Failing test `tests/graph/test_pivot_node.py::test_adds_one_hypothesis_within_pivot_max_15`. Plus `test_pivot_does_not_re_enter_planner`. -- [ ] **W3.D.1.b** — Implement. Bounded `pivot_max=15` in `InvestigationPlan.pivot_budget`. -- [ ] **W3.D.1.c** — Commit: `feat(graph): pivot_node distinct from replan_node (max=15) [W3.D.1]` +- [x] **W3.D.1.a** — Failing test `tests/graph/test_pivot_node.py::test_adds_one_hypothesis_within_pivot_max_15`. Plus `test_pivot_does_not_re_enter_planner`. +- [x] **W3.D.1.b** — Implement. Bounded `pivot_max=15` in `InvestigationPlan.pivot_budget`. +- [x] **W3.D.1.c** — Commit: `feat(graph): pivot_node distinct from replan_node (max=15) [W3.D.1]` ### W3.D.2 — `replan_max=3` explicit budget on `InvestigationPlan` -- [ ] **W3.D.2.a** — Failing test: `replan_budget` field defaults to 3. -- [ ] **W3.D.2.b** — Implement. -- [ ] **W3.D.2.c** — Commit: `feat(schema): InvestigationPlan.replan_budget=3 explicit [W3.D.2]` +- [x] **W3.D.2.a** — Failing test: `replan_budget` field defaults to 3. +- [x] **W3.D.2.b** — Implement. +- [x] **W3.D.2.c** — Commit: `feat(schema): InvestigationPlan.replan_budget=3 explicit [W3.D.2]` ### W3.D.3 — `unverifiable_finalize_node` -- [ ] **W3.D.3.a** — Failing test `tests/graph/test_unverifiable_finalize.py::test_writes_unverifiable_finding_at_replan_iteration_4`. Plus `test_writes_exhausted_replan_ledger_event`, `test_calls_interrupt`, and `test_resume_does_not_duplicate_exhausted_replan_ledger_entry`. -- [ ] **W3.D.3.b** — Implement. Use deterministic idempotency key `case_id + chain_id + hypothesis_id + replan_iteration + "exhausted_replan"`; before writing the ledger entry, check whether that key already exists and skip the write on resume. No non-idempotent side effects may occur before `interrupt()`. -- [ ] **W3.D.3.c** — Commit: `feat(graph): unverifiable_finalize_node + exhausted_replan event [W3.D.3]` +- [x] **W3.D.3.a** — Failing test `tests/graph/test_unverifiable_finalize.py::test_writes_unverifiable_finding_at_replan_iteration_4`. Plus `test_writes_exhausted_replan_ledger_event`, `test_calls_interrupt`, and `test_resume_does_not_duplicate_exhausted_replan_ledger_entry`. +- [x] **W3.D.3.b** — Implement. Use deterministic idempotency key `case_id + chain_id + hypothesis_id + replan_iteration + "exhausted_replan"`; before writing the ledger entry, check whether that key already exists and skip the write on resume. No non-idempotent side effects may occur before `interrupt()`. +- [x] **W3.D.3.c** — Commit: `feat(graph): unverifiable_finalize_node + exhausted_replan event [W3.D.3]` ### W3.D.4 — Wire `interrupt()` properly - [ ] **W3.D.4.a** — Failing test: analyst can `update_state` and resume after interrupt. -- [ ] **W3.D.4.b** — Implement `verdict/graph/interrupt.py` with helpers for resume-from-interrupt path. +- [ ] **W3.D.4.b** — Implement `src/verdict/graph/interrupt.py` with helpers for resume-from-interrupt path. - [ ] **W3.D.4.c** — Commit: `feat(graph): interrupt() helpers for HITL resume [W3.D.4]` ## Phase W3.E — Checkpointing (Beaver, ~1 day) ### W3.E.1 — `SqliteSaver` with WAL + synchronous=FULL -- [ ] **W3.E.1.a** — Failing test `tests/graph/test_checkpoint.py::test_pragma_journal_mode_wal`. Plus `test_pragma_synchronous_full`. -- [ ] **W3.E.1.b** — Implement `verdict/graph/checkpoint.py` with `PRAGMA journal_mode=WAL; PRAGMA synchronous=FULL;`. -- [ ] **W3.E.1.c** — Commit: `feat(graph): SqliteSaver with WAL + synchronous=FULL [W3.E.1]` +- [x] **W3.E.1.a** — Failing test `tests/graph/test_checkpoint.py::test_pragma_journal_mode_wal`. Plus `test_pragma_synchronous_full`. +- [x] **W3.E.1.b** — Implement `src/verdict/graph/checkpoint.py` with `PRAGMA journal_mode=WAL; PRAGMA synchronous=FULL;`. +- [x] **W3.E.1.c** — Commit: `feat(graph): SqliteSaver with WAL + synchronous=FULL [W3.E.1]` ### W3.E.2 — `thread_id = case_id` everywhere - [ ] **W3.E.2.a** — Failing test: gateway invocation passes `config={"configurable": {"thread_id": case_id}}`. @@ -856,7 +879,7 @@ If RED: drop W2.D.3 (planner CoT capture, push to W3) → drop W2.E.3-4 (plaso/H ### W3.E.5 — `trace_id` ↔ ledger cross-link - [ ] **W3.E.5.a** — Failing test `tests/observability/test_trace_link.py::test_ledger_entry_has_langfuse_trace_id`. Plus `test_langfuse_span_has_ledger_entry_id_attribute`. -- [ ] **W3.E.5.b** — Implement `verdict/observability/trace_link.py` — bi-directional linking. +- [ ] **W3.E.5.b** — Implement `src/verdict/observability/trace_link.py` — bi-directional linking. - [ ] **W3.E.5.c** — Commit: `feat(observability): trace_id ↔ ledger bidirectional cross-link [W3.E.5]` ### W3.E.6 — Kill-9 chaos test @@ -868,7 +891,7 @@ If RED: drop W2.D.3 (planner CoT capture, push to W3) → drop W2.E.3-4 (plaso/H ### W3.F.1 — `/health` endpoint - [ ] **W3.F.1.a** — Failing test: returns `{mode, components: {langfuse, sglang, microsandbox, ledger}, last_healthcheck_utc}`. -- [ ] **W3.F.1.b** — Implement `verdict/cli/health.py`. +- [ ] **W3.F.1.b** — Implement `src/verdict/cli/health.py`. - [ ] **W3.F.1.c** — Commit: `feat(cli): /health endpoint [W3.F.1]` ### W3.F.2 — Continuous healthcheck loop (30s interval) @@ -929,7 +952,7 @@ If RED: drop W3.E.6 (chaos test, ship without quantified guarantee) → drop W3. ## Phase W4.B — `lolbins.yml` knowledge file (KP, ~0.5 day) -### W4.B.1 — `verdict/knowledge/lolbins.yml` +### W4.B.1 — `src/verdict/knowledge/lolbins.yml` - [ ] **W4.B.1.a** — Failing test: ≥6 LOLBins with shape patterns + MITRE technique mapping (T1218 sub-techniques). - [ ] **W4.B.1.b** — Author per Appendix C.5. - [ ] **W4.B.1.c** — Commit: `feat(knowledge): lolbins.yml with cmdline shapes + T1218 mapping [W4.B.1]` @@ -996,12 +1019,12 @@ If RED: drop W3.E.6 (chaos test, ship without quantified guarantee) → drop W3. ## Phase W4.F — Prompt engineering (Beaver + KP, ~1 day) ### W4.F.1 — Negative-hypothesis few-shot examples -- [ ] **W4.F.1.a** — Author `verdict/planning/prompts/negative_hypothesis_examples.md` with 5 high-quality few-shots demonstrating: T1547 ruling out T1055; T1543.003 ruling out T1543.001; etc. +- [ ] **W4.F.1.a** — Author `src/verdict/planning/prompts/negative_hypothesis_examples.md` with 5 high-quality few-shots demonstrating: T1547 ruling out T1055; T1543.003 ruling out T1543.001; etc. - [ ] **W4.F.1.b** — Wire into planner system prompt. - [ ] **W4.F.1.c** — Commit: `feat(prompts): 5 negative-hypothesis few-shot examples [W4.F.1]` ### W4.F.2 — Adversarial-reasoning prompt -- [ ] **W4.F.2.a** — Author `verdict/planning/prompts/adversarial_reasoning.md`. Inject "if I were the attacker, where would I hide?" — Scheduled Tasks `\Microsoft\Windows\` namespace, WMI event subscriptions, IFEO debugger keys (per project MEMORY.md persistence top-5). +- [ ] **W4.F.2.a** — Author `src/verdict/planning/prompts/adversarial_reasoning.md`. Inject "if I were the attacker, where would I hide?" — Scheduled Tasks `\Microsoft\Windows\` namespace, WMI event subscriptions, IFEO debugger keys (per project MEMORY.md persistence top-5). - [ ] **W4.F.2.b** — Wire into planner system prompt. - [ ] **W4.F.2.c** — Commit: `feat(prompts): adversarial-reasoning planner injection [W4.F.2]` @@ -1067,7 +1090,7 @@ If RED: drop W4.B (LOLBin catalog → push to W5) → drop W4.F.2 (adversarial r ### W5.B.1 — OpenCTI MCP integration - [ ] **W5.B.1.a** — Failing integration test `tests/sandboxes/test_malware_vm_tsi.py::test_opencti_enrichment_via_tsi_keeps_key_out_of_vm`. Assertions: `tcpdump_capture(microvm_iface).bearer_count == 0` AND `tcpdump_capture(host_egress_to_opencti).bearer_count == 1` AND the resulting `Finding.enrichment` dict contains the OpenCTI threat-actor metadata. -- [ ] **W5.B.1.b** — Implement `verdict/adapters/opencti_mcp.py`. +- [ ] **W5.B.1.b** — Implement `src/verdict/adapters/opencti_mcp.py`. - [ ] **W5.B.1.c** — Commit: `feat(adapters): OpenCTI MCP via TSI [W5.B.1]` ### W5.B.2 — Velociraptor MCP via socfortress server @@ -1118,7 +1141,7 @@ If RED: drop W4.B (LOLBin catalog → push to W5) → drop W4.F.2 (adversarial r - [ ] **W5.E.3.c** — Commit: `feat(observability): Langfuse demo dashboard [W5.E.3]` ### W5.E.4 — HMAC-signed approval flow -- [ ] **W5.E.4.a** — Failing test: `verdict approve ` produces ledger entry with HMAC sig over Finding+approver+timestamp. +- [ ] **W5.E.4.a** — Failing test: `verdict approve --approver ` produces ledger entry with HMAC sig over Finding+approver+timestamp. - [ ] **W5.E.4.b** — Implement. - [ ] **W5.E.4.c** — Commit: `feat(cli): verdict approve with HMAC signing [W5.E.4]` @@ -1192,10 +1215,10 @@ If RED: drop W5.C optional adapters first → drop W5.B.3 (REMnux) → drop W5.E - [ ] **W6.C.6** — 500-word writeup for Devpost summarizing: problem, architecture, three innovations, accuracy results, demo video. References all 6 official judging criteria explicitly per `DEVPOST_COMPLIANCE.md` Part 3. Commit: `docs: Devpost submission writeup [W6.C.6]` ### W6.C.7 — rendered architecture visual (Devpost-required) -- [ ] **W6.C.7.a** — Author Mermaid or draw.io source covering: Examiner CLI, FastMCP gateway, Mode autodetect, Planner Protocol (CloudPlanner/LocalPlanner), planner_critique_node, comprehension_gate, executor_fanout (4 branches), executor_work split (DenyRuleWrapper → ToolExecutor → LedgerEmitter), pivot_node, quorum_node, replan/unverifiable_finalize, Microsandbox VMs, Evidence Vault (chattr +i, read-only mount), HMAC ledger, Langfuse, SqliteSaver checkpoint, optional out-of-band services. +- [x] **W6.C.7.a** — Author Mermaid or draw.io source covering: Examiner CLI, FastMCP gateway, Mode autodetect, Planner Protocol (CloudPlanner/LocalPlanner), planner_critique_node, comprehension_gate, executor_fanout (4 branches), executor_work split (DenyRuleWrapper → ToolExecutor → LedgerEmitter), pivot_node, quorum_node, replan/unverifiable_finalize, Microsandbox VMs, Evidence Vault (chattr +i, read-only mount), HMAC ledger, Langfuse, SqliteSaver checkpoint, optional out-of-band services. - [ ] **W6.C.7.b** — Render to SVG + PNG fallback for the final submission package. - [ ] **W6.C.7.c** — Reference from README + ARCHITECTURE.md + Devpost form. -- [ ] **W6.C.7.d** — Commit: `docs(submission): add rendered architecture visual [W6.C.7]` +- [x] **W6.C.7.d** — Commit: `docs(submission): add rendered architecture visual [W6.C.7]` ### W6.C.8 — `docs/RELEASE.md` evidence dataset section (Devpost-required) - [ ] **W6.C.8.a** — Author. Sections: (1) Datasets used (NIST CFReDS Hacking Case, Honeynet ransomware, 3 engineered cases). (2) Source attribution per dataset (URL, license, hash). (3) What VERDICT was tested against per case. (4) What VERDICT found per case (with finding_ids referencing accuracy report). (5) Limitations: Windows-only; no live-response; no Win11/macOS/Linux/network. @@ -1209,7 +1232,7 @@ If RED: drop W5.C optional adapters first → drop W5.B.3 (REMnux) → drop W5.E - [ ] **W6.C.9.d** — Commit: `feat(cli): export execution-logs format for Devpost compliance [W6.C.9]` ### W6.C.10 — `docs/RELEASE.md` novel contribution section (Devpost-required) -- [ ] **W6.C.10.a** — Author. Sections: (1) Project timeline (started 2026-05-02; substantially new work per Devpost rules §4 New & Existing). (2) What we built (mode-aware verifier, three-layer immutability, encoded forensic discipline, planner_critique CoVe, pivot vs replan distinction, schema-enforced caveat acknowledgment, DKOM/T1014 auto-detection, Hunt Evil masquerade catch, LOLBin matcher, agentskills.io skill bundle, custom Inspect AI scorers). (3) Pre-existing open source enumerated with license + source URL each (SIFT, Volatility 3, Hayabusa, plaso, EZ Tools, Microsandbox, SGLang, vLLM, LangGraph, Langfuse, OpenLLMetry, Inspect AI, Pydantic, Pydantic-AI, FastMCP, NeMo Guardrails, Claude Agent SDK, blake3). (4) What we extended vs replaced. +- [ ] **W6.C.10.a** — Author. Sections: (1) Project timeline (started 2026-05-02; substantially new work per Devpost rules §4 New & Existing). (2) What we built (mode-aware verifier, three-layer immutability, encoded forensic discipline, planner_critique CoVe, pivot vs replan distinction, schema-enforced caveat acknowledgment, DKOM/T1014 auto-detection, Hunt Evil masquerade catch, LOLBin matcher, agentskills.io skill bundle, custom Inspect AI scorers). (3) Pre-existing open source enumerated with license + source URL each (SIFT, Volatility 3, Hayabusa, plaso, EZ Tools, Microsandbox, SGLang, vLLM, LangGraph, Langfuse, OpenLLMetry, Inspect AI, Pydantic, FastMCP, Claude Agent SDK, blake3). (4) What we extended vs replaced. - [ ] **W6.C.10.b** — Cross-reference from README + Devpost form. - [ ] **W6.C.10.c** — Commit: `docs: NOVEL_CONTRIBUTION.md [W6.C.10]` @@ -1419,7 +1442,7 @@ If RED: drop W5.C optional adapters first → drop W5.B.3 (REMnux) → drop W5.E | Examiner Portal | v4.5 line 802 | CLI + JSONL + Langfuse UI is sufficient | | Multi-tenant deployment | this doc | Single-host v1 | | OpenSearch evidence indexing | (Valhuntir comparison) | Not needed for 3 demo cases | -| Anthropic Constitutional Classifiers integration | (research consideration) | NeMo Guardrails sufficient v1 | +| Anthropic Constitutional Classifiers integration | (research consideration) | DenyRuleWrapper + sanitization.py sufficient v1 | --- @@ -1446,7 +1469,7 @@ When in doubt, fall back to v4.5 for architecture rationale, v4.6 for schema pat # Appendix A — Schema bundle (copy-paste ready) -## A.1 — `verdict/schemas/artifact_class.py` +## A.1 — `src/verdict/schemas/artifact_class.py` ```python from enum import Enum @@ -1470,7 +1493,7 @@ class ArtifactClass(str, Enum): SIGMA_HIT = "sigma_hit" ``` -## A.2 — `verdict/schemas/caveat_id.py` +## A.2 — `src/verdict/schemas/caveat_id.py` ```python from enum import Enum @@ -1487,7 +1510,7 @@ class CaveatID(str, Enum): SYSMON_PROCESSGUID_OVER_PID = "sysmon_processguid_correlation_key_not_pid" ``` -## A.3 — `verdict/schemas/evidence.py` +## A.3 — `src/verdict/schemas/evidence.py` ```python from datetime import datetime @@ -1511,7 +1534,7 @@ class EvidenceManifest(BaseModel): schema_version: int = 1 ``` -## A.4 — `verdict/schemas/tool_output.py` +## A.4 — `src/verdict/schemas/tool_output.py` ```python from pathlib import Path @@ -1526,7 +1549,7 @@ class Artifact(BaseModel): class ToolOutput(BaseModel): tool_name: str # "vol3.windows.pslist" - tool_version: str # "vol3 2.10.0" + tool_version: str # "vol3 2.28.0" invocation_args: list[str] invocation_hash: str # blake3(name + version + args + evidence_hash) stdout_hash: str # SHA-256 of raw stdout @@ -1538,23 +1561,15 @@ class ToolOutput(BaseModel): schema_version: int = 1 ``` -## A.5 — `verdict/verification/cloud_self_consistency.py` +## A.5 — `src/verdict/verification/cloud_self_consistency.py` + +`derive_seeds` is already shipped in `src/verdict/verification/derive_seeds.py` +(W1.C.1). Implement `CloudSelfConsistency` here to use it: ```python from blake3 import blake3 import asyncio - -def derive_seeds(case_id: str) -> tuple[int, int, int]: - """Three different seeds, deterministic per case for reproducibility, - distinct so n=3 actually samples three different reasoning paths. - Wang et al. 2022 (arXiv:2203.11171) requires diverse paths — temp 0 - + same seed = identical output = n=1 in disguise.""" - h = blake3(case_id.encode()) - return ( - int.from_bytes(h.derive_key("seed_a").digest()[:4], "big"), - int.from_bytes(h.derive_key("seed_b").digest()[:4], "big"), - int.from_bytes(h.derive_key("seed_c").digest()[:4], "big"), - ) +from verdict.verification.derive_seeds import derive_seeds class CloudSelfConsistency: async def verify(self, plan, evidence_hash): @@ -1566,13 +1581,17 @@ class CloudSelfConsistency: return await self.usc_judge(samples, plan) # Chen 2023 ``` +`derive_seeds` uses a keyed blake3 digest (`key=b"VERDICT self-consistency seeds\0\0"`, +`digest(length=12)`) — do not inline a copy. See `docs/ARCHITECTURE.md` §1 for the +rationale and the shipping implementation. + (Other schemas — Hypothesis, InvestigationPlan, Finding, LedgerEntry — are large; reference v4.5 lines 195–290 + v4.6 schema patch sections.) --- # Appendix B — System prompt templates -## B.1 — `verdict/planning/prompts/examiner_caveats.md` +## B.1 — `src/verdict/planning/prompts/examiner_caveats.md` ```markdown # Examiner Caveats — Tier-1 (always loaded) @@ -1639,7 +1658,7 @@ LOLBin cmdline-shape catalog. Includes regsvr32 (T1218.010), rundll32 (T1218.011 # Appendix C — Playbook + knowledge YAMLs -## C.1 — `verdict/playbooks/memory.yml` +## C.1 — `src/verdict/playbooks/memory.yml` ```yaml evidence_type: memory @@ -1661,7 +1680,7 @@ steps: - {order: 11, tool: vol3.windows.callbacks, mitre_technique_hint: T1014} ``` -## C.2 — `verdict/playbooks/disk.yml` +## C.2 — `src/verdict/playbooks/disk.yml` ```yaml evidence_type: disk_image @@ -1685,7 +1704,7 @@ steps: - {order: 12, tool: bulk_extractor, depends_on: [4]} ``` -## C.3 — `verdict/playbooks/triage.yml` +## C.3 — `src/verdict/playbooks/triage.yml` ```yaml evidence_type: triage @@ -1702,11 +1721,11 @@ steps: - {order: 8, tool: bulk_extractor, depends_on: [1]} ``` -## C.4 — `verdict/knowledge/hunt_evil.yml` +## C.4 — `src/verdict/knowledge/hunt_evil.yml` (Per W1.F.9 task body. 8 entries: svchost, lsass, csrss, winlogon, services, wininit, explorer, smss.) -## C.5 — `verdict/knowledge/lolbins.yml` +## C.5 — `src/verdict/knowledge/lolbins.yml` ```yaml - binary: regsvr32.exe diff --git a/docs/CASE_ISOLATION.md b/docs/CASE_ISOLATION.md index 5ca8503..e03172f 100644 --- a/docs/CASE_ISOLATION.md +++ b/docs/CASE_ISOLATION.md @@ -47,7 +47,7 @@ The fork point is before `planner_node`, not at quorum. Reverify reruns the full |---|---| | `verdict show ` | Lists all `chain_id` values, modes, statuses, created timestamps, and finding counts | | `verdict export ` | Defaults to `chain_id={case_id}-original`; accepts `--chain-id ` or `--chain-id all` | -| `verdict approve ` | If the finding ID exists in more than one chain, fails and requires `--chain-id ` | +| `verdict approve --approver ` | If the finding ID exists in more than one chain, fails and requires `--chain-id ` | | `verdict validate ` | Validates every chain unless `--chain-id ` narrows scope | | `verdict resume ` | Resumes only the latest interrupted chain if unambiguous; otherwise requires `--chain-id ` | diff --git a/docs/DEVPOST_COMPLIANCE.md b/docs/DEVPOST_COMPLIANCE.md index b87bf4b..d9a3ea4 100644 --- a/docs/DEVPOST_COMPLIANCE.md +++ b/docs/DEVPOST_COMPLIANCE.md @@ -33,7 +33,7 @@ Every item must be present or the submission is incomplete. | Required artifact | Source rule | Where in our doc set | Owner | Status | |---|---|---|---|---| -| **Public repository URL** | "Provide a URL to your code repository" | GitHub repo (TBD: `github.com//verdict`); set up W6.D.1 | Tim | TODO | +| **Public repository URL** | "Provide a URL to your code repository" | `github.com/TimothyVang/Verdict`; remaining W6.D.1 tasks: About-section license badge, confirm public visibility | Tim | TODO | | **Public + open source** | "The repository must be public and open source" | Repo settings: Public; LICENSE file with MIT | Tim | TODO | | **MIT or Apache 2.0 license** | "by including an MIT or Apache 2.0 open source license file" | `LICENSE` file at repo root with MIT text | Tim (W6.C.4) | GATED | | **License visible at top of repo (About section)** | "This license should be detectable and visible at the top of the repository page (in the About section)" | GitHub repo About metadata: License = "MIT" badge displayed | Tim (NEW: W6.D.0) | **MISSING — added below** | @@ -46,7 +46,7 @@ Every item must be present or the submission is incomplete. | **Demo shows ≥1 self-correction sequence** | "including at least one self-correction sequence" | Air-gap mode hero beat ⓹ — Qwen3-vs-GLM disagreement → CONTESTED → replan → VETTED_AIRGAP. Narrator calls it out: "this is self-correction." | Beaver (W6.A.2) | **MUST verify on every cut** | | **Demo on YouTube/Vimeo/Youku, public** | "must be uploaded to and made publicly visible on YouTube, Vimeo, or Youku" | YouTube unlisted-then-public on Jun 14 | Tim (W6.D.3) | GATED | | **Demo no third-party trademarks/copyrighted music** | "must not include third party trademarks, or copyrighted music or other material unless the Entrant has permission" | Use royalty-free / CC0 audio. No corporate logos beyond fair-use citation of Volatility/Hayabusa logos for tool identification. | Beaver (W6.A.2) | GATED | -| **Architecture Diagram (visual file)** | "Include an Architecture Diagram — A clear visual showing how components connect — the agent, SIFT tools, MCP servers, evidence sources, output pipeline" | `docs/ARCHITECTURE_DIAGRAM.svg` rendered visual (NOT just ASCII). Shows: Examiner CLI → Gateway → Mode selector → Planner/Verifier → Microsandbox VMs → 12 SIFT tools → Evidence Vault + Ledger + Langfuse. | Tim (NEW: W6.C.7) | **MISSING — added below** | +| **Architecture Diagram (visual file)** | "Include an Architecture Diagram — A clear visual showing how components connect — the agent, SIFT tools, MCP servers, evidence sources, output pipeline" | `docs/ARCHITECTURE_DIAGRAM.svg` rendered visual (NOT just ASCII). Shows: Examiner CLI → Gateway → Mode selector → Planner/Verifier → Microsandbox VMs → 12 SIFT tools → Evidence Vault + Ledger + Langfuse. | Tim (W6.C.7) | **SVG present** (`docs/ARCHITECTURE_DIAGRAM.svg`); ARCHITECTURE.md embed present; README reference pending W6.C.7.c; PNG fallback pending W6.C.7.b | | **Evidence Dataset Documentation** | "Evidence Dataset Documentation — What the agent was tested against, source of the data, and what the agent found" | `docs/RELEASE.md` dataset section covering NIST CFReDS Hacking Case, Honeynet ransomware image, 3 engineered cases, source attribution, and summary findings. | KP (NEW: W6.C.8) | GATED | | **Accuracy Report** | "Self-assessment of findings accuracy. False positives, missed artifacts, hallucinated claims identified during testing. Honesty valued over perfection." | `docs/RELEASE.md` accuracy section with per-mode hallucination rate, false positives, missed artifacts, MITRE sub-technique precision, and Qwen3-vs-GLM disagreement correlation. | KP (W5.E.1) | GATED | | **Agent Execution Logs** | "Structured logs showing the full agent communication and tool execution sequence... Multi-agent: agent-to-agent message logs with timestamps. Single-agent: tool execution logs with timestamps and token usage. Persistent loop: iteration-over-iteration traces showing how the agent's approach changed. Judges must be able to trace any finding back to the specific tool execution that produced it." | VERDICT is **multi-agent** (Qwen3 + GLM + Claude in dual mode). Required: agent-to-agent logs with timestamps. Plus tool execution logs with timestamps + token usage. Plus replan iteration traces. **Package as `submission/execution-logs/.jsonl`** — distilled view of HMAC ledger + Langfuse traces + planner CoT, formatted for judge consumption. NOT just a tar of the raw ledger. | Tim (NEW: W6.C.9) | **MISSING — added below** | @@ -63,10 +63,10 @@ The rules list **six equally weighted** criteria. Earlier doc-set passes claimed **How VERDICT scores:** - Mode-aware verifier strategy (cloud-only/airgap/dual auto-detected, mode-locked at case_init) -- Plan-then-Execute LangGraph with 9 nodes including planner_critique_node (CoVe), comprehension_gate, pivot_node, replan_node, unverifiable_finalize_node +- Plan-then-Execute LangGraph with 8 registered nodes (planner, planner_critique, comprehension_gate, executor_fanout, pivot, quorum, replan, finalize); `unverifiable_finalize_node` is a helper called from `replan_node`, not a registered graph node - Self-correction via cross-engine quorum CONTESTED → replan loop - Bounded recovery: pivot_max=15 (cheap), replan_max=3 (expensive), then explicit UNVERIFIABLE + interrupt() -- Tool-call argument hallucination caught by Pydantic-AI args_validator before sandbox spawn (W2.E.1) +- Tool-call argument hallucination caught by args_validator (Pydantic v2) before sandbox spawn (W2.E.1) **Demo segment:** air-gap hero beat ⓹ (Qwen3-vs-GLM disagreement → replan → re-converge). @@ -90,7 +90,7 @@ The rules list **six equally weighted** criteria. Earlier doc-set passes claimed **How VERDICT scores:** - **Explicit scope decision: Windows-DFIR-depth-first.** macOS / Linux / Win11 SRUM-ETW / ESXi / FOR572 network forensics deferred to v2 with named architectural extension points (5th `net_executor` and `live_executor` fanout branches). -- 23 tool wrappers (10 vol3 plugins + Hayabusa split into csv-timeline+filter + plaso split into extract+filter + 9 Sleuth Kit/EZ Tools/bulk_extractor/exiftool/capa) +- 24 tool wrappers (11 vol3 plugins + Hayabusa split into csv-timeline+filter + plaso split into extract+filter + 9 Sleuth Kit/EZ Tools/bulk_extractor/exiftool/capa) - Three evidence types covered with depth: memory image, disk image, triage zip. Three playbook YAMLs encode SANS-canonical sequencing per type. - 50 ground-truth indicators across 3 engineered cases (lol-bins, credential theft, ransomware) - DKOM/T1014 detection via pslist+psscan divergence — encoded, not LLM-recalled @@ -115,7 +115,7 @@ The rules list **six equally weighted** criteria. Earlier doc-set passes claimed - Mode lock at case_init: refuses to advance if resume detects mode mismatch - chattr +i on evidence vault at case_init - Sanitization scanner on tool output for prompt-injection patterns (IGNORE PREVIOUS, SYSTEM:, etc.) -- Pydantic-AI args_validator rejects unknown flags before sandbox spawn +- args_validator (Pydantic v2) rejects unknown flags before sandbox spawn - Periodic evidence re-hash check (every 10 super-steps) catches anything that bypasses the three-layer gate - HMAC key TPM-backed if /dev/tpmrm0 present, else gpg-encrypted with passphrase @@ -146,7 +146,7 @@ The rules list **six equally weighted** criteria. Earlier doc-set passes claimed - `scripts/install.sh` with cloud credential detection (`CLAUDE_CODE_OAUTH_TOKEN`, interactive Claude Code OAuth, `ANTHROPIC_API_KEY`, optional host-side `OPENROUTER_API_KEY`); auto-detects and configures without passing secrets into microVMs - `verdict doctor` pre-flight (W5.A.4) reports each component status before first use - `docs/RELEASE.md` reproducible-build section verified from a fresh SIFT VM and second VM in W6.C.3 -- Full CLI surface: `verdict {init, resume, reverify, status, ls, show, export, validate, mode, gc, health, doctor}` +- Full CLI surface: `verdict {doctor, mode, init, run-tool, run-case, resume, reverify, status, ls, show, export, validate, approve, gc, package-check, health}` - Consolidated documentation set: README, ARCHITECTURE, DEVPOST_COMPLIANCE, RELEASE, FAILURE_MODES, CASE_ISOLATION, and the docs wiki index - Conventional Commits with task ID embedded (e.g. `feat(schema): foo [W1.B.1]`) — git log archeology trivial - Architecture diagram as rendered visual (`docs/ARCHITECTURE_DIAGRAM.svg`) @@ -196,7 +196,7 @@ Three hours before pushing the v-submit tag, verify ALL of the following are TRU - [ ] Repo About section shows MIT badge - [ ] README.md at repo root with quickstart - [ ] `docs/RELEASE.md` reproducible-build section verified from fresh SIFT VM and second VM -- [ ] `docs/ARCHITECTURE_DIAGRAM.svg` rendered visual present +- [x] `docs/ARCHITECTURE_DIAGRAM.svg` rendered visual present - [ ] `docs/RELEASE.md` documents what we tested against + sources + findings - [ ] `docs/RELEASE.md` includes honest false-positive / missed-artifact / hallucination tally per mode - [ ] `submission/execution-logs/case_001.jsonl`, `case_002.jsonl`, `case_003.jsonl` distilled execution-log artifacts @@ -227,10 +227,10 @@ These amendments have been patched into `BUILD_PLAN.md`; they remain here so eve - [ ] **W6.D.0.e** — Commit if any docs reference the repo URL: `chore(release): GitHub repo public + MIT badge in About [W6.D.0]` ### W6.C.7 — `docs/ARCHITECTURE_DIAGRAM.svg` rendered visual -- [ ] **W6.C.7.a** — Author Mermaid or draw.io source for system diagram covering: Examiner CLI, FastMCP gateway, Mode autodetect, Planner Protocol (CloudPlanner/LocalPlanner), planner_critique_node, comprehension_gate, executor_fanout (4 branches; each branch composes DenyRuleWrapper → ToolExecutor → LedgerEmitter as the per-branch `executor_work` sub-state), pivot_node, quorum_node, replan/unverifiable_finalize, Microsandbox VMs (per-tool), Evidence Vault (chattr +i, read-only mount), HMAC ledger, Langfuse, SqliteSaver checkpoint, optional out-of-band services (Velociraptor, OpenCTI, REMnux). +- [x] **W6.C.7.a** — Author Mermaid or draw.io source for system diagram covering: Examiner CLI, FastMCP gateway, Mode autodetect, Planner Protocol (CloudPlanner/LocalPlanner), planner_critique_node, comprehension_gate, executor_fanout (4 branches; each branch composes DenyRuleWrapper → ToolExecutor → LedgerEmitter as the per-branch `executor_work` sub-state), pivot_node, quorum_node, replan/unverifiable_finalize, Microsandbox VMs (per-tool), Evidence Vault (chattr +i, read-only mount), HMAC ledger, Langfuse, SqliteSaver checkpoint, optional out-of-band services (Velociraptor, OpenCTI, REMnux). - [ ] **W6.C.7.b** — Render to SVG (preferred — scales) and PNG fallback. Place at `docs/ARCHITECTURE_DIAGRAM.svg` and `docs/ARCHITECTURE_DIAGRAM.png`. - [ ] **W6.C.7.c** — Reference from README + ARCHITECTURE.md. -- [ ] **W6.C.7.d** — Commit: `docs: ARCHITECTURE_DIAGRAM.svg rendered visual [W6.C.7]` +- [x] **W6.C.7.d** — Commit: `docs: ARCHITECTURE_DIAGRAM.svg rendered visual [W6.C.7]` ### W6.C.8 — `docs/RELEASE.md` dataset section - [ ] **W6.C.8.a** — Author. Sections: (1) Datasets used (NIST CFReDS Hacking Case, Honeynet ransomware image, 3 engineered cases). (2) Source attribution per dataset (URL, license, hash). (3) What VERDICT was tested against per case. (4) What VERDICT found per case (summary of findings, with finding_ids referencing the accuracy report). (5) Limitations: Windows-only; no live-response; no Win11-specific; no macOS/Linux. @@ -244,7 +244,7 @@ These amendments have been patched into `BUILD_PLAN.md`; they remain here so eve - [ ] **W6.C.9.d** — Commit: `feat(cli): export execution-logs format for Devpost compliance [W6.C.9]` ### W6.C.10 — `docs/RELEASE.md` novelty section -- [ ] **W6.C.10.a** — Author. Sections: (1) Project timeline (started 2026-05-02; substantially new work per Devpost rules). (2) What we built (mode-aware verifier, three-layer immutability, encoded forensic discipline, planner_critique CoVe, pivot vs replan, schema-enforced caveat acknowledgment, DKOM/T1014 auto-detection, Hunt Evil masquerade catch, LOLBin matcher with T1218 sub-techniques, agentskills.io skill bundle, custom Inspect AI scorers including step_efficiency + mitre_subtechnique_precision + negative_hypothesis_quality + Qwen3-vs-GLM disagreement-correlation analysis). (3) What was pre-existing open source (with license + source URL each: SIFT Workstation, Volatility 3, Hayabusa, plaso, EZ Tools, Microsandbox, SGLang, vLLM, LangGraph, Langfuse, OpenLLMetry, Inspect AI, Pydantic + Pydantic-AI, FastMCP, NeMo Guardrails, Claude Agent SDK, blake3). (4) What we extended vs replaced. +- [ ] **W6.C.10.a** — Author. Sections: (1) Project timeline (started 2026-05-02; substantially new work per Devpost rules). (2) What we built (mode-aware verifier, three-layer immutability, encoded forensic discipline, planner_critique CoVe, pivot vs replan, schema-enforced caveat acknowledgment, DKOM/T1014 auto-detection, Hunt Evil masquerade catch, LOLBin matcher with T1218 sub-techniques, agentskills.io skill bundle, custom Inspect AI scorers including step_efficiency + mitre_subtechnique_precision + negative_hypothesis_quality + Qwen3-vs-GLM disagreement-correlation analysis). (3) What was pre-existing open source (with license + source URL each: SIFT Workstation, Volatility 3, Hayabusa, plaso, EZ Tools, Microsandbox, SGLang, vLLM, LangGraph, Langfuse, OpenLLMetry, Inspect AI, Pydantic, FastMCP, Claude Agent SDK, blake3). (4) What we extended vs replaced. - [ ] **W6.C.10.b** — Cross-reference from README. - [ ] **W6.C.10.c** — Commit: `docs(release): document novel contribution [W6.C.10]` diff --git a/docs/DFIR_MEMORY.md b/docs/DFIR_MEMORY.md index efb54d9..8c7013f 100644 --- a/docs/DFIR_MEMORY.md +++ b/docs/DFIR_MEMORY.md @@ -1,5 +1,7 @@ # DFIR Self-Evolving Memory (SANS-Aligned) +> **Wiki:** [Index](README.md) · [Architecture](ARCHITECTURE.md) · [Failure Modes](FAILURE_MODES.md) · [Case Isolation](CASE_ISOLATION.md) · root [CLAUDE.md](../CLAUDE.md) + This document defines VERDICT's memory model for **evidence-first** incident response. It is intentionally conservative: memory is allowed to evolve, but only through governed, auditable mutations. ## Goals @@ -29,7 +31,7 @@ Only these operations are allowed: For persistent classes (`technique`, `pattern`, `meta`): -- At least one `evidence_ref` is required. +- At least one `evidence_refs` entry is required. - `last_validated_at` must be on/after `created_at`. - `expiry`, when present, must be after `created_at`. - Versioning is monotonic (`version >= 1`) and lineage is retained. diff --git a/docs/DOCS_ACCURACY_REPORT.md b/docs/DOCS_ACCURACY_REPORT.md index d4aba17..e84859d 100644 --- a/docs/DOCS_ACCURACY_REPORT.md +++ b/docs/DOCS_ACCURACY_REPORT.md @@ -167,7 +167,7 @@ These were checked and hold up. - **Judging window** Jun 19 – Jul 3 2026, winners ~Jul 8 — ✓ matches. - **SANS Institute** as sponsor — ✓ matches. - **6 equally weighted judging criteria** — ✓ matches Devpost rules; the prior "5 of 6" framing is correctly called out as wrong (DEVPOST_COMPLIANCE Part 3 intro). -- **License attributions** for the stack table (`ARCHITECTURE.md` §7) — spot-checked SGLang Apache-2.0, LangGraph MIT, Pydantic MIT, FastMCP Apache-2.0, NeMo Guardrails Apache-2.0, Inspect AI MIT — all match upstream. +- **License attributions** for the stack table (`ARCHITECTURE.md` §7) — spot-checked SGLang Apache-2.0, LangGraph MIT, Pydantic MIT, FastMCP Apache-2.0, Inspect AI MIT — all match upstream. - **License "hard nos"** — Daytona AGPL-3.0, REMnux GPL-3.0 — match upstream. - **6-week window math** — May 2 → Jun 14 EOD against a Jun 15 23:45 EDT deadline = ~28h buffer. ✓ correct. @@ -200,13 +200,13 @@ All 11 punchlist items closed in two passes: | C1 | `T1014.001` → `T1014` | ✅ closed | `README.md`, `docs/ARCHITECTURE.md`, `docs/DEVPOST_COMPLIANCE.md` | | C2 | "RFC-2119 standard MIT" → "OSI-canonical MIT" | ✅ closed | `docs/DEVPOST_COMPLIANCE.md` W6.D.0.b | | C3 | vol3 example version `2.10.0` → `2.28.0` (with build-time pin note) | ✅ closed | `docs/ARCHITECTURE.md` Pattern 1 | -| H1 | vol3 plugin count: dropped `windows.info` from typed list (10 plugins, matches BUILD_PLAN). Note: `windows.info` is still invokable through the dynamic vol3 allow-list (`§6 Tool-call argument validation`). | ✅ closed | `docs/ARCHITECTURE.md` §6 | -| H2 | "19 wrappers" → "23 wrappers" (10 vol3 + 2 hayabusa + 2 plaso + 9 other) | ✅ closed | `docs/ARCHITECTURE.md` §6 header, `docs/DEVPOST_COMPLIANCE.md` line 89 | +| H1 | vol3 plugin count: `windows.info` reinstated in typed list as registered `ExternalToolSpec` in `src/verdict/tools/registry.py`; count is 11 plugins (info, pslist, psscan, pstree, cmdline, dlllist, malfind, netscan, svcscan, handles, callbacks). | ✅ closed | `docs/ARCHITECTURE.md` §6, `src/verdict/tools/registry.py` | +| H2 | "19 wrappers" → "24 wrappers" (11 vol3 + 2 hayabusa + 2 plaso + 9 other); count updated after `windows.info` reinstatement in H1. | ✅ closed | `docs/ARCHITECTURE.md` §6 header, `docs/DEVPOST_COMPLIANCE.md` line 93 | | H3 | 9-node count: aligned by removing "executor_work" as a separate node in CLAUDE.md and BUILD_PLAN.md (it's the in-fanout composition); clarified `clarify_node` is a sub-state of `comprehension_gate` in ARCHITECTURE.md §2 | ✅ closed | `CLAUDE.md` §4, `docs/BUILD_PLAN.md` Week 2 critical-path, `docs/ARCHITECTURE.md` §2 | | H4 | "12 documentation files" → "16 documentation files" | ✅ closed | `docs/DEVPOST_COMPLIANCE.md` line 146 | | H5 | Old checklist-count wording → "every checklist item ticked" | ✅ closed | `docs/DEVPOST_COMPLIANCE.md` W6.D.4.a + `docs/BUILD_PLAN.md` W6.D.3 | | H6 | README W2 roadmap "9 vol3 wrappers" reference removed during 2026-05-02 doc refactor | ✅ closed | `README.md` (no current occurrence) | -| M1 | `blake3` API: `blake3(enc, derive_key_context=...)` constructor pattern (not `instance.derive_key`) | ✅ closed | `docs/ARCHITECTURE.md` §1 `derive_seeds` snippet | +| M1 | `blake3` API: keyed-hash mode `blake3(data, key=_SEED_DERIVATION_KEY)` (not `instance.derive_key()`); 32-byte key held in module constant | ✅ closed | `docs/ARCHITECTURE.md` §1 `derive_seeds` snippet; `src/verdict/verification/derive_seeds.py` | | M3 | Pattern 2 TSI vs Pattern 1 `network=False`: explicit one-paragraph clarification added | ✅ closed | `docs/ARCHITECTURE.md` Pattern 2 | | M4 | `verdict reverify` surfaced in README CLI block with parallel-chain note | ✅ closed | `README.md` Quick reference | diff --git a/docs/FAILURE_MODES.md b/docs/FAILURE_MODES.md index 0ddfbcf..8343acf 100644 --- a/docs/FAILURE_MODES.md +++ b/docs/FAILURE_MODES.md @@ -20,13 +20,13 @@ |---|---|---|---|---|---| | Evidence vault | SHA-256 mismatch against `EvidenceManifest` | Periodic re-hash every 10 super-steps | None; stop immediately | `evidence_hash_recheck` with mismatch detail | Raise `HashMismatchError`; halt case | | Mode lock | Current `detect_mode()` differs from `mode_at_case_init` on resume | `verdict resume` pre-flight | Operator runs `verdict reverify --mode ` | `mode_lock` | Exit 2 with documented `ModeLockedError` stderr | -| Tool args | Invalid tool name, flag, PID, filter, or timeline option | Pydantic-AI `args_validator` before sandbox spawn | `ModelRetry`, max 2 | `tool_call` with validation error | After retry exhaustion, `Finding(status=UNVERIFIABLE, failure_reason="tool_args_failed_validation_after_2_retries")` | +| Tool args | Invalid tool name, flag, PID, filter, or timeline option | `ArgsValidator` (Pydantic v2) before sandbox spawn | `ModelRetry`, max 2 | `tool_call` with validation error | After retry exhaustion, `Finding(status=UNVERIFIABLE, failure_reason="tool_args_failed_validation_after_2_retries")` | | Microsandbox spawn | libkrun/KVM unavailable, image missing, host disk full, resource exhaustion | Exception or spawn timeout | No retry for kernel/resource failures | `sandbox_failure` with `error_detail` | `UNVERIFIABLE`, `failure_reason="sandbox_spawn_failed"` | | Tool execution | Tool exits nonzero with stderr | Process exit status | One retry only if wrapper marks failure transient | `tool_call` with `is_error=true` | Retry exhaustion produces `UNVERIFIABLE`, `failure_reason="tool_execution_failed"` | | Branch timeout | One executor branch hangs | Branch wall-clock reaches `branch_timeout=900s` | Reducer proceeds with timeout output | `sandbox_failure` or `tool_call` with timeout detail | Timed-out branch emits `ToolOutput(status=TIMEOUT, parsed_artifacts=[])`; quorum treats as disagreement | | TSI proxy | Proxy origin unavailable or credential injection fails | `NetworkProxyError`, connection refused, DNS fail | Counts against `tool_arg_retry_max=2` only for TSI-enabled tools | `tool_call` with `error_detail` | Retry exhaustion produces `UNVERIFIABLE`, `failure_reason="tsi_proxy_unreachable"` | | Verifier disagreement | Engines disagree on artifact set or MITRE technique | Quorum dispatch table | Route to `replan_node`, max 3 | `finding` with `status=CONTESTED` or `exhausted_replan` | Iteration 4 routes to `unverifiable_finalize_node` | -| Langfuse unavailable | Health check or span write fails | Exception / non-200 health | Fail open for investigation; write local ledger anyway | `tool_call` payload notes `langfuse_write_failed=true` | Case continues; `verdict doctor` fails before new case if still down | +| Langfuse unavailable | Health check or span write fails | Exception / non-200 health | Fail open for investigation; write local ledger anyway | `tool_call` payload notes `langfuse_write_failed=true` | Case continues; local ledger written; Langfuse traces unavailable until service restored | | Ledger write | Write, fsync, or verify-readback fails | `LedgerEmitter` post-write readback | None; preserving chain integrity wins | Best-effort stderr only if write failed before event persisted | Halt case; operator must repair filesystem or start a new case | ## Fanout Reducer Rule @@ -35,7 +35,9 @@ ## UNVERIFIABLE Schema Exemption -`Finding.artifact_paths` and `Finding.artifact_classes` keep `min_length=2` for positive findings. A finding may omit artifacts only when `status == UNVERIFIABLE` and `failure_reason` is set to one of: +`Finding.artifact_paths` and `Finding.artifact_classes` currently enforce `min_length=2` for **all** findings, including those with `status == UNVERIFIABLE`. The `Finding` schema does not yet have a `failure_reason` field or a `_unverifiable_relaxes_corroboration` validator. + +**Planned:** add `failure_reason: str | None` to `Finding` with a validator that relaxes the `min_length=2` constraint when `status == UNVERIFIABLE` and `failure_reason` is one of: - `tool_args_failed_validation_after_2_retries` - `sandbox_spawn_failed` @@ -43,4 +45,4 @@ - `tsi_proxy_unreachable` - `branch_timeout` -This is an explicit failure claim, not evidence of absence. +Until that field is implemented, UNVERIFIABLE findings from tool or sandbox failures must cite whatever artifact paths were collected before the failure, and case-level terminal failures should use `CaseConclusion(status="UNVERIFIABLE")` instead. diff --git a/docs/MCP_FRAMEWORK.md b/docs/MCP_FRAMEWORK.md index fa3945f..0599ca6 100644 --- a/docs/MCP_FRAMEWORK.md +++ b/docs/MCP_FRAMEWORK.md @@ -89,7 +89,7 @@ grep MICROSANDBOX_NETWORK_DEFAULT .env # Expect: MICROSANDBOX_NETWORK_DEFAULT=false # 4. Verdict doctor pre-flight -verdict doctor # checks all of the above + SGLang + Langfuse + HMAC key +verdict doctor # checks all of the above + SGLang + HMAC key ``` ## 5. Install pattern diff --git a/docs/README.md b/docs/README.md index 2160656..efed21e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -44,6 +44,7 @@ Code wins over docs. If code is right and a doc is wrong, fix the doc — don't | [`DEVPOST_COMPLIANCE.md`](DEVPOST_COMPLIANCE.md) | Submission rule-to-artifact mapping. Every Devpost requirement traced to the file/commit that satisfies it. | Before any submission packaging; before any merge that touches a submission deliverable. | | [`FAILURE_MODES.md`](FAILURE_MODES.md) | Runtime failure matrix: sandbox spawn, tool errors, fanout timeout, TSI failure, ledger write failures, and UNVERIFIABLE semantics. | Before implementing error paths or explaining graceful degradation to judges. | | [`CASE_ISOLATION.md`](CASE_ISOLATION.md) | Case, chain, checkpoint, reverify, export, approval, and mode-lock boundaries. | Before implementing `verdict reverify`, `resume`, `export`, `approve`, or `validate`. | +| [`DFIR_MEMORY.md`](DFIR_MEMORY.md) | VERDICT's self-evolving memory model — evidence-first mutation rules, memory layers (case / technique / pattern / meta), and governance constraints. | Before implementing anything under `src/verdict/memory/`. | ### Release deliverables @@ -77,6 +78,12 @@ Each of these explicitly subordinates itself to `BUILD_PLAN.md` and `../CLAUDE.m | [`hackathon/RULES.md`](hackathon/RULES.md) | Official SANS *FIND EVIL!* 2026 rules, scraped from Devpost on 2026-05-02. | Before any submission decision — these are the upstream of `DEVPOST_COMPLIANCE.md`. | | [`hackathon/OVERVIEW.md`](hackathon/OVERVIEW.md) | Hackathon overview + resource links (judge bios, prize structure, timeline). | Context-setting; not load-bearing. | +### Internal working plans + +| File | Role | When to read | +|------|------|--------------| +| [`superpowers/plans/2026-05-04-cloud-proof-harness.md`](superpowers/plans/2026-05-04-cloud-proof-harness.md) | Implementation plan for the cloud-only Claude Agent SDK v0 proof path. Agentic worker task checklist with acceptance gates. | Before implementing anything under `src/verdict/planning/` or `src/verdict/proof/`. | + ### Frozen archive (audit history) [`spec/`](spec/) — point-in-time decision records. **Do not edit.** Cite from `ARCHITECTURE.md` instead. See [`spec/README.md`](spec/README.md) for what each numbered audit captured. Files: `01-audit-v4.3.md`, `02-audit-v4.4.md`, `03-audit-v4.5.md`, `04-spec-plan-v4.6.md`. (The original v4.6 TL;DR was promoted to `TLDR.md` and removed from the archive on 2026-05-02 to avoid drift.) diff --git a/docs/RELEASE.md b/docs/RELEASE.md index 529aea4..56ab510 100644 --- a/docs/RELEASE.md +++ b/docs/RELEASE.md @@ -78,19 +78,22 @@ Planned v1 command surface. Commands remain documentation-only until their imple | Command | Purpose | Status | |---|---|---| -| `verdict init ` | Create case, hash evidence, detect/lock mode | Roadmap W1/W5 | -| `verdict resume ` | Resume interrupted chain from checkpoint | Roadmap W3 | -| `verdict reverify --mode ` | Create parallel verification chain | Roadmap W5 | -| `verdict status ` | Show current graph/checkpoint status | Roadmap | -| `verdict ls` | List local cases | Roadmap | -| `verdict show ` | Render findings/chains | Roadmap | -| `verdict export ` | Export report, JSONL, or execution logs | Roadmap | -| `verdict validate ` | Verify ledger HMAC/chain and hashes | Roadmap W3 | -| `verdict mode` | Explain detected mode and prerequisites | Roadmap W5 | -| `verdict gc` | Rotate local logs and old traces | Roadmap | -| `verdict health` | Machine-readable health endpoint/check | Roadmap | -| `verdict doctor` | Human pre-flight for dependencies/secrets/services | Roadmap W5 | -| `verdict approve ` | HMAC-sign human approval | Roadmap W5 | +| `verdict init ` | Create case, hash evidence, detect/lock mode | Implemented | +| `verdict resume ` | Resume interrupted chain from checkpoint | Implemented | +| `verdict reverify --mode ` | Create parallel verification chain | Implemented | +| `verdict status ` | Show current graph/checkpoint status | Implemented | +| `verdict ls` | List local cases | Implemented | +| `verdict show ` | Render findings/chains | Implemented | +| `verdict export ` | Export report, JSONL, or execution logs | Implemented | +| `verdict validate ` | Verify ledger HMAC/chain and hashes | Implemented | +| `verdict mode` | Explain detected mode and prerequisites | Implemented | +| `verdict gc` | List local cases eligible for manual cleanup | Implemented | +| `verdict health` | Machine-readable health endpoint/check | Implemented | +| `verdict doctor` | Human pre-flight for dependencies/secrets/services | Implemented | +| `verdict approve --approver ` | HMAC-sign human approval | Implemented | +| `verdict run-tool ` | Run a single registered SIFT tool | Implemented | +| `verdict run-case ` | Run canonical real-tool triage sequence | Implemented | +| `verdict package-check [--output ]` | Validate and optionally zip Devpost artifacts | Implemented | Export formats: @@ -232,9 +235,8 @@ Pre-existing open source used: | Langfuse | Trace UI | TODO | | OpenLLMetry | OTel instrumentation | TODO | | Inspect AI | Evaluation harness | TODO | -| Pydantic / Pydantic-AI | Schemas and typed retries | TODO | +| Pydantic v2 | Schemas and typed retries | TODO | | FastMCP | MCP gateway | TODO | -| NeMo Guardrails | Rails | TODO | | Claude Agent SDK / Claude Code | Cloud execution lane | TODO | | blake3 | Invocation/ledger hashing | TODO | @@ -247,8 +249,10 @@ Required release paths: - `README.md` - `LICENSE` - `docs/ARCHITECTURE.md` -- `docs/ARCHITECTURE.md` +- `docs/ARCHITECTURE_DIAGRAM.svg` - `docs/DEVPOST_COMPLIANCE.md` +- `docs/FAILURE_MODES.md` +- `docs/CASE_ISOLATION.md` - `docs/RELEASE.md` - `submission/execution-logs/case_001.jsonl` - `submission/execution-logs/case_002.jsonl` diff --git a/docs/SKILLS_FRAMEWORK.md b/docs/SKILLS_FRAMEWORK.md index 6694cce..2575f2c 100644 --- a/docs/SKILLS_FRAMEWORK.md +++ b/docs/SKILLS_FRAMEWORK.md @@ -12,7 +12,7 @@ ## 1. Why a tandem framework -Sixteen skills under `.claude/skills/` is more than any single task should pull on. Without composition discipline, a generic agent will skip planning, jump to code, and skip the review/commit gate. The framework below pins down **which skill fires when**, **which Verdict hard rule each skill enforces**, and **which subagent_type runs it**, so a session opened against the Verdict charter executes the same loop every time. +Eighteen skills under `.claude/skills/` is more than any single task should pull on. Without composition discipline, a generic agent will skip planning, jump to code, and skip the review/commit gate. The framework below pins down **which skill fires when**, **which Verdict hard rule each skill enforces**, and **which subagent_type runs it**, so a session opened against the Verdict charter executes the same loop every time. The composition is a single-pass pipeline. Each phase has a defined input, output, the skill that runs it, and the gate that admits it to the next phase. Iteration happens **inside** a phase (e.g. RED → GREEN → REFACTOR inside test-driven-development), not by jumping back across the pipeline. @@ -112,6 +112,6 @@ The framework is intentionally Phase 0 — skills are vendored and documented, b ## 7. Anti-patterns (what NOT to do) - **Don't edit vendored skill files in place.** All Verdict-specific behavior goes in `verdict-house-rules` so `git pull upstream main` re-vendoring stays mechanical. -- **Don't pile on more skills.** The Reddit consensus is "2–3 plugins max + a precise CLAUDE.md beats 50 generic plugins". We have 16; that is already at the upper end. New skills require the §3.8 audit gate **and** a clear pipeline-phase justification. +- **Don't pile on more skills.** The Reddit consensus is "2–3 plugins max + a precise CLAUDE.md beats 50 generic plugins". We have 18; that is already at the upper end. New skills require the §3.8 audit gate **and** a clear pipeline-phase justification. - **Don't skip phases.** The pipeline is single-pass on purpose. If you find yourself jumping from phase 1 to phase 7, you're rationalizing vibe-coding. - **Don't mock to satisfy a skill.** `test-driven-development` will pass with mocked tests; `verdict-house-rules` will fail the commit. The mock is the bug. diff --git a/docs/SKILLS_LICENSE_AUDIT.md b/docs/SKILLS_LICENSE_AUDIT.md index f7ec40b..1ddaafd 100644 --- a/docs/SKILLS_LICENSE_AUDIT.md +++ b/docs/SKILLS_LICENSE_AUDIT.md @@ -13,6 +13,8 @@ Per CLAUDE.md §3.8, every vendored skill, hook, MCP, or external artifact requi |---|---|---|---|---|---| | Superpowers | [obra/superpowers](https://github.com/obra/superpowers) | `e7a2d16` (2026-04-27) | **MIT** © Jesse Vincent | ✅ | **Vendored** — 14 skills under `.claude/skills/` | | mattpocock skills | [mattpocock/skills](https://github.com/mattpocock/skills) | `b843cb5` (2026-04-30) | **MIT** © Matt Pocock | ✅ | **Vendored** — `grill-me/`, `grill-with-docs/` | +| qc (custom) | Verdict — no upstream | n/a | **MIT** (Verdict) | ✅ | **Custom** — Verdict-specific quick-commit + docs-sweep slash command; `.claude/skills/qc/` | +| verdict-house-rules (custom) | Verdict — no upstream | n/a | **MIT** (Verdict) | ✅ | **Custom** — Session-start overlay enforcing CLAUDE.md §3 hard rules; wins on conflict with any vendored skill; `.claude/skills/verdict-house-rules/` | | tdd-guard | [nizos/tdd-guard](https://github.com/nizos/tdd-guard) | `2c82daa` (2026-05-02, v1.6.7) | **MIT** © Nizar Selander | ✅ | **Deferred** — Node tool + hook (separate workstream; not yet integrated, will be wired via `scripts/bootstrap-dev.sh` + `.claude/settings.json`) | | filesystem MCP | [modelcontextprotocol/servers](https://github.com/modelcontextprotocol/servers/tree/main/src/filesystem) | npm package resolved by `.mcp.cloud.json` / `.mcp.dual.json`; pin exact package version before W6 release | **MIT** | ✅ | **Allowed with safety caveat** — excluded from safe default because upstream exposes write-capable tools; see `MCP_FRAMEWORK.md` §2 | | fetch MCP | [modelcontextprotocol/servers](https://github.com/modelcontextprotocol/servers) | npm package resolved by `.mcp.cloud.json` / `.mcp.dual.json`; pin exact package version before W6 release | **MIT** | ✅ | **Allowed with mode caveat** — cloud/dual configs only; omitted from `.mcp.json` and `.mcp.airgap.json` | diff --git a/docs/TLDR.md b/docs/TLDR.md index 26885cc..c6dd6ec 100644 --- a/docs/TLDR.md +++ b/docs/TLDR.md @@ -285,7 +285,8 @@ WEEK 6 ── Jun 6-14 ── DEMO + DOCS + SUBMIT │ • OpenLLMetry / Langfuse instrumentation │ │ • TSI secret injection + tcpdump demo │ │ • Mode autodetect + verdict CLI │ -│ • THREAT_MODEL.md + FAILURE_MODES.md + CLI.md │ +│ • Threat model (ARCHITECTURE.md §9) + FAILURE_MODES.md│ +│ + CLI reference (RELEASE.md) │ │ • Devpost packaging + submission │ └────────────────────────────────────────────────────────┘ @@ -294,7 +295,7 @@ WEEK 6 ── Jun 6-14 ── DEMO + DOCS + SUBMIT │ ~22 teammate-days │ │ GIAC: GNFA, GCFA, GMLE · runs HW+local LLMs at home │ │ │ -│ • Plan-then-Execute LangGraph topology (9 nodes) │ +│ • Plan-then-Execute LangGraph topology (8 nodes) │ │ • Three verifier strategies (Cloud/Airgap/Dual) │ │ • Seed-derivation fix (n=3 actually diverse paths) │ │ • planner_critique_node (CoVe) │ @@ -560,14 +561,14 @@ Three layers — **toolchain** (host), **services** (lab/cloud), **agent surface | **SGLang** serving Qwen3-30B-A3B-Thinking-2507 (Apache-2.0) | Air-gap + dual modes (planner/executor) | `sglang_server_v1 --model-path … --tool-call-parser qwen --port 30000` | | **SGLang** serving GLM-4.5-Air (MIT) | Air-gap + dual modes (verifier only) | `sglang_server_v1 --model-path … --tool-call-parser glm --port 30001` | | **Anthropic / OpenRouter API** | Cloud + dual modes | `ANTHROPIC_API_KEY` preferred; `OPENROUTER_API_KEY` optional host-side AI-agent fallback. OAuth/API tokens are per-contributor and never enter microVMs. | -| **Langfuse v2 (self-host)** | Trace observability, ledger ↔ trace cross-link | `docker-compose up -d` | +| **Langfuse v2 (self-host)** | Trace observability, ledger ↔ trace cross-link | `docker-compose -f infra/langfuse/docker-compose.yml up -d` | | **HMAC signing key** | Ledger integrity (CLAUDE.md §3.9) | TPM (`/dev/tpmrm0`) when available, else gpg-encrypted at `~/.verdict/key.gpg` | -`verdict doctor` is the one-command pre-flight: API reachable, SGLang up, microsandbox installed, Langfuse healthy, HMAC key resolvable. CI fails closed if it fails. +`verdict doctor` is the one-command pre-flight: API reachable, SGLang up, microsandbox installed, HMAC key resolvable. CI fails closed if it fails. ### Skills — `.claude/skills/` (auto-loaded by Claude Code) -17 vendored skills compose into a Plan → TDD → Subagent-driven-dev → Review → Commit pipeline (`docs/SKILLS_FRAMEWORK.md`). Skill and MCP licenses are tracked in `docs/SKILLS_LICENSE_AUDIT.md`. +18 skills compose into a Plan → TDD → Subagent-driven-dev → Review → Commit pipeline (`docs/SKILLS_FRAMEWORK.md`). Skill and MCP licenses are tracked in `docs/SKILLS_LICENSE_AUDIT.md`. ``` verdict-house-rules ← Verdict (custom). Re-states CLAUDE.md §3 hard rules @@ -592,6 +593,10 @@ using-superpowers ← obra/superpowers — index of the framework grill-me ← mattpocock/skills — relentless interview on a plan grill-with-docs ← mattpocock/skills — same, but cross-checks ARCH.md + +qc ← Verdict (custom). Quick-commit + docs-sweep + push. + Drafts Conventional Commit per §3.7, commits, sweeps + docs/ for drift triggered by the diff, then pushes. ``` ### MCPs — mode-scoped configs (MIT/Apache-2.0 only) @@ -637,7 +642,7 @@ export GITHUB_TOKEN="ghp_..." # Required for air-gap / dual mode (after SGLang is up) export SGLANG_BASE_URL="http://localhost:30000" -export SGLANG_VERIFIER_BASE_URL="http://localhost:30001" +export SGLANG_GLM_BASE_URL="http://localhost:30001" # Required: microsandbox network closed by default export MICROSANDBOX_NETWORK_DEFAULT=false @@ -655,7 +660,7 @@ Full template in `.env.example`. **Never** commit `.env` — `.gitignore` covers ```bash bash scripts/bootstrap-dev.sh # toolchain (uv, rustup, nvm, microsandbox), pinned versions, idempotent uv sync # Python deps -docker-compose up -d # Langfuse v2 +docker-compose -f infra/langfuse/docker-compose.yml up -d # Langfuse v2 verdict doctor # pre-flight: all of the above must be green ``` diff --git a/docs/superpowers/plans/2026-05-04-cloud-proof-harness.md b/docs/superpowers/plans/2026-05-04-cloud-proof-harness.md index db73126..2f9ef38 100644 --- a/docs/superpowers/plans/2026-05-04-cloud-proof-harness.md +++ b/docs/superpowers/plans/2026-05-04-cloud-proof-harness.md @@ -1,5 +1,7 @@ # Cloud Proof Harness Implementation Plan +> **Wiki:** [Index](../../README.md) · [Architecture](../../ARCHITECTURE.md) · [Build Plan](../../BUILD_PLAN.md) · root [CLAUDE.md](../../../CLAUDE.md) + > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Build a cloud-only Claude Agent SDK v0 proof path that generates a schema-valid VERDICT investigation plan and writes visual-proof artifacts under `proof/runs/`. @@ -56,7 +58,7 @@ - [ ] Run `uv run pytest tests/proof tests/planning/test_cloud_planner.py -v`. - [ ] Run `uv run pytest tests -v`. - [ ] Run `uv run ruff check src tests scripts`. -- [ ] Run `uv run python scripts/run-cloud-proof.py` if cloud credentials are configured; otherwise confirm it writes a blocker proof run without leaking secrets. +- [ ] Run `uv run python scripts/run_cloud_proof.py` if cloud credentials are configured; otherwise confirm it writes a blocker proof run without leaking secrets. ---