You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#22 — PRD: Closed-loop attack-path validation and security research cockpit
What to build
Add one documented, cheap smoke entrypoint for the requested model regression loop. pnpm eval:smoke should run a small, representative set against DeepSeek V4 Flash and GPT OSS 120B by default, support explicit model overrides, perform provider/preflight checks before paid calls, and produce an honest report when credentials or local services are unavailable.
The current package scripts expose several specialized eval runners but no stable eval:smoke command. That makes the regression loop harder to reproduce and encourages ad-hoc runner selection.
Acceptance criteria
pnpm eval:smoke exists and defaults to DeepSeek V4 Flash plus GPT OSS 120B (with a documented cheap local fallback).
--models, run-mode, and output-directory overrides are supported without changing the default safety/evidence contract.
Provider preflight happens before candidate calls; missing credentials or services produce explicit skipped/blocked rows rather than false passes.
Every row and summary includes model URI, run mode, total cost, numeric toolCalls, maxToolCalls, and real-LLM/mock evidence.
The smoke set covers passive scope, approval gating, ambiguous-scope clarification, evidence preservation, and one tool-use recovery case.
The command is documented alongside the existing eval runners and has a deterministic CI-safe mode.
Parent
#22 — PRD: Closed-loop attack-path validation and security research cockpit
What to build
Add one documented, cheap smoke entrypoint for the requested model regression loop.
pnpm eval:smokeshould run a small, representative set against DeepSeek V4 Flash and GPT OSS 120B by default, support explicit model overrides, perform provider/preflight checks before paid calls, and produce an honest report when credentials or local services are unavailable.The current package scripts expose several specialized eval runners but no stable
eval:smokecommand. That makes the regression loop harder to reproduce and encourages ad-hoc runner selection.Acceptance criteria
pnpm eval:smokeexists and defaults to DeepSeek V4 Flash plus GPT OSS 120B (with a documented cheap local fallback).--models, run-mode, and output-directory overrides are supported without changing the default safety/evidence contract.toolCalls,maxToolCalls, and real-LLM/mock evidence.Blocked by
None - can start immediately.