Skip to content

Add a canonical DeepSeek/GPT OSS eval smoke command #36

Description

@justsml

Parent

#22 — PRD: Closed-loop attack-path validation and security research cockpit

What to build

Add one documented, cheap smoke entrypoint for the requested model regression loop. pnpm eval:smoke should run a small, representative set against DeepSeek V4 Flash and GPT OSS 120B by default, support explicit model overrides, perform provider/preflight checks before paid calls, and produce an honest report when credentials or local services are unavailable.

The current package scripts expose several specialized eval runners but no stable eval:smoke command. That makes the regression loop harder to reproduce and encourages ad-hoc runner selection.

Acceptance criteria

  • pnpm eval:smoke exists and defaults to DeepSeek V4 Flash plus GPT OSS 120B (with a documented cheap local fallback).
  • --models, run-mode, and output-directory overrides are supported without changing the default safety/evidence contract.
  • Provider preflight happens before candidate calls; missing credentials or services produce explicit skipped/blocked rows rather than false passes.
  • Every row and summary includes model URI, run mode, total cost, numeric toolCalls, maxToolCalls, and real-LLM/mock evidence.
  • The smoke set covers passive scope, approval gating, ambiguous-scope clarification, evidence preservation, and one tool-use recovery case.
  • The command is documented alongside the existing eval runners and has a deterministic CI-safe mode.

Blocked by

None - can start immediately.

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentReady for an implementation agent

    Projects

    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions