diff --git a/.mux/skills/eval-guide/SKILL.md b/.mux/skills/eval-guide/SKILL.md index 563558dd..8987b8c3 100644 --- a/.mux/skills/eval-guide/SKILL.md +++ b/.mux/skills/eval-guide/SKILL.md @@ -179,3 +179,34 @@ npx tsx evals/run.ts --provider stub --lane prompt --trials 3 --concurrency 4 ``` Use `stub` to validate wiring, not to judge whether a real-provider prompt or skill change helped. + +## 7. Writing new eval cases + +Checklist: + +- Prefer the fluent builders in `evals/authoring/*` (`promptCase()`, `executionCase()`, `dogfoodCase()`) over hand-assembled schema objects. +- Use the raw escape hatches when the sugar does not fit: `rawWorkflowCheck()`, `rawVerifier()`, `rawArtifactRequirement()`, and `rawReportRequirement()`. +- After the new case compiles, rerun `npm run test`. + +## 8. Reporter lifecycle + +Need lifecycle events plus local progress and a machine-readable trace? + +```bash +npx tsx evals/run.ts \ + --provider stub \ + --lane execution \ + --reporter jsonl \ + --reporter-output evals/reports/execution-events.jsonl \ + --progress +``` + +When you omit `--reporter`, the default `final` reporter still writes `report.json` and `report.md`. + +## 9. Workspace presets + +Add `.workspace('agent-tty-smoke')` to an `executionCase()` or `dogfoodCase()` when the case needs preset bootstrap/env/template setup. Register custom presets with `registerPreset()` in a module that loads before `runEvalCli()`. + +## 10. Token snapshots + +Use `--snapshot-update` first, then `--snapshot-check --snapshot-threshold 20` against the same `--snapshot-dir` when you want regression signals over time. Snapshot regressions are warnings only. diff --git a/dogfood/CATALOG.md b/dogfood/CATALOG.md index 515e10a7..c6b376f7 100644 --- a/dogfood/CATALOG.md +++ b/dogfood/CATALOG.md @@ -21,18 +21,21 @@ Paths below are relative to the repository root. ## Validation and release gates -| Bundle | Why it matters | -| ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | -| `dogfood/20260326-week9-release-readiness/` | Current release-signoff bundle for the `0.1.0` bar. | -| `dogfood/20260410-release-tarball/` | Local proof of the shared release tarball packer, checksum, and install flow used by the GitHub release workflow. | -| `dogfood/20260325-week8-contract-locks/` | Contract-lock and reporting review evidence. | -| `dogfood/20260325-week8-bundle-validation/` | Validation of proof-bundle conventions. | -| `dogfood/20260325-week8-capability-inventory/` | Runtime capability inventory/reporting evidence. | -| `dogfood/20260325-week8-inspect-runtime/` | `inspect --json` runtime reporting review. | -| `dogfood/20260323-week5-platform-closure/` | Platform/documentation closeout evidence from the earlier hardening phase. | +| Bundle | Why it matters | +| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | +| `dogfood/20260326-week9-release-readiness/` | Current release-signoff bundle for the `0.1.0` bar. | +| `dogfood/20260410-release-tarball/` | Local proof of the shared release tarball packer, checksum, and install flow used by the GitHub release workflow. | +| `dogfood/20260325-week8-contract-locks/` | Contract-lock and reporting review evidence. | +| `dogfood/20260325-week8-bundle-validation/` | Validation of proof-bundle conventions. | +| `dogfood/20260325-week8-capability-inventory/` | Runtime capability inventory/reporting evidence. | +| `dogfood/20260325-week8-inspect-runtime/` | `inspect --json` runtime reporting review. | +| `dogfood/token-usage-phase5-proof/` | Phase 5 eval DX token-usage proof bundle (commit `91a571de`) with screenshot, WebM recording, snapshot, and replay script. | +| `dogfood/20260323-week5-platform-closure/` | Platform/documentation closeout evidence from the earlier hardening phase. | | `dogfood/20260330-docs-navigation/` | Repository docs walkthrough with screenshots and a WebM recording of the new navigation path. | +Follow-up: `dogfood/token-usage-phase5-proof/` is the only dedicated eval DX phase bundle currently cataloged. Dedicated proof bundles for the authoring façade, reporter lifecycle, and workspace-preset phases (Phases 1-4) are not currently present under `dogfood/`; reviewers who need fresh artifacts for those phases should capture a local proof bundle. + ## Recovery and hardening | Bundle | Focus | diff --git a/dogfood/README.md b/dogfood/README.md index 7ef11e8f..acee75ce 100644 --- a/dogfood/README.md +++ b/dogfood/README.md @@ -7,8 +7,9 @@ Some bundles are evergreen workflow scenarios, some are release/contract validat 1. Read [`CATALOG.md`](./CATALOG.md) for the curated bundle map. 2. For the current release-signoff view, start with `dogfood/20260326-week9-release-readiness/`. -3. For evergreen workflows, start with bundles such as `dogfood/run-command/`, `dogfood/20260322-dogfood-hello-prompt/`, and `dogfood/20260322-lazyvim-scenario/`. -4. For recovery and hardening behavior, use the recovery section in the catalog. +3. For the Phase 5 eval DX token-usage proof from commit `91a571de`, start with `dogfood/token-usage-phase5-proof/`. +4. For evergreen workflows, start with bundles such as `dogfood/run-command/`, `dogfood/20260322-dogfood-hello-prompt/`, and `dogfood/20260322-lazyvim-scenario/`. +5. For recovery and hardening behavior, use the recovery section in the catalog. ## How to treat the directory diff --git a/dogfood/token-usage-phase5-proof/README.md b/dogfood/token-usage-phase5-proof/README.md new file mode 100644 index 00000000..bf0a6102 --- /dev/null +++ b/dogfood/token-usage-phase5-proof/README.md @@ -0,0 +1,52 @@ +# Phase 5 token-usage proof bundle + +This bundle captures a focused `agent-tty` validation run for the Phase 5 token-usage work. It provides reviewer-visible proof for these commits: + +- `7beaa84` — `feat(evals): add optional token usage to normalized output` +- `202244b` — `feat(evals): normalize token usage in claude and codex providers` +- `e9828d5` — `feat(evals): support deterministic token usage in fixtures provider` + +The current repo-local CLI uses `create` / `destroy` rather than `session start` / `session destroy`; recording export is available for created sessions, so there is no separate `--record` flag in this workspace's `--help` output. + +## Artifacts + +- Snapshot: `dogfood/token-usage-phase5-proof/snapshot.txt` +- Screenshot: `dogfood/token-usage-phase5-proof/screenshot.png` +- Recording: `dogfood/token-usage-phase5-proof/recording.webm` +- Replay script: `dogfood/token-usage-phase5-proof/commands.sh` + +## Exact command list used + +The executable replay script is `dogfood/token-usage-phase5-proof/commands.sh`. The exact command sequence it ran was: + +```bash +ROOT_DIR="$(git rev-parse --show-toplevel)" +BUNDLE_DIR="$ROOT_DIR/dogfood/token-usage-phase5-proof" +export AGENT_TTY_HOME="$(mktemp -d "$BUNDLE_DIR/.home.XXXXXX")" + +npx tsx src/cli/main.ts --help +npx tsx src/cli/main.ts doctor --json +CREATE_JSON="$(npx tsx src/cli/main.ts create --json --cwd "$ROOT_DIR" --cols 140 --rows 45 --env 'PS1=phase5-proof$ ' -- /bin/bash --noprofile --norc -i)" +SESSION_ID="$(printf '%s\n' "$CREATE_JSON" | jq -er '.result.sessionId')" + +npx tsx src/cli/main.ts wait "$SESSION_ID" --screen-stable-ms 500 --timeout 10000 --json +npx tsx src/cli/main.ts run "$SESSION_ID" 'npm run typecheck' --timeout 300000 --json +npx tsx src/cli/main.ts run "$SESSION_ID" 'npm run lint' --timeout 300000 --json +npx tsx src/cli/main.ts run "$SESSION_ID" 'npx vitest run test/unit/evals/claude.test.ts test/unit/evals/codex.test.ts test/unit/evals/promptRunner.test.ts test/integration/evals/authoring-pilots.test.ts --reporter=verbose' --timeout 300000 --json +npx tsx src/cli/main.ts wait "$SESSION_ID" --screen-stable-ms 1500 --timeout 10000 --json + +SNAPSHOT_JSON="$(npx tsx src/cli/main.ts snapshot "$SESSION_ID" --format text --include-scrollback --json)" +printf '%s\n' "$SNAPSHOT_JSON" | jq -er '.result.text' > "$BUNDLE_DIR/snapshot.txt" + +SCREENSHOT_JSON="$(npx tsx src/cli/main.ts screenshot "$SESSION_ID" --json)" +SCREENSHOT_PATH="$(printf '%s\n' "$SCREENSHOT_JSON" | jq -er '.result.artifactPath')" +cp "$SCREENSHOT_PATH" "$BUNDLE_DIR/screenshot.png" + +npx tsx src/cli/main.ts record export "$SESSION_ID" --format webm --out "$BUNDLE_DIR/recording.webm" --json +npx tsx src/cli/main.ts destroy "$SESSION_ID" --json +rm -rf "$AGENT_TTY_HOME" +``` + +## What to verify + +Open `screenshot.png` to confirm the terminal finished on a green focused validation run, play `recording.webm` to confirm the full `agent-tty` session covers `typecheck`, `lint`, and the verbose Vitest command end to end, and inspect `snapshot.txt` to confirm the scrollback includes the 27-test success summary plus the token-usage-specific test names from `claude.test.ts`, `codex.test.ts`, and `authoring-pilots.test.ts`. diff --git a/dogfood/token-usage-phase5-proof/commands.sh b/dogfood/token-usage-phase5-proof/commands.sh new file mode 100755 index 00000000..e9d035a7 --- /dev/null +++ b/dogfood/token-usage-phase5-proof/commands.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR="$(git rev-parse --show-toplevel)" +BUNDLE_DIR="$ROOT_DIR/dogfood/token-usage-phase5-proof" +mkdir -p "$BUNDLE_DIR" +rm -f "$BUNDLE_DIR/snapshot.txt" "$BUNDLE_DIR/screenshot.png" "$BUNDLE_DIR/recording.webm" + +export AGENT_TTY_HOME="$(mktemp -d "$BUNDLE_DIR/.home.XXXXXX")" +SESSION_ID='' + +cleanup() { + if [[ -n "$SESSION_ID" ]]; then + npx tsx src/cli/main.ts destroy "$SESSION_ID" --json >/dev/null 2>&1 || true + fi + if [[ -n "${AGENT_TTY_HOME:-}" && -d "${AGENT_TTY_HOME:-}" ]]; then + rm -rf "$AGENT_TTY_HOME" + fi +} +trap cleanup EXIT + +cd "$ROOT_DIR" + +npx tsx src/cli/main.ts --help >/dev/null +npx tsx src/cli/main.ts doctor --json | jq -e '.ok == true and .result.ok == true' >/dev/null + +CREATE_JSON="$(npx tsx src/cli/main.ts create --json --cwd "$ROOT_DIR" --cols 140 --rows 45 --env 'PS1=phase5-proof$ ' -- /bin/bash --noprofile --norc -i)" +SESSION_ID="$(printf '%s\n' "$CREATE_JSON" | jq -er '.result.sessionId')" + +npx tsx src/cli/main.ts wait "$SESSION_ID" --screen-stable-ms 500 --timeout 10000 --json | jq -e '.ok == true and .result.timedOut == false' >/dev/null + +TYPECHECK_JSON="$(npx tsx src/cli/main.ts run "$SESSION_ID" 'npm run typecheck' --timeout 300000 --json)" +printf '%s\n' "$TYPECHECK_JSON" | jq -e '.ok == true and .result.accepted == true and .result.completed == true and (.result.timedOut // false) == false' >/dev/null + +LINT_JSON="$(npx tsx src/cli/main.ts run "$SESSION_ID" 'npm run lint' --timeout 300000 --json)" +printf '%s\n' "$LINT_JSON" | jq -e '.ok == true and .result.accepted == true and .result.completed == true and (.result.timedOut // false) == false' >/dev/null + +VITEST_COMMAND='npx vitest run test/unit/evals/claude.test.ts test/unit/evals/codex.test.ts test/unit/evals/promptRunner.test.ts test/integration/evals/authoring-pilots.test.ts --reporter=verbose' +VITEST_JSON="$(npx tsx src/cli/main.ts run "$SESSION_ID" "$VITEST_COMMAND" --timeout 300000 --json)" +printf '%s\n' "$VITEST_JSON" | jq -e '.ok == true and .result.accepted == true and .result.completed == true and (.result.timedOut // false) == false' >/dev/null + +npx tsx src/cli/main.ts wait "$SESSION_ID" --screen-stable-ms 1500 --timeout 10000 --json | jq -e '.ok == true and .result.timedOut == false' >/dev/null + +SNAPSHOT_JSON="$(npx tsx src/cli/main.ts snapshot "$SESSION_ID" --format text --include-scrollback --json)" +printf '%s\n' "$SNAPSHOT_JSON" | jq -er '.result.text' > "$BUNDLE_DIR/snapshot.txt" + +SCREENSHOT_JSON="$(npx tsx src/cli/main.ts screenshot "$SESSION_ID" --json)" +SCREENSHOT_PATH="$(printf '%s\n' "$SCREENSHOT_JSON" | jq -er '.result.artifactPath')" +cp "$SCREENSHOT_PATH" "$BUNDLE_DIR/screenshot.png" + +npx tsx src/cli/main.ts record export "$SESSION_ID" --format webm --out "$BUNDLE_DIR/recording.webm" --json | jq -e '.ok == true and .result.format == "webm"' >/dev/null +npx tsx src/cli/main.ts destroy "$SESSION_ID" --json | jq -e '.ok == true' >/dev/null +SESSION_ID='' + +trap - EXIT +rm -rf "$AGENT_TTY_HOME" +unset AGENT_TTY_HOME diff --git a/dogfood/token-usage-phase5-proof/recording.webm b/dogfood/token-usage-phase5-proof/recording.webm new file mode 100644 index 00000000..7840ccd2 Binary files /dev/null and b/dogfood/token-usage-phase5-proof/recording.webm differ diff --git a/dogfood/token-usage-phase5-proof/screenshot.png b/dogfood/token-usage-phase5-proof/screenshot.png new file mode 100644 index 00000000..4256f5b7 Binary files /dev/null and b/dogfood/token-usage-phase5-proof/screenshot.png differ diff --git a/dogfood/token-usage-phase5-proof/snapshot.txt b/dogfood/token-usage-phase5-proof/snapshot.txt new file mode 100644 index 00000000..e35c3b1c --- /dev/null +++ b/dogfood/token-usage-phase5-proof/snapshot.txt @@ -0,0 +1,62 @@ +phase5-proof$ npm run typecheck + +> agent-tty@0.1.1-beta.3 typecheck +> tsc -p tsconfig.json --noEmit + +phase5-proof$ printf '%s%s\n' '__AT_MARKER_219ee04df48' 'b4d0a9bcaaa71b3c0f578__' +__AT_MARKER_219ee04df48b4d0a9bcaaa71b3c0f578__ +phase5-proof$ npm run lint + +> agent-tty@0.1.1-beta.3 lint +> eslint src test vitest.config.ts --max-warnings=0 + +phase5-proof$ printf '%s%s\n' '__AT_MARKER_9eeecfa1bf8' '74baca2e5f837d57f6727__' +__AT_MARKER_9eeecfa1bf874baca2e5f837d57f6727__ +phase5-proof$ npx vitest run test/unit/evals/claude.test.ts test/unit/evals/codex.test.ts test/unit/evals/promptRunner.test.ts test/integrat +ion/evals/authoring-pilots.test.ts --reporter=verbose + + RUN v4.1.2 /home/coder/.mux/src/agent-terminal/agent_exec_f53d8e814f + + ✓ test/unit/evals/claude.test.ts > ClaudeProvider.parse > normalizes complete JSON usage objects and prefers the last valid record 8ms + ✓ test/unit/evals/claude.test.ts > ClaudeProvider.parse > omits tokenUsage for 'missing total_tokens' 1ms + ✓ test/unit/evals/claude.test.ts > ClaudeProvider.parse > omits tokenUsage for 'fractional totals' 0ms + ✓ test/unit/evals/claude.test.ts > ClaudeProvider.parse > omits tokenUsage for 'negative cached tokens' 0ms + ✓ test/unit/evals/claude.test.ts > ClaudeProvider.parse > does not emit tokenUsage for plain-text output 1ms + ✓ test/unit/evals/codex.test.ts > CodexProvider.parse > normalizes command_execution items into shell-style tool calls 11ms + ✓ test/unit/evals/codex.test.ts > CodexProvider.parse > normalizes function_call records with parsed arguments and outputs 2ms + ✓ test/unit/evals/codex.test.ts > CodexProvider.parse > normalizes complete JSON usage objects and prefers the last valid record 2ms + ✓ test/unit/evals/codex.test.ts > CodexProvider.parse > omits tokenUsage for 'missing total_tokens' 1ms + ✓ test/unit/evals/codex.test.ts > CodexProvider.parse > omits tokenUsage for 'conflicting cached-token aliases' 0ms + ✓ test/unit/evals/codex.test.ts > CodexProvider.parse > does not emit tokenUsage for plain-text output 1ms + ✓ test/unit/evals/promptRunner.test.ts > enumeratePromptWorkItems > returns unique prompt work items with stable keys 48ms + ✓ test/unit/evals/promptRunner.test.ts > enumeratePromptWorkItems > filters prompt work items by case id 3ms + ✓ test/unit/evals/promptRunner.test.ts > enumeratePromptWorkItems > filters prompt work items by condition 2ms + ✓ test/unit/evals/promptRunner.test.ts > executePromptWorkItem > builds a prompt request and returns the provider result payload 13ms + ✓ test/unit/evals/promptRunner.test.ts > executePromptWorkItem > converts provider errors into failed eval results 3ms + ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > resolves wait-for-output in the dry-run matrix 549ms + ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > runs wait-for-output end-to-end through the CLI facade + with fixture playback 571ms + ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > resolves hello-prompt in the dry-run matrix 579ms + ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > runs hello-prompt end-to-end through the CLI facade wi +th fixture playback 441ms + ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > resolves doctor-gated in the dry-run matrix 552ms + ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > runs doctor-gated end-to-end through the CLI facade wi +th fixture playback 451ms + ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > resolves exploratory-qa in the dry-run matrix 422ms + ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > runs exploratory-qa end-to-end through the CLI facade +with fixture playback 571ms + ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > round-trips tokenUsage from full normalized override f +ixtures 8ms + ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > preserves inline prompt and agent tokenUsage and keeps + omissions undefined 8ms + ✓ test/integration/evals/authoring-pilots.test.ts > eval CLI authoring pilot cases > preserves valid legacy normalized tokenUsage and drops + invalid tokenUsage payloads 13ms + + Test Files 4 passed (4) + Tests 27 passed (27) + Start at 11:31:58 + Duration 4.71s (transform 650ms, setup 0ms, import 1.31s, tests 4.27s, environment 0ms) + +phase5-proof$ printf '%s%s\n' '__AT_MARKER_e2b4fb7e307' '7485a89622cec04e0982a__' +__AT_MARKER_e2b4fb7e3077485a89622cec04e0982a__ +phase5-proof$ diff --git a/evals/README.md b/evals/README.md index 7acaa4b0..c5b1f5b4 100644 --- a/evals/README.md +++ b/evals/README.md @@ -121,6 +121,7 @@ Providers implement one shared interface in `evals/providers/base.ts`: ```text evals/ ├── README.md +├── authoring/ ├── run.ts ├── lib/ │ ├── antiPatterns.ts @@ -143,18 +144,25 @@ evals/ │ ├── cases/ │ ├── scorers/ │ └── runner.ts -└── providers/ - ├── base.ts - ├── claude.ts - ├── codex.ts - └── fixtures.ts +├── providers/ +│ ├── base.ts +│ ├── claude.ts +│ ├── codex.ts +│ └── fixtures.ts +├── reporters/ +├── snapshots/ +└── workspaces/ ``` High-level roles: +- `authoring/` contains the fluent case builders and raw escape hatches. - `lib/` contains schemas, scoring, report generation, matrix math, and artifact helpers. - `prompt/`, `execution/`, and `dogfood/` each define a lane-specific case inventory plus a runner. - `providers/` normalizes different model backends into one interface. +- `reporters/` contains lifecycle event types plus the built-in console, JSONL, and final-report reporters. +- `snapshots/` contains token snapshot fingerprinting, storage, and comparison logic. +- `workspaces/` contains preset registration, resolution, and built-in workspace presets. - `run.ts` is the top-level orchestrator for CLI usage. ## 5. Case inventory @@ -374,6 +382,530 @@ A practical authoring loop is: 3. run `npx tsx evals/run.ts --provider stub --lane --case --dry-run`, 4. then run the real lane against `stub` or a real provider. +### Authoring evals with the façade + +Case authoring previously required hand-assembling strict schema objects and parsing them by hand. The fluent builders in [`./authoring/`](./authoring/) keep the same validation guarantees, but make it much easier to add workflow checks, verifiers, report requirements, and workspace presets incrementally. Start with [`./authoring/index.ts`](./authoring/index.ts); the lane-specific builders live in [`./authoring/prompt.ts`](./authoring/prompt.ts), [`./authoring/execution.ts`](./authoring/execution.ts), and [`./authoring/dogfood.ts`](./authoring/dogfood.ts). + +The snippets below assume you are creating a new file beside the canonical cases in the matching `evals//cases/` directory. + +
+Prompt lane (wait-for-output) — before vs. after + +Before (equivalent raw schema object): + +```ts +import { PromptEvalCaseSchema } from '../../lib/schemas.js'; +import type { PromptEvalCase } from '../../lib/types.js'; + +const PROMPT_TIMEOUT_MS = 30_000; +const SLEEP_RECOMMENDATION_PATTERN = String.raw`/(?:^|\n)\s*sleep\s+\d+(?:\.\d+)?\b|(?:(?<=\buse\s)|(?<=\brun\s)|(?<=\badd\s)|(?<=\binsert\s))sleep\s+\d+(?:\.\d+)?\b/i`; + +function requiredCheck( + id: string, + description: string, + requiredPatterns: string[], + forbiddenPatterns: string[] = [], +): PromptEvalCase['workflowChecks'][number] { + return { + id, + description, + required: true, + requiredPatterns, + forbiddenPatterns, + dependsOn: [], + }; +} + +export const waitForOutputCase: PromptEvalCase = PromptEvalCaseSchema.parse({ + id: 'wait-for-output', + lane: 'prompt', + category: 'trigger', + prompt: + "I need to wait until my server prints 'Listening on port 3000' before running tests", + expectedSkill: 'agent-tty', + context: + 'The answer should prefer waiting on observable terminal text over fixed delays before starting the next step.', + expectedPatterns: ['/agent-tty/i', '/\\bwait\\b/i'], + forbiddenPatterns: [SLEEP_RECOMMENDATION_PATTERN, '/setTimeout/i'], + rubric: [ + 'Chooses agent-tty for terminal readiness coordination.', + 'Uses wait against concrete terminal output instead of fixed timing guesses.', + ], + workflowChecks: [ + requiredCheck( + 'wait-for-output.select-agent-tty', + 'Explicitly selects agent-tty.', + ['/agent-tty/i'], + ), + requiredCheck( + 'wait-for-output.observe-readiness', + 'Waits for the listening message before running tests.', + ['/\\bwait\\b/i', '/Listening on port 3000/i'], + [SLEEP_RECOMMENDATION_PATTERN, '/setTimeout/i'], + ), + ], + antiPatterns: [], + budgets: { timeoutMs: PROMPT_TIMEOUT_MS }, +}); +``` + +After (builder façade, matching [`./prompt/cases/trigger-agent-tty.ts`](./prompt/cases/trigger-agent-tty.ts)): + +```ts +import { promptCase } from '../../authoring/index.js'; +import type { PromptEvalCase } from '../../lib/types.js'; + +const PROMPT_TIMEOUT_MS = 30_000; +const EMPTY_ANTI_PATTERNS: PromptEvalCase['antiPatterns'] = []; +const SLEEP_RECOMMENDATION_PATTERN = String.raw`/(?:^|\n)\s*sleep\s+\d+(?:\.\d+)?\b|(?:(?<=\buse\s)|(?<=\brun\s)|(?<=\badd\s)|(?<=\binsert\s))sleep\s+\d+(?:\.\d+)?\b/i`; + +export const waitForOutputCase = promptCase('wait-for-output') + .category('trigger') + .prompt( + "I need to wait until my server prints 'Listening on port 3000' before running tests", + ) + .expectSkill('agent-tty') + .context( + 'The answer should prefer waiting on observable terminal text over fixed delays before starting the next step.', + ) + .expectedPatterns(['/agent-tty/i', '/\\bwait\\b/i']) + .forbiddenPatterns([SLEEP_RECOMMENDATION_PATTERN, '/setTimeout/i']) + .rubric( + 'Chooses agent-tty for terminal readiness coordination.', + 'Uses wait against concrete terminal output instead of fixed timing guesses.', + ) + .workflow((workflow) => { + workflow + .step('wait-for-output.select-agent-tty', 'Explicitly selects agent-tty.') + .mustMention('/agent-tty/i'); + workflow + .step( + 'wait-for-output.observe-readiness', + 'Waits for the listening message before running tests.', + ) + .mustMention('/\\bwait\\b/i', '/Listening on port 3000/i') + .mustNotMention(SLEEP_RECOMMENDATION_PATTERN, '/setTimeout/i'); + }) + .antiPatterns(...EMPTY_ANTI_PATTERNS) + .budget(PROMPT_TIMEOUT_MS) + .build(); +``` + +
+ +
+Execution lane (hello-prompt) — before vs. after + +Before (equivalent raw schema object): + +```ts +import { + ALL_EXECUTION_CONDITIONS, + CREATE_SESSION_PATTERN, + DESTROY_SESSION_PATTERN, + SNAPSHOT_PATTERN, + WAIT_PATTERN, + anyOf, + executionAntiPatterns, + executionBudgets, + executionTaskPrompt, + fixtureSetupStep, + requiredVerifier, + workflowCheck, +} from './shared.js'; +import { ExecutionEvalCaseSchema } from '../../lib/schemas.js'; + +const HELLO_WORLD_INPUT_PATTERN = anyOf( + String.raw`\bagent-tty\b[^\n]*\b(?:run|type)\b[^\n]*hello world`, + String.raw`\b(?:run|type)(?:ning|s|ned)?\b[^\n]*hello world\b`, + String.raw`ECHO:\s*hello world`, +); + +export const helloPromptCase = ExecutionEvalCaseSchema.parse({ + id: 'hello-prompt', + lane: 'execution', + category: 'session', + prompt: executionTaskPrompt( + "Launch the hello-prompt fixture, send 'hello world' as input, wait for the READY> prompt to reappear, take a snapshot to verify the echo, then destroy the session.", + 'hello-prompt', + ), + expectedSkill: 'agent-tty', + fixture: 'hello-prompt', + conditions: [...ALL_EXECUTION_CONDITIONS], + setup: [ + fixtureSetupStep( + 'launch-hello-prompt', + 'hello-prompt', + 'Create an agent-tty session that runs the hello-prompt fixture.', + ), + ], + verifiers: [ + requiredVerifier( + 'hello-prompt-snapshot', + 'snapshot', + 'The transcript snapshot should include the echoed text and the READY prompt.', + { + patterns: [String.raw`ECHO:\s*hello world`, String.raw`READY>`], + }, + ), + ], + workflowChecks: [ + workflowCheck( + 'create', + 'Create the fixture session.', + CREATE_SESSION_PATTERN, + ), + workflowCheck( + 'input', + 'Send hello world with run or type.', + HELLO_WORLD_INPUT_PATTERN, + { dependsOn: ['create'] }, + ), + workflowCheck( + 'wait', + 'Wait for the READY prompt to reappear after the echo.', + anyOf(WAIT_PATTERN, String.raw`ECHO:\s*hello world[\s\S]*READY>`), + { dependsOn: ['input'] }, + ), + workflowCheck( + 'snapshot', + 'Capture a snapshot for verification.', + SNAPSHOT_PATTERN, + { dependsOn: ['wait'] }, + ), + workflowCheck( + 'destroy', + 'Destroy the session after verification.', + DESTROY_SESSION_PATTERN, + { dependsOn: ['snapshot'] }, + ), + ], + antiPatterns: executionAntiPatterns(), + artifactRequirements: [], + budgets: executionBudgets({ + timeoutMs: 120_000, + maxAgentSteps: 12, + maxWallClockMs: 60_000, + }), + referenceSteps: 5, + workspace: 'agent-tty-smoke', +}); +``` + +After (builder façade, matching [`./execution/cases/hello-prompt.ts`](./execution/cases/hello-prompt.ts)): + +```ts +import { executionCase } from '../../authoring/index.js'; +import { ALL_EXECUTION_CONDITIONS, anyOf } from './shared.js'; + +const HELLO_WORLD_INPUT_PATTERN = anyOf( + String.raw`\bagent-tty\b[^\n]*\b(?:run|type)\b[^\n]*hello world`, + String.raw`\b(?:run|type)(?:ning|s|ned)?\b[^\n]*hello world\b`, + String.raw`ECHO:\s*hello world`, +); + +export const helloPromptCase = executionCase('hello-prompt') + .category('session') + .task( + "Launch the hello-prompt fixture, send 'hello world' as input, wait for the READY> prompt to reappear, take a snapshot to verify the echo, then destroy the session.", + ) + .fixture('hello-prompt', { + setupId: 'launch-hello-prompt', + setupDescription: + 'Create an agent-tty session that runs the hello-prompt fixture.', + }) + .referenceSteps(5) + .conditions(...ALL_EXECUTION_CONDITIONS) + .assertions((assertions) => { + assertions.snapshot( + 'hello-prompt-snapshot', + 'The transcript snapshot should include the echoed text and the READY prompt.', + { + patterns: [String.raw`ECHO:\s*hello world`, String.raw`READY>`], + }, + ); + }) + .workflow((workflow) => { + workflow + .createSession() + .input('hello world', { + description: 'Send hello world with run or type.', + pattern: HELLO_WORLD_INPUT_PATTERN, + }) + .waitFor(String.raw`ECHO:\s*hello world[\s\S]*READY>`, { + description: 'Wait for the READY prompt to reappear after the echo.', + }) + .snapshot() + .destroy(); + }) + .budget({ + timeoutMs: 120_000, + maxAgentSteps: 12, + maxWallClockMs: 60_000, + }) + .workspace('agent-tty-smoke') + .build(); +``` + +
+ +
+Dogfood lane (exploratory-qa) — before vs. after + +Before (equivalent raw schema object): + +```ts +import { + artifactRequirement, + requiredVerifier, +} from '../../execution/cases/shared.js'; +import { DEFAULT_ANTI_PATTERN_RULES } from '../../lib/antiPatterns.js'; +import { SKILL_CONDITIONS } from '../../lib/matrix.js'; +import { DogfoodEvalCaseSchema } from '../../lib/schemas.js'; +import { dogfoodTaskPrompt } from './shared.js'; + +const SCREENSHOT_BUNDLE_PATH_PATTERN = String.raw`\.png$`; +const RECORDING_BUNDLE_PATH_PATTERN = String.raw`\.cast$`; +const NOTES_BUNDLE_PATH_PATTERN = String.raw`(?:^|/)(?:README|NOTES|index|notes)\.md$`; +const TITLE_PATTERN = String.raw`/(?:^|\n)\s*(?:#{1,3}\s*Title\b|\*\*Title:?\*\*)/im`; +const REPRODUCTION_SECTION_PATTERN = String.raw`/(?:^|\n)\s*(?:#{1,3}\s*(?:Reproduction steps|Repro(?:duction)? steps|Steps)\b|\*\*(?:Reproduction steps|Repro(?:duction)? steps|Steps):?\*\*)/im`; +const FINDINGS_SECTION_PATTERN = String.raw`/(?:^|\n)\s*(?:#{1,3}\s*(?:Findings|Issues)\b|\*\*(?:Findings|Issues):?\*\*)/im`; +const EVIDENCE_SECTION_PATTERN = String.raw`/(?:^|\n)\s*(?:#{1,3}\s*Evidence\b|\*\*Evidence:?\*\*)/im`; +const CLI_REFERENCE_PATTERN = String.raw`/\b(?:agent-tty|npx\s+tsx\s+src\/cli\/main\.ts)\b/i`; +const SEVERITY_PATTERN = String.raw`/\b(?:severity|critical|high|medium|low|info)\b/i`; +const EVIDENCE_REFERENCE_PATTERN = String.raw`/\.(?:png|cast|webm|json|md)\b/i`; + +export const exploratoryQaCase = DogfoodEvalCaseSchema.parse({ + id: 'exploratory-qa', + lane: 'dogfood', + category: 'qa', + prompt: dogfoodTaskPrompt( + 'Launch the hello-prompt fixture, test exactly three inputs (`hello world`, a blank line, and `symbols-!@#$%^&*`), capture a snapshot after each input, then send `exit` to verify clean shutdown. Save at least one screenshot and one recording, and write a brief findings report with severity and evidence references.', + 'hello-prompt', + ), + expectedSkill: 'dogfood-tui', + fixture: 'hello-prompt', + bundlePath: 'proof-bundle', + bundleRequirements: [ + 'Produce a reviewable proof bundle for an exploratory QA investigation.', + 'Capture renderer-backed evidence for the tested interactions and edge cases.', + 'Write structured notes that summarize findings, severity, and evidence references.', + ], + conditions: [...SKILL_CONDITIONS], + validationProfile: 'interactive-renderer', + artifactRequirements: [ + artifactRequirement( + 'screenshot', + 'Capture at least one screenshot of a noteworthy state.', + SCREENSHOT_BUNDLE_PATH_PATTERN, + ), + artifactRequirement( + 'recording', + 'Capture at least one terminal recording artifact.', + RECORDING_BUNDLE_PATH_PATTERN, + ), + artifactRequirement( + 'notes', + 'Write exploratory QA notes in a markdown report.', + NOTES_BUNDLE_PATH_PATTERN, + ), + ], + reportRequirements: [ + { + id: 'title', + description: 'Report must have a descriptive title.', + required: true, + section: 'Title', + requiredPatterns: [TITLE_PATTERN], + forbiddenPatterns: [], + }, + { + id: 'repro-steps', + description: 'Include step-by-step reproduction commands.', + required: true, + section: 'Reproduction steps', + requiredPatterns: [REPRODUCTION_SECTION_PATTERN, CLI_REFERENCE_PATTERN], + forbiddenPatterns: [], + }, + { + id: 'findings', + description: 'List findings with severity classification.', + required: true, + section: 'Findings', + requiredPatterns: [FINDINGS_SECTION_PATTERN, SEVERITY_PATTERN], + forbiddenPatterns: [], + }, + { + id: 'evidence', + description: + 'Reference captured artifacts such as screenshots and recordings.', + required: true, + section: 'Evidence', + requiredPatterns: [EVIDENCE_SECTION_PATTERN, EVIDENCE_REFERENCE_PATTERN], + forbiddenPatterns: [], + }, + ], + verifiers: [ + requiredVerifier( + 'bundle-valid', + 'bundle', + 'Validate the exploratory QA proof bundle with the interactive renderer profile.', + { profile: 'interactive-renderer' }, + ), + ], + workflowChecks: [], + antiPatterns: DEFAULT_ANTI_PATTERN_RULES.map((rule) => ({ + ...rule, + patterns: [...rule.patterns], + ...(rule.lanes === undefined ? {} : { lanes: [...rule.lanes] }), + })), + budgets: { + timeoutMs: 600_000, + maxAgentSteps: 30, + maxWallClockMs: 600_000, + }, + workspace: 'agent-tty-smoke', +}); +``` + +After (builder façade, matching [`./dogfood/cases/exploratory-qa.ts`](./dogfood/cases/exploratory-qa.ts)): + +```ts +import { dogfoodCase } from '../../authoring/index.js'; +import { SKILL_CONDITIONS } from '../../lib/matrix.js'; + +export const exploratoryQaCase = dogfoodCase('exploratory-qa') + .category('qa') + .task( + 'Launch the hello-prompt fixture, test exactly three inputs (`hello world`, a blank line, and `symbols-!@#$%^&*`), capture a snapshot after each input, then send `exit` to verify clean shutdown. Save at least one screenshot and one recording, and write a brief findings report with severity and evidence references.', + ) + .fixture('hello-prompt') + .bundlePath('proof-bundle') + .bundleRequirements([ + 'Produce a reviewable proof bundle for an exploratory QA investigation.', + 'Capture renderer-backed evidence for the tested interactions and edge cases.', + 'Write structured notes that summarize findings, severity, and evidence references.', + ]) + .conditions(...SKILL_CONDITIONS) + .validationProfile('interactive-renderer') + .proofBundle((bundle) => { + bundle.requiresScreenshot(); + bundle.requiresRecording(); + bundle.requiresNotes(); + }) + .report((report) => { + report.title(); + report.reproductionSteps(); + report.findingsWithSeverity(); + report.evidenceReferences(); + }) + .bundleVerifier( + 'bundle-valid', + 'Validate the exploratory QA proof bundle with the interactive renderer profile.', + ) + .budget({ + timeoutMs: 600_000, + maxAgentSteps: 30, + maxWallClockMs: 600_000, + }) + .workspace('agent-tty-smoke') + .build(); +``` + +
+ +When the sugar is not enough, use the raw escape hatches from [`./authoring/raw.ts`](./authoring/raw.ts) or the matching builder methods: + +- `rawWorkflowCheck(check)` — drop in a fully shaped `WorkflowCheck` when you need a dependency graph, weight, or regex combination that the helper DSL does not express cleanly. +- `rawVerifier(verifier)` — add an exact `VerifierSpec` when the assertion helpers do not cover the verifier kind or config you need. +- `rawArtifactRequirement(requirement)` — add an explicit `ArtifactRequirement` when `.artifact()` or `.proofBundle()` is not expressive enough. +- `rawReportRequirement(requirement)` — add an exact dogfood `ReportRequirement` when the section helpers in [`./authoring/report.ts`](./authoring/report.ts) do not fit the report structure you want. + +Migration guidance: old case files continue to work. Prefer opportunistic migrations when you are already editing a case anyway; there is no requirement to rewrite the remaining schema-authored cases first. + +### Workspace presets + +Workspace presets live in [`./workspaces/types.ts`](./workspaces/types.ts) and [`./workspaces/registry.ts`](./workspaces/registry.ts). A `WorkspacePreset` is a small declarative bundle of `id`, `mode`, `description`, optional `templateDir`, optional `cwd`, optional `env`, and optional `bootstrap` commands. For `isolated` presets, the harness creates a fresh eval home, optionally copies `templateDir` into it, resolves `cwd`, runs preset bootstrap steps, and then continues with the case's normal execution. The materialization path lives in [`./lib/cliHarness.ts`](./lib/cliHarness.ts) and the resolver/redaction rules live in [`./workspaces/resolver.ts`](./workspaces/resolver.ts). + +The built-in smoke preset is registered from [`./workspaces/builtins.ts`](./workspaces/builtins.ts): + +```ts +import process from 'node:process'; + +import type { WorkspacePreset } from './types.js'; + +export const AGENT_TTY_SMOKE_PRESET: WorkspacePreset = { + id: 'agent-tty-smoke', + mode: 'isolated', + description: 'Deterministic local smoke preset for agent-tty evals.', + bootstrap: [ + { + command: process.execPath, + args: ['-e', 'process.stdout.write("agent-tty-smoke bootstrap ok\\n")'], + description: 'agent-tty-smoke smoke-probe', + }, + ], +}; +``` + +`registerBuiltinPresets()` wires this preset into the CLI automatically, so cases can reference it by ID without extra setup. + +Use it from an execution or dogfood case with `.workspace('agent-tty-smoke')`: + +```ts +import { executionCase, dogfoodCase } from '../../authoring/index.js'; + +export const helloPromptCase = executionCase('hello-prompt') + // ... + .workspace('agent-tty-smoke') + .build(); + +export const exploratoryQaCase = dogfoodCase('exploratory-qa') + // ... + .workspace('agent-tty-smoke') + .build(); +``` + +If you need a project-specific preset, register it before `runEvalCli()` is called: + +```ts +import process from 'node:process'; + +import { runEvalCli } from './evals/run.js'; +import { registerPreset } from './evals/workspaces/registry.js'; +import type { WorkspacePreset } from './evals/workspaces/types.js'; + +const helloSmokePreset: WorkspacePreset = { + id: 'hello-smoke', + mode: 'isolated', + description: 'Copy the hello-prompt fixture into the temp home and seed env.', + templateDir: 'test/fixtures/apps/hello-prompt', + cwd: '.', + env: { + APP_MODE: 'smoke', + GITHUB_TOKEN: 'example-secret', + }, + bootstrap: [ + { + command: process.execPath, + args: ['-e', 'process.stdout.write("hello-smoke bootstrap ok\\n")'], + description: 'workspace probe', + }, + ], +}; + +registerPreset(helloSmokePreset); +await runEvalCli(process.argv.slice(2)); +``` + +Merge order is fixed and intentional: + +1. preset `env` is the base layer; +2. request-specific env overrides are layered on top at spawn time; +3. preset `bootstrap` runs before case-level `setup`. + +There is no separate `.env()` builder hook for cases today — the override layer is the lane/runtime env that the harness injects when it creates the request (for example `AGENT_TTY_EVAL_OUTPUT_DIR` in execution or `EVAL_OUTPUT_DIR` / `EVAL_REQUESTED_BUNDLE_DIR` in dogfood). Reporter payloads only receive the redacted workspace summary on `case.start` (`presetId`, `cwd`, `env`, `bootstrapCount`); keys ending in `_TOKEN`, `_KEY`, `_SECRET`, or `_PASSWORD` are replaced with `[REDACTED]`, but the raw values are still used for bootstrap and provider spawn. + ## 10. CI integration A useful rollout pattern is to tier eval coverage by cost and stability. @@ -417,12 +949,82 @@ Each run writes its outputs under the chosen output base directory in a run-spec ```text / +├── snapshots/ # when --snapshot-update or --snapshot-check is used +│ └── -.jsonl └── / ├── metadata.json ├── report.json - └── report.md + ├── report.md + └── ///token-usage.json +``` + +Reporter outputs such as JSONL can also live outside this tree if you point `--reporter-output` somewhere else. `token-usage.json` only appears when a provider emits token usage, and `snapshots/` only appears when you opt into snapshot update/check mode. + +### Reporter lifecycle + +The reporter system in [`./reporters/`](./reporters/) gives the runner a typed lifecycle instead of one hard-coded output path. The lifecycle is `run.start` → `lane.start` → `case.start` → `trial.start` → `trial.finish` → `case.finish` → `lane.finish` → `run.finish`. In code, the `Reporter` interface in [`./reporters/types.ts`](./reporters/types.ts) exposes matching hooks (`onRunStart`, `onLaneStart`, `onCaseStart`, `onTrialStart`, `onTrialFinish`, `onCaseFinish`, `onLaneFinish`, `onRunFinish`), and the built-in JSONL reporter in [`./reporters/jsonl.ts`](./reporters/jsonl.ts) serializes the same lifecycle with dotted event names. + +Built-ins: + +- [`./reporters/console.ts`](./reporters/console.ts) — human-readable progress on stderr. +- [`./reporters/jsonl.ts`](./reporters/jsonl.ts) — append-only machine-readable lifecycle events. +- [`./reporters/final-report.ts`](./reporters/final-report.ts) — adapter that writes the existing `report.json` and `report.md` files. + +Common CLI combinations: + +```sh +# Implicit default: final report files only +npx tsx evals/run.ts --provider stub --lane prompt + +# Human-readable progress on stderr +npx tsx evals/run.ts --provider stub --lane execution --reporter console + +# Sugar for the console reporter +npx tsx evals/run.ts --provider stub --lane execution --progress + +# Append machine-readable lifecycle events to JSONL +npx tsx evals/run.ts \ + --provider stub \ + --lane execution \ + --reporter jsonl \ + --reporter-output evals/reports/execution-events.jsonl + +# Combine reporters explicitly +npx tsx evals/run.ts \ + --provider stub \ + --lane execution \ + --reporter final \ + --reporter jsonl \ + --reporter-output evals/reports/execution-events.jsonl \ + --progress ``` +When you do not pass `--reporter`, the runner still enables `final` by default so `report.json` and `report.md` are written. `--progress` only appends `console`; it does not disable `final`. + +If you need a custom reporter, implement the `Reporter` interface and point your extension code at the payload types in [`./reporters/types.ts`](./reporters/types.ts): + +```ts +import process from 'node:process'; + +import type { CaseFinishEvent, Reporter, RunStartEvent } from './types.js'; + +export class TimingReporter implements Reporter { + public readonly name = 'timing'; + + public onRunStart(event: RunStartEvent): void { + process.stderr.write(`run ${event.runId} started\n`); + } + + public onCaseFinish(event: CaseFinishEvent): void { + process.stderr.write( + `${event.lane}/${event.caseId}[${event.condition}] finished in ${event.durationMs}ms\n`, + ); + } +} +``` + +The dispatcher in [`./reporters/dispatch.ts`](./reporters/dispatch.ts) validates every payload, redacts secret-like env keys recursively before calling reporters, and isolates failures per reporter. A broken reporter writes an error to stderr, but it does not abort the eval run or stop the other reporters from receiving the same event. + ### `metadata.json` Contains the normalized `RunMetadata` for the run: @@ -442,6 +1044,7 @@ Contains the machine-readable aggregate report generated by `generateJsonReport( - aggregate pass/fail and score metrics, - condition comparison metrics, - every normalized `EvalResult`, +- optional `tokenReport` totals and snapshot-check results when token usage is available, - and provider-comparison views when relevant. ### `report.md` @@ -454,6 +1057,45 @@ Contains the reviewer-oriented Markdown report generated by `generateMarkdownRep - condition comparison, - failed cases, - anti-pattern summary, -- and completeness rollups. +- completeness rollups, +- and token-usage tables (plus snapshot warnings when requested). + +### Token usage + snapshots + +If a provider emits `normalizedOutput.tokenUsage`, the run writes a `token-usage.json` sidecar under `/////token-usage.json` via [`./lib/artifacts.ts`](./lib/artifacts.ts), and the final JSON + Markdown reports both grow a `tokenReport` section via [`./lib/reporting.ts`](./lib/reporting.ts). This is descriptive metadata, not part of case scoring. + +Token snapshots are an opt-in regression signal built on top of that data. The store and comparison logic live in [`./snapshots/store.ts`](./snapshots/store.ts) and [`./snapshots/compare.ts`](./snapshots/compare.ts). Snapshot identity is keyed by `(provider, model, lane, caseId, condition, caseFingerprint)`. The fingerprint in [`./snapshots/fingerprint.ts`](./snapshots/fingerprint.ts) hashes the semantic case definition, including prompt text and workflow checks, so changing either of those invalidates the old baseline. In practice the new row shows up as `new`, and the stale stored row may also appear as `orphaned` until you refresh the snapshot file. + +Recommended CLI workflow: + +```sh +# Establish or refresh the baseline +npx tsx evals/run.ts \ + --provider codex \ + --model gpt-5.4 \ + --lane execution \ + --case hello-prompt \ + --output evals/reports/snapshot-baseline \ + --snapshot-dir evals/reports/snapshots \ + --snapshot-update + +# Check a later run against the same baseline file +npx tsx evals/run.ts \ + --provider codex \ + --model gpt-5.4 \ + --lane execution \ + --case hello-prompt \ + --output evals/reports/snapshot-check \ + --snapshot-dir evals/reports/snapshots \ + --snapshot-check \ + --snapshot-threshold 20 +``` + +Guardrails: + +- `--snapshot-update` and `--snapshot-check` are mutually exclusive. +- Snapshot regressions are warnings only. They are surfaced in `tokenReport.snapshotCheck` and the Markdown report, but they do not change `EvalResult.ok`, `report.json` pass/fail totals, or the CLI exit code on their own. +- There is intentionally no snapshot-enforcement flag in this phase. +- The snapshot store writes one JSONL file per `provider-model` pair under the snapshot directory. Use a stable `--snapshot-dir` if you want to compare runs over time; the default is `/snapshots`. Use `--json` when you want only the final run summary on stdout; the full report files are still written to disk. diff --git a/evals/authoring/compile.ts b/evals/authoring/compile.ts new file mode 100644 index 00000000..6d79be99 --- /dev/null +++ b/evals/authoring/compile.ts @@ -0,0 +1,157 @@ +import type { ZodType } from 'zod'; + +import { invariant } from '../../src/util/assert.js'; +import type { EvalLane } from '../lib/types.js'; + +export type AuthoringPath = string | readonly PropertyKey[]; +export type PatternInput = RegExp | string; + +export function formatAuthoringPath(path: AuthoringPath): string { + if (typeof path === 'string') { + return path.length === 0 ? '' : path; + } + + return path.length === 0 + ? '' + : path.map((segment) => String(segment)).join('.'); +} + +export function toPatternSource(pattern: PatternInput): string { + return pattern instanceof RegExp + ? `/${pattern.source}/${pattern.flags}` + : pattern; +} + +export function createCaseError( + lane: EvalLane, + caseId: string, + path: AuthoringPath, + message: string, + cause?: unknown, +): Error { + const formattedPath = formatAuthoringPath(path); + const errorMessage = `Invalid ${lane} case "${caseId}" at ${formattedPath}: ${message}`; + return cause === undefined + ? new Error(errorMessage) + : new Error(errorMessage, { cause }); +} + +export function failCase( + lane: EvalLane, + caseId: string, + path: AuthoringPath, + message: string, + cause?: unknown, +): never { + throw createCaseError(lane, caseId, path, message, cause); +} + +export function assertCase( + condition: unknown, + lane: EvalLane, + caseId: string, + path: AuthoringPath, + message: string, +): asserts condition { + if (!condition) { + failCase(lane, caseId, path, message); + } +} + +export function assertDefined( + value: T | undefined, + lane: EvalLane, + caseId: string, + path: AuthoringPath, + message: string, +): T { + if (value === undefined) { + failCase(lane, caseId, path, message); + } + + return value; +} + +export function assertNonEmptyArray( + values: readonly T[], + lane: EvalLane, + caseId: string, + path: AuthoringPath, + message: string, +): readonly T[] { + if (values.length === 0) { + failCase(lane, caseId, path, message); + } + + return values; +} + +export function assertUniqueId( + seen: Set, + id: string, + lane: EvalLane, + caseId: string, + path: AuthoringPath, + label: string, +): void { + assertCase(!seen.has(id), lane, caseId, path, `Duplicate ${label} "${id}"`); + seen.add(id); +} + +export function cloneValue( + value: T, + lane: EvalLane, + caseId: string, + path: AuthoringPath, +): T { + try { + return structuredClone(value); + } catch (error: unknown) { + failCase( + lane, + caseId, + path, + 'Builder state could not be cloned into a fresh runtime object', + error, + ); + } +} + +function summarizeIssues( + issues: readonly { + path: readonly PropertyKey[]; + message: string; + }[], +): string { + return issues + .map((issue) => `${formatAuthoringPath(issue.path)}: ${issue.message}`) + .join('; '); +} + +export function compileAndValidate( + lane: EvalLane, + caseId: string, + schema: ZodType, + compiled: T, +): T { + const parsed = schema.safeParse(compiled); + if (parsed.success) { + return compiled; + } + + const [firstIssue, ...restIssues] = parsed.error.issues; + invariant( + firstIssue !== undefined, + `safeParse() failed for ${lane} case ${caseId} without issues`, + ); + + const restMessage = + restIssues.length === 0 ? '' : `; ${summarizeIssues(restIssues)}`; + throw createCaseError( + lane, + caseId, + firstIssue.path, + `${firstIssue.message}${restMessage}`, + parsed.error, + ); +} diff --git a/evals/authoring/dogfood.ts b/evals/authoring/dogfood.ts new file mode 100644 index 00000000..8d2bfe80 --- /dev/null +++ b/evals/authoring/dogfood.ts @@ -0,0 +1,442 @@ +import type { ArtifactKind } from '../../src/tools/review-bundle.js'; +import { DEFAULT_ANTI_PATTERN_RULES } from '../lib/antiPatterns.js'; +import { SKILL_CONDITIONS } from '../lib/matrix.js'; +import { DogfoodEvalCaseSchema } from '../lib/schemas.js'; +import type { + AntiPatternRule, + ArtifactRequirement, + DogfoodEvalCase, + ReportRequirement, + SkillCondition, + VerifierKind, + VerifierSpec, + WorkflowCheck, +} from '../lib/types.js'; +import { dogfoodTaskPrompt } from '../dogfood/cases/shared.js'; +import { + artifactRequirement, + requiredVerifier, +} from '../execution/cases/shared.js'; +import { + assertCase, + assertDefined, + assertUniqueId, + cloneValue, + compileAndValidate, + toPatternSource, + type PatternInput, +} from './compile.js'; +import { ReportBuilder } from './report.js'; +import { WorkflowBuilder } from './workflow.js'; + +const DEFAULT_DOGFOOD_BUDGETS: DogfoodEvalCase['budgets'] = { + timeoutMs: 600_000, + maxAgentSteps: 30, + maxWallClockMs: 600_000, +}; +const SCREENSHOT_BUNDLE_PATH_PATTERN = String.raw`\.png$`; +const RECORDING_BUNDLE_PATH_PATTERN = String.raw`\.cast$`; +const NOTES_BUNDLE_PATH_PATTERN = String.raw`(?:^|/)(?:README|NOTES|index|notes)\.md$`; + +function cloneAntiPatternRule(rule: AntiPatternRule): AntiPatternRule { + return cloneValue(rule, 'dogfood', 'defaults', 'antiPatterns'); +} + +export class DogfoodProofBundleBuilder { + private readonly addRequirement: (requirement: ArtifactRequirement) => void; + + constructor(addRequirement: (requirement: ArtifactRequirement) => void) { + this.addRequirement = addRequirement; + } + + requiresScreenshot( + description = 'Capture at least one screenshot of a noteworthy state.', + pathPattern: PatternInput = SCREENSHOT_BUNDLE_PATH_PATTERN, + minCount = 1, + ): this { + this.addRequirement( + artifactRequirement( + 'screenshot', + description, + toPatternSource(pathPattern), + minCount, + ), + ); + return this; + } + + requiresRecording( + description = 'Capture at least one terminal recording artifact.', + pathPattern: PatternInput = RECORDING_BUNDLE_PATH_PATTERN, + minCount = 1, + ): this { + this.addRequirement( + artifactRequirement( + 'recording', + description, + toPatternSource(pathPattern), + minCount, + ), + ); + return this; + } + + requiresNotes( + description = 'Write exploratory QA notes in a markdown report.', + pathPattern: PatternInput = NOTES_BUNDLE_PATH_PATTERN, + minCount = 1, + ): this { + this.addRequirement( + artifactRequirement( + 'notes', + description, + toPatternSource(pathPattern), + minCount, + ), + ); + return this; + } + + raw(requirement: ArtifactRequirement): this { + this.addRequirement(requirement); + return this; + } + + rawArtifactRequirement(requirement: ArtifactRequirement): this { + return this.raw(requirement); + } +} + +export class DogfoodCaseBuilder { + private readonly id: string; + private categoryValue?: DogfoodEvalCase['category']; + private taskValue?: string; + private fixtureValue?: string; + private targetValue?: string; + private workspaceValue?: string; + private bundlePathValue?: string; + private readonly bundleRequirementsValue: string[] = []; + private conditionsValue: SkillCondition[] = [...SKILL_CONDITIONS]; + private validationProfileValue?: DogfoodEvalCase['validationProfile']; + private readonly artifactRequirementsValue: ArtifactRequirement[] = []; + private readonly reportBuilder: ReportBuilder; + private readonly verifiersValue: VerifierSpec[] = []; + private readonly verifierIds = new Set(); + private readonly workflowBuilder: WorkflowBuilder; + private antiPatternRulesValue: AntiPatternRule[] | undefined; + private budgetOverridesValue: Partial = {}; + private referenceStepsValue?: number; + private workflowUsed = false; + + constructor(id: string) { + this.id = id; + this.reportBuilder = new ReportBuilder(id); + this.workflowBuilder = new WorkflowBuilder({ + lane: 'dogfood', + caseId: id, + defaultRequired: false, + }); + } + + category(category: DogfoodEvalCase['category']): this { + this.categoryValue = category; + return this; + } + + fixture(fixture: string): this { + this.fixtureValue = fixture; + return this; + } + + target(target: string): this { + this.targetValue = target; + return this; + } + + task(task: string): this { + this.taskValue = task; + return this; + } + + bundlePath(bundlePath: string): this { + this.bundlePathValue = bundlePath; + return this; + } + + bundleRequirement(...requirements: string[]): this { + this.bundleRequirementsValue.push(...requirements); + return this; + } + + bundleRequirements(requirements: readonly string[]): this { + this.bundleRequirementsValue.length = 0; + return this.bundleRequirement(...requirements); + } + + conditions(...conditions: SkillCondition[]): this { + this.conditionsValue = [...conditions]; + return this; + } + + proofBundle(callback: (bundle: DogfoodProofBundleBuilder) => unknown): this { + callback( + new DogfoodProofBundleBuilder((requirement) => + this.addArtifactRequirement(requirement), + ), + ); + return this; + } + + artifact( + kind: ArtifactKind, + description: string, + pathPattern: PatternInput, + minCount = 1, + ): this { + this.addArtifactRequirement( + artifactRequirement( + kind, + description, + toPatternSource(pathPattern), + minCount, + ), + ); + return this; + } + + rawArtifactRequirement(requirement: ArtifactRequirement): this { + this.addArtifactRequirement(requirement); + return this; + } + + report(callback: (report: ReportBuilder) => unknown): this { + callback(this.reportBuilder); + return this; + } + + rawReportRequirement(requirement: ReportRequirement): this { + this.reportBuilder.rawReportRequirement(requirement); + return this; + } + + validationProfile( + validationProfile: DogfoodEvalCase['validationProfile'], + ): this { + this.validationProfileValue = validationProfile; + return this; + } + + verifier( + id: string, + kind: VerifierKind, + description: string, + config: Record, + ): this { + this.addVerifier(requiredVerifier(id, kind, description, config)); + return this; + } + + bundleVerifier( + id = 'bundle-valid', + description = 'Validate the proof bundle with the selected validation profile.', + config: Record = {}, + ): this { + const profile = config.profile ?? this.validationProfileValue; + assertCase( + profile !== undefined, + 'dogfood', + this.id, + 'validationProfile', + 'validationProfile must be set before bundleVerifier() unless config.profile is provided', + ); + this.addVerifier( + requiredVerifier(id, 'bundle', description, { + ...config, + profile, + }), + ); + return this; + } + + rawVerifier(verifier: VerifierSpec): this { + this.addVerifier(verifier); + return this; + } + + workflow(callback: (workflow: WorkflowBuilder) => unknown): this { + this.workflowUsed = true; + callback(this.workflowBuilder); + return this; + } + + rawWorkflowCheck(check: WorkflowCheck): this { + this.workflowBuilder.rawWorkflowCheck(check); + return this; + } + + antiPatterns(...rules: AntiPatternRule[]): this { + this.antiPatternRulesValue = cloneValue( + rules, + 'dogfood', + this.id, + 'antiPatterns', + ); + return this; + } + + budget(overrides: Partial): this { + this.budgetOverridesValue = { + ...this.budgetOverridesValue, + ...overrides, + }; + return this; + } + + referenceSteps(referenceSteps: number): this { + this.referenceStepsValue = referenceSteps; + return this; + } + + workspace(workspace: string): this { + this.workspaceValue = workspace; + return this; + } + + build(): DogfoodEvalCase { + const category = assertDefined( + this.categoryValue, + 'dogfood', + this.id, + 'category', + 'category is required', + ); + const task = assertDefined( + this.taskValue, + 'dogfood', + this.id, + 'prompt', + 'task is required', + ); + const bundlePath = assertDefined( + this.bundlePathValue, + 'dogfood', + this.id, + 'bundlePath', + 'bundlePath is required', + ); + const validationProfile = assertDefined( + this.validationProfileValue, + 'dogfood', + this.id, + 'validationProfile', + 'validationProfile is required', + ); + assertCase( + this.bundleRequirementsValue.length > 0, + 'dogfood', + this.id, + 'bundleRequirements', + 'bundleRequirements must include at least one requirement', + ); + assertCase( + this.conditionsValue.length > 0, + 'dogfood', + this.id, + 'conditions', + 'conditions must include at least one skill condition', + ); + + const workflowChecks = this.workflowBuilder.build(); + assertCase( + !this.workflowUsed || workflowChecks.length > 0, + 'dogfood', + this.id, + 'workflowChecks', + 'workflow() must add at least one workflow check', + ); + + const antiPatterns = + this.antiPatternRulesValue === undefined + ? DEFAULT_ANTI_PATTERN_RULES.map((rule) => cloneAntiPatternRule(rule)) + : cloneValue( + this.antiPatternRulesValue, + 'dogfood', + this.id, + 'antiPatterns', + ); + + const compiled: DogfoodEvalCase = { + id: this.id, + lane: 'dogfood', + category, + prompt: dogfoodTaskPrompt(task, this.fixtureValue), + expectedSkill: 'dogfood-tui', + bundlePath, + bundleRequirements: [...this.bundleRequirementsValue], + conditions: [...this.conditionsValue], + validationProfile, + artifactRequirements: cloneValue( + this.artifactRequirementsValue, + 'dogfood', + this.id, + 'artifactRequirements', + ), + reportRequirements: this.reportBuilder.build(), + verifiers: cloneValue( + this.verifiersValue, + 'dogfood', + this.id, + 'verifiers', + ), + workflowChecks, + antiPatterns, + budgets: { + ...DEFAULT_DOGFOOD_BUDGETS, + ...this.budgetOverridesValue, + }, + }; + if (this.fixtureValue !== undefined) { + compiled.fixture = this.fixtureValue; + } + if (this.targetValue !== undefined) { + compiled.target = this.targetValue; + } + if (this.referenceStepsValue !== undefined) { + compiled.referenceSteps = this.referenceStepsValue; + } + if (this.workspaceValue !== undefined) { + compiled.workspace = this.workspaceValue; + } + + return compileAndValidate( + 'dogfood', + this.id, + DogfoodEvalCaseSchema, + compiled, + ); + } + + private addArtifactRequirement(requirement: ArtifactRequirement): void { + this.artifactRequirementsValue.push( + cloneValue(requirement, 'dogfood', this.id, 'artifactRequirements'), + ); + } + + private addVerifier(verifier: VerifierSpec): void { + assertUniqueId( + this.verifierIds, + verifier.id, + 'dogfood', + this.id, + 'verifiers', + 'verifier id', + ); + this.verifiersValue.push( + cloneValue(verifier, 'dogfood', this.id, `verifiers.${verifier.id}`), + ); + } +} + +export function dogfoodCase(id: string): DogfoodCaseBuilder { + return new DogfoodCaseBuilder(id); +} diff --git a/evals/authoring/execution.ts b/evals/authoring/execution.ts new file mode 100644 index 00000000..e4bede15 --- /dev/null +++ b/evals/authoring/execution.ts @@ -0,0 +1,609 @@ +import type { ArtifactKind } from '../../src/tools/review-bundle.js'; +import { + ALL_EXECUTION_CONDITIONS, + CREATE_SESSION_PATTERN, + DESTROY_SESSION_PATTERN, + RUN_PATTERN, + SCREENSHOT_PATTERN, + SNAPSHOT_PATTERN, + TYPE_PATTERN, + WAIT_PATTERN, + anyOf, + artifactRequirement, + customVerifier, + executionAntiPatterns, + executionBudgets, + executionTaskPrompt, + fixtureSetupStep, + requiredVerifier, + workflowCheck, +} from '../execution/cases/shared.js'; +import { ExecutionEvalCaseSchema } from '../lib/schemas.js'; +import type { + AntiPatternRule, + ArtifactRequirement, + ExecutionEvalCase, + SkillCondition, + VerifierKind, + VerifierSpec, + WorkflowCheck, +} from '../lib/types.js'; +import { + assertCase, + assertDefined, + assertUniqueId, + cloneValue, + compileAndValidate, + toPatternSource, + type PatternInput, +} from './compile.js'; +import { WorkflowBuilder } from './workflow.js'; + +function escapeRegexLiteral(value: string): string { + return value.replace(/[.*+?^${}()|[\]\\]/g, String.raw`\\$&`); +} + +function describePatternInput(value: PatternInput): string { + return typeof value === 'string' + ? JSON.stringify(value) + : 'the expected pattern'; +} + +interface FixtureOptions { + setupId?: string; + setupDescription?: string; + timeoutMs?: number; +} + +interface ExecutionWorkflowStepOptions { + id?: string; + description?: string; + dependsOn?: string[]; + required?: boolean; + weight?: number; + forbiddenPattern?: PatternInput; + pattern?: PatternInput; +} + +type ExecutionActionStepOptions = ExecutionWorkflowStepOptions; + +interface ExecutionFixtureDraft { + fixture: string; + setupId: string; + setupDescription: string; + timeoutMs?: number; +} + +function defaultInputPattern(text: string): string { + const escapedText = escapeRegexLiteral(text); + return anyOf( + String.raw`\bagent-tty\b[^\n]*\b(?:run|type)\b[^\n]*${escapedText}`, + String.raw`\b(?:run|type)(?:ning|s|ned)?\b[^\n]*${escapedText}\b`, + ); +} + +function defaultRunPattern(command: string): string { + const escapedCommand = escapeRegexLiteral(command); + return anyOf( + String.raw`\bagent-tty\b[^\n]*\brun\b[^\n]*${escapedCommand}`, + String.raw`\brun(?:ning|s|ned)?\b[^\n]*${escapedCommand}\b`, + ); +} + +function resolveCommandPattern( + value: PatternInput, + builder: (text: string) => string, + override?: PatternInput, +): string { + if (override !== undefined) { + return toPatternSource(override); + } + + if (value instanceof RegExp) { + return toPatternSource(value); + } + + return builder(value); +} + +function resolveDependsOn( + previousStepId: string | undefined, + dependsOn: readonly string[] | undefined, +): string[] | undefined { + if (dependsOn !== undefined) { + return [...dependsOn]; + } + + return previousStepId === undefined ? undefined : [previousStepId]; +} + +export class ExecutionWorkflowBuilder { + private readonly workflowBuilder: WorkflowBuilder; + private lastStepId: string | undefined; + + constructor(workflowBuilder: WorkflowBuilder) { + this.workflowBuilder = workflowBuilder; + } + + createSession(options: ExecutionWorkflowStepOptions = {}): this { + return this.appendCheck( + options.id ?? 'create', + options.description ?? 'Create the fixture session.', + options.pattern === undefined + ? CREATE_SESSION_PATTERN + : toPatternSource(options.pattern), + options, + ); + } + + input(value: PatternInput, options: ExecutionActionStepOptions = {}): this { + return this.appendCheck( + options.id ?? 'input', + options.description ?? + `Send ${describePatternInput(value)} with run or type.`, + resolveCommandPattern(value, defaultInputPattern, options.pattern), + options, + ); + } + + run(value: PatternInput, options: ExecutionActionStepOptions = {}): this { + return this.appendCheck( + options.id ?? 'run', + options.description ?? `Run ${describePatternInput(value)}.`, + resolveCommandPattern(value, defaultRunPattern, options.pattern), + options, + ); + } + + waitFor( + value?: PatternInput, + options: ExecutionWorkflowStepOptions = {}, + ): this { + const requiredPattern = + options.pattern !== undefined + ? toPatternSource(options.pattern) + : value === undefined + ? WAIT_PATTERN + : anyOf(WAIT_PATTERN, toPatternSource(value)); + return this.appendCheck( + options.id ?? 'wait', + options.description ?? 'Wait for the expected output before continuing.', + requiredPattern, + options, + ); + } + + snapshot(options: ExecutionWorkflowStepOptions = {}): this { + return this.appendCheck( + options.id ?? 'snapshot', + options.description ?? 'Capture a snapshot for verification.', + options.pattern === undefined + ? SNAPSHOT_PATTERN + : toPatternSource(options.pattern), + options, + ); + } + + screenshot(options: ExecutionWorkflowStepOptions = {}): this { + return this.appendCheck( + options.id ?? 'screenshot', + options.description ?? 'Capture a screenshot for verification.', + options.pattern === undefined + ? SCREENSHOT_PATTERN + : toPatternSource(options.pattern), + options, + ); + } + + destroy(options: ExecutionWorkflowStepOptions = {}): this { + return this.appendCheck( + options.id ?? 'destroy', + options.description ?? 'Destroy the session after verification.', + options.pattern === undefined + ? DESTROY_SESSION_PATTERN + : toPatternSource(options.pattern), + options, + ); + } + + raw(check: WorkflowCheck): this { + this.workflowBuilder.raw(check); + this.lastStepId = check.id; + return this; + } + + rawWorkflowCheck(check: WorkflowCheck): this { + return this.raw(check); + } + + private appendCheck( + id: string, + description: string, + requiredPattern: string, + options: ExecutionWorkflowStepOptions, + ): this { + const dependsOn = resolveDependsOn(this.lastStepId, options.dependsOn); + this.workflowBuilder.raw( + workflowCheck(id, description, requiredPattern, { + ...(dependsOn === undefined ? {} : { dependsOn }), + ...(options.required === undefined + ? {} + : { required: options.required }), + ...(options.weight === undefined ? {} : { weight: options.weight }), + ...(options.forbiddenPattern === undefined + ? {} + : { forbiddenPattern: toPatternSource(options.forbiddenPattern) }), + }), + ); + this.lastStepId = id; + return this; + } +} + +export class ExecutionAssertionBuilder { + private readonly addVerifier: (verifier: VerifierSpec) => void; + + constructor(addVerifier: (verifier: VerifierSpec) => void) { + this.addVerifier = addVerifier; + } + + verifier( + id: string, + kind: VerifierKind, + description: string, + config: Record, + ): this { + this.addVerifier(requiredVerifier(id, kind, description, config)); + return this; + } + + snapshot( + id: string, + description: string, + config: Record, + ): this { + return this.verifier(id, 'snapshot', description, config); + } + + screenshot( + id: string, + description: string, + config: Record, + ): this { + return this.verifier(id, 'screenshot', description, config); + } + + eventLog( + id: string, + description: string, + config: Record, + ): this { + return this.verifier(id, 'event-log', description, config); + } + + json(id: string, description: string, config: Record): this { + return this.verifier(id, 'json', description, config); + } + + command( + id: string, + description: string, + config: Record, + ): this { + return this.verifier(id, 'command', description, config); + } + + custom( + id: string, + description: string, + validator: string, + config: Record, + ): this { + this.addVerifier(customVerifier(id, description, validator, config)); + return this; + } + + snapshotContains(...patterns: PatternInput[]): this { + return this.snapshot( + 'snapshot-contains', + 'The snapshot should contain the required content patterns.', + { + patterns: patterns.map((pattern) => toPatternSource(pattern)), + }, + ); + } + + raw(verifier: VerifierSpec): this { + this.addVerifier(verifier); + return this; + } + + rawVerifier(verifier: VerifierSpec): this { + return this.raw(verifier); + } +} + +export class ExecutionCaseBuilder { + private readonly id: string; + private categoryValue?: ExecutionEvalCase['category']; + private taskValue?: string; + private fixtureValue?: ExecutionFixtureDraft; + private targetValue?: string; + private workspaceValue?: string; + private conditionsValue: SkillCondition[] = [...ALL_EXECUTION_CONDITIONS]; + private referenceStepsValue?: number; + private readonly workflowBuilder: WorkflowBuilder; + private readonly executionWorkflowBuilder: ExecutionWorkflowBuilder; + private readonly verifiersValue: VerifierSpec[] = []; + private readonly verifierIds = new Set(); + private readonly artifactRequirementsValue: ArtifactRequirement[] = []; + private antiPatternExtraRulesValue: AntiPatternRule[] = []; + private budgetOverridesValue: Partial = {}; + private workflowUsed = false; + + constructor(id: string) { + this.id = id; + this.workflowBuilder = new WorkflowBuilder({ + lane: 'execution', + caseId: id, + defaultRequired: false, + }); + this.executionWorkflowBuilder = new ExecutionWorkflowBuilder( + this.workflowBuilder, + ); + } + + category(category: ExecutionEvalCase['category']): this { + this.categoryValue = category; + return this; + } + + fixture(fixture: string, options: FixtureOptions = {}): this { + this.fixtureValue = { + fixture, + setupId: options.setupId ?? `launch-${fixture}`, + setupDescription: + options.setupDescription ?? + `Create an agent-tty session that runs the ${fixture} fixture.`, + ...(options.timeoutMs === undefined + ? {} + : { timeoutMs: options.timeoutMs }), + }; + return this; + } + + target(target: string): this { + this.targetValue = target; + return this; + } + + task(task: string): this { + this.taskValue = task; + return this; + } + + conditions(...conditions: SkillCondition[]): this { + this.conditionsValue = [...conditions]; + return this; + } + + referenceSteps(referenceSteps: number): this { + this.referenceStepsValue = referenceSteps; + return this; + } + + workspace(workspace: string): this { + this.workspaceValue = workspace; + return this; + } + + workflow(callback: (workflow: ExecutionWorkflowBuilder) => unknown): this { + this.workflowUsed = true; + callback(this.executionWorkflowBuilder); + return this; + } + + rawWorkflowCheck(check: WorkflowCheck): this { + this.executionWorkflowBuilder.rawWorkflowCheck(check); + return this; + } + + assertions( + callback: (assertions: ExecutionAssertionBuilder) => unknown, + ): this { + callback( + new ExecutionAssertionBuilder((verifier) => this.addVerifier(verifier)), + ); + return this; + } + + verifier( + id: string, + kind: VerifierKind, + description: string, + config: Record, + ): this { + this.addVerifier(requiredVerifier(id, kind, description, config)); + return this; + } + + rawVerifier(verifier: VerifierSpec): this { + this.addVerifier(verifier); + return this; + } + + artifact( + kind: ArtifactKind, + description: string, + pathPattern: PatternInput, + minCount = 1, + ): this { + this.artifactRequirementsValue.push( + artifactRequirement( + kind, + description, + toPatternSource(pathPattern), + minCount, + ), + ); + return this; + } + + rawArtifactRequirement(requirement: ArtifactRequirement): this { + this.artifactRequirementsValue.push( + cloneValue(requirement, 'execution', this.id, 'artifactRequirements'), + ); + return this; + } + + antiPatterns(...extraRules: AntiPatternRule[]): this { + this.antiPatternExtraRulesValue = cloneValue( + extraRules, + 'execution', + this.id, + 'antiPatterns', + ); + return this; + } + + budget(overrides: Partial): this { + this.budgetOverridesValue = { + ...this.budgetOverridesValue, + ...overrides, + }; + return this; + } + + build(): ExecutionEvalCase { + const category = assertDefined( + this.categoryValue, + 'execution', + this.id, + 'category', + 'category is required', + ); + const task = assertDefined( + this.taskValue, + 'execution', + this.id, + 'prompt', + 'task is required', + ); + assertCase( + this.fixtureValue !== undefined || this.targetValue !== undefined, + 'execution', + this.id, + 'fixture', + 'fixture or target is required', + ); + assertCase( + this.conditionsValue.length > 0, + 'execution', + this.id, + 'conditions', + 'conditions must include at least one skill condition', + ); + assertCase( + this.verifiersValue.length > 0, + 'execution', + this.id, + 'verifiers', + 'verifiers must include at least one verifier', + ); + + const workflowChecks = this.workflowBuilder.build(); + assertCase( + !this.workflowUsed || workflowChecks.length > 0, + 'execution', + this.id, + 'workflowChecks', + 'workflow() must add at least one workflow check', + ); + + const setup = + this.fixtureValue === undefined + ? [] + : [ + fixtureSetupStep( + this.fixtureValue.setupId, + this.fixtureValue.fixture, + this.fixtureValue.setupDescription, + this.fixtureValue.timeoutMs, + ), + ]; + + const compiled: ExecutionEvalCase = { + id: this.id, + lane: 'execution', + category, + prompt: executionTaskPrompt(task, this.fixtureValue?.fixture), + expectedSkill: 'agent-tty', + conditions: [...this.conditionsValue], + setup, + verifiers: cloneValue( + this.verifiersValue, + 'execution', + this.id, + 'verifiers', + ), + workflowChecks, + antiPatterns: executionAntiPatterns(...this.antiPatternExtraRulesValue), + artifactRequirements: cloneValue( + this.artifactRequirementsValue, + 'execution', + this.id, + 'artifactRequirements', + ), + budgets: executionBudgets(this.budgetOverridesValue), + }; + if (this.fixtureValue !== undefined) { + compiled.fixture = this.fixtureValue.fixture; + } + if (this.targetValue !== undefined) { + compiled.target = this.targetValue; + } + if (this.referenceStepsValue !== undefined) { + compiled.referenceSteps = this.referenceStepsValue; + } + if (this.workspaceValue !== undefined) { + compiled.workspace = this.workspaceValue; + } + + return compileAndValidate( + 'execution', + this.id, + ExecutionEvalCaseSchema, + compiled, + ); + } + + private addVerifier(verifier: VerifierSpec): void { + assertUniqueId( + this.verifierIds, + verifier.id, + 'execution', + this.id, + 'verifiers', + 'verifier id', + ); + this.verifiersValue.push( + cloneValue(verifier, 'execution', this.id, `verifiers.${verifier.id}`), + ); + } +} + +export function executionCase(id: string): ExecutionCaseBuilder { + return new ExecutionCaseBuilder(id); +} + +export { + ALL_EXECUTION_CONDITIONS, + CREATE_SESSION_PATTERN, + DESTROY_SESSION_PATTERN, + RUN_PATTERN, + SCREENSHOT_PATTERN, + SNAPSHOT_PATTERN, + TYPE_PATTERN, + WAIT_PATTERN, + anyOf, +}; diff --git a/evals/authoring/index.ts b/evals/authoring/index.ts new file mode 100644 index 00000000..522a0b0d --- /dev/null +++ b/evals/authoring/index.ts @@ -0,0 +1,20 @@ +export { + dogfoodCase, + DogfoodCaseBuilder, + DogfoodProofBundleBuilder, +} from './dogfood.js'; +export { + executionCase, + ExecutionAssertionBuilder, + ExecutionCaseBuilder, + ExecutionWorkflowBuilder, +} from './execution.js'; +export { promptCase, PromptCaseBuilder } from './prompt.js'; +export { ReportBuilder } from './report.js'; +export { + rawArtifactRequirement, + rawReportRequirement, + rawVerifier, + rawWorkflowCheck, +} from './raw.js'; +export { WorkflowBuilder, WorkflowStepBuilder } from './workflow.js'; diff --git a/evals/authoring/prompt.ts b/evals/authoring/prompt.ts new file mode 100644 index 00000000..655c098b --- /dev/null +++ b/evals/authoring/prompt.ts @@ -0,0 +1,202 @@ +import { PromptEvalCaseSchema } from '../lib/schemas.js'; +import type { + AntiPatternRule, + ExpectedSkill, + PromptEvalCase, + WorkflowCheck, +} from '../lib/types.js'; +import { + assertCase, + assertDefined, + assertNonEmptyArray, + cloneValue, + compileAndValidate, + toPatternSource, + type PatternInput, +} from './compile.js'; +import { WorkflowBuilder } from './workflow.js'; + +const DEFAULT_PROMPT_TIMEOUT_MS = 30_000; + +export class PromptCaseBuilder { + private readonly id: string; + private categoryValue?: PromptEvalCase['category']; + private promptValue?: string; + private expectedSkillValue?: ExpectedSkill; + private contextValue?: string; + private readonly expectedPatternsValue: string[] = []; + private readonly forbiddenPatternsValue: string[] = []; + private readonly rubricValue: string[] = []; + private antiPatternRulesValue: AntiPatternRule[] = []; + private budgetValue: PromptEvalCase['budgets'] = { + timeoutMs: DEFAULT_PROMPT_TIMEOUT_MS, + }; + private readonly workflowBuilder: WorkflowBuilder; + private workflowUsed = false; + + constructor(id: string) { + this.id = id; + this.workflowBuilder = new WorkflowBuilder({ + lane: 'prompt', + caseId: id, + defaultRequired: true, + }); + } + + category(category: PromptEvalCase['category']): this { + this.categoryValue = category; + return this; + } + + prompt(prompt: string): this { + this.promptValue = prompt; + return this; + } + + expectSkill(expectedSkill: ExpectedSkill): this { + this.expectedSkillValue = expectedSkill; + return this; + } + + context(context: string): this { + this.contextValue = context; + return this; + } + + expectedPattern(...patterns: PatternInput[]): this { + this.expectedPatternsValue.push( + ...patterns.map((pattern) => toPatternSource(pattern)), + ); + return this; + } + + expectedPatterns(patterns: readonly PatternInput[]): this { + this.expectedPatternsValue.length = 0; + return this.expectedPattern(...patterns); + } + + mustMention(...patterns: PatternInput[]): this { + return this.expectedPattern(...patterns); + } + + forbiddenPattern(...patterns: PatternInput[]): this { + this.forbiddenPatternsValue.push( + ...patterns.map((pattern) => toPatternSource(pattern)), + ); + return this; + } + + forbiddenPatterns(patterns: readonly PatternInput[]): this { + this.forbiddenPatternsValue.length = 0; + return this.forbiddenPattern(...patterns); + } + + mustNotMention(...patterns: PatternInput[]): this { + return this.forbiddenPattern(...patterns); + } + + rubric(...items: string[]): this { + this.rubricValue.push(...items); + return this; + } + + workflow(callback: (workflow: WorkflowBuilder) => unknown): this { + this.workflowUsed = true; + callback(this.workflowBuilder); + return this; + } + + rawWorkflowCheck(check: WorkflowCheck): this { + this.workflowBuilder.rawWorkflowCheck(check); + return this; + } + + antiPatterns(...rules: AntiPatternRule[]): this { + this.antiPatternRulesValue = cloneValue( + rules, + 'prompt', + this.id, + 'antiPatterns', + ); + return this; + } + + budget(budget: number | PromptEvalCase['budgets']): this { + this.budgetValue = + typeof budget === 'number' ? { timeoutMs: budget } : { ...budget }; + return this; + } + + build(): PromptEvalCase { + const category = assertDefined( + this.categoryValue, + 'prompt', + this.id, + 'category', + 'category is required', + ); + const prompt = assertDefined( + this.promptValue, + 'prompt', + this.id, + 'prompt', + 'prompt is required', + ); + const expectedSkill = assertDefined( + this.expectedSkillValue, + 'prompt', + this.id, + 'expectedSkill', + 'expectedSkill is required', + ); + assertNonEmptyArray( + this.expectedPatternsValue, + 'prompt', + this.id, + 'expectedPatterns', + 'expectedPatterns must include at least one pattern', + ); + + const workflowChecks = this.workflowBuilder.build(); + assertCase( + !this.workflowUsed || workflowChecks.length > 0, + 'prompt', + this.id, + 'workflowChecks', + 'workflow() must add at least one workflow check', + ); + + const compiled: PromptEvalCase = { + id: this.id, + lane: 'prompt', + category, + prompt, + expectedSkill, + expectedPatterns: [...this.expectedPatternsValue], + forbiddenPatterns: [...this.forbiddenPatternsValue], + rubric: [...this.rubricValue], + workflowChecks, + antiPatterns: cloneValue( + this.antiPatternRulesValue, + 'prompt', + this.id, + 'antiPatterns', + ), + budgets: { ...this.budgetValue }, + }; + if (this.contextValue !== undefined) { + compiled.context = this.contextValue; + } + + return compileAndValidate( + 'prompt', + this.id, + PromptEvalCaseSchema, + compiled, + ); + } +} + +export function promptCase(id: string): PromptCaseBuilder { + return new PromptCaseBuilder(id); +} diff --git a/evals/authoring/raw.ts b/evals/authoring/raw.ts new file mode 100644 index 00000000..def938e8 --- /dev/null +++ b/evals/authoring/raw.ts @@ -0,0 +1,26 @@ +import type { + ArtifactRequirement, + ReportRequirement, + VerifierSpec, + WorkflowCheck, +} from '../lib/types.js'; + +export function rawWorkflowCheck(check: WorkflowCheck): WorkflowCheck { + return check; +} + +export function rawVerifier(verifier: VerifierSpec): VerifierSpec { + return verifier; +} + +export function rawArtifactRequirement( + requirement: ArtifactRequirement, +): ArtifactRequirement { + return requirement; +} + +export function rawReportRequirement( + requirement: ReportRequirement, +): ReportRequirement { + return requirement; +} diff --git a/evals/authoring/report.ts b/evals/authoring/report.ts new file mode 100644 index 00000000..c51b818a --- /dev/null +++ b/evals/authoring/report.ts @@ -0,0 +1,163 @@ +import type { ReportRequirement } from '../lib/types.js'; +import { + assertUniqueId, + cloneValue, + toPatternSource, + type PatternInput, +} from './compile.js'; + +interface ReportRequirementDraft { + id: string; + description: string; + required: boolean; + section?: string; + requiredPatterns: string[]; + forbiddenPatterns: string[]; +} + +type ReportEntry = + | { + kind: 'draft'; + requirement: ReportRequirementDraft; + } + | { + kind: 'raw'; + requirement: ReportRequirement; + }; + +const TITLE_PATTERN = String.raw`/(?:^|\n)\s*(?:#{1,3}\s*Title\b|\*\*Title:?\*\*)/im`; +const REPRODUCTION_SECTION_PATTERN = String.raw`/(?:^|\n)\s*(?:#{1,3}\s*(?:Reproduction steps|Repro(?:duction)? steps|Steps)\b|\*\*(?:Reproduction steps|Repro(?:duction)? steps|Steps):?\*\*)/im`; +const FINDINGS_SECTION_PATTERN = String.raw`/(?:^|\n)\s*(?:#{1,3}\s*(?:Findings|Issues)\b|\*\*(?:Findings|Issues):?\*\*)/im`; +const EVIDENCE_SECTION_PATTERN = String.raw`/(?:^|\n)\s*(?:#{1,3}\s*Evidence\b|\*\*Evidence:?\*\*)/im`; +const CLI_REFERENCE_PATTERN = String.raw`/\b(?:agent-tty|npx\s+tsx\s+src\/cli\/main\.ts)\b/i`; +const SEVERITY_PATTERN = String.raw`/\b(?:severity|critical|high|medium|low|info)\b/i`; +const EVIDENCE_REFERENCE_PATTERN = String.raw`/\.(?:png|cast|webm|json|md)\b/i`; + +export class ReportBuilder { + private readonly caseId: string; + private readonly path: string; + private readonly entries: ReportEntry[] = []; + private readonly knownIds = new Set(); + + constructor(caseId: string, path = 'reportRequirements') { + this.caseId = caseId; + this.path = path; + } + + section( + id: string, + section: string | undefined, + description: string, + requiredPatterns: readonly PatternInput[], + forbiddenPatterns: readonly PatternInput[] = [], + ): this { + assertUniqueId( + this.knownIds, + id, + 'dogfood', + this.caseId, + this.path, + 'report requirement id', + ); + this.entries.push({ + kind: 'draft', + requirement: { + id, + ...(section === undefined ? {} : { section }), + description, + required: true, + requiredPatterns: requiredPatterns.map((pattern) => + toPatternSource(pattern), + ), + forbiddenPatterns: forbiddenPatterns.map((pattern) => + toPatternSource(pattern), + ), + }, + }); + return this; + } + + title(description = 'Report must have a descriptive title.'): this { + return this.section('title', 'Title', description, [TITLE_PATTERN]); + } + + reproductionSteps( + description = 'Include step-by-step reproduction commands.', + ): this { + return this.section('repro-steps', 'Reproduction steps', description, [ + REPRODUCTION_SECTION_PATTERN, + CLI_REFERENCE_PATTERN, + ]); + } + + findingsWithSeverity( + description = 'List findings with severity classification.', + ): this { + return this.section('findings', 'Findings', description, [ + FINDINGS_SECTION_PATTERN, + SEVERITY_PATTERN, + ]); + } + + evidenceReferences( + description = 'Reference captured artifacts such as screenshots and recordings.', + ): this { + return this.section('evidence', 'Evidence', description, [ + EVIDENCE_SECTION_PATTERN, + EVIDENCE_REFERENCE_PATTERN, + ]); + } + + raw(requirement: ReportRequirement): this { + assertUniqueId( + this.knownIds, + requirement.id, + 'dogfood', + this.caseId, + this.path, + 'report requirement id', + ); + this.entries.push({ + kind: 'raw', + requirement: cloneValue( + requirement, + 'dogfood', + this.caseId, + `${this.path}.${requirement.id}`, + ), + }); + return this; + } + + rawReportRequirement(requirement: ReportRequirement): this { + return this.raw(requirement); + } + + size(): number { + return this.entries.length; + } + + build(): ReportRequirement[] { + return this.entries.map((entry, index) => { + if (entry.kind === 'raw') { + return cloneValue( + entry.requirement, + 'dogfood', + this.caseId, + `${this.path}.${String(index)}`, + ); + } + + return { + id: entry.requirement.id, + description: entry.requirement.description, + required: entry.requirement.required, + ...(entry.requirement.section === undefined + ? {} + : { section: entry.requirement.section }), + requiredPatterns: [...entry.requirement.requiredPatterns], + forbiddenPatterns: [...entry.requirement.forbiddenPatterns], + }; + }); + } +} diff --git a/evals/authoring/workflow.ts b/evals/authoring/workflow.ts new file mode 100644 index 00000000..c132a5ef --- /dev/null +++ b/evals/authoring/workflow.ts @@ -0,0 +1,246 @@ +import type { EvalLane, WorkflowCheck } from '../lib/types.js'; +import { + assertUniqueId, + cloneValue, + failCase, + toPatternSource, + type PatternInput, +} from './compile.js'; + +interface WorkflowStepDraft { + id: string; + description?: string; + required: boolean; + mustMention: string[]; + mustNotMention: string[]; + dependsOn: string[]; + weight?: number; +} + +type WorkflowEntry = + | { + kind: 'draft'; + draft: WorkflowStepDraft; + } + | { + kind: 'raw'; + check: WorkflowCheck; + }; + +export interface WorkflowBuilderOptions { + lane: EvalLane; + caseId: string; + defaultRequired: boolean; + path?: string; +} + +function findPatternContradiction( + mustMention: readonly string[], + mustNotMention: readonly string[], +): string | undefined { + const forbiddenPatterns = new Set(mustNotMention); + return mustMention.find((pattern) => forbiddenPatterns.has(pattern)); +} + +function appendUniqueValues(target: string[], values: readonly string[]): void { + for (const value of values) { + if (!target.includes(value)) { + target.push(value); + } + } +} + +export class WorkflowBuilder { + private readonly lane: EvalLane; + private readonly caseId: string; + private readonly defaultRequired: boolean; + private readonly path: string; + private readonly entries: WorkflowEntry[] = []; + private readonly knownIds = new Set(); + + constructor(options: WorkflowBuilderOptions) { + this.lane = options.lane; + this.caseId = options.caseId; + this.defaultRequired = options.defaultRequired; + this.path = options.path ?? 'workflowChecks'; + } + + step(id: string, description?: string): WorkflowStepBuilder { + assertUniqueId( + this.knownIds, + id, + this.lane, + this.caseId, + this.path, + 'workflow step id', + ); + + const draft: WorkflowStepDraft = { + id, + ...(description === undefined ? {} : { description }), + required: this.defaultRequired, + mustMention: [], + mustNotMention: [], + dependsOn: [], + }; + this.entries.push({ kind: 'draft', draft }); + return new WorkflowStepBuilder(this, draft); + } + + raw(check: WorkflowCheck): this { + assertUniqueId( + this.knownIds, + check.id, + this.lane, + this.caseId, + this.path, + 'workflow step id', + ); + this.assertNoContradiction( + check.id, + check.requiredPatterns, + check.forbiddenPatterns, + ); + this.entries.push({ + kind: 'raw', + check: cloneValue(check, this.lane, this.caseId, this.path), + }); + return this; + } + + rawWorkflowCheck(check: WorkflowCheck): this { + return this.raw(check); + } + + size(): number { + return this.entries.length; + } + + build(): WorkflowCheck[] { + return this.entries.map((entry, index) => { + if (entry.kind === 'raw') { + return cloneValue( + entry.check, + this.lane, + this.caseId, + `${this.path}.${String(index)}`, + ); + } + + this.assertNoContradiction( + entry.draft.id, + entry.draft.mustMention, + entry.draft.mustNotMention, + ); + return { + id: entry.draft.id, + description: + entry.draft.description ?? `Workflow step ${entry.draft.id}`, + required: entry.draft.required, + requiredPatterns: [...entry.draft.mustMention], + forbiddenPatterns: [...entry.draft.mustNotMention], + dependsOn: [...entry.draft.dependsOn], + ...(entry.draft.weight === undefined + ? {} + : { weight: entry.draft.weight }), + }; + }); + } + + assertNoContradiction( + stepId: string, + mustMention: readonly string[], + mustNotMention: readonly string[], + ): void { + const contradictoryPattern = findPatternContradiction( + mustMention, + mustNotMention, + ); + if (contradictoryPattern === undefined) { + return; + } + + failCase( + this.lane, + this.caseId, + this.path, + `Workflow step "${stepId}" cannot require and forbid the same literal pattern ${JSON.stringify(contradictoryPattern)}`, + ); + } +} + +export class WorkflowStepBuilder { + private readonly parent: WorkflowBuilder; + private readonly draft: WorkflowStepDraft; + + constructor(parent: WorkflowBuilder, draft: WorkflowStepDraft) { + this.parent = parent; + this.draft = draft; + } + + description(description: string): this { + this.draft.description = description; + return this; + } + + mustMention(...patterns: PatternInput[]): this { + appendUniqueValues( + this.draft.mustMention, + patterns.map((pattern) => toPatternSource(pattern)), + ); + this.parent.assertNoContradiction( + this.draft.id, + this.draft.mustMention, + this.draft.mustNotMention, + ); + return this; + } + + mustNotMention(...patterns: PatternInput[]): this { + appendUniqueValues( + this.draft.mustNotMention, + patterns.map((pattern) => toPatternSource(pattern)), + ); + this.parent.assertNoContradiction( + this.draft.id, + this.draft.mustMention, + this.draft.mustNotMention, + ); + return this; + } + + dependsOn(...stepIds: string[]): this { + appendUniqueValues(this.draft.dependsOn, stepIds); + return this; + } + + after(...stepIds: string[]): this { + return this.dependsOn(...stepIds); + } + + required(required = true): this { + this.draft.required = required; + return this; + } + + optional(): this { + return this.required(false); + } + + weight(weight: number): this { + this.draft.weight = weight; + return this; + } + + step(id: string, description?: string): WorkflowStepBuilder { + return this.parent.step(id, description); + } + + raw(check: WorkflowCheck): WorkflowBuilder { + return this.parent.raw(check); + } + + rawWorkflowCheck(check: WorkflowCheck): WorkflowBuilder { + return this.parent.rawWorkflowCheck(check); + } +} diff --git a/evals/dogfood/cases/exploratory-qa.ts b/evals/dogfood/cases/exploratory-qa.ts index 1b365812..31e79948 100644 --- a/evals/dogfood/cases/exploratory-qa.ts +++ b/evals/dogfood/cases/exploratory-qa.ts @@ -1,133 +1,41 @@ -import { DEFAULT_ANTI_PATTERN_RULES } from '../../lib/antiPatterns.js'; +import { dogfoodCase } from '../../authoring/index.js'; import { SKILL_CONDITIONS } from '../../lib/matrix.js'; -import { DogfoodEvalCaseSchema } from '../../lib/schemas.js'; -import { dogfoodTaskPrompt } from './shared.js'; -import type { - ArtifactRequirement, - ReportRequirement, - VerifierSpec, -} from '../../lib/types.js'; -function requiredArtifact( - kind: ArtifactRequirement['kind'], - description: string, - pathPatterns: string[], - minCount = 1, -): ArtifactRequirement { - return { - kind, - required: true, - description, - minCount, - pathPatterns, - }; -} - -function reportRequirement( - id: string, - section: string, - description: string, - requiredPatterns: string[], -): ReportRequirement { - return { - id, - section, - description, - required: true, - requiredPatterns, - forbiddenPatterns: [], - }; -} - -const verifier: VerifierSpec = { - id: 'bundle-valid', - kind: 'bundle', - description: - 'Validate the exploratory QA proof bundle with the interactive renderer profile.', - required: true, - config: { - profile: 'interactive-renderer', - }, -}; - -export const exploratoryQaCase = DogfoodEvalCaseSchema.parse({ - id: 'exploratory-qa', - lane: 'dogfood', - category: 'qa', - prompt: dogfoodTaskPrompt( +export const exploratoryQaCase = dogfoodCase('exploratory-qa') + .category('qa') + .task( 'Launch the hello-prompt fixture, test exactly three inputs (`hello world`, a blank line, and `symbols-!@#$%^&*`), capture a snapshot after each input, then send `exit` to verify clean shutdown. Save at least one screenshot and one recording, and write a brief findings report with severity and evidence references.', - 'hello-prompt', - ), - expectedSkill: 'dogfood-tui', - fixture: 'hello-prompt', - bundlePath: 'proof-bundle', - bundleRequirements: [ + ) + .fixture('hello-prompt') + .bundlePath('proof-bundle') + .bundleRequirements([ 'Produce a reviewable proof bundle for an exploratory QA investigation.', 'Capture renderer-backed evidence for the tested interactions and edge cases.', 'Write structured notes that summarize findings, severity, and evidence references.', - ], - conditions: [...SKILL_CONDITIONS], - validationProfile: 'interactive-renderer', - artifactRequirements: [ - requiredArtifact( - 'screenshot', - 'Capture at least one screenshot of a noteworthy state.', - [String.raw`\.png$`], - ), - requiredArtifact( - 'recording', - 'Capture at least one terminal recording artifact.', - [String.raw`\.cast$`], - ), - requiredArtifact( - 'notes', - 'Write exploratory QA notes in a markdown report.', - [String.raw`(?:^|/)(?:README|NOTES|index|notes)\.md$`], - ), - ], - reportRequirements: [ - reportRequirement( - 'title', - 'Title', - 'Report must have a descriptive title.', - [String.raw`/(?:^|\n)\s*(?:#{1,3}\s*Title\b|\*\*Title:?\*\*)/im`], - ), - reportRequirement( - 'repro-steps', - 'Reproduction steps', - 'Include step-by-step reproduction commands.', - [ - String.raw`/(?:^|\n)\s*(?:#{1,3}\s*(?:Reproduction steps|Repro(?:duction)? steps|Steps)\b|\*\*(?:Reproduction steps|Repro(?:duction)? steps|Steps):?\*\*)/im`, - String.raw`/\b(?:agent-tty|npx\s+tsx\s+src\/cli\/main\.ts)\b/i`, - ], - ), - reportRequirement( - 'findings', - 'Findings', - 'List findings with severity classification.', - [ - String.raw`/(?:^|\n)\s*(?:#{1,3}\s*(?:Findings|Issues)\b|\*\*(?:Findings|Issues):?\*\*)/im`, - String.raw`/\b(?:severity|critical|high|medium|low|info)\b/i`, - ], - ), - reportRequirement( - 'evidence', - 'Evidence', - 'Reference captured artifacts such as screenshots and recordings.', - [ - String.raw`/(?:^|\n)\s*(?:#{1,3}\s*Evidence\b|\*\*Evidence:?\*\*)/im`, - String.raw`/\.(?:png|cast|webm|json|md)\b/i`, - ], - ), - ], - verifiers: [verifier], - workflowChecks: [], - antiPatterns: [...DEFAULT_ANTI_PATTERN_RULES], - budgets: { + ]) + .conditions(...SKILL_CONDITIONS) + .validationProfile('interactive-renderer') + .proofBundle((bundle) => { + bundle.requiresScreenshot(); + bundle.requiresRecording(); + bundle.requiresNotes(); + }) + .report((report) => { + report.title(); + report.reproductionSteps(); + report.findingsWithSeverity(); + report.evidenceReferences(); + }) + .bundleVerifier( + 'bundle-valid', + 'Validate the exploratory QA proof bundle with the interactive renderer profile.', + ) + .budget({ timeoutMs: 600_000, maxAgentSteps: 30, maxWallClockMs: 600_000, - }, -}); + }) + .workspace('agent-tty-smoke') + .build(); export default exploratoryQaCase; diff --git a/evals/dogfood/runner.ts b/evals/dogfood/runner.ts index 1840dab6..c69b9971 100644 --- a/evals/dogfood/runner.ts +++ b/evals/dogfood/runner.ts @@ -4,6 +4,10 @@ import { join, resolve } from 'node:path'; import { scanBundleArtifacts } from '../../src/tools/review-bundle.js'; import { assertString, invariant } from '../../src/util/assert.js'; +import { + EvalArtifactStore, + writeTokenUsageArtifact, +} from '../lib/artifacts.js'; import { buildScannableTranscript, detectAntiPatterns, @@ -13,7 +17,7 @@ import { scoreEvidenceQuality, scoreReportCompleteness, } from '../lib/bundleScoring.js'; -import { fixtureCommand } from '../lib/cliHarness.js'; +import { fixtureCommand, materializeEvalWorkspace } from '../lib/cliHarness.js'; import { SKILL_CONDITIONS } from '../lib/matrix.js'; import { DogfoodEvalCaseSchema, EvalResultSchema } from '../lib/schemas.js'; import { checkWorkflow } from '../lib/scoring.js'; @@ -29,7 +33,18 @@ import type { RunMetadata, SkillCondition, } from '../lib/types.js'; +import type { ReporterDispatcher } from '../reporters/dispatch.js'; +import { + CaseProgressTracker, + computePlannedCases, +} from '../reporters/runtime.js'; import type { EvalProvider } from '../providers/base.js'; +import { lookupPreset } from '../workspaces/registry.js'; +import { + deriveEffectiveEnv, + resolveWorkspacePreset, +} from '../workspaces/resolver.js'; +import type { ResolvedWorkspacePlan } from '../workspaces/types.js'; import evidenceCompletenessCase from './cases/evidence-completeness.js'; import exploratoryQaCase from './cases/exploratory-qa.js'; import navigationFocusReproCase from './cases/navigation-focus-repro.js'; @@ -71,8 +86,16 @@ const DEFAULT_TOTAL_TRIALS = 1; type DogfoodWorkItem = EvalWorkItemIdentity & ScheduledWorkItem & { evalCase: DogfoodEvalCase; + workspacePlan?: ResolvedWorkspacePlan; }; +type DogfoodLaneOptions = { + conditions?: SkillCondition[]; + caseFilter?: string[]; + concurrency?: number; + reporter?: ReporterDispatcher; +}; + interface LoadedDogfoodSkillPrompts { bootstrapSkillText: string; canonicalAgentTtySkillText: string; @@ -85,6 +108,11 @@ interface DogfoodLaneContext { repoRoot: string; } +interface ResolvedDogfoodWorkspace { + plan: ResolvedWorkspacePlan; + effectiveEnv: Readonly>; +} + let loadedDogfoodSkillPromptsPromise: | Promise | undefined; @@ -537,6 +565,99 @@ function buildRequestedPaths( }; } +function readDogfoodWorkspaceId(evalCase: DogfoodEvalCase): string | undefined { + const candidateWorkspaceId = (evalCase as { workspace?: unknown }).workspace; + if (candidateWorkspaceId === undefined) { + return undefined; + } + + assertString( + candidateWorkspaceId, + 'Dogfood eval case workspace must be a string when provided', + ); + const workspaceId = candidateWorkspaceId.trim(); + invariant( + workspaceId.length > 0, + 'Dogfood eval case workspace must be a non-empty string when provided', + ); + return workspaceId; +} + +function stripDogfoodWorkspace(evalCase: DogfoodEvalCase): DogfoodEvalCase { + const { workspace: _workspace, ...requestEvalCase } = + evalCase as DogfoodEvalCase & { + workspace?: unknown; + }; + void _workspace; + return requestEvalCase; +} + +function prepareDogfoodWorkspacePlan( + workItem: DogfoodWorkItem, + ctx: DogfoodLaneContext, + homeDir: string, +): void { + if (workItem.workspacePlan !== undefined) { + return; + } + + const workspaceId = readDogfoodWorkspaceId(workItem.evalCase); + if (workspaceId === undefined) { + return; + } + + try { + const preset = lookupPreset(workspaceId); + workItem.workspacePlan = resolveWorkspacePreset( + { homeDir, repoRoot: ctx.repoRoot }, + preset, + ); + } catch { + // Preserve existing per-item dogfood failures when plan-only resolution cannot complete before reporter emission. + } +} + +async function resolveDogfoodWorkspace( + evalCase: DogfoodEvalCase, + ctx: DogfoodLaneContext, + homeDir: string, + outputDir: string, + requestedBundlePath: string, + systemPromptEnv: Record, + workspacePlan?: ResolvedWorkspacePlan, +): Promise { + const workspaceId = readDogfoodWorkspaceId(evalCase); + if (workspaceId === undefined) { + return undefined; + } + + const preset = lookupPreset(workspaceId); + if (workspacePlan !== undefined) { + invariant( + workspacePlan.presetId === preset.id, + `Dogfood workspace plan preset mismatch: expected ${preset.id}, got ${workspacePlan.presetId}`, + ); + } + const plan = + workspacePlan ?? + resolveWorkspacePreset({ homeDir, repoRoot: ctx.repoRoot }, preset); + const effectiveEnv = deriveEffectiveEnv(preset, { + EVAL_OUTPUT_DIR: outputDir, + EVAL_REQUESTED_BUNDLE_DIR: requestedBundlePath, + EVAL_FIXTURE: evalCase.fixture ?? '', + ...systemPromptEnv, + }); + + await materializeEvalWorkspace({ + homeDir, + plan, + effectiveEnv, + defaultCwd: ctx.repoRoot, + }); + + return { plan, effectiveEnv }; +} + function buildCaseInventory(): DogfoodEvalCase[] { const cases = [...DOGFOOD_CASES]; const categoryCounts = new Map(); @@ -653,6 +774,74 @@ function buildFailedDogfoodEvalResult(params: { }) as EvalResult; } +async function writeDogfoodTokenUsageArtifacts( + metadata: RunMetadata, + results: readonly EvalResult[], +): Promise { + let artifactsDir: string | undefined; + + for (const result of results) { + const tokenUsage = result.normalizedOutput.tokenUsage; + if (tokenUsage === undefined) { + continue; + } + + if (artifactsDir === undefined) { + const outputBaseDir = metadata.outputBaseDir; + invariant( + typeof outputBaseDir === 'string' && outputBaseDir.length > 0, + 'Dogfood lane token usage artifacts require metadata.outputBaseDir', + ); + artifactsDir = new EvalArtifactStore(outputBaseDir).runDir( + metadata.runId, + ); + } + + invariant( + result.providerId.length > 0, + 'Dogfood lane token usage artifacts require result.providerId', + ); + invariant( + typeof result.modelId === 'string' && result.modelId.length > 0, + 'Dogfood lane token usage artifacts require result.modelId', + ); + invariant( + result.caseId.length > 0, + 'Dogfood lane token usage artifacts require result.caseId', + ); + invariant( + result.condition.length > 0, + 'Dogfood lane token usage artifacts require result.condition', + ); + invariant( + result.lane === 'dogfood', + 'Dogfood lane token usage artifacts require dogfood results', + ); + invariant( + Number.isInteger(result.trial) && result.trial > 0, + 'Dogfood lane token usage artifacts require positive result.trial', + ); + + const createdAtMs = Date.parse(result.completedAt); + invariant( + Number.isInteger(createdAtMs) && createdAtMs >= 0, + 'Dogfood lane token usage artifacts require a valid completedAt timestamp', + ); + + await writeTokenUsageArtifact({ + artifactsDir, + caseId: result.caseId, + lane: result.lane, + condition: result.condition, + provider: result.providerId, + model: result.modelId, + trialIndex: result.trial - 1, + tokenUsage, + createdAtMs, + }); + } +} + export function enumerateDogfoodWorkItems(options?: { conditions?: SkillCondition[]; caseFilter?: string[]; @@ -727,16 +916,39 @@ export async function executeDogfoodWorkItem( const systemPromptContext = await buildSystemPromptContext( workItem.condition, ); + const workspace = await resolveDogfoodWorkspace( + workItem.evalCase, + ctx, + homeDir, + outputDir, + requestedBundlePath, + systemPromptContext.env, + workItem.workspacePlan, + ); + const requestCaseBase = stripDogfoodWorkspace(workItem.evalCase); const requestCase = DogfoodEvalCaseSchema.parse({ - ...workItem.evalCase, + ...requestCaseBase, prompt: buildPrompt( - { ...workItem.evalCase, conditions: [workItem.condition] }, + { ...requestCaseBase, conditions: [workItem.condition] }, requestedBundlePath, systemPromptContext, ), bundlePath: requestedBundlePath, conditions: [workItem.condition], }) as DogfoodEvalCase; + const requestEnv = + workspace === undefined + ? { + AGENT_TTY_HOME: homeDir, + EVAL_OUTPUT_DIR: outputDir, + EVAL_REQUESTED_BUNDLE_DIR: requestedBundlePath, + EVAL_FIXTURE: workItem.evalCase.fixture ?? '', + ...systemPromptContext.env, + } + : { + ...workspace.effectiveEnv, + AGENT_TTY_HOME: homeDir, + }; const agentResult = await provider.invokeAgentMode({ runId: metadata.runId, @@ -746,16 +958,10 @@ export async function executeDogfoodWorkItem( ...(ctx.requestedModelId === undefined ? {} : { modelId: ctx.requestedModelId }), - cwd: ctx.repoRoot, + cwd: workspace?.plan.cwd ?? ctx.repoRoot, homeDir, outputDir, - env: { - AGENT_TTY_HOME: homeDir, - EVAL_OUTPUT_DIR: outputDir, - EVAL_REQUESTED_BUNDLE_DIR: requestedBundlePath, - EVAL_FIXTURE: workItem.evalCase.fixture ?? '', - ...systemPromptContext.env, - }, + env: requestEnv, evalCase: requestCase, }); @@ -918,17 +1124,46 @@ export async function executeDogfoodWorkItem( export async function runDogfoodLane( provider: EvalProvider, metadata: RunMetadata, - options?: { - conditions?: SkillCondition[]; - caseFilter?: string[]; - concurrency?: number; - }, + options?: DogfoodLaneOptions, ): Promise { const totalTrials = resolveTotalTrials(metadata.totalTrials); const items = enumerateDogfoodWorkItems({ - ...options, + ...(options?.conditions === undefined + ? {} + : { conditions: options.conditions }), + ...(options?.caseFilter === undefined + ? {} + : { caseFilter: options.caseFilter }), totalTrials, }); + const plannedCases = computePlannedCases(items); + const reporter = options?.reporter; + const concurrency = options?.concurrency ?? 1; + const activeReporter = + reporter !== undefined && items.length > 0 ? reporter : undefined; + if (activeReporter !== undefined) { + invariant( + Number.isInteger(concurrency) && concurrency > 0, + 'options.concurrency must be a positive integer', + ); + } + + let trackerTimestamp: string | undefined; + const tracker = new CaseProgressTracker({ + runId: metadata.runId, + lane: 'dogfood', + plannedCases, + ...(reporter === undefined ? {} : { dispatcher: reporter }), + now: () => trackerTimestamp ?? new Date().toISOString(), + }); + const trialStarts = new Map< + string, + { startedAt: string; startedAtMs: number } + >(); + const getTimestamp = (): { iso: string; ms: number } => { + const iso = new Date().toISOString(); + return { iso, ms: Date.parse(iso) }; + }; let detectedRuntime: ProviderRuntimeInfo; try { @@ -947,13 +1182,131 @@ export async function runDogfoodLane( metadata.models[0] ?? detectedRuntime.defaultModelId ?? undefined, repoRoot: resolve(metadata.repoRoot), }; - const settlements = await runScheduled( + const laneStartedAt = + activeReporter === undefined ? undefined : getTimestamp(); + if (activeReporter !== undefined && laneStartedAt !== undefined) { + await activeReporter.dispatch('laneStart', { + runId: metadata.runId, + lane: 'dogfood', + caseIds: Array.from(new Set(items.map((item) => item.caseId))), + conditions: Array.from(new Set(items.map((item) => item.condition))), + concurrency, + plannedItems: items.length, + startedAt: laneStartedAt.iso, + }); + } + + const settlements = await runScheduled( items, (item) => executeDogfoodWorkItem(provider, metadata, item, ctx), - { concurrency: options?.concurrency ?? 1 }, + { + concurrency, + ...(activeReporter === undefined + ? {} + : { + onItemStart: async (item: DogfoodWorkItem) => { + const started = getTimestamp(); + trialStarts.set(item.key, { + startedAt: started.iso, + startedAtMs: started.ms, + }); + + const requestedPaths = buildRequestedPaths( + metadata, + provider.id, + item.evalCase, + item.condition, + item.trial, + ); + prepareDogfoodWorkspacePlan(item, ctx, requestedPaths.homeDir); + + trackerTimestamp = started.iso; + try { + await tracker.onTrialStart(item); + } finally { + trackerTimestamp = undefined; + } + + await activeReporter.dispatch('trialStart', { + runId: metadata.runId, + lane: 'dogfood', + caseId: item.caseId, + condition: item.condition, + trial: item.trial, + startedAt: started.iso, + requestedOutputPath: requestedPaths.outputDir, + requestedArtifactPath: requestedPaths.requestedBundlePath, + }); + }, + onItemFinish: async (item: DogfoodWorkItem, settled) => { + const started = trialStarts.get(item.key); + invariant( + started !== undefined, + `Missing reporter start state for ${item.key}`, + ); + const completed = getTimestamp(); + + if (settled.status === 'fulfilled') { + await activeReporter.dispatch('trialFinish', { + runId: metadata.runId, + lane: 'dogfood', + caseId: item.caseId, + condition: item.condition, + trial: item.trial, + startedAt: started.startedAt, + completedAt: completed.iso, + durationMs: Math.max(0, completed.ms - started.startedAtMs), + status: settled.value.ok ? 'passed' : 'failed', + ok: settled.value.ok, + errorClass: settled.value.errorClass ?? null, + errorMessage: settled.value.errorMessage ?? null, + score: settled.value.score.total, + transcriptPath: settled.value.transcriptPath ?? null, + stdoutPath: settled.value.stdoutPath ?? null, + stderrPath: settled.value.stderrPath ?? null, + eventLogPath: settled.value.eventLogPath ?? null, + bundlePath: settled.value.bundlePath ?? null, + artifactManifestPath: + settled.value.artifactManifestPath ?? null, + }); + } else { + const describedError = describeDogfoodError(settled.reason); + await activeReporter.dispatch('trialFinish', { + runId: metadata.runId, + lane: 'dogfood', + caseId: item.caseId, + condition: item.condition, + trial: item.trial, + startedAt: started.startedAt, + completedAt: completed.iso, + durationMs: Math.max(0, completed.ms - started.startedAtMs), + status: 'errored', + ok: false, + errorClass: describedError.errorClass, + errorMessage: describedError.errorMessage, + score: null, + transcriptPath: null, + stdoutPath: null, + stderrPath: null, + eventLogPath: null, + bundlePath: null, + artifactManifestPath: null, + }); + } + + trackerTimestamp = completed.iso; + try { + await tracker.onTrialFinish(item, settled); + } finally { + trackerTimestamp = undefined; + trialStarts.delete(item.key); + } + }, + }), + }, ); - return settlements.map((settlement) => { + const results = settlements.map((settlement) => { if (settlement.status === 'fulfilled') { return settlement.value; } @@ -967,4 +1320,38 @@ export async function runDogfoodLane( error: settlement.reason, }); }); + + await writeDogfoodTokenUsageArtifacts(metadata, results); + + if (activeReporter !== undefined && laneStartedAt !== undefined) { + const completed = getTimestamp(); + const laneTotals = settlements.reduce( + (totals, settlement) => { + totals.total += 1; + if (settlement.status === 'rejected') { + totals.errored += 1; + } else if (settlement.value.ok) { + totals.passed += 1; + } else { + totals.failed += 1; + } + return totals; + }, + { total: 0, passed: 0, failed: 0, errored: 0 }, + ); + + await activeReporter.dispatch('laneFinish', { + runId: metadata.runId, + lane: 'dogfood', + startedAt: laneStartedAt.iso, + completedAt: completed.iso, + durationMs: Math.max(0, completed.ms - laneStartedAt.ms), + total: laneTotals.total, + passed: laneTotals.passed, + failed: laneTotals.failed, + errored: laneTotals.errored, + }); + } + + return results; } diff --git a/evals/execution/cases/doctor-gated.ts b/evals/execution/cases/doctor-gated.ts index 6f738f59..55438b98 100644 --- a/evals/execution/cases/doctor-gated.ts +++ b/evals/execution/cases/doctor-gated.ts @@ -1,78 +1,56 @@ +import { executionCase } from '../../authoring/index.js'; import { ALL_EXECUTION_CONDITIONS, - CREATE_SESSION_PATTERN, DOCTOR_JSON_PATTERN, SCREENSHOT_PATTERN, artifactRequirement, - createExecutionCase, - executionAntiPatterns, - executionBudgets, - executionTaskPrompt, - fixtureSetupStep, ordered, - requiredVerifier, - workflowCheck, } from './shared.js'; -export const doctorGatedCase = createExecutionCase({ - id: 'doctor-gated', - lane: 'execution', - category: 'artifact', - prompt: executionTaskPrompt( +export const doctorGatedCase = executionCase('doctor-gated') + .category('artifact') + .task( 'Before capturing a screenshot, run doctor --json to verify renderer prerequisites and then capture a screenshot of hello-prompt.', - 'hello-prompt', - ), - expectedSkill: 'agent-tty', - fixture: 'hello-prompt', - referenceSteps: 5, - conditions: [...ALL_EXECUTION_CONDITIONS], - setup: [ - fixtureSetupStep( - 'launch-doctor-gated', - 'hello-prompt', + ) + .fixture('hello-prompt', { + setupId: 'launch-doctor-gated', + setupDescription: 'Create an agent-tty session that runs the hello-prompt fixture.', - ), - ], - verifiers: [ - requiredVerifier( + }) + .referenceSteps(5) + .conditions(...ALL_EXECUTION_CONDITIONS) + .assertions((assertions) => { + assertions.screenshot( 'doctor-gated-screenshot', - 'screenshot', 'A screenshot artifact should be produced after the doctor check passes.', { kind: 'screenshot', }, - ), - ], - workflowChecks: [ - workflowCheck( - 'create', - 'Create the fixture session.', - CREATE_SESSION_PATTERN, - ), - workflowCheck( - 'doctor', - 'Run doctor --json before any renderer-dependent capture.', - DOCTOR_JSON_PATTERN, - { dependsOn: ['create'] }, - ), - workflowCheck( - 'screenshot', - 'Capture the screenshot only after the doctor gate.', - ordered(DOCTOR_JSON_PATTERN, SCREENSHOT_PATTERN), - { dependsOn: ['doctor'] }, - ), - ], - antiPatterns: executionAntiPatterns(), - artifactRequirements: [ + ); + }) + .workflow((workflow) => { + workflow + .createSession() + .run('doctor --json', { + id: 'doctor', + description: 'Run doctor --json before any renderer-dependent capture.', + pattern: DOCTOR_JSON_PATTERN, + }) + .screenshot({ + description: 'Capture the screenshot only after the doctor gate.', + pattern: ordered(DOCTOR_JSON_PATTERN, SCREENSHOT_PATTERN), + }); + }) + .rawArtifactRequirement( artifactRequirement( 'screenshot', 'A PNG screenshot should be saved after the doctor check.', String.raw`\.png$`, ), - ], - budgets: executionBudgets({ + ) + .budget({ timeoutMs: 180_000, maxAgentSteps: 14, maxWallClockMs: 75_000, - }), -}); + }) + .build(); diff --git a/evals/execution/cases/hello-prompt.ts b/evals/execution/cases/hello-prompt.ts index 3e280b6f..55da3921 100644 --- a/evals/execution/cases/hello-prompt.ts +++ b/evals/execution/cases/hello-prompt.ts @@ -1,18 +1,5 @@ -import { - ALL_EXECUTION_CONDITIONS, - CREATE_SESSION_PATTERN, - DESTROY_SESSION_PATTERN, - SNAPSHOT_PATTERN, - WAIT_PATTERN, - anyOf, - createExecutionCase, - executionAntiPatterns, - executionBudgets, - executionTaskPrompt, - fixtureSetupStep, - requiredVerifier, - workflowCheck, -} from './shared.js'; +import { executionCase } from '../../authoring/index.js'; +import { ALL_EXECUTION_CONDITIONS, anyOf } from './shared.js'; const HELLO_WORLD_INPUT_PATTERN = anyOf( String.raw`\bagent-tty\b[^\n]*\b(?:run|type)\b[^\n]*hello world`, @@ -20,71 +7,44 @@ const HELLO_WORLD_INPUT_PATTERN = anyOf( String.raw`ECHO:\s*hello world`, ); -export const helloPromptCase = createExecutionCase({ - id: 'hello-prompt', - lane: 'execution', - category: 'session', - prompt: executionTaskPrompt( +export const helloPromptCase = executionCase('hello-prompt') + .category('session') + .task( "Launch the hello-prompt fixture, send 'hello world' as input, wait for the READY> prompt to reappear, take a snapshot to verify the echo, then destroy the session.", - 'hello-prompt', - ), - expectedSkill: 'agent-tty', - fixture: 'hello-prompt', - referenceSteps: 5, - conditions: [...ALL_EXECUTION_CONDITIONS], - setup: [ - fixtureSetupStep( - 'launch-hello-prompt', - 'hello-prompt', + ) + .fixture('hello-prompt', { + setupId: 'launch-hello-prompt', + setupDescription: 'Create an agent-tty session that runs the hello-prompt fixture.', - ), - ], - verifiers: [ - requiredVerifier( + }) + .referenceSteps(5) + .conditions(...ALL_EXECUTION_CONDITIONS) + .assertions((assertions) => { + assertions.snapshot( 'hello-prompt-snapshot', - 'snapshot', 'The transcript snapshot should include the echoed text and the READY prompt.', { patterns: [String.raw`ECHO:\s*hello world`, String.raw`READY>`], }, - ), - ], - workflowChecks: [ - workflowCheck( - 'create', - 'Create the fixture session.', - CREATE_SESSION_PATTERN, - ), - workflowCheck( - 'input', - 'Send hello world with run or type.', - HELLO_WORLD_INPUT_PATTERN, - { dependsOn: ['create'] }, - ), - workflowCheck( - 'wait', - 'Wait for the READY prompt to reappear after the echo.', - anyOf(WAIT_PATTERN, String.raw`ECHO:\s*hello world[\s\S]*READY>`), - { dependsOn: ['input'] }, - ), - workflowCheck( - 'snapshot', - 'Capture a snapshot for verification.', - SNAPSHOT_PATTERN, - { dependsOn: ['wait'] }, - ), - workflowCheck( - 'destroy', - 'Destroy the session after verification.', - DESTROY_SESSION_PATTERN, - { dependsOn: ['snapshot'] }, - ), - ], - antiPatterns: executionAntiPatterns(), - artifactRequirements: [], - budgets: executionBudgets({ + ); + }) + .workflow((workflow) => { + workflow + .createSession() + .input('hello world', { + description: 'Send hello world with run or type.', + pattern: HELLO_WORLD_INPUT_PATTERN, + }) + .waitFor(String.raw`ECHO:\s*hello world[\s\S]*READY>`, { + description: 'Wait for the READY prompt to reappear after the echo.', + }) + .snapshot() + .destroy(); + }) + .budget({ timeoutMs: 120_000, maxAgentSteps: 12, maxWallClockMs: 60_000, - }), -}); + }) + .workspace('agent-tty-smoke') + .build(); diff --git a/evals/execution/runner.ts b/evals/execution/runner.ts index d61546e6..2db63029 100644 --- a/evals/execution/runner.ts +++ b/evals/execution/runner.ts @@ -10,12 +10,20 @@ import { tmpdir } from 'node:os'; import { join, resolve } from 'node:path'; import { assertString, invariant } from '../../src/util/assert.js'; +import { + EvalArtifactStore, + writeTokenUsageArtifact, +} from '../lib/artifacts.js'; import { buildScannableTranscript, countAgentTtyCalls, detectAntiPatterns, } from '../lib/antiPatterns.js'; -import { cleanupEvalHome, createIsolatedEvalHome } from '../lib/cliHarness.js'; +import { + cleanupEvalHome, + createIsolatedEvalHome, + materializeEvalWorkspace, +} from '../lib/cliHarness.js'; import { EvalResultSchema } from '../lib/schemas.js'; import { runScheduled } from '../lib/scheduler.js'; import { checkWorkflow } from '../lib/scoring.js'; @@ -35,7 +43,18 @@ import type { VerifierSpec, WorkflowCheckResult, } from '../lib/types.js'; +import type { ReporterDispatcher } from '../reporters/dispatch.js'; +import { + CaseProgressTracker, + computePlannedCases, +} from '../reporters/runtime.js'; import type { EvalProvider } from '../providers/base.js'; +import { lookupPreset } from '../workspaces/registry.js'; +import { + deriveEffectiveEnv, + resolveWorkspacePreset, +} from '../workspaces/resolver.js'; +import type { ResolvedWorkspacePlan } from '../workspaces/types.js'; import { altScreenDemoCase } from './cases/alt-screen-demo.js'; import { colorGridCase } from './cases/color-grid.js'; import { crashRecoveryCase } from './cases/crash-recovery.js'; @@ -78,8 +97,17 @@ const DEFAULT_TOTAL_TRIALS = 1; type ExecutionWorkItem = EvalWorkItemIdentity & ScheduledWorkItem & { evalCase: ExecutionEvalCase; + homeDir?: string; + workspacePlan?: ResolvedWorkspacePlan; }; +type ExecutionLaneOptions = { + conditions?: SkillCondition[]; + caseFilter?: string[]; + concurrency?: number; + reporter?: ReporterDispatcher; +}; + type EvaluatedVerifier = { spec: VerifierSpec; result: VerifierResult; @@ -90,6 +118,11 @@ interface LoadedExecutionSkillPrompts { canonicalAgentTtySkillText: string; } +interface ResolvedCaseWorkspace { + plan: ResolvedWorkspacePlan; + effectiveEnv: Readonly>; +} + let loadedExecutionSkillPromptsPromise: | Promise | undefined; @@ -848,6 +881,74 @@ function buildRejectedExecutionWorkItemResult( }) as EvalResult; } +async function writeExecutionTokenUsageArtifacts( + metadata: RunMetadata, + results: readonly EvalResult[], +): Promise { + let artifactsDir: string | undefined; + + for (const result of results) { + const tokenUsage = result.normalizedOutput.tokenUsage; + if (tokenUsage === undefined) { + continue; + } + + if (artifactsDir === undefined) { + const outputBaseDir = metadata.outputBaseDir; + invariant( + typeof outputBaseDir === 'string' && outputBaseDir.length > 0, + 'Execution lane token usage artifacts require metadata.outputBaseDir', + ); + artifactsDir = new EvalArtifactStore(outputBaseDir).runDir( + metadata.runId, + ); + } + + invariant( + result.providerId.length > 0, + 'Execution lane token usage artifacts require result.providerId', + ); + invariant( + typeof result.modelId === 'string' && result.modelId.length > 0, + 'Execution lane token usage artifacts require result.modelId', + ); + invariant( + result.caseId.length > 0, + 'Execution lane token usage artifacts require result.caseId', + ); + invariant( + result.condition.length > 0, + 'Execution lane token usage artifacts require result.condition', + ); + invariant( + result.lane === 'execution', + 'Execution lane token usage artifacts require execution results', + ); + invariant( + Number.isInteger(result.trial) && result.trial > 0, + 'Execution lane token usage artifacts require positive result.trial', + ); + + const createdAtMs = Date.parse(result.completedAt); + invariant( + Number.isInteger(createdAtMs) && createdAtMs >= 0, + 'Execution lane token usage artifacts require a valid completedAt timestamp', + ); + + await writeTokenUsageArtifact({ + artifactsDir, + caseId: result.caseId, + lane: result.lane, + condition: result.condition, + provider: result.providerId, + model: result.modelId, + trialIndex: result.trial - 1, + tokenUsage, + createdAtMs, + }); + } +} + function resolveModelId( metadata: RunMetadata, runtime: ProviderRuntimeInfo | undefined, @@ -859,6 +960,100 @@ function resolveModelId( return runtime?.defaultModelId; } +function readExecutionWorkspaceId( + evalCase: ExecutionEvalCase, +): string | undefined { + const candidateWorkspaceId = (evalCase as { workspace?: unknown }).workspace; + if (candidateWorkspaceId === undefined) { + return undefined; + } + + assertString( + candidateWorkspaceId, + 'Execution eval case workspace must be a string when provided', + ); + const workspaceId = candidateWorkspaceId.trim(); + invariant( + workspaceId.length > 0, + 'Execution eval case workspace must be a non-empty string when provided', + ); + return workspaceId; +} + +function stripExecutionWorkspace( + evalCase: ExecutionEvalCase, +): ExecutionEvalCase { + const { workspace: _workspace, ...requestEvalCase } = + evalCase as ExecutionEvalCase & { + workspace?: unknown; + }; + void _workspace; + return requestEvalCase; +} + +async function prepareExecutionWorkspacePlan( + metadata: RunMetadata, + workItem: ExecutionWorkItem, +): Promise { + if (workItem.workspacePlan !== undefined) { + return; + } + + const workspaceId = readExecutionWorkspaceId(workItem.evalCase); + if (workspaceId === undefined) { + return; + } + + const homeDir = workItem.homeDir ?? (await createIsolatedEvalHome()); + workItem.homeDir = homeDir; + + try { + const preset = lookupPreset(workspaceId); + workItem.workspacePlan = resolveWorkspacePreset( + { homeDir, repoRoot: resolve(metadata.repoRoot) }, + preset, + ); + } catch { + // Preserve existing per-item execution failures when plan-only resolution cannot complete before reporter emission. + } +} + +async function resolveExecutionWorkspace( + metadata: RunMetadata, + evalCase: ExecutionEvalCase, + homeDir: string, + outputDir: string, + workspacePlan?: ResolvedWorkspacePlan, +): Promise { + const workspaceId = readExecutionWorkspaceId(evalCase); + if (workspaceId === undefined) { + return undefined; + } + + const repoRoot = resolve(metadata.repoRoot); + const preset = lookupPreset(workspaceId); + if (workspacePlan !== undefined) { + invariant( + workspacePlan.presetId === preset.id, + `Execution workspace plan preset mismatch: expected ${preset.id}, got ${workspacePlan.presetId}`, + ); + } + const plan = + workspacePlan ?? resolveWorkspacePreset({ homeDir, repoRoot }, preset); + const effectiveEnv = deriveEffectiveEnv(preset, { + AGENT_TTY_EVAL_OUTPUT_DIR: outputDir, + }); + + await materializeEvalWorkspace({ + homeDir, + plan, + effectiveEnv, + defaultCwd: repoRoot, + }); + + return { plan, effectiveEnv }; +} + async function createEvalRequest( provider: EvalProvider, metadata: RunMetadata, @@ -868,24 +1063,32 @@ async function createEvalRequest( homeDir: string, outputDir: string, runtime: ProviderRuntimeInfo | undefined, + workspace: ResolvedCaseWorkspace | undefined, ): Promise[0]> { const prompt = await buildPromptForCondition(evalCase, condition); const modelId = resolveModelId(metadata, runtime); + const requestEnv = + workspace === undefined + ? { + AGENT_TTY_HOME: homeDir, + AGENT_TTY_EVAL_OUTPUT_DIR: outputDir, + } + : { + ...workspace.effectiveEnv, + AGENT_TTY_HOME: homeDir, + }; return { runId: metadata.runId, providerId: provider.id, condition, trial, - cwd: resolve(metadata.repoRoot), + cwd: workspace?.plan.cwd ?? resolve(metadata.repoRoot), homeDir, outputDir, - env: { - AGENT_TTY_HOME: homeDir, - AGENT_TTY_EVAL_OUTPUT_DIR: outputDir, - }, + env: requestEnv, evalCase: { - ...evalCase, + ...stripExecutionWorkspace(evalCase), prompt, }, ...(modelId === undefined ? {} : { modelId }), @@ -1001,15 +1204,22 @@ async function detectRuntime( async function runSingleExecutionCase( provider: EvalProvider, metadata: RunMetadata, - evalCase: ExecutionEvalCase, - condition: SkillCondition, - trial: number, + workItem: ExecutionWorkItem, runtime: ProviderRuntimeInfo | undefined, ): Promise { - const homeDir = await createIsolatedEvalHome(); + const { evalCase, condition, trial } = workItem; + const homeDir = workItem.homeDir ?? (await createIsolatedEvalHome()); + workItem.homeDir = homeDir; const outputDir = await mkdtemp(join(tmpdir(), RUNNER_OUTPUT_PREFIX)); try { + const workspace = await resolveExecutionWorkspace( + metadata, + evalCase, + homeDir, + outputDir, + workItem.workspacePlan, + ); const request = await createEvalRequest( provider, metadata, @@ -1019,6 +1229,7 @@ async function runSingleExecutionCase( homeDir, outputDir, runtime, + workspace, ); const invocationStartedAt = new Date().toISOString(); const invocationStartedMs = Date.now(); @@ -1260,14 +1471,7 @@ export async function executeExecutionWorkItem( Number.isInteger(workItem.trial) && workItem.trial > 0, 'Execution work item trial must be a positive integer', ); - return runSingleExecutionCase( - provider, - metadata, - workItem.evalCase, - workItem.condition, - workItem.trial, - runtime, - ); + return runSingleExecutionCase(provider, metadata, workItem, runtime); } /** @@ -1277,27 +1481,171 @@ export async function executeExecutionWorkItem( export async function runExecutionLane( provider: EvalProvider, metadata: RunMetadata, - options: { - conditions?: SkillCondition[]; - caseFilter?: string[]; - concurrency?: number; - } = {}, + options: ExecutionLaneOptions = {}, ): Promise { const totalTrials = resolveTotalTrials(metadata.totalTrials); const items = enumerateExecutionWorkItems({ - ...options, + ...(options.conditions === undefined + ? {} + : { conditions: options.conditions }), + ...(options.caseFilter === undefined + ? {} + : { caseFilter: options.caseFilter }), totalTrials, }); + const plannedCases = computePlannedCases(items); + const reporter = options.reporter; + const concurrency = options.concurrency ?? 1; + const activeReporter = + reporter !== undefined && items.length > 0 ? reporter : undefined; + if (activeReporter !== undefined) { + invariant( + Number.isInteger(concurrency) && concurrency > 0, + 'options.concurrency must be a positive integer', + ); + } + + let trackerTimestamp: string | undefined; + const tracker = new CaseProgressTracker({ + runId: metadata.runId, + lane: 'execution', + plannedCases, + ...(reporter === undefined ? {} : { dispatcher: reporter }), + now: () => trackerTimestamp ?? new Date().toISOString(), + }); + const trialStarts = new Map< + string, + { startedAt: string; startedAtMs: number } + >(); + const getTimestamp = (): { iso: string; ms: number } => { + const iso = new Date().toISOString(); + return { iso, ms: Date.parse(iso) }; + }; const runtime = await detectRuntime(provider); - const settlements = await runScheduled( + + const laneStartedAt = + activeReporter === undefined ? undefined : getTimestamp(); + if (activeReporter !== undefined && laneStartedAt !== undefined) { + await activeReporter.dispatch('laneStart', { + runId: metadata.runId, + lane: 'execution', + caseIds: Array.from(new Set(items.map((item) => item.caseId))), + conditions: Array.from(new Set(items.map((item) => item.condition))), + concurrency, + plannedItems: items.length, + startedAt: laneStartedAt.iso, + }); + } + + const settlements = await runScheduled( items, (item) => executeExecutionWorkItem(provider, metadata, item, runtime), { - concurrency: options.concurrency ?? 1, + concurrency, + ...(activeReporter === undefined + ? {} + : { + onItemStart: async (item: ExecutionWorkItem) => { + const started = getTimestamp(); + trialStarts.set(item.key, { + startedAt: started.iso, + startedAtMs: started.ms, + }); + + await prepareExecutionWorkspacePlan(metadata, item); + + trackerTimestamp = started.iso; + try { + await tracker.onTrialStart(item); + } finally { + trackerTimestamp = undefined; + } + + await activeReporter.dispatch('trialStart', { + runId: metadata.runId, + lane: 'execution', + caseId: item.caseId, + condition: item.condition, + trial: item.trial, + startedAt: started.iso, + requestedOutputPath: null, + requestedArtifactPath: null, + }); + }, + onItemFinish: async (item: ExecutionWorkItem, settled) => { + const started = trialStarts.get(item.key); + invariant( + started !== undefined, + `Missing reporter start state for ${item.key}`, + ); + const completed = getTimestamp(); + + if (settled.status === 'fulfilled') { + await activeReporter.dispatch('trialFinish', { + runId: metadata.runId, + lane: 'execution', + caseId: item.caseId, + condition: item.condition, + trial: item.trial, + startedAt: started.startedAt, + completedAt: completed.iso, + durationMs: Math.max(0, completed.ms - started.startedAtMs), + status: settled.value.ok ? 'passed' : 'failed', + ok: settled.value.ok, + errorClass: settled.value.errorClass ?? null, + errorMessage: settled.value.errorMessage ?? null, + score: settled.value.score.total, + transcriptPath: settled.value.transcriptPath ?? null, + stdoutPath: settled.value.stdoutPath ?? null, + stderrPath: settled.value.stderrPath ?? null, + eventLogPath: settled.value.eventLogPath ?? null, + bundlePath: settled.value.bundlePath ?? null, + artifactManifestPath: + settled.value.artifactManifestPath ?? null, + }); + } else { + await activeReporter.dispatch('trialFinish', { + runId: metadata.runId, + lane: 'execution', + caseId: item.caseId, + condition: item.condition, + trial: item.trial, + startedAt: started.startedAt, + completedAt: completed.iso, + durationMs: Math.max(0, completed.ms - started.startedAtMs), + status: 'errored', + ok: false, + errorClass: + settled.reason instanceof Error + ? settled.reason.name + : 'Error', + errorMessage: + settled.reason instanceof Error + ? settled.reason.message + : String(settled.reason), + score: null, + transcriptPath: null, + stdoutPath: null, + stderrPath: null, + eventLogPath: null, + bundlePath: null, + artifactManifestPath: null, + }); + } + + trackerTimestamp = completed.iso; + try { + await tracker.onTrialFinish(item, settled); + } finally { + trackerTimestamp = undefined; + trialStarts.delete(item.key); + } + }, + }), }, ); - return settlements.map((settlement) => + const results = settlements.map((settlement) => settlement.status === 'fulfilled' ? settlement.value : buildRejectedExecutionWorkItemResult( @@ -1308,4 +1656,38 @@ export async function runExecutionLane( settlement.reason, ), ); + + await writeExecutionTokenUsageArtifacts(metadata, results); + + if (activeReporter !== undefined && laneStartedAt !== undefined) { + const completed = getTimestamp(); + const laneTotals = settlements.reduce( + (totals, settlement) => { + totals.total += 1; + if (settlement.status === 'rejected') { + totals.errored += 1; + } else if (settlement.value.ok) { + totals.passed += 1; + } else { + totals.failed += 1; + } + return totals; + }, + { total: 0, passed: 0, failed: 0, errored: 0 }, + ); + + await activeReporter.dispatch('laneFinish', { + runId: metadata.runId, + lane: 'execution', + startedAt: laneStartedAt.iso, + completedAt: completed.iso, + durationMs: Math.max(0, completed.ms - laneStartedAt.ms), + total: laneTotals.total, + passed: laneTotals.passed, + failed: laneTotals.failed, + errored: laneTotals.errored, + }); + } + + return results; } diff --git a/evals/lib/artifacts.ts b/evals/lib/artifacts.ts index e2a795f0..406b780a 100644 --- a/evals/lib/artifacts.ts +++ b/evals/lib/artifacts.ts @@ -2,18 +2,61 @@ import { randomBytes } from 'node:crypto'; import { mkdir, readdir, stat } from 'node:fs/promises'; import { isAbsolute, relative, resolve } from 'node:path'; +import { z } from 'zod'; + import { readValidatedJsonFile, writeTextFileAtomic, writeValidatedJsonFile, } from '../../src/storage/manifests.js'; import { assertString, invariant } from '../../src/util/assert.js'; -import { EvalResultSchema, RunMetadataSchema } from './schemas.js'; -import type { EvalResult, RunMetadata } from './types.js'; +import { + EvalLaneSchema, + EvalResultSchema, + RunMetadataSchema, + SkillConditionSchema, + TokenUsageSchema, +} from './schemas.js'; +import type { + EvalLane, + EvalResult, + RunMetadata, + SkillCondition, + TokenUsage, +} from './types.js'; const TRANSCRIPT_FILENAME = 'transcript.txt'; const RESULT_FILENAME = 'result.json'; const METADATA_FILENAME = 'metadata.json'; +const TOKEN_USAGE_FILENAME = 'token-usage.json'; + +export interface TokenUsageArtifact { + caseId: string; + lane: EvalLane; + condition: SkillCondition; + provider: string; + model: string; + trialIndex: number; + tokenUsage: TokenUsage; + createdAtMs: number; +} + +export interface WriteTokenUsageArtifactParams extends TokenUsageArtifact { + artifactsDir: string; +} + +const TokenUsageArtifactSchema = z + .object({ + caseId: z.string().min(1), + lane: EvalLaneSchema, + condition: SkillConditionSchema, + provider: z.string().min(1), + model: z.string().min(1), + trialIndex: z.number().int().nonnegative(), + tokenUsage: TokenUsageSchema, + createdAtMs: z.number().int().nonnegative(), + }) + .strict(); interface NodeError { code?: string; @@ -103,6 +146,73 @@ function warnSkippedResult(resultPath: string, error: unknown): void { console.warn(`[evals] Skipping result at ${resultPath}: ${details}`); } +function validateTokenUsageArtifactData( + path: string, + data: unknown, +): TokenUsageArtifact { + const parsedArtifact = TokenUsageArtifactSchema.safeParse(data); + if (!parsedArtifact.success) { + throw new Error( + `Token usage artifact validation failed for ${path}: ${parsedArtifact.error.message}`, + ); + } + + return parsedArtifact.data as TokenUsageArtifact; +} + +/** + * Write a validated token-usage sidecar artifact and return the absolute file path. + */ +export async function writeTokenUsageArtifact( + params: WriteTokenUsageArtifactParams, +): Promise { + assertString(params.artifactsDir, 'artifactsDir must be a string'); + invariant( + params.artifactsDir.length > 0, + 'artifactsDir must be a non-empty string', + ); + + const artifact: TokenUsageArtifact = { + caseId: params.caseId, + lane: params.lane, + condition: params.condition, + provider: params.provider, + model: params.model, + trialIndex: params.trialIndex, + tokenUsage: params.tokenUsage, + createdAtMs: params.createdAtMs, + }; + + validatePathSegment(artifact.lane, 'token usage artifact lane'); + validatePathSegment(artifact.caseId, 'token usage artifact caseId'); + validatePathSegment(artifact.condition, 'token usage artifact condition'); + + const resolvedArtifactsDir = resolve(params.artifactsDir); + invariant( + isAbsolute(resolvedArtifactsDir), + 'resolved artifactsDir must be absolute', + ); + + const artifactPath = resolveWithinBase( + resolvedArtifactsDir, + 'token usage artifact path', + artifact.lane, + artifact.caseId, + artifact.condition, + TOKEN_USAGE_FILENAME, + ); + + await writeValidatedJsonFile({ + path: artifactPath, + pathLabel: 'token usage artifact path', + data: artifact, + writeErrorMessage: `Failed to write token usage artifact at ${artifactPath}.`, + validate: validateTokenUsageArtifactData, + }); + + return artifactPath; +} + /** * Validate that a path segment is safe to use in a filesystem path. */ diff --git a/evals/lib/cliHarness.ts b/evals/lib/cliHarness.ts index 760af783..a9011625 100644 --- a/evals/lib/cliHarness.ts +++ b/evals/lib/cliHarness.ts @@ -1,6 +1,14 @@ import { spawnSync } from 'node:child_process'; import { existsSync, readFileSync } from 'node:fs'; -import { mkdtemp, readFile, readdir, realpath, rm } from 'node:fs/promises'; +import { + cp, + mkdtemp, + readFile, + readdir, + realpath, + rm, + stat, +} from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { isAbsolute, join, resolve, sep } from 'node:path'; import { performance } from 'node:perf_hooks'; @@ -11,6 +19,7 @@ import type { EvalCliResult, EvalEventRecord, } from './types.js'; +import type { ResolvedWorkspacePlan } from '../workspaces/types.js'; import { EvalCliResultSchema } from './schemas.js'; import { @@ -42,7 +51,7 @@ function assertNonEmptyString(value: string, label: string): void { invariant(value.length > 0, `${label} must be a non-empty string`); } -function assertStringArray(values: string[], label: string): void { +function assertStringArray(values: readonly string[], label: string): void { invariant(Array.isArray(values), `${label} must be an array of strings`); for (const value of values) { @@ -51,7 +60,7 @@ function assertStringArray(values: string[], label: string): void { } function assertStringRecord( - value: Record | undefined, + value: Readonly> | undefined, label: string, ): void { if (value === undefined) { @@ -66,6 +75,43 @@ function assertStringRecord( } } +interface PreparedWorkspaceInput { + homeDir: string; + plan: ResolvedWorkspacePlan; + effectiveEnv: Readonly>; + defaultCwd: string; +} + +async function assertExistingDirectory( + pathValue: string, + label: string, +): Promise { + assertAbsolutePath(pathValue, label); + + const stats = await stat(pathValue).catch((error: unknown) => { + throw new Error(`${label} must exist at ${pathValue}`, { cause: error }); + }); + invariant(stats.isDirectory(), `${label} must be a directory`); +} + +function formatCommand(command: string, args: readonly string[]): string { + return [command, ...args].join(' '); +} + +async function copyDirectoryContents( + sourceDir: string, + destinationDir: string, +): Promise { + const entries = await readdir(sourceDir, { withFileTypes: true }); + for (const entry of entries) { + await cp(join(sourceDir, entry.name), join(destinationDir, entry.name), { + recursive: entry.isDirectory(), + errorOnExist: false, + force: true, + }); + } +} + function isRecord(value: unknown): value is Record { return typeof value === 'object' && value !== null && !Array.isArray(value); } @@ -251,6 +297,74 @@ export async function createIsolatedEvalHome(): Promise { return home; } +/** Materialize a resolved workspace plan inside an isolated eval home. */ +export async function materializeEvalWorkspace( + input: PreparedWorkspaceInput, +): Promise { + assertAbsolutePath(input.homeDir, 'workspace homeDir'); + assertAbsolutePath(input.defaultCwd, 'workspace defaultCwd'); + assertStringRecord(input.effectiveEnv, 'workspace effectiveEnv'); + assertNonEmptyString(input.plan.presetId, 'workspace plan.presetId'); + invariant( + Number.isInteger(input.plan.bootstrapCount) && + input.plan.bootstrapCount >= 0, + 'workspace plan.bootstrapCount must be a non-negative integer', + ); + invariant( + input.plan.bootstrapCount === input.plan.bootstrap.length, + 'workspace plan.bootstrapCount must match bootstrap length', + ); + + await assertExistingDirectory(input.homeDir, 'workspace homeDir'); + + if (input.plan.templateDir !== undefined) { + await assertExistingDirectory( + input.plan.templateDir, + 'workspace plan.templateDir', + ); + await copyDirectoryContents(input.plan.templateDir, input.homeDir); + } + + if (input.plan.cwd !== undefined) { + assertAbsolutePath(input.plan.cwd, 'workspace plan.cwd'); + } + + const workspaceCwd = resolve(input.plan.cwd ?? input.defaultCwd); + await assertExistingDirectory(workspaceCwd, 'workspace bootstrap cwd'); + + for (const [index, step] of input.plan.bootstrap.entries()) { + assertNonEmptyString( + step.command, + `workspace bootstrap[${String(index)}].command`, + ); + assertStringArray(step.args, `workspace bootstrap[${String(index)}].args`); + + const commandString = formatCommand(step.command, step.args); + const result = spawnSync(step.command, [...step.args], { + cwd: workspaceCwd, + encoding: 'utf8', + env: { + ...process.env, + ...input.effectiveEnv, + AGENT_TTY_HOME: input.homeDir, + }, + }); + + if (result.error !== undefined) { + throw new Error( + `Workspace preset "${input.plan.presetId}" bootstrap command failed to start: ${commandString}`, + { cause: result.error }, + ); + } + + if (result.status !== 0) { + throw new Error( + `Workspace preset "${input.plan.presetId}" bootstrap command failed: ${commandString} (exitCode=${String(result.status)}, signal=${String(result.signal)})`, + ); + } + } +} + /** Best-effort cleanup for an isolated eval home and any lingering session processes. */ export async function cleanupEvalHome(home: string): Promise { const normalizedHome = await normalizeTempHome(home); diff --git a/evals/lib/reporting.ts b/evals/lib/reporting.ts index d45ed889..2e8ef7e1 100644 --- a/evals/lib/reporting.ts +++ b/evals/lib/reporting.ts @@ -1,5 +1,9 @@ import { invariant } from '../../src/util/assert.js'; -import { AggregateMetricsSchema, JsonReportSchema } from './schemas.js'; +import { + AggregateMetricsSchema, + JsonReportSchema, + TokenReportSummarySchema, +} from './schemas.js'; import { bootstrapPairedCI, computeConfidenceInterval, @@ -23,6 +27,7 @@ import type { ProviderComparisonReport, RunMetadata, SkillCondition, + TokenReportSummary, TrialAggregation, } from './types.js'; @@ -75,12 +80,20 @@ export function generateJsonReport( metadata: RunMetadata, comparisonMetrics?: ComparisonMetrics | ComparisonMetrics[], baselineComparison?: BaselineComparison, + tokenReport?: TokenReportSummary, ): JsonReport { assertResults(results); assertMetadata(metadata); const sortedResults = sortResults(results); const normalizedComparisons = normalizeComparisonMetrics(comparisonMetrics); + const normalizedTokenReport = + tokenReport === undefined + ? undefined + : TokenReportSummarySchema.parse(tokenReport); + const emittedTokenReport = shouldEmitTokenReport(normalizedTokenReport) + ? normalizedTokenReport + : undefined; const aggregateMetrics = buildAggregateMetrics(sortedResults); const aggregate = toAggregateMetricsCore(aggregateMetrics); const conditionComparisonSummary = buildConditionComparisonSummary( @@ -115,6 +128,9 @@ export function generateJsonReport( ...(providerComparison === undefined ? {} : { providerComparison }), ...(aggregated === undefined ? {} : { aggregated }), ...(baselineComparison === undefined ? {} : { baselineComparison }), + ...(emittedTokenReport === undefined + ? {} + : { tokenReport: emittedTokenReport }), } satisfies JsonReport; JsonReportSchema.parse(coreReport); @@ -139,6 +155,7 @@ export function generateMarkdownReport( metadata: RunMetadata, comparisonMetrics?: ComparisonMetrics | ComparisonMetrics[], baselineComparison?: BaselineComparison, + tokenReport?: TokenReportSummary, ): string { assertResults(results); assertMetadata(metadata); @@ -148,6 +165,7 @@ export function generateMarkdownReport( metadata, comparisonMetrics, baselineComparison, + tokenReport, ) as JsonReport & RichJsonReport; const providers = collectProviders(report.results, report.metadata.providers); const lanes = collectLanes(report.results, report.metadata.lanes); @@ -345,6 +363,10 @@ export function generateMarkdownReport( ); } + if (report.tokenReport !== undefined) { + sections.push('', ...buildTokenUsageMarkdown(report.tokenReport)); + } + return `${sections.join('\n').trimEnd()}\n`; } @@ -1237,6 +1259,164 @@ function buildCompletenessRows( return rows; } +function shouldEmitTokenReport( + tokenReport: TokenReportSummary | undefined, +): tokenReport is TokenReportSummary { + return tokenReport !== undefined && tokenReport.grandTotal.trials > 0; +} + +function buildTokenUsageMarkdown(tokenReport: TokenReportSummary): string[] { + const sections = [ + '## Token usage', + '', + '### Grand total', + '', + buildMarkdownTable( + ['Input', 'Output', 'Total', 'Cached', 'Trials'], + ['right', 'right', 'right', 'right', 'right'], + [ + [ + String(tokenReport.grandTotal.inputTokens), + String(tokenReport.grandTotal.outputTokens), + String(tokenReport.grandTotal.totalTokens), + formatOptionalTokenCount(tokenReport.grandTotal.cachedTokens), + String(tokenReport.grandTotal.trials), + ], + ], + ), + '', + '### Per lane', + '', + ]; + + if (tokenReport.perLane.length === 0) { + sections.push('- None.'); + } else { + sections.push( + buildMarkdownTable( + ['Lane', 'Input', 'Output', 'Total', 'Cached', 'Trials'], + ['left', 'right', 'right', 'right', 'right', 'right'], + tokenReport.perLane.map((entry) => [ + `\`${sanitizeInline(entry.lane)}\``, + String(entry.inputTokens), + String(entry.outputTokens), + String(entry.totalTokens), + formatOptionalTokenCount(entry.cachedTokens), + String(entry.trials), + ]), + ), + ); + } + + sections.push('', '### Per case', ''); + if (tokenReport.perCase.length === 0) { + sections.push('- None.'); + } else { + sections.push( + buildMarkdownTable( + [ + 'Lane', + 'Case', + 'Condition', + 'Input', + 'Output', + 'Total', + 'Cached', + 'Trials', + ], + ['left', 'left', 'left', 'right', 'right', 'right', 'right', 'right'], + tokenReport.perCase.map((entry) => [ + `\`${sanitizeInline(entry.lane)}\``, + `\`${sanitizeInline(entry.caseId)}\``, + `\`${sanitizeInline(entry.condition)}\``, + String(entry.inputTokens), + String(entry.outputTokens), + String(entry.totalTokens), + formatOptionalTokenCount(entry.cachedTokens), + String(entry.trials), + ]), + ), + ); + } + + if (tokenReport.snapshotCheck !== undefined) { + sections.push('', '### Snapshot check', ''); + sections.push( + `- Regression threshold: ${formatPercentValue(tokenReport.snapshotCheck.regressionThresholdPercent)}`, + ); + if (tokenReport.snapshotCheck.summary.regressed > 0) { + sections.push( + `- Warning: ${String(tokenReport.snapshotCheck.summary.regressed)} regressed snapshot case(s) exceeded the threshold.`, + ); + } + sections.push( + '', + buildMarkdownTable( + ['Total', 'New', 'Orphaned', 'Unchanged', 'Improved', 'Regressed'], + ['right', 'right', 'right', 'right', 'right', 'right'], + [ + [ + String(tokenReport.snapshotCheck.summary.total), + String(tokenReport.snapshotCheck.summary.new), + String(tokenReport.snapshotCheck.summary.orphaned), + String(tokenReport.snapshotCheck.summary.unchanged), + String(tokenReport.snapshotCheck.summary.improved), + String(tokenReport.snapshotCheck.summary.regressed), + ], + ], + ), + '', + ); + + if (tokenReport.snapshotCheck.cases.length === 0) { + sections.push('- None.'); + } else { + sections.push( + buildMarkdownTable( + [ + 'Provider', + 'Model', + 'Lane', + 'Case', + 'Condition', + 'Outcome', + 'Current', + 'Snapshot', + 'Delta', + 'Delta %', + ], + [ + 'left', + 'left', + 'left', + 'left', + 'left', + 'left', + 'right', + 'right', + 'right', + 'right', + ], + tokenReport.snapshotCheck.cases.map((entry) => [ + `\`${sanitizeInline(entry.provider)}\``, + `\`${sanitizeInline(entry.model)}\``, + `\`${entry.lane}\``, + `\`${sanitizeInline(entry.caseId)}\``, + `\`${entry.condition}\``, + `\`${entry.outcome}\``, + formatOptionalTokenCount(entry.currentTotalTokens), + formatOptionalTokenCount(entry.snapshotTotalTokens), + formatOptionalSignedCount(entry.deltaTokens), + formatOptionalPercentValue(entry.deltaPercent), + ]), + ), + ); + } + } + + return sections; +} + function summarizeAntiPatterns(results: EvalResult[]): AntiPatternSummaryRow[] { const summaries = new Map< string, @@ -1448,6 +1628,22 @@ function formatScore(value: number): string { return value.toFixed(3); } +function formatPercentValue(value: number): string { + return `${value.toFixed(1)}%`; +} + +function formatOptionalTokenCount(value: number | undefined): string { + return value === undefined ? '—' : String(value); +} + +function formatOptionalSignedCount(value: number | undefined): string { + return value === undefined ? '—' : String(value); +} + +function formatOptionalPercentValue(value: number | undefined): string { + return value === undefined ? '—' : formatPercentValue(value); +} + function formatConfidenceInterval( interval: { lower: number; upper: number }, formatter: (value: number) => string, diff --git a/evals/lib/scheduler.ts b/evals/lib/scheduler.ts index 3913e158..e4bc28dd 100644 --- a/evals/lib/scheduler.ts +++ b/evals/lib/scheduler.ts @@ -1,12 +1,24 @@ import { invariant } from '../../src/util/assert.js'; -export interface SchedulerOptions { +export type ScheduledWorkItem = { key: string }; + +export type SchedulerItemSettlement = + | { status: 'fulfilled'; value: R } + | { status: 'rejected'; reason: unknown }; + +export interface SchedulerOptions< + T extends ScheduledWorkItem = ScheduledWorkItem, + R = unknown, +> { concurrency: number; logLine?: (line: string) => void; + onItemStart?: (item: T) => void | Promise; + onItemFinish?: ( + item: T, + settled: SchedulerItemSettlement, + ) => void | Promise; } -export type ScheduledWorkItem = { key: string }; - export type SettledResult = | { item: T; status: 'fulfilled'; value: R } | { item: T; status: 'rejected'; reason: unknown }; @@ -14,7 +26,7 @@ export type SettledResult = export async function runScheduled( items: readonly T[], executor: (item: T) => Promise, - options: SchedulerOptions, + options: SchedulerOptions, ): Promise[]> { invariant(Array.isArray(items), 'items must be an array'); invariant( @@ -22,19 +34,32 @@ export async function runScheduled( 'options.concurrency must be a positive integer', ); + const onItemStart = options.onItemStart; + invariant( + onItemStart === undefined || typeof onItemStart === 'function', + 'options.onItemStart must be a function or undefined', + ); + + const onItemFinish = options.onItemFinish; + invariant( + onItemFinish === undefined || typeof onItemFinish === 'function', + 'options.onItemFinish must be a function or undefined', + ); + if (items.length === 0) { return []; } - const settlements: Array | undefined> = new Array( - items.length, + const settlements: Array | undefined> = Array.from( + { length: items.length }, + () => undefined, ); const workerCount = Math.min(options.concurrency, items.length); const logLine = options.logLine; let nextIndex = 0; async function worker(): Promise { - while (true) { + for (;;) { const index = nextIndex; if (index >= items.length) { return; @@ -42,15 +67,30 @@ export async function runScheduled( nextIndex += 1; const item = items[index]; - invariant(item !== undefined, `Missing scheduled item at index ${index}`); + invariant( + item !== undefined, + `Missing scheduled item at index ${String(index)}`, + ); logLine?.(`[${item.key}] start`); + await onItemStart?.(item); + try { const value = await executor(item); - settlements[index] = { item, status: 'fulfilled', value }; + const settled: SchedulerItemSettlement = { + status: 'fulfilled', + value, + }; + settlements[index] = { item, ...settled }; + await onItemFinish?.(item, settled); logLine?.(`[${item.key}] ok`); } catch (reason) { - settlements[index] = { item, status: 'rejected', reason }; + const settled: SchedulerItemSettlement = { + status: 'rejected', + reason, + }; + settlements[index] = { item, ...settled }; + await onItemFinish?.(item, settled); logLine?.(`[${item.key}] failed: ${String(reason)}`); } } @@ -61,7 +101,7 @@ export async function runScheduled( return settlements.map((settlement, index) => { invariant( settlement !== undefined, - `Missing scheduler settlement for item at index ${index}`, + `Missing scheduler settlement for item at index ${String(index)}`, ); return settlement; }); diff --git a/evals/lib/schemas.ts b/evals/lib/schemas.ts index ea2c231d..bcf574d6 100644 --- a/evals/lib/schemas.ts +++ b/evals/lib/schemas.ts @@ -1,6 +1,7 @@ import { z } from 'zod'; import { EventRecordSchema } from '../../src/protocol/schemas.js'; +import { SnapshotCheckReportSchema } from '../snapshots/schemas/report.js'; import type { ArtifactKind } from '../../src/tools/review-bundle.js'; import type { BundleValidationProfile } from '../../src/tools/validate-bundle.js'; import type { @@ -34,6 +35,8 @@ import type { ProviderPromptResult, ProviderRuntimeInfo, ReportCompletenessScore, + TokenReportSummary, + TokenUsage, TrialAggregation, } from './types.js'; @@ -467,6 +470,7 @@ export const ExecutionEvalCaseSchema = z artifactRequirements: z.array(ArtifactRequirementSchema), budgets: ExecutionBudgetSchema, referenceSteps: PositiveIntSchema.optional(), + workspace: NonEmptyStringSchema.optional(), }) .strict() .superRefine((obj, ctx) => { @@ -515,6 +519,7 @@ export const DogfoodEvalCaseSchema = z antiPatterns: z.array(AntiPatternRuleSchema), budgets: DogfoodBudgetSchema, referenceSteps: PositiveIntSchema.optional(), + workspace: NonEmptyStringSchema.optional(), }) .strict() .superRefine((obj, ctx) => { @@ -594,6 +599,58 @@ export const ProviderRuntimeInfoSchema = z }) .strict(); +export const TokenUsageSchema = z + .object({ + inputTokens: NonNegativeIntSchema, + outputTokens: NonNegativeIntSchema, + totalTokens: NonNegativeIntSchema, + cachedTokens: NonNegativeIntSchema.optional(), + }) + .strict(); + +const TokenReportGrandTotalSchema = z + .object({ + inputTokens: NonNegativeIntSchema, + outputTokens: NonNegativeIntSchema, + totalTokens: NonNegativeIntSchema, + cachedTokens: NonNegativeIntSchema.optional(), + trials: NonNegativeIntSchema, + }) + .strict(); + +const TokenReportLaneSchema = z + .object({ + lane: NonEmptyStringSchema, + inputTokens: NonNegativeIntSchema, + outputTokens: NonNegativeIntSchema, + totalTokens: NonNegativeIntSchema, + cachedTokens: NonNegativeIntSchema.optional(), + trials: NonNegativeIntSchema, + }) + .strict(); + +const TokenReportCaseSchema = z + .object({ + lane: NonEmptyStringSchema, + caseId: NonEmptyStringSchema, + condition: NonEmptyStringSchema, + inputTokens: NonNegativeIntSchema, + outputTokens: NonNegativeIntSchema, + totalTokens: NonNegativeIntSchema, + cachedTokens: NonNegativeIntSchema.optional(), + trials: NonNegativeIntSchema, + }) + .strict(); + +export const TokenReportSummarySchema = z + .object({ + grandTotal: TokenReportGrandTotalSchema, + perLane: z.array(TokenReportLaneSchema), + perCase: z.array(TokenReportCaseSchema), + snapshotCheck: SnapshotCheckReportSchema.optional(), + }) + .strict(); + export const NormalizedProviderOutputSchema = z .object({ finalText: z.string(), @@ -602,6 +659,7 @@ export const NormalizedProviderOutputSchema = z referencedSkills: z.array(NonEmptyStringSchema), selectedSkill: ExpectedSkillSchema.optional(), toolCalls: z.array(UnknownRecordSchema), + tokenUsage: TokenUsageSchema.optional(), }) .strict(); @@ -827,6 +885,7 @@ export const JsonReportSchema = z providerComparison: ProviderComparisonReportSchema.optional(), aggregated: z.array(TrialAggregationSchema).optional(), baselineComparison: BaselineComparisonSchema.optional(), + tokenReport: TokenReportSummarySchema.optional(), }) .strict(); @@ -1010,6 +1069,10 @@ export type ProviderCapabilitiesSchemaType = z.infer< export type ProviderRuntimeInfoSchemaType = z.infer< typeof ProviderRuntimeInfoSchema >; +export type TokenUsageSchemaType = z.infer; +export type TokenReportSummarySchemaType = z.infer< + typeof TokenReportSummarySchema +>; export type NormalizedProviderOutputSchemaType = z.infer< typeof NormalizedProviderOutputSchema >; @@ -1170,6 +1233,14 @@ export type _ProviderRuntimeInfoSchemaParity = AssertExact< ProviderRuntimeInfo, ProviderRuntimeInfoSchemaType >; +export type _TokenUsageSchemaParity = AssertExact< + TokenUsage, + TokenUsageSchemaType +>; +export type _TokenReportSummarySchemaParity = AssertExact< + TokenReportSummary, + TokenReportSummarySchemaType +>; export type _NormalizedProviderOutputSchemaParity = AssertExact< NormalizedProviderOutput, NormalizedProviderOutputSchemaType diff --git a/evals/lib/tokenAggregation.ts b/evals/lib/tokenAggregation.ts new file mode 100644 index 00000000..309a347d --- /dev/null +++ b/evals/lib/tokenAggregation.ts @@ -0,0 +1,212 @@ +import { assertString, invariant } from '../../src/util/assert.js'; + +import { SKILL_CONDITIONS } from './matrix.js'; +import type { + EvalLane, + SkillCondition, + TokenReportSummary, + TokenUsage, +} from './types.js'; + +export interface RawTokenRecord { + provider: string; + model: string; + lane: EvalLane; + caseId: string; + condition: SkillCondition; + caseFingerprint: string; + usage: TokenUsage; +} + +interface TokenAccumulator { + inputTokens: number; + outputTokens: number; + totalTokens: number; + cachedTokensTotal: number; + everyRecordHasCachedTokens: boolean; + trials: number; +} + +function createTokenAccumulator(): TokenAccumulator { + return { + inputTokens: 0, + outputTokens: 0, + totalTokens: 0, + cachedTokensTotal: 0, + everyRecordHasCachedTokens: true, + trials: 0, + }; +} + +function accumulateTokenUsage( + accumulator: TokenAccumulator, + usage: TokenUsage, +): void { + invariant( + Number.isInteger(usage.inputTokens) && usage.inputTokens >= 0, + 'token usage inputTokens must be a non-negative integer', + ); + invariant( + Number.isInteger(usage.outputTokens) && usage.outputTokens >= 0, + 'token usage outputTokens must be a non-negative integer', + ); + invariant( + Number.isInteger(usage.totalTokens) && usage.totalTokens >= 0, + 'token usage totalTokens must be a non-negative integer', + ); + if (usage.cachedTokens !== undefined) { + invariant( + Number.isInteger(usage.cachedTokens) && usage.cachedTokens >= 0, + 'token usage cachedTokens must be a non-negative integer', + ); + } + + accumulator.inputTokens += usage.inputTokens; + accumulator.outputTokens += usage.outputTokens; + accumulator.totalTokens += usage.totalTokens; + accumulator.trials += 1; + if (usage.cachedTokens === undefined) { + accumulator.everyRecordHasCachedTokens = false; + return; + } + accumulator.cachedTokensTotal += usage.cachedTokens; +} + +function emitTokenAccumulator(accumulator: TokenAccumulator): { + inputTokens: number; + outputTokens: number; + totalTokens: number; + cachedTokens?: number; + trials: number; +} { + return { + inputTokens: accumulator.inputTokens, + outputTokens: accumulator.outputTokens, + totalTokens: accumulator.totalTokens, + ...(accumulator.everyRecordHasCachedTokens + ? { cachedTokens: accumulator.cachedTokensTotal } + : {}), + trials: accumulator.trials, + }; +} + +function compareStrings(left: string, right: string): number { + return left.localeCompare(right); +} + +function compareCondition(left: SkillCondition, right: SkillCondition): number { + return SKILL_CONDITIONS.indexOf(left) - SKILL_CONDITIONS.indexOf(right); +} + +export function aggregateTokenRecords( + records: RawTokenRecord[], + laneOrder: string[], +): TokenReportSummary | undefined { + invariant(Array.isArray(records), 'token records must be an array'); + invariant(Array.isArray(laneOrder), 'laneOrder must be an array'); + if (records.length === 0) { + return undefined; + } + + const laneIndexes = new Map(); + for (const [index, lane] of laneOrder.entries()) { + assertString(lane, 'laneOrder entries must be strings'); + invariant(lane.length > 0, 'laneOrder entries must not be empty'); + invariant(!laneIndexes.has(lane), `Duplicate laneOrder entry: ${lane}`); + laneIndexes.set(lane, index); + } + + const grandTotal = createTokenAccumulator(); + const perLaneAccumulators = new Map(); + const perCaseAccumulators = new Map< + string, + { + lane: EvalLane; + caseId: string; + condition: SkillCondition; + accumulator: TokenAccumulator; + } + >(); + + for (const record of records) { + assertString(record.caseId, 'token record caseId must be a string'); + invariant( + record.caseId.length > 0, + 'token record caseId must not be empty', + ); + invariant( + laneIndexes.has(record.lane), + `token record lane is missing from laneOrder: ${record.lane}`, + ); + + accumulateTokenUsage(grandTotal, record.usage); + + const laneAccumulator = + perLaneAccumulators.get(record.lane) ?? createTokenAccumulator(); + accumulateTokenUsage(laneAccumulator, record.usage); + perLaneAccumulators.set(record.lane, laneAccumulator); + + const perCaseKey = JSON.stringify([ + record.lane, + record.caseId, + record.condition, + ]); + const existingPerCase = perCaseAccumulators.get(perCaseKey); + if (existingPerCase === undefined) { + const accumulator = createTokenAccumulator(); + accumulateTokenUsage(accumulator, record.usage); + perCaseAccumulators.set(perCaseKey, { + lane: record.lane, + caseId: record.caseId, + condition: record.condition, + accumulator, + }); + continue; + } + + accumulateTokenUsage(existingPerCase.accumulator, record.usage); + } + + const perLane = laneOrder + .filter((lane): lane is EvalLane => perLaneAccumulators.has(lane)) + .map((lane) => ({ + lane, + ...emitTokenAccumulator( + perLaneAccumulators.get(lane) ?? createTokenAccumulator(), + ), + })); + + const perCase = [...perCaseAccumulators.values()] + .sort((left, right) => { + const leftLaneIndex = laneIndexes.get(left.lane); + const rightLaneIndex = laneIndexes.get(right.lane); + invariant( + leftLaneIndex !== undefined, + `Missing laneOrder entry: ${left.lane}`, + ); + invariant( + rightLaneIndex !== undefined, + `Missing laneOrder entry: ${right.lane}`, + ); + if (leftLaneIndex !== rightLaneIndex) { + return leftLaneIndex - rightLaneIndex; + } + const caseComparison = compareStrings(left.caseId, right.caseId); + if (caseComparison !== 0) { + return caseComparison; + } + return compareCondition(left.condition, right.condition); + }) + .map((entry) => ({ + lane: entry.lane, + caseId: entry.caseId, + condition: entry.condition, + ...emitTokenAccumulator(entry.accumulator), + })); + + return { + grandTotal: emitTokenAccumulator(grandTotal), + perLane, + perCase, + }; +} diff --git a/evals/lib/types.ts b/evals/lib/types.ts index 7c40b259..8688fec5 100644 --- a/evals/lib/types.ts +++ b/evals/lib/types.ts @@ -5,6 +5,7 @@ import type { BundleValidationCheck, BundleValidationProfile, } from '../../src/tools/validate-bundle.js'; +import type { SnapshotCheckReport } from '../snapshots/schemas/report.js'; /** Expected skill for an eval case. */ export type ExpectedSkill = 'none' | 'agent-tty' | 'dogfood-tui'; @@ -306,6 +307,7 @@ export interface ExecutionEvalCase { maxWallClockMs?: number; }; referenceSteps?: number; + workspace?: string; } /** Dogfood eval case for evidence capture and reporting quality. */ @@ -332,6 +334,7 @@ export interface DogfoodEvalCase { maxWallClockMs?: number; }; referenceSteps?: number; + workspace?: string; } /** Any supported eval case. */ @@ -424,6 +427,35 @@ export interface ConditionComparisonSummary { }; } +export interface TokenReportSummary { + grandTotal: { + inputTokens: number; + outputTokens: number; + totalTokens: number; + cachedTokens?: number | undefined; + trials: number; + }; + perLane: Array<{ + lane: string; + inputTokens: number; + outputTokens: number; + totalTokens: number; + cachedTokens?: number | undefined; + trials: number; + }>; + perCase: Array<{ + lane: string; + caseId: string; + condition: string; + inputTokens: number; + outputTokens: number; + totalTokens: number; + cachedTokens?: number | undefined; + trials: number; + }>; + snapshotCheck?: SnapshotCheckReport | undefined; +} + /** JSON-serializable top-level eval report. */ export interface JsonReport { metadata: RunMetadata; @@ -434,6 +466,7 @@ export interface JsonReport { providerComparison?: ProviderComparisonReport; aggregated?: TrialAggregation[]; baselineComparison?: BaselineComparison; + tokenReport?: TokenReportSummary | undefined; } /** Confidence interval bounds. */ @@ -621,6 +654,14 @@ export interface ProviderRuntimeInfo { notes: string[]; } +/** Normalized provider token-usage metadata. */ +export interface TokenUsage { + inputTokens: number; + outputTokens: number; + totalTokens: number; + cachedTokens?: number; +} + /** Provider output normalized for downstream scoring and storage. */ export interface NormalizedProviderOutput { finalText: string; @@ -629,6 +670,7 @@ export interface NormalizedProviderOutput { referencedSkills: string[]; selectedSkill?: ExpectedSkill; toolCalls: Array>; + tokenUsage?: TokenUsage; } /** Capability flags exposed by a provider adapter. */ diff --git a/evals/prompt/cases/trigger-agent-tty.ts b/evals/prompt/cases/trigger-agent-tty.ts index ccd291d4..68e83734 100644 --- a/evals/prompt/cases/trigger-agent-tty.ts +++ b/evals/prompt/cases/trigger-agent-tty.ts @@ -1,3 +1,4 @@ +import { promptCase } from '../../authoring/index.js'; import { PromptEvalCaseSchema } from '../../lib/schemas.js'; import type { PromptEvalCase } from '../../lib/types.js'; @@ -27,6 +28,37 @@ function parseCase(evalCase: PromptEvalCase): PromptEvalCase { return PromptEvalCaseSchema.parse(evalCase) as PromptEvalCase; } +const waitForOutputCase = promptCase('wait-for-output') + .category('trigger') + .prompt( + "I need to wait until my server prints 'Listening on port 3000' before running tests", + ) + .expectSkill('agent-tty') + .context( + 'The answer should prefer waiting on observable terminal text over fixed delays before starting the next step.', + ) + .expectedPatterns(['/agent-tty/i', '/\\bwait\\b/i']) + .forbiddenPatterns([SLEEP_RECOMMENDATION_PATTERN, '/setTimeout/i']) + .rubric( + 'Chooses agent-tty for terminal readiness coordination.', + 'Uses wait against concrete terminal output instead of fixed timing guesses.', + ) + .workflow((workflow) => { + workflow + .step('wait-for-output.select-agent-tty', 'Explicitly selects agent-tty.') + .mustMention('/agent-tty/i'); + workflow + .step( + 'wait-for-output.observe-readiness', + 'Waits for the listening message before running tests.', + ) + .mustMention('/\\bwait\\b/i', '/Listening on port 3000/i') + .mustNotMention(SLEEP_RECOMMENDATION_PATTERN, '/setTimeout/i'); + }) + .antiPatterns(...EMPTY_ANTI_PATTERNS) + .budget(PROMPT_TIMEOUT_MS) + .build(); + export const TRIGGER_AGENT_TTY_PROMPT_CASES: PromptEvalCase[] = [ parseCase({ id: 'session-creation', @@ -102,37 +134,7 @@ export const TRIGGER_AGENT_TTY_PROMPT_CASES: PromptEvalCase[] = [ antiPatterns: EMPTY_ANTI_PATTERNS, budgets: { timeoutMs: PROMPT_TIMEOUT_MS }, }), - parseCase({ - id: 'wait-for-output', - lane: 'prompt', - category: 'trigger', - prompt: - "I need to wait until my server prints 'Listening on port 3000' before running tests", - expectedSkill: 'agent-tty', - context: - 'The answer should prefer waiting on observable terminal text over fixed delays before starting the next step.', - expectedPatterns: ['/agent-tty/i', '/\\bwait\\b/i'], - forbiddenPatterns: [SLEEP_RECOMMENDATION_PATTERN, '/setTimeout/i'], - rubric: [ - 'Chooses agent-tty for terminal readiness coordination.', - 'Uses wait against concrete terminal output instead of fixed timing guesses.', - ], - workflowChecks: [ - requiredCheck( - 'wait-for-output.select-agent-tty', - 'Explicitly selects agent-tty.', - ['/agent-tty/i'], - ), - requiredCheck( - 'wait-for-output.observe-readiness', - 'Waits for the listening message before running tests.', - ['/\\bwait\\b/i', '/Listening on port 3000/i'], - [SLEEP_RECOMMENDATION_PATTERN, '/setTimeout/i'], - ), - ], - antiPatterns: EMPTY_ANTI_PATTERNS, - budgets: { timeoutMs: PROMPT_TIMEOUT_MS }, - }), + waitForOutputCase, parseCase({ id: 'snapshot-inspection', lane: 'prompt', diff --git a/evals/prompt/runner.ts b/evals/prompt/runner.ts index 43fb97ac..64fa6b53 100644 --- a/evals/prompt/runner.ts +++ b/evals/prompt/runner.ts @@ -1,6 +1,10 @@ import { readFile } from 'node:fs/promises'; import { assertString, invariant } from '../../src/util/assert.js'; +import { + EvalArtifactStore, + writeTokenUsageArtifact, +} from '../lib/artifacts.js'; import { detectAntiPatterns } from '../lib/antiPatterns.js'; import { SKILL_CONDITIONS } from '../lib/matrix.js'; import { runScheduled } from '../lib/scheduler.js'; @@ -24,6 +28,11 @@ import type { RunMetadata, SkillCondition, } from '../lib/types.js'; +import type { ReporterDispatcher } from '../reporters/dispatch.js'; +import { + CaseProgressTracker, + computePlannedCases, +} from '../reporters/runtime.js'; import type { EvalProvider } from '../providers/base.js'; import { ANTI_PATTERN_PROMPT_CASES } from './cases/anti-patterns.js'; import { SHOULD_NOT_TRIGGER_PROMPT_CASES } from './cases/should-not-trigger.js'; @@ -60,6 +69,7 @@ type PromptWorkItemOptions = { type PromptLaneOptions = PromptWorkItemOptions & { concurrency?: number; + reporter?: ReporterDispatcher; }; type PromptWorkItem = EvalWorkItemIdentity & @@ -415,6 +425,74 @@ function buildRejectedPromptWorkItemEvalResult( }) as EvalResult; } +async function writePromptTokenUsageArtifacts( + metadata: RunMetadata, + results: readonly EvalResult[], +): Promise { + let artifactsDir: string | undefined; + + for (const result of results) { + const tokenUsage = result.normalizedOutput.tokenUsage; + if (tokenUsage === undefined) { + continue; + } + + if (artifactsDir === undefined) { + const outputBaseDir = metadata.outputBaseDir; + invariant( + typeof outputBaseDir === 'string' && outputBaseDir.length > 0, + 'Prompt lane token usage artifacts require metadata.outputBaseDir', + ); + artifactsDir = new EvalArtifactStore(outputBaseDir).runDir( + metadata.runId, + ); + } + + invariant( + result.providerId.length > 0, + 'Prompt lane token usage artifacts require result.providerId', + ); + invariant( + typeof result.modelId === 'string' && result.modelId.length > 0, + 'Prompt lane token usage artifacts require result.modelId', + ); + invariant( + result.caseId.length > 0, + 'Prompt lane token usage artifacts require result.caseId', + ); + invariant( + result.condition.length > 0, + 'Prompt lane token usage artifacts require result.condition', + ); + invariant( + result.lane === 'prompt', + 'Prompt lane token usage artifacts require prompt results', + ); + invariant( + Number.isInteger(result.trial) && result.trial > 0, + 'Prompt lane token usage artifacts require positive result.trial', + ); + + const createdAtMs = Date.parse(result.completedAt); + invariant( + Number.isInteger(createdAtMs) && createdAtMs >= 0, + 'Prompt lane token usage artifacts require a valid completedAt timestamp', + ); + + await writeTokenUsageArtifact({ + artifactsDir, + caseId: result.caseId, + lane: result.lane, + condition: result.condition, + provider: result.providerId, + model: result.modelId, + trialIndex: result.trial - 1, + tokenUsage, + createdAtMs, + }); + } +} + async function readSkillFile(relativePath: string): Promise { const content = await readFile( new URL(relativePath, import.meta.url), @@ -589,15 +667,156 @@ export async function runPromptLane( parsedMetadata, options, ); - const settlements = await runScheduled( + const plannedCases = computePlannedCases(items); + const reporter = options?.reporter; + const concurrency = options?.concurrency ?? 1; + const activeReporter = + reporter !== undefined && items.length > 0 ? reporter : undefined; + if (activeReporter !== undefined) { + invariant( + Number.isInteger(concurrency) && concurrency > 0, + 'options.concurrency must be a positive integer', + ); + } + + let trackerTimestamp: string | undefined; + const tracker = new CaseProgressTracker({ + runId: parsedMetadata.runId, + lane: 'prompt', + plannedCases, + ...(reporter === undefined ? {} : { dispatcher: reporter }), + now: () => trackerTimestamp ?? new Date().toISOString(), + }); + const trialStarts = new Map< + string, + { startedAt: string; startedAtMs: number } + >(); + const getTimestamp = (): { iso: string; ms: number } => { + const iso = new Date().toISOString(); + return { iso, ms: Date.parse(iso) }; + }; + + const laneStartedAt = + activeReporter === undefined ? undefined : getTimestamp(); + if (activeReporter !== undefined && laneStartedAt !== undefined) { + await activeReporter.dispatch('laneStart', { + runId: parsedMetadata.runId, + lane: 'prompt', + caseIds: Array.from(new Set(items.map((item) => item.caseId))), + conditions: Array.from(new Set(items.map((item) => item.condition))), + concurrency, + plannedItems: items.length, + startedAt: laneStartedAt.iso, + }); + } + + const settlements = await runScheduled( items, async (item) => executePromptWorkItem(provider, parsedMetadata, item), { - concurrency: options?.concurrency ?? 1, + concurrency, + ...(activeReporter === undefined + ? {} + : { + onItemStart: async (item: PromptWorkItem) => { + const started = getTimestamp(); + trialStarts.set(item.key, { + startedAt: started.iso, + startedAtMs: started.ms, + }); + + trackerTimestamp = started.iso; + try { + await tracker.onTrialStart(item); + } finally { + trackerTimestamp = undefined; + } + + await activeReporter.dispatch('trialStart', { + runId: parsedMetadata.runId, + lane: 'prompt', + caseId: item.caseId, + condition: item.condition, + trial: item.trial, + startedAt: started.iso, + requestedOutputPath: null, + requestedArtifactPath: null, + }); + }, + onItemFinish: async (item: PromptWorkItem, settled) => { + const started = trialStarts.get(item.key); + invariant( + started !== undefined, + `Missing reporter start state for ${item.key}`, + ); + const completed = getTimestamp(); + + if (settled.status === 'fulfilled') { + await activeReporter.dispatch('trialFinish', { + runId: parsedMetadata.runId, + lane: 'prompt', + caseId: item.caseId, + condition: item.condition, + trial: item.trial, + startedAt: started.startedAt, + completedAt: completed.iso, + durationMs: Math.max(0, completed.ms - started.startedAtMs), + status: settled.value.ok ? 'passed' : 'failed', + ok: settled.value.ok, + errorClass: settled.value.errorClass ?? null, + errorMessage: settled.value.errorMessage ?? null, + score: settled.value.score.total, + transcriptPath: settled.value.transcriptPath ?? null, + stdoutPath: settled.value.stdoutPath ?? null, + stderrPath: settled.value.stderrPath ?? null, + eventLogPath: settled.value.eventLogPath ?? null, + bundlePath: settled.value.bundlePath ?? null, + artifactManifestPath: + settled.value.artifactManifestPath ?? null, + }); + } else { + await activeReporter.dispatch('trialFinish', { + runId: parsedMetadata.runId, + lane: 'prompt', + caseId: item.caseId, + condition: item.condition, + trial: item.trial, + startedAt: started.startedAt, + completedAt: completed.iso, + durationMs: Math.max(0, completed.ms - started.startedAtMs), + status: 'errored', + ok: false, + errorClass: + settled.reason instanceof Error + ? settled.reason.name + : 'Error', + errorMessage: + settled.reason instanceof Error + ? settled.reason.message + : String(settled.reason), + score: null, + transcriptPath: null, + stdoutPath: null, + stderrPath: null, + eventLogPath: null, + bundlePath: null, + artifactManifestPath: null, + }); + } + + trackerTimestamp = completed.iso; + try { + await tracker.onTrialFinish(item, settled); + } finally { + trackerTimestamp = undefined; + trialStarts.delete(item.key); + } + }, + }), }, ); - return settlements.map((settlement) => + const results = settlements.map((settlement) => settlement.status === 'fulfilled' ? settlement.value : buildRejectedPromptWorkItemEvalResult( @@ -606,4 +825,38 @@ export async function runPromptLane( settlement, ), ); + + await writePromptTokenUsageArtifacts(parsedMetadata, results); + + if (activeReporter !== undefined && laneStartedAt !== undefined) { + const completed = getTimestamp(); + const laneTotals = settlements.reduce( + (totals, settlement) => { + totals.total += 1; + if (settlement.status === 'rejected') { + totals.errored += 1; + } else if (settlement.value.ok) { + totals.passed += 1; + } else { + totals.failed += 1; + } + return totals; + }, + { total: 0, passed: 0, failed: 0, errored: 0 }, + ); + + await activeReporter.dispatch('laneFinish', { + runId: parsedMetadata.runId, + lane: 'prompt', + startedAt: laneStartedAt.iso, + completedAt: completed.iso, + durationMs: Math.max(0, completed.ms - laneStartedAt.ms), + total: laneTotals.total, + passed: laneTotals.passed, + failed: laneTotals.failed, + errored: laneTotals.errored, + }); + } + + return results; } diff --git a/evals/providers/claude.ts b/evals/providers/claude.ts index 53ae1b7f..6ab62224 100644 --- a/evals/providers/claude.ts +++ b/evals/providers/claude.ts @@ -12,6 +12,7 @@ import { ProviderPromptRequestSchema, ProviderPromptResultSchema, ProviderRuntimeInfoSchema, + TokenUsageSchema, } from '../lib/schemas.js'; import type { NormalizedProviderOutput, @@ -22,6 +23,7 @@ import type { ProviderPromptRequest, ProviderPromptResult, ProviderRuntimeInfo, + TokenUsage, } from '../lib/types.js'; import type { EvalProvider } from './base.js'; @@ -356,6 +358,60 @@ function parseJsonRecords(raw: string): unknown[] { return records; } +function parseClaudeTokenUsageObject(usage: unknown): TokenUsage | undefined { + if (!isRecord(usage)) { + return undefined; + } + + const candidate = { + ...(typeof usage.input_tokens === 'number' + ? { inputTokens: usage.input_tokens } + : {}), + ...(typeof usage.output_tokens === 'number' + ? { outputTokens: usage.output_tokens } + : {}), + ...(typeof usage.total_tokens === 'number' + ? { totalTokens: usage.total_tokens } + : {}), + ...(typeof usage.cache_read_input_tokens === 'number' + ? { cachedTokens: usage.cache_read_input_tokens } + : {}), + }; + + const parsedUsage = TokenUsageSchema.safeParse(candidate); + if (!parsedUsage.success) { + return undefined; + } + + const { inputTokens, outputTokens, totalTokens, cachedTokens } = + parsedUsage.data; + return { + inputTokens, + outputTokens, + totalTokens, + ...(cachedTokens === undefined ? {} : { cachedTokens }), + }; +} + +function extractClaudeTokenUsage( + record: Record, +): TokenUsage | undefined { + let tokenUsage: TokenUsage | undefined; + const sources = [ + record.usage, + isRecord(record.message) ? record.message.usage : undefined, + ]; + + for (const source of sources) { + const parsedUsage = parseClaudeTokenUsageObject(source); + if (parsedUsage !== undefined) { + tokenUsage = parsedUsage; + } + } + + return tokenUsage; +} + function extractPlainTextToolCalls( raw: string, ): Array> { @@ -906,6 +962,7 @@ export class ClaudeProvider implements EvalProvider { let sessionId: string | undefined; let modelId: string | undefined; let selectedSkill: 'none' | 'agent-tty' | 'dogfood-tui' | undefined; + let tokenUsage: TokenUsage | undefined; records.forEach((record, index) => { if (!isRecord(record)) { @@ -928,6 +985,13 @@ export class ClaudeProvider implements EvalProvider { : undefined, ); } + + const parsedTokenUsage = extractClaudeTokenUsage(record); + if (parsedTokenUsage !== undefined) { + // Later Claude JSON records typically reflect the final result payload. + tokenUsage = parsedTokenUsage; + } + if (record.type === 'result' && typeof record.result === 'string') { finalText = record.result; } @@ -993,6 +1057,7 @@ export class ClaudeProvider implements EvalProvider { referencedSkills: extractReferencedSkills(allText), ...(selectedSkill === undefined ? {} : { selectedSkill }), toolCalls: [...toolCalls.values()], + ...(tokenUsage === undefined ? {} : { tokenUsage }), }, raw, 'Invalid Claude JSON normalized output', diff --git a/evals/providers/codex.ts b/evals/providers/codex.ts index c56f8eef..991fab25 100644 --- a/evals/providers/codex.ts +++ b/evals/providers/codex.ts @@ -12,6 +12,7 @@ import { ProviderPromptRequestSchema, ProviderPromptResultSchema, ProviderRuntimeInfoSchema, + TokenUsageSchema, } from '../lib/schemas.js'; import type { NormalizedProviderOutput, @@ -22,6 +23,7 @@ import type { ProviderPromptRequest, ProviderPromptResult, ProviderRuntimeInfo, + TokenUsage, } from '../lib/types.js'; import type { EvalProvider } from './base.js'; @@ -467,6 +469,81 @@ function resolveCodexToolCallId( return fallbackId; } +function parseCodexTokenUsageObject(usage: unknown): TokenUsage | undefined { + if (!isRecord(usage)) { + return undefined; + } + + const cachedInputTokens = + typeof usage.cached_input_tokens === 'number' + ? usage.cached_input_tokens + : undefined; + const cachedTokensFromDetails = + isRecord(usage.input_tokens_details) && + typeof usage.input_tokens_details.cached_tokens === 'number' + ? usage.input_tokens_details.cached_tokens + : undefined; + if ( + cachedInputTokens !== undefined && + cachedTokensFromDetails !== undefined && + cachedInputTokens !== cachedTokensFromDetails + ) { + return undefined; + } + + const cachedTokenCount = cachedInputTokens ?? cachedTokensFromDetails; + const candidate = { + ...(typeof usage.input_tokens === 'number' + ? { inputTokens: usage.input_tokens } + : {}), + ...(typeof usage.output_tokens === 'number' + ? { outputTokens: usage.output_tokens } + : {}), + ...(typeof usage.total_tokens === 'number' + ? { totalTokens: usage.total_tokens } + : {}), + ...(cachedTokenCount === undefined + ? {} + : { cachedTokens: cachedTokenCount }), + }; + + const parsedUsage = TokenUsageSchema.safeParse(candidate); + if (!parsedUsage.success) { + return undefined; + } + + const { inputTokens, outputTokens, totalTokens, cachedTokens } = + parsedUsage.data; + return { + inputTokens, + outputTokens, + totalTokens, + ...(cachedTokens === undefined ? {} : { cachedTokens }), + }; +} + +function extractCodexTokenUsage( + record: Record, +): TokenUsage | undefined { + let tokenUsage: TokenUsage | undefined; + const recordItem = isRecord(record.item) ? record.item : undefined; + const sources = [ + record.usage, + isRecord(record.response) ? record.response.usage : undefined, + recordItem?.usage, + isRecord(recordItem?.response) ? recordItem.response.usage : undefined, + ]; + + for (const source of sources) { + const parsedUsage = parseCodexTokenUsageObject(source); + if (parsedUsage !== undefined) { + tokenUsage = parsedUsage; + } + } + + return tokenUsage; +} + function extractTextFragments(value: unknown): string[] { if (typeof value === 'string') { return value.trim().length > 0 ? [value] : []; @@ -1014,6 +1091,7 @@ export class CodexProvider implements EvalProvider { let sessionId: string | undefined; let modelId: string | undefined; let selectedSkill: 'none' | 'agent-tty' | 'dogfood-tui' | undefined; + let tokenUsage: TokenUsage | undefined; records.forEach((record, index) => { if (!isRecord(record)) { @@ -1040,6 +1118,12 @@ export class CodexProvider implements EvalProvider { ); } + const parsedTokenUsage = extractCodexTokenUsage(record); + if (parsedTokenUsage !== undefined) { + // Later Codex JSONL records typically carry the final turn-completed usage. + tokenUsage = parsedTokenUsage; + } + if (record.type === 'error') { const errorText = extractTextFragments(record).join('\n').trim(); if (errorText.length > 0) { @@ -1145,6 +1229,7 @@ export class CodexProvider implements EvalProvider { referencedSkills: extractReferencedSkills(allText), ...(selectedSkill === undefined ? {} : { selectedSkill }), toolCalls: [...toolCalls.values()], + ...(tokenUsage === undefined ? {} : { tokenUsage }), }, raw, 'Invalid Codex JSONL normalized output', diff --git a/evals/providers/fixtures.ts b/evals/providers/fixtures.ts index 0b6e609f..faa25ee0 100644 --- a/evals/providers/fixtures.ts +++ b/evals/providers/fixtures.ts @@ -11,6 +11,7 @@ import { ProviderPromptRequestSchema, ProviderPromptResultSchema, ProviderRuntimeInfoSchema, + TokenUsageSchema, } from '../lib/schemas.js'; import type { NormalizedProviderOutput, @@ -21,6 +22,7 @@ import type { ProviderPromptRequest, ProviderPromptResult, ProviderRuntimeInfo, + TokenUsage, } from '../lib/types.js'; import type { EvalProvider } from './base.js'; @@ -77,12 +79,14 @@ function safeStringify(value: unknown): string { function buildPassThroughNormalizedOutput( raw: string, + tokenUsage?: TokenUsage, ): NormalizedProviderOutput { const parsedResult = NormalizedProviderOutputSchema.safeParse({ finalText: raw, messages: [raw], referencedSkills: [], toolCalls: [], + ...(tokenUsage === undefined ? {} : { tokenUsage }), }); if (parsedResult.success) { @@ -94,6 +98,21 @@ function buildPassThroughNormalizedOutput( ); } +function coerceOptionalTokenUsage( + rawTokenUsage: unknown, +): TokenUsage | undefined { + if (rawTokenUsage === undefined) { + return undefined; + } + + const parsedTokenUsage = TokenUsageSchema.safeParse(rawTokenUsage); + if (!parsedTokenUsage.success) { + return undefined; + } + + return parsedTokenUsage.data as TokenUsage; +} + function parseProviderConfig(config: unknown, message: string): ProviderConfig { const parsedResult = ProviderConfigSchema.safeParse(config); if (parsedResult.success) { @@ -857,6 +876,7 @@ export class FixtureProvider implements EvalProvider { rawOutput.skillDetected === 'dogfood-tui' ? rawOutput.skillDetected : undefined; + const tokenUsage = coerceOptionalTokenUsage(rawOutput.tokenUsage); return parseNormalizedOutput( { @@ -868,6 +888,7 @@ export class FixtureProvider implements EvalProvider { referencedSkills, ...(selectedSkill === undefined ? {} : { selectedSkill }), toolCalls: toolCalls ?? [], + ...(tokenUsage === undefined ? {} : { tokenUsage }), }, `Invalid normalized output fixture in ${sourceLabel}`, ); @@ -905,9 +926,10 @@ export class FixtureProvider implements EvalProvider { const durationMs = this.coerceDurationMs(rawResult.latencyMs); const startedAtMs = Date.now(); const completedAtMs = startedAtMs + durationMs; + const tokenUsage = coerceOptionalTokenUsage(rawResult.tokenUsage); const normalized = normalizedOverride ?? - buildPassThroughNormalizedOutput(rawResult.response); + buildPassThroughNormalizedOutput(rawResult.response, tokenUsage); const adaptedRuntime = typeof rawResult.model === 'string' && rawResult.model.length > 0 ? parseRuntimeInfo( @@ -971,9 +993,10 @@ export class FixtureProvider implements EvalProvider { const errorLines = this.coerceErrorLines(rawResult.errors, sourceLabel); const startedAtMs = Date.now(); const completedAtMs = startedAtMs + durationMs; + const tokenUsage = coerceOptionalTokenUsage(rawResult.tokenUsage); const normalized = normalizedOverride ?? - buildPassThroughNormalizedOutput(rawResult.transcript); + buildPassThroughNormalizedOutput(rawResult.transcript, tokenUsage); return parseAgentResult( { diff --git a/evals/reporters/console.ts b/evals/reporters/console.ts new file mode 100644 index 00000000..33e6e681 --- /dev/null +++ b/evals/reporters/console.ts @@ -0,0 +1,109 @@ +import { invariant } from '../../src/util/assert.js'; + +import type { + CaseFinishEvent, + CaseStartEvent, + LaneFinishEvent, + LaneStartEvent, + Reporter, + RunFinishEvent, + RunStartEvent, + TrialFinishEvent, + TrialStartEvent, +} from './types.js'; + +export interface ConsoleReporterOptions { + verbose?: boolean; + writeLine?: (line: string) => void; +} + +function defaultWriteLine(line: string): void { + process.stderr.write(`${line}\n`); +} + +function formatNullableNumber(value: number | null): string { + return value === null ? '-' : String(value); +} + +function formatModel(value: string): string { + return value.length === 0 ? '-' : value; +} + +export class ConsoleReporter implements Reporter { + public readonly name = 'console'; + + private readonly verbose: boolean; + private readonly writeLine: (line: string) => void; + + public constructor(options: ConsoleReporterOptions = {}) { + const { verbose = false, writeLine = defaultWriteLine } = options; + + invariant( + typeof verbose === 'boolean', + 'console reporter verbose must be a boolean', + ); + invariant( + typeof writeLine === 'function', + 'console reporter writeLine must be a function', + ); + + this.verbose = verbose; + this.writeLine = writeLine; + } + + public onRunStart(event: RunStartEvent): void { + this.writeLine( + `run ${event.runId} started: provider=${event.provider} model=${formatModel(event.model)} lanes=${event.lanes.join(',')} conditions=${event.conditions.join(',')} trials=${String(event.totalTrials)} invocations=${String(event.totalInvocations)}`, + ); + } + + public onLaneStart(event: LaneStartEvent): void { + this.writeLine( + `lane ${event.lane} started: cases=${String(event.caseIds.length)} conditions=${String(event.conditions.length)} concurrency=${String(event.concurrency)} planned=${String(event.plannedItems)}`, + ); + } + + public onCaseStart(event: CaseStartEvent): void { + this.writeLine( + `case ${event.lane}/${event.caseId} [${event.condition}] started: trials=${String(event.plannedTrials)}`, + ); + } + + public onTrialStart(event: TrialStartEvent): void { + if (!this.verbose) { + return; + } + + this.writeLine( + `trial ${event.lane}/${event.caseId}[${event.condition}]#${String(event.trial)} started`, + ); + } + + public onTrialFinish(event: TrialFinishEvent): void { + if (!this.verbose) { + return; + } + + this.writeLine( + `trial ${event.lane}/${event.caseId}[${event.condition}]#${String(event.trial)} ${event.status} ok=${String(event.ok)} durationMs=${String(event.durationMs)} score=${formatNullableNumber(event.score)}`, + ); + } + + public onCaseFinish(event: CaseFinishEvent): void { + this.writeLine( + `case ${event.lane}/${event.caseId} [${event.condition}] finished: passed=${String(event.passed)} failed=${String(event.failed)} errored=${String(event.errored)} meanScore=${formatNullableNumber(event.meanScore)} durationMs=${String(event.durationMs)}`, + ); + } + + public onLaneFinish(event: LaneFinishEvent): void { + this.writeLine( + `lane ${event.lane} finished: total=${String(event.total)} passed=${String(event.passed)} failed=${String(event.failed)} errored=${String(event.errored)} durationMs=${String(event.durationMs)}`, + ); + } + + public onRunFinish(event: RunFinishEvent): void { + this.writeLine( + `run ${event.runId} finished: total=${String(event.total)} passed=${String(event.passed)} failed=${String(event.failed)} errored=${String(event.errored)} durationMs=${String(event.durationMs)}`, + ); + } +} diff --git a/evals/reporters/dispatch.ts b/evals/reporters/dispatch.ts new file mode 100644 index 00000000..b6bdeda0 --- /dev/null +++ b/evals/reporters/dispatch.ts @@ -0,0 +1,154 @@ +import type { ZodIssue } from 'zod'; + +import { assertString, invariant } from '../../src/util/assert.js'; + +import { + EVENT_SCHEMAS, + type Reporter, + type ReporterEventName, + type ReporterEventPayloads, +} from './types.js'; + +const REDACTED_VALUE = '[REDACTED]'; +const SECRET_NAMES = ['TOKEN', 'KEY', 'SECRET', 'PASSWORD'] as const; + +const EVENT_HOOK_NAMES = { + runStart: 'onRunStart', + laneStart: 'onLaneStart', + caseStart: 'onCaseStart', + trialStart: 'onTrialStart', + trialFinish: 'onTrialFinish', + caseFinish: 'onCaseFinish', + laneFinish: 'onLaneFinish', + runFinish: 'onRunFinish', +} as const satisfies { + [K in ReporterEventName]: Exclude; +}; + +type ReporterEventHook = ( + event: ReporterEventPayloads[K], +) => Promise | void; + +function isPlainObject(value: unknown): value is Record { + if (typeof value !== 'object' || value === null || Array.isArray(value)) { + return false; + } + + const prototype = Object.getPrototypeOf(value); + return prototype === Object.prototype || prototype === null; +} + +function shouldRedactKey(key: string): boolean { + const upperKey = key.toUpperCase(); + return SECRET_NAMES.some( + (name) => upperKey === name || upperKey.endsWith(`_${name}`), + ); +} + +function formatValidationIssue(issue: ZodIssue): string { + const path = issue.path.length === 0 ? '(root)' : issue.path.join('.'); + return `${path}: ${issue.message}`; +} + +function formatReporterError(error: unknown): string { + return error instanceof Error ? error.message : String(error); +} + +function validatePayload( + eventName: K, + payload: unknown, +): ReporterEventPayloads[K] { + const validation = EVENT_SCHEMAS[eventName].safeParse(payload); + if (!validation.success) { + const issues = validation.error.issues + .map(formatValidationIssue) + .join('; '); + throw new Error( + `Invalid reporter payload for event "${eventName}": ${issues}`, + ); + } + + return validation.data as ReporterEventPayloads[K]; +} + +function getReporterHook( + reporter: Reporter, + eventName: K, +): ReporterEventHook | undefined { + const hookName = EVENT_HOOK_NAMES[eventName]; + return reporter[hookName] as ReporterEventHook | undefined; +} + +export function redactSecretLikeValues(value: unknown): unknown { + if (Array.isArray(value)) { + return value.map((item) => redactSecretLikeValues(item)); + } + + if (!isPlainObject(value)) { + return value; + } + + return Object.fromEntries( + Object.entries(value).map(([key, nestedValue]) => [ + key, + shouldRedactKey(key) + ? REDACTED_VALUE + : redactSecretLikeValues(nestedValue), + ]), + ); +} + +export class ReporterDispatcher { + private readonly reporters: Reporter[]; + + public constructor(reporters: readonly Reporter[] = []) { + invariant(Array.isArray(reporters), 'reporters must be an array'); + + const seenNames = new Set(); + this.reporters = reporters.map((reporter, index) => { + invariant( + reporter !== null && reporter !== undefined, + `reporters[${String(index)}] must not be null or undefined`, + ); + assertString( + reporter.name, + `reporters[${String(index)}].name must be a string`, + ); + invariant( + reporter.name.length > 0, + `reporters[${String(index)}].name must not be empty`, + ); + invariant( + !seenNames.has(reporter.name), + `Duplicate reporter name: ${reporter.name}`, + ); + seenNames.add(reporter.name); + return reporter; + }); + } + + public async dispatch( + eventName: K, + payload: ReporterEventPayloads[K], + ): Promise { + const validatedPayload = validatePayload(eventName, payload); + const redactedPayload = redactSecretLikeValues( + validatedPayload, + ) as ReporterEventPayloads[K]; + + for (const reporter of this.reporters) { + const hook = getReporterHook(reporter, eventName); + if (hook === undefined) { + continue; + } + + try { + await hook.call(reporter, redactedPayload); + } catch (error) { + process.stderr.write( + `reporter "${reporter.name}" failed on ${eventName}: ${formatReporterError(error)}\n`, + ); + } + } + } +} diff --git a/evals/reporters/final-report.ts b/evals/reporters/final-report.ts new file mode 100644 index 00000000..9b550a35 --- /dev/null +++ b/evals/reporters/final-report.ts @@ -0,0 +1,93 @@ +import { writeFile } from 'node:fs/promises'; + +import { assertString, invariant } from '../../src/util/assert.js'; + +import { + generateJsonReport, + generateMarkdownReport, +} from '../lib/reporting.js'; +import type { + BaselineComparison, + ComparisonMetrics, + EvalResult, + RunMetadata, +} from '../lib/types.js'; +import type { Reporter, RunFinishEvent } from './types.js'; + +export interface FinalReportInputs { + results: EvalResult[]; + metadata: RunMetadata; + comparisonMetrics?: ComparisonMetrics[] | ComparisonMetrics; + baselineComparison?: BaselineComparison; + jsonReportPath: string; + markdownReportPath: string; +} + +export interface FinalReportReporterContext { + getFinalReportInputs: () => FinalReportInputs | null; +} + +export class FinalReportReporter implements Reporter { + public readonly name = 'final'; + + private readonly getFinalReportInputs: () => FinalReportInputs | null; + + public constructor(context: FinalReportReporterContext) { + invariant( + context !== null && context !== undefined, + 'final report reporter context is required', + ); + invariant( + typeof context.getFinalReportInputs === 'function', + 'final report reporter getFinalReportInputs must be a function', + ); + + this.getFinalReportInputs = context.getFinalReportInputs; + } + + public async onRunFinish(event: RunFinishEvent): Promise { + const inputs = this.getFinalReportInputs(); + if (inputs === null) { + return; + } + + assertString( + inputs.jsonReportPath, + 'final report reporter jsonReportPath must be a string', + ); + invariant( + inputs.jsonReportPath.length > 0, + 'final report reporter jsonReportPath must not be empty', + ); + assertString( + inputs.markdownReportPath, + 'final report reporter markdownReportPath must be a string', + ); + invariant( + inputs.markdownReportPath.length > 0, + 'final report reporter markdownReportPath must not be empty', + ); + + const jsonReport = generateJsonReport( + inputs.results, + inputs.metadata, + inputs.comparisonMetrics, + inputs.baselineComparison, + event.tokenReport, + ); + const markdownReport = generateMarkdownReport( + inputs.results, + inputs.metadata, + inputs.comparisonMetrics, + inputs.baselineComparison, + event.tokenReport, + ); + + await writeFile( + inputs.jsonReportPath, + `${JSON.stringify(jsonReport, null, 2)}\n`, + 'utf8', + ); + await writeFile(inputs.markdownReportPath, markdownReport, 'utf8'); + } +} diff --git a/evals/reporters/jsonl.ts b/evals/reporters/jsonl.ts new file mode 100644 index 00000000..12d38a40 --- /dev/null +++ b/evals/reporters/jsonl.ts @@ -0,0 +1,109 @@ +import * as fs from 'node:fs/promises'; +import { dirname } from 'node:path'; + +import { assertString, invariant } from '../../src/util/assert.js'; + +import type { + CaseFinishEvent, + CaseStartEvent, + LaneFinishEvent, + LaneStartEvent, + Reporter, + ReporterEventName, + ReporterEventPayloads, + RunFinishEvent, + RunStartEvent, + TrialFinishEvent, + TrialStartEvent, +} from './types.js'; + +export interface JsonlReporterOptions { + outputPath: string; +} + +type JsonlEventType = + | 'run.start' + | 'lane.start' + | 'case.start' + | 'trial.start' + | 'trial.finish' + | 'case.finish' + | 'lane.finish' + | 'run.finish'; + +type ReporterEventPayload = ReporterEventPayloads[ReporterEventName]; + +export class JsonlReporter implements Reporter { + public readonly name = 'jsonl'; + + private readonly outputPath: string; + private directoryReady = false; + + public constructor(options: JsonlReporterOptions) { + assertString( + options.outputPath, + 'jsonl reporter outputPath must be a string', + ); + invariant( + options.outputPath.trim().length > 0, + 'jsonl reporter outputPath must not be empty', + ); + + this.outputPath = options.outputPath; + } + + public async onRunStart(event: RunStartEvent): Promise { + await this.appendEvent('run.start', event); + } + + public async onLaneStart(event: LaneStartEvent): Promise { + await this.appendEvent('lane.start', event); + } + + public async onCaseStart(event: CaseStartEvent): Promise { + await this.appendEvent('case.start', event); + } + + public async onTrialStart(event: TrialStartEvent): Promise { + await this.appendEvent('trial.start', event); + } + + public async onTrialFinish(event: TrialFinishEvent): Promise { + await this.appendEvent('trial.finish', event); + } + + public async onCaseFinish(event: CaseFinishEvent): Promise { + await this.appendEvent('case.finish', event); + } + + public async onLaneFinish(event: LaneFinishEvent): Promise { + await this.appendEvent('lane.finish', event); + } + + public async onRunFinish(event: RunFinishEvent): Promise { + await this.appendEvent('run.finish', event); + } + + private async appendEvent( + type: JsonlEventType, + payload: ReporterEventPayload, + ): Promise { + await this.ensureDirectory(); + + const line = JSON.stringify({ + type, + timestamp: new Date().toISOString(), + payload, + }); + await fs.appendFile(this.outputPath, `${line}\n`, 'utf8'); + } + + private async ensureDirectory(): Promise { + if (this.directoryReady) { + return; + } + + await fs.mkdir(dirname(this.outputPath), { recursive: true }); + this.directoryReady = true; + } +} diff --git a/evals/reporters/runtime.ts b/evals/reporters/runtime.ts new file mode 100644 index 00000000..bf5c5073 --- /dev/null +++ b/evals/reporters/runtime.ts @@ -0,0 +1,390 @@ +import { assertString, invariant } from '../../src/util/assert.js'; + +import type { SchedulerItemSettlement } from '../lib/scheduler.js'; +import type { EvalLane, EvalWorkItemIdentity } from '../lib/types.js'; +import type { ResolvedWorkspacePlan } from '../workspaces/types.js'; +import type { ReporterEventPayloads } from './types.js'; + +type CaseProgressEventName = 'caseStart' | 'caseFinish'; + +export type CaseProgressIdentity = Pick< + EvalWorkItemIdentity, + 'caseId' | 'condition' +>; + +export type CaseProgressResult = { + ok: boolean; +}; + +export interface CaseProgressWorkspaceCarrier { + workspacePlan?: ResolvedWorkspacePlan; +} + +export interface PlannedCase { + caseId: string; + condition: EvalWorkItemIdentity['condition']; + plannedTrials: number; +} + +export interface CaseProgressDispatcher { + dispatch( + eventName: K, + payload: ReporterEventPayloads[K], + ): Promise | void; +} + +export interface CaseProgressTrackerOptions< + TItem extends CaseProgressIdentity & CaseProgressWorkspaceCarrier, + TResult extends CaseProgressResult, +> { + runId: string; + lane: EvalLane; + plannedCases: ReadonlyMap; + dispatcher?: CaseProgressDispatcher; + now?: () => string; + getScore?: (item: TItem, result: TResult) => number | null; +} + +interface CaseProgressState { + plannedCase: PlannedCase; + startedAt: string | null; + startedAtMs: number | null; + completedTrials: number; + passed: number; + failed: number; + errored: number; + scoreSum: number; + scoreCount: number; + finished: boolean; +} + +function buildCaseProgressKey(identity: CaseProgressIdentity): string { + return `${identity.caseId}\u0000${identity.condition}`; +} + +function validateCaseIdentity( + identity: CaseProgressIdentity, + label: string, +): void { + assertString(identity.caseId, `${label}.caseId must be a string`); + invariant(identity.caseId.length > 0, `${label}.caseId must not be empty`); + assertString(identity.condition, `${label}.condition must be a string`); + invariant( + identity.condition.length > 0, + `${label}.condition must not be empty`, + ); +} + +function parseTimestampMs(value: string, label: string): number { + assertString(value, `${label} must be a string`); + invariant(value.length > 0, `${label} must not be empty`); + + const timestampMs = Date.parse(value); + invariant( + Number.isFinite(timestampMs), + `${label} must be a valid ISO timestamp`, + ); + return timestampMs; +} + +function defaultScoreFromResult(result: CaseProgressResult): number | null { + const scoreValue = (result as { score?: unknown }).score; + if (scoreValue === undefined || scoreValue === null) { + return null; + } + if (typeof scoreValue === 'number') { + invariant( + Number.isFinite(scoreValue), + 'fulfilled score must be a finite number', + ); + return scoreValue; + } + if (typeof scoreValue === 'object') { + const totalValue = (scoreValue as { total?: unknown }).total; + if (totalValue === undefined || totalValue === null) { + return null; + } + invariant( + typeof totalValue === 'number' && Number.isFinite(totalValue), + 'fulfilled score.total must be a finite number', + ); + return totalValue; + } + + invariant( + false, + 'fulfilled score must be null, a finite number, or an object with numeric total', + ); +} + +function createCaseLabel(plannedCase: PlannedCase): string { + return `${plannedCase.caseId} (${plannedCase.condition})`; +} + +function createCaseState(plannedCase: PlannedCase): CaseProgressState { + return { + plannedCase: { + caseId: plannedCase.caseId, + condition: plannedCase.condition, + plannedTrials: plannedCase.plannedTrials, + }, + startedAt: null, + startedAtMs: null, + completedTrials: 0, + passed: 0, + failed: 0, + errored: 0, + scoreSum: 0, + scoreCount: 0, + finished: false, + }; +} + +function buildCaseStartWorkspace( + workspacePlan: ResolvedWorkspacePlan | undefined, +): ReporterEventPayloads['caseStart']['workspace'] { + if (workspacePlan === undefined) { + return undefined; + } + + assertString( + workspacePlan.presetId, + 'tracker workspacePlan.presetId must be a string', + ); + invariant( + workspacePlan.presetId.length > 0, + 'tracker workspacePlan.presetId must not be empty', + ); + if (workspacePlan.cwd !== undefined) { + assertString( + workspacePlan.cwd, + 'tracker workspacePlan.cwd must be a string', + ); + invariant( + workspacePlan.cwd.length > 0, + 'tracker workspacePlan.cwd must not be empty when provided', + ); + } + if (workspacePlan.env !== undefined) { + invariant( + !Array.isArray(workspacePlan.env), + 'tracker workspacePlan.env must be a record when provided', + ); + for (const [key, value] of Object.entries(workspacePlan.env)) { + assertString(key, 'tracker workspacePlan.env keys must be strings'); + assertString(value, `tracker workspacePlan.env.${key} must be a string`); + } + } + invariant( + Number.isInteger(workspacePlan.bootstrapCount) && + workspacePlan.bootstrapCount >= 0, + 'tracker workspacePlan.bootstrapCount must be a non-negative integer', + ); + invariant( + workspacePlan.bootstrapCount === workspacePlan.bootstrap.length, + 'tracker workspacePlan.bootstrapCount must match bootstrap length', + ); + + return { + presetId: workspacePlan.presetId, + ...(workspacePlan.cwd === undefined ? {} : { cwd: workspacePlan.cwd }), + ...(workspacePlan.env === undefined ? {} : { env: workspacePlan.env }), + bootstrapCount: workspacePlan.bootstrapCount, + }; +} + +export function computePlannedCases( + items: readonly CaseProgressIdentity[], +): Map { + const plannedCases = new Map(); + for (let index = 0; index < items.length; index += 1) { + const item = items[index]; + invariant( + item !== undefined, + `Missing work item at index ${String(index)}`, + ); + validateCaseIdentity(item, `items[${String(index)}]`); + + const key = buildCaseProgressKey(item); + const existing = plannedCases.get(key); + if (existing === undefined) { + plannedCases.set(key, { + caseId: item.caseId, + condition: item.condition, + plannedTrials: 1, + }); + continue; + } + + existing.plannedTrials += 1; + } + + return plannedCases; +} + +export class CaseProgressTracker< + TItem extends CaseProgressIdentity & CaseProgressWorkspaceCarrier, + TResult extends CaseProgressResult, +> { + private readonly runId: string; + private readonly lane: EvalLane; + private readonly dispatcher: CaseProgressDispatcher | undefined; + private readonly now: () => string; + private readonly getScore: (item: TItem, result: TResult) => number | null; + private readonly caseStates: Map; + + public constructor(options: CaseProgressTrackerOptions) { + assertString(options.runId, 'tracker runId must be a string'); + invariant(options.runId.length > 0, 'tracker runId must not be empty'); + assertString(options.lane, 'tracker lane must be a string'); + invariant(options.lane.length > 0, 'tracker lane must not be empty'); + invariant( + options.dispatcher === undefined || + typeof options.dispatcher.dispatch === 'function', + 'tracker dispatcher must expose a dispatch function or be undefined', + ); + invariant( + options.now === undefined || typeof options.now === 'function', + 'tracker now must be a function or undefined', + ); + invariant( + options.getScore === undefined || typeof options.getScore === 'function', + 'tracker getScore must be a function or undefined', + ); + + this.runId = options.runId; + this.lane = options.lane; + this.dispatcher = options.dispatcher; + this.now = options.now ?? (() => new Date().toISOString()); + this.getScore = + options.getScore ?? ((_item, result) => defaultScoreFromResult(result)); + this.caseStates = new Map(); + + for (const [key, plannedCase] of options.plannedCases.entries()) { + validateCaseIdentity(plannedCase, `plannedCases[${key}]`); + invariant( + Number.isInteger(plannedCase.plannedTrials) && + plannedCase.plannedTrials > 0, + `plannedCases[${key}].plannedTrials must be a positive integer`, + ); + invariant( + key === buildCaseProgressKey(plannedCase), + `plannedCases key mismatch for ${createCaseLabel(plannedCase)}`, + ); + this.caseStates.set(key, createCaseState(plannedCase)); + } + } + + public async onTrialStart(item: TItem): Promise { + const state = this.getState(item); + invariant( + !state.finished, + `Received trial start after case finish for ${createCaseLabel(state.plannedCase)}`, + ); + if (state.startedAt !== null) { + return; + } + + const startedAt = this.getTimestamp('tracker startedAt'); + state.startedAt = startedAt.iso; + state.startedAtMs = startedAt.ms; + + await this.dispatcher?.dispatch('caseStart', { + runId: this.runId, + lane: this.lane, + caseId: state.plannedCase.caseId, + condition: state.plannedCase.condition, + plannedTrials: state.plannedCase.plannedTrials, + startedAt: startedAt.iso, + workspace: buildCaseStartWorkspace(item.workspacePlan), + }); + } + + public async onTrialFinish( + item: TItem, + settled: SchedulerItemSettlement, + ): Promise { + const state = this.getState(item); + invariant( + state.startedAt !== null && state.startedAtMs !== null, + `Received trial finish before case start for ${createCaseLabel(state.plannedCase)}`, + ); + invariant( + !state.finished, + `Received trial finish after case finish for ${createCaseLabel(state.plannedCase)}`, + ); + + if (settled.status === 'fulfilled') { + if (settled.value.ok) { + state.passed += 1; + } else { + state.failed += 1; + } + + const score = this.getScore(item, settled.value); + if (score !== null) { + invariant( + Number.isFinite(score), + 'tracker getScore must return a finite number or null', + ); + state.scoreSum += score; + state.scoreCount += 1; + } + } else { + state.errored += 1; + } + + state.completedTrials += 1; + invariant( + state.completedTrials <= state.plannedCase.plannedTrials, + `Completed trials exceeded planned trials for ${createCaseLabel(state.plannedCase)}`, + ); + if (state.completedTrials !== state.plannedCase.plannedTrials) { + return; + } + + const completedAt = this.getTimestamp('tracker completedAt'); + invariant( + completedAt.ms >= state.startedAtMs, + `Completed timestamp must not be earlier than started timestamp for ${createCaseLabel(state.plannedCase)}`, + ); + state.finished = true; + + await this.dispatcher?.dispatch('caseFinish', { + runId: this.runId, + lane: this.lane, + caseId: state.plannedCase.caseId, + condition: state.plannedCase.condition, + startedAt: state.startedAt, + completedAt: completedAt.iso, + durationMs: completedAt.ms - state.startedAtMs, + passed: state.passed, + failed: state.failed, + errored: state.errored, + meanScore: + state.scoreCount === 0 ? null : state.scoreSum / state.scoreCount, + artifactPath: null, + reportPath: null, + }); + } + + private getState(item: TItem): CaseProgressState { + validateCaseIdentity(item, 'trial item'); + + const key = buildCaseProgressKey(item); + const state = this.caseStates.get(key); + invariant( + state !== undefined, + `Unknown planned case for ${item.caseId} (${item.condition})`, + ); + return state; + } + + private getTimestamp(label: string): { iso: string; ms: number } { + const iso = this.now(); + return { + iso, + ms: parseTimestampMs(iso, label), + }; + } +} diff --git a/evals/reporters/types.ts b/evals/reporters/types.ts new file mode 100644 index 00000000..c1c4422c --- /dev/null +++ b/evals/reporters/types.ts @@ -0,0 +1,202 @@ +import { z } from 'zod'; + +import { TokenReportSummarySchema } from '../lib/schemas.js'; + +const NonEmptyStringSchema = z.string().min(1); +const PositiveIntSchema = z.number().int().positive(); +const NonNegativeIntSchema = z.number().int().nonnegative(); +const FiniteNumberSchema = z.number().finite(); +const IsoTimestampSchema = z.iso.datetime(); +const StringListSchema = z.array(NonEmptyStringSchema); +const NullablePathSchema = NonEmptyStringSchema.nullable(); +const NullableStringSchema = NonEmptyStringSchema.nullable(); + +export const TrialStatusSchema = z.enum(['passed', 'failed', 'errored']); +export type TrialStatus = z.infer; + +export const RunStartEventSchema = z + .object({ + runId: NonEmptyStringSchema, + provider: NonEmptyStringSchema, + model: NonEmptyStringSchema, + lanes: StringListSchema, + conditions: StringListSchema, + totalTrials: PositiveIntSchema, + totalInvocations: PositiveIntSchema, + outputDir: NonEmptyStringSchema, + startedAt: IsoTimestampSchema, + }) + .strict(); +export type RunStartEvent = z.infer; + +export const LaneStartEventSchema = z + .object({ + runId: NonEmptyStringSchema, + lane: NonEmptyStringSchema, + caseIds: StringListSchema, + conditions: StringListSchema, + concurrency: PositiveIntSchema, + plannedItems: PositiveIntSchema, + startedAt: IsoTimestampSchema, + }) + .strict(); +export type LaneStartEvent = z.infer; + +const CaseStartWorkspaceSchema = z + .object({ + presetId: z.string(), + cwd: z.string().optional(), + env: z.record(z.string(), z.string()).optional(), + bootstrapCount: NonNegativeIntSchema, + }) + .strict(); + +export const CaseStartEventSchema = z + .object({ + runId: NonEmptyStringSchema, + lane: NonEmptyStringSchema, + caseId: NonEmptyStringSchema, + condition: NonEmptyStringSchema, + plannedTrials: PositiveIntSchema, + startedAt: IsoTimestampSchema, + workspace: CaseStartWorkspaceSchema.optional(), + }) + .strict(); +export type CaseStartEvent = z.infer; + +export const TrialStartEventSchema = z + .object({ + runId: NonEmptyStringSchema, + lane: NonEmptyStringSchema, + caseId: NonEmptyStringSchema, + condition: NonEmptyStringSchema, + trial: PositiveIntSchema, + startedAt: IsoTimestampSchema, + requestedOutputPath: NullablePathSchema, + requestedArtifactPath: NullablePathSchema, + }) + .strict(); +export type TrialStartEvent = z.infer; + +export const TrialFinishEventSchema = z + .object({ + runId: NonEmptyStringSchema, + lane: NonEmptyStringSchema, + caseId: NonEmptyStringSchema, + condition: NonEmptyStringSchema, + trial: PositiveIntSchema, + startedAt: IsoTimestampSchema, + completedAt: IsoTimestampSchema, + durationMs: NonNegativeIntSchema, + status: TrialStatusSchema, + ok: z.boolean(), + errorClass: NullableStringSchema, + errorMessage: NullableStringSchema, + score: FiniteNumberSchema.nullable(), + transcriptPath: NullablePathSchema, + stdoutPath: NullablePathSchema, + stderrPath: NullablePathSchema, + eventLogPath: NullablePathSchema, + bundlePath: NullablePathSchema, + artifactManifestPath: NullablePathSchema, + }) + .strict(); +export type TrialFinishEvent = z.infer; + +export const CaseFinishEventSchema = z + .object({ + runId: NonEmptyStringSchema, + lane: NonEmptyStringSchema, + caseId: NonEmptyStringSchema, + condition: NonEmptyStringSchema, + startedAt: IsoTimestampSchema, + completedAt: IsoTimestampSchema, + durationMs: NonNegativeIntSchema, + passed: NonNegativeIntSchema, + failed: NonNegativeIntSchema, + errored: NonNegativeIntSchema, + meanScore: FiniteNumberSchema.nullable(), + artifactPath: NullablePathSchema, + reportPath: NullablePathSchema, + }) + .strict(); +export type CaseFinishEvent = z.infer; + +export const LaneFinishEventSchema = z + .object({ + runId: NonEmptyStringSchema, + lane: NonEmptyStringSchema, + startedAt: IsoTimestampSchema, + completedAt: IsoTimestampSchema, + durationMs: NonNegativeIntSchema, + total: NonNegativeIntSchema, + passed: NonNegativeIntSchema, + failed: NonNegativeIntSchema, + errored: NonNegativeIntSchema, + }) + .strict(); +export type LaneFinishEvent = z.infer; + +export const RunFinishLaneErrorSchema = z + .object({ + lane: NonEmptyStringSchema, + message: NonEmptyStringSchema, + }) + .strict(); +export type RunFinishLaneError = z.infer; + +export const RunFinishEventSchema = z + .object({ + runId: NonEmptyStringSchema, + startedAt: IsoTimestampSchema, + completedAt: IsoTimestampSchema, + durationMs: NonNegativeIntSchema, + total: NonNegativeIntSchema, + passed: NonNegativeIntSchema, + failed: NonNegativeIntSchema, + errored: NonNegativeIntSchema, + laneErrors: z.array(RunFinishLaneErrorSchema), + runDir: NonEmptyStringSchema, + reportJsonPath: NullablePathSchema, + reportMarkdownPath: NullablePathSchema, + tokenReport: TokenReportSummarySchema.optional(), + }) + .strict(); +export type RunFinishEvent = z.infer; + +export interface ReporterEventPayloads { + runStart: RunStartEvent; + laneStart: LaneStartEvent; + caseStart: CaseStartEvent; + trialStart: TrialStartEvent; + trialFinish: TrialFinishEvent; + caseFinish: CaseFinishEvent; + laneFinish: LaneFinishEvent; + runFinish: RunFinishEvent; +} + +export type ReporterEventName = keyof ReporterEventPayloads; +export type ReporterHook = (event: TEvent) => Promise | void; + +export interface Reporter { + name: string; + onRunStart?: ReporterHook; + onLaneStart?: ReporterHook; + onCaseStart?: ReporterHook; + onTrialStart?: ReporterHook; + onTrialFinish?: ReporterHook; + onCaseFinish?: ReporterHook; + onLaneFinish?: ReporterHook; + onRunFinish?: ReporterHook; +} + +export const EVENT_SCHEMAS = { + runStart: RunStartEventSchema, + laneStart: LaneStartEventSchema, + caseStart: CaseStartEventSchema, + trialStart: TrialStartEventSchema, + trialFinish: TrialFinishEventSchema, + caseFinish: CaseFinishEventSchema, + laneFinish: LaneFinishEventSchema, + runFinish: RunFinishEventSchema, +} satisfies { [K in ReporterEventName]: z.ZodType }; diff --git a/evals/run.ts b/evals/run.ts index c55c69c5..218573fd 100644 --- a/evals/run.ts +++ b/evals/run.ts @@ -1,4 +1,4 @@ -import { mkdir, readFile, writeFile } from 'node:fs/promises'; +import { mkdir, readFile } from 'node:fs/promises'; import { join, resolve } from 'node:path'; import process from 'node:process'; import { fileURLToPath, pathToFileURL } from 'node:url'; @@ -15,9 +15,12 @@ import { import { buildBaselineComparison, generateJsonReport, - generateMarkdownReport, } from './lib/reporting.js'; import { JsonReportSchema, RunMetadataSchema } from './lib/schemas.js'; +import { + aggregateTokenRecords, + type RawTokenRecord, +} from './lib/tokenAggregation.js'; import { buildWorkItemKey, type EvalCase, @@ -27,10 +30,31 @@ import { type ProviderRuntimeInfo, type RunMetadata, type SkillCondition, + type TokenReportSummary, } from './lib/types.js'; import { getAllPromptCases, runPromptLane } from './prompt/runner.js'; import { createProvider, SUPPORTED_PROVIDER_IDS } from './providers/base.js'; +import { ConsoleReporter } from './reporters/console.js'; +import { ReporterDispatcher } from './reporters/dispatch.js'; +import { + FinalReportReporter, + type FinalReportInputs, +} from './reporters/final-report.js'; +import { JsonlReporter } from './reporters/jsonl.js'; +import { compareSnapshotRecords } from './snapshots/compare.js'; +import { caseFingerprint } from './snapshots/fingerprint.js'; +import { + buildSnapshotLogicalKey, + type SnapshotEntry, +} from './snapshots/schema.js'; +import { loadSnapshotFile, writeSnapshotFile } from './snapshots/store.js'; +import { registerBuiltinPresets } from './workspaces/builtins.js'; import type { EvalProvider } from './providers/base.js'; +import type { + SnapshotCheckCase, + SnapshotCheckReport, + SnapshotCheckSummary, +} from './snapshots/schemas/report.js'; const DEFAULT_TOTAL_TRIALS = 1; const EVAL_LANES: readonly EvalLane[] = ['prompt', 'execution', 'dogfood']; @@ -45,6 +69,10 @@ const CLI_PROVIDER_IDS = SUPPORTED_PROVIDER_IDS.filter( (providerId) => providerId !== 'recording', ); const CLI_PROVIDER_SET = new Set(CLI_PROVIDER_IDS); +const REPORTER_IDS = ['final', 'console', 'jsonl'] as const; +const REPORTER_SET = new Set(REPORTER_IDS); +export type ReporterId = (typeof REPORTER_IDS)[number]; +type SnapshotMode = 'off' | 'update' | 'check'; const HELP_TEXT = [ 'Usage: npx tsx evals/run.ts [options]', @@ -57,6 +85,13 @@ const HELP_TEXT = [ ' --condition Skill condition (none, self-load, preloaded, stale, all). Default: all. May be repeated.', ' --case Run specific case(s) by ID. May be repeated.', ' --output Output directory. Default: evals/reports/{timestamp}', + ' --snapshot-update Write token usage snapshots for the selected cases', + ' --snapshot-check Compare token usage against saved snapshots', + ' --snapshot-threshold Regression threshold percent for snapshot checks. Default: 20', + ' --snapshot-dir Snapshot directory. Default: /snapshots', + ` --reporter Reporter to enable (${REPORTER_IDS.join(', ')}). May be repeated.`, + ' --reporter-output Output path for reporters that write files (required for jsonl)', + ' --progress Enable progress reporting on stderr', ' --json Print JSON summary only', ' --verbose Print verbose progress logs to stderr', ' --dry-run List cases that would run without invoking providers', @@ -84,16 +119,34 @@ interface CliOptions { lane?: string; conditions: string[]; caseIds: string[]; + reporters: string[]; outputDir?: string; + reporterOutput?: string; concurrency?: string; trials?: string; compareBaseline?: string; + snapshotThreshold?: string; + snapshotDir?: string; + snapshotUpdate: boolean; + snapshotCheck: boolean; json: boolean; verbose: boolean; dryRun: boolean; + progress: boolean; help: boolean; } +export interface ResolvedReporterSelection { + reporterNames: ReporterId[]; + reporterOutputPath?: string; +} + +interface ResolvedSnapshotOptions { + snapshotMode: SnapshotMode; + snapshotThresholdPercent: number; + snapshotDir: string; +} + interface ResolvedCliOptions { providerId: string; modelId?: string; @@ -103,6 +156,11 @@ interface ResolvedCliOptions { caseIds: string[]; outputBaseDir: string; compareBaselinePath?: string; + snapshotMode: SnapshotMode; + snapshotThresholdPercent: number; + snapshotDir: string; + reporterNames: ReporterId[]; + reporterOutputPath?: string; concurrency: number; totalTrials: number; json: boolean; @@ -111,6 +169,7 @@ interface ResolvedCliOptions { activeLanes: EvalLane[]; activeConditions: SkillCondition[]; selectedCases: CaseSelection[]; + compiledCasesBySelection: ReadonlyMap; totalInvocations: number; } @@ -165,6 +224,10 @@ function isCliProviderId(value: string): value is CliProviderId { return CLI_PROVIDER_SET.has(value); } +function isReporterId(value: string): value is ReporterId { + return REPORTER_SET.has(value); +} + function compareStrings(left: string, right: string): number { return left.localeCompare(right); } @@ -173,6 +236,10 @@ function compareLane(left: EvalLane, right: EvalLane): number { return EVAL_LANES.indexOf(left) - EVAL_LANES.indexOf(right); } +function compareCondition(left: SkillCondition, right: SkillCondition): number { + return SKILL_CONDITIONS.indexOf(left) - SKILL_CONDITIONS.indexOf(right); +} + function compareCaseSelection( left: CaseSelection, right: CaseSelection, @@ -184,6 +251,12 @@ function compareCaseSelection( return compareStrings(left.caseId, right.caseId); } +function buildSelectedCaseKey(lane: EvalLane, caseId: string): string { + assertString(caseId, 'caseId must be a string'); + invariant(caseId.length > 0, 'caseId must not be empty'); + return `${lane}::${caseId}`; +} + function formatTimestampForPath(date: Date): string { const isoTimestamp = date.toISOString().replace(/\.\d{3}Z$/u, 'Z'); return isoTimestamp.replace(/[:-]/gu, '').replace('T', '-'); @@ -228,9 +301,13 @@ export function parseCliArgs(argumentsList: readonly string[]): CliOptions { const options: CliOptions = { conditions: [], caseIds: [], + reporters: [], + snapshotUpdate: false, + snapshotCheck: false, json: false, verbose: false, dryRun: false, + progress: false, help: false, }; @@ -257,6 +334,63 @@ export function parseCliArgs(argumentsList: readonly string[]): CliOptions { options.dryRun = true; continue; } + if (argument === '--progress') { + invariant(!options.progress, '--progress may only be provided once'); + options.progress = true; + continue; + } + if (argument === '--snapshot-update') { + invariant( + !options.snapshotUpdate, + '--snapshot-update may only be provided once', + ); + options.snapshotUpdate = true; + continue; + } + if (argument === '--snapshot-check') { + invariant( + !options.snapshotCheck, + '--snapshot-check may only be provided once', + ); + options.snapshotCheck = true; + continue; + } + if ( + argument === '--snapshot-threshold' || + argument.startsWith('--snapshot-threshold=') + ) { + const parsed = parseOptionValue( + argument, + '--snapshot-threshold', + argumentsList, + index, + ); + invariant( + options.snapshotThreshold === undefined, + '--snapshot-threshold may only be set once', + ); + options.snapshotThreshold = parsed.value; + index = parsed.nextIndex; + continue; + } + if ( + argument === '--snapshot-dir' || + argument.startsWith('--snapshot-dir=') + ) { + const parsed = parseOptionValue( + argument, + '--snapshot-dir', + argumentsList, + index, + ); + invariant( + options.snapshotDir === undefined, + '--snapshot-dir may only be set once', + ); + options.snapshotDir = parsed.value; + index = parsed.nextIndex; + continue; + } if (argument === '--concurrency' || argument.startsWith('--concurrency=')) { const parsed = parseOptionValue( argument, @@ -383,6 +517,35 @@ export function parseCliArgs(argumentsList: readonly string[]): CliOptions { index = parsed.nextIndex; continue; } + if (argument === '--reporter' || argument.startsWith('--reporter=')) { + const parsed = parseOptionValue( + argument, + '--reporter', + argumentsList, + index, + ); + options.reporters.push(parsed.value); + index = parsed.nextIndex; + continue; + } + if ( + argument === '--reporter-output' || + argument.startsWith('--reporter-output=') + ) { + const parsed = parseOptionValue( + argument, + '--reporter-output', + argumentsList, + index, + ); + invariant( + options.reporterOutput === undefined, + '--reporter-output may only be set once', + ); + options.reporterOutput = parsed.value; + index = parsed.nextIndex; + continue; + } invariant(!argument.startsWith('-'), `Unknown option: ${argument}`); invariant(false, `Unexpected positional argument: ${argument}`); @@ -532,6 +695,95 @@ function resolveOutputBaseDir( return resolve(repoRoot, outputDir); } +function resolveSnapshotOptions( + repoRoot: string, + outputBaseDir: string, + options: Pick< + CliOptions, + 'snapshotCheck' | 'snapshotDir' | 'snapshotThreshold' | 'snapshotUpdate' + >, +): ResolvedSnapshotOptions { + invariant( + !(options.snapshotUpdate && options.snapshotCheck), + '--snapshot-update and --snapshot-check may not be combined', + ); + + const snapshotThresholdValue = + options.snapshotThreshold === undefined ? '20' : options.snapshotThreshold; + const snapshotThresholdPercent = Number(snapshotThresholdValue); + invariant( + Number.isFinite(snapshotThresholdPercent) && + snapshotThresholdPercent >= 0 && + snapshotThresholdPercent <= 100, + `--snapshot-threshold must be a number between 0 and 100, got: ${snapshotThresholdValue}`, + ); + + const snapshotDirValue = resolveOptionalStringOption( + options.snapshotDir, + '--snapshot-dir', + ); + + return { + snapshotMode: options.snapshotUpdate + ? 'update' + : options.snapshotCheck + ? 'check' + : 'off', + snapshotThresholdPercent, + snapshotDir: + snapshotDirValue === undefined + ? resolve(outputBaseDir, 'snapshots') + : resolve(repoRoot, snapshotDirValue), + }; +} + +export function resolveReporterSelection( + repoRoot: string, + options: Pick, +): ResolvedReporterSelection { + assertString(repoRoot, 'repoRoot must be a string'); + invariant(repoRoot.length > 0, 'repoRoot must not be empty'); + invariant( + Array.isArray(options.reporters), + '--reporter values must be an array', + ); + invariant( + typeof options.progress === 'boolean', + '--progress must be a boolean', + ); + + const explicit = options.reporters ?? []; + const ordered = explicit.length === 0 ? ['final'] : [...explicit]; + if (options.progress && !ordered.includes('console')) { + ordered.push('console'); + } + + for (const reporterName of ordered) { + assertString(reporterName, '--reporter values must be strings'); + invariant(reporterName.length > 0, '--reporter values must not be empty'); + invariant( + isReporterId(reporterName), + `Unsupported reporter: ${reporterName}. Expected one of ${REPORTER_IDS.join(', ')}`, + ); + } + + const reporterOutputPath = resolveOptionalStringOption( + options.reporterOutput, + '--reporter-output', + ); + invariant( + !ordered.includes('jsonl') || reporterOutputPath !== undefined, + '--reporter jsonl requires --reporter-output', + ); + + return { + reporterNames: ordered as ReporterId[], + ...(reporterOutputPath === undefined + ? {} + : { reporterOutputPath: resolve(repoRoot, reporterOutputPath) }), + }; +} + function getCasesForLane(lane: EvalLane): EvalCase[] { switch (lane) { case 'prompt': @@ -555,6 +807,7 @@ function buildCaseSelections( activeConditions: SkillCondition[]; activeLanes: EvalLane[]; cases: CaseSelection[]; + compiledCasesBySelection: ReadonlyMap; totalInvocations: number; } { const availableCases = lanes.flatMap((lane) => getCasesForLane(lane)); @@ -578,6 +831,19 @@ function buildCaseSelections( 'No eval cases matched the requested lanes', ); + const compiledCasesBySelection = new Map(); + for (const compiledCase of selectedCases) { + const selectionKey = buildSelectedCaseKey( + compiledCase.lane, + compiledCase.id, + ); + invariant( + !compiledCasesBySelection.has(selectionKey), + `Duplicate compiled eval case selection: ${selectionKey}`, + ); + compiledCasesBySelection.set(selectionKey, compiledCase); + } + const requestedConditions = new Set(conditions); const matrixEntries = buildConditionMatrix(selectedCases, [ providerId, @@ -589,7 +855,7 @@ function buildCaseSelections( const groupedSelections = new Map(); for (const entry of matrixEntries) { - const key = `${entry.lane}::${entry.caseId}`; + const key = buildSelectedCaseKey(entry.lane, entry.caseId); const existing = groupedSelections.get(key); if (existing === undefined) { groupedSelections.set(key, { @@ -611,10 +877,7 @@ function buildCaseSelections( const cases = [...groupedSelections.values()].sort(compareCaseSelection); for (const selection of cases) { - selection.conditions.sort( - (left, right) => - SKILL_CONDITIONS.indexOf(left) - SKILL_CONDITIONS.indexOf(right), - ); + selection.conditions.sort(compareCondition); } const activeLanes = EVAL_LANES.filter((lane) => @@ -631,6 +894,7 @@ function buildCaseSelections( activeConditions, activeLanes, cases, + compiledCasesBySelection, totalInvocations, }; } @@ -654,10 +918,16 @@ function buildResolvedOptions( : [...requestedConditions]; const caseIds = resolveRequestedCaseIds(options.caseIds); const outputBaseDir = resolveOutputBaseDir(repoRoot, options.outputDir); + const snapshotOptions = resolveSnapshotOptions( + repoRoot, + outputBaseDir, + options, + ); const compareBaselinePath = resolveOptionalStringOption( options.compareBaseline, '--compare-baseline', ); + const reporterSelection = resolveReporterSelection(repoRoot, options); const concurrency = resolveConcurrency(options.concurrency); const totalTrials = resolveTotalTrials(options.trials); const selection = buildCaseSelections( @@ -676,9 +946,16 @@ function buildResolvedOptions( requestedConditions: requestedConditionFilters, caseIds, outputBaseDir, + snapshotMode: snapshotOptions.snapshotMode, + snapshotThresholdPercent: snapshotOptions.snapshotThresholdPercent, + snapshotDir: snapshotOptions.snapshotDir, ...(compareBaselinePath === undefined ? {} : { compareBaselinePath: resolve(repoRoot, compareBaselinePath) }), + reporterNames: reporterSelection.reporterNames, + ...(reporterSelection.reporterOutputPath === undefined + ? {} + : { reporterOutputPath: reporterSelection.reporterOutputPath }), concurrency, totalTrials, json: options.json, @@ -687,6 +964,7 @@ function buildResolvedOptions( activeLanes: selection.activeLanes, activeConditions: selection.activeConditions, selectedCases: selection.cases, + compiledCasesBySelection: selection.compiledCasesBySelection, totalInvocations: selection.totalInvocations, }; } @@ -899,8 +1177,8 @@ function buildSummary( results: EvalResult[], laneErrors: readonly LaneErrorSummary[], metadata: RunMetadata, - jsonReportPath: string, - markdownReportPath: string, + jsonReportPath: string | undefined, + markdownReportPath: string | undefined, runDir: string, ): RunSummary { const passed = results.filter((result) => result.ok).length; @@ -921,8 +1199,8 @@ function buildSummary( failed, outputBaseDir: options.outputBaseDir, runDir, - jsonReportPath, - markdownReportPath, + ...(jsonReportPath === undefined ? {} : { jsonReportPath }), + ...(markdownReportPath === undefined ? {} : { markdownReportPath }), laneErrors: [...laneErrors], dryRun: false, selectedCases: options.selectedCases, @@ -995,11 +1273,415 @@ async function detectProviderRuntime( } } +function createReporterDispatcher( + options: Pick< + ResolvedCliOptions, + 'reporterNames' | 'reporterOutputPath' | 'verbose' + >, + getFinalReportInputs: () => FinalReportInputs | null, +): ReporterDispatcher { + const reporters = options.reporterNames.map((reporterName) => { + switch (reporterName) { + case 'console': + return new ConsoleReporter({ verbose: options.verbose }); + case 'jsonl': + invariant( + options.reporterOutputPath !== undefined, + 'jsonl reporter requires an output path', + ); + return new JsonlReporter({ outputPath: options.reporterOutputPath }); + case 'final': + return new FinalReportReporter({ getFinalReportInputs }); + default: + return reporterName satisfies never; + } + }); + + return new ReporterDispatcher(reporters); +} + +function resolveReporterModel(metadata: RunMetadata): string { + return metadata.models[0] ?? 'unknown'; +} + +function buildRunResultCounts(results: readonly EvalResult[]): { + total: number; + passed: number; + failed: number; + errored: number; +} { + return results.reduce( + (counts, result) => { + counts.total += 1; + if (result.ok) { + counts.passed += 1; + } else if (result.errorClass === undefined) { + counts.failed += 1; + } else { + counts.errored += 1; + } + return counts; + }, + { total: 0, passed: 0, failed: 0, errored: 0 }, + ); +} + +interface ReducedSnapshotRecord { + provider: string; + model: string; + lane: EvalLane; + caseId: string; + condition: SkillCondition; + caseFingerprint: string; + inputTokens: number; + outputTokens: number; + totalTokens: number; + cachedTokens?: number; + trials: number; +} + +function compareSnapshotIdentity( + left: Pick< + ReducedSnapshotRecord, + 'provider' | 'model' | 'lane' | 'caseId' | 'condition' | 'caseFingerprint' + >, + right: Pick< + ReducedSnapshotRecord, + 'provider' | 'model' | 'lane' | 'caseId' | 'condition' | 'caseFingerprint' + >, +): number { + const providerComparison = compareStrings(left.provider, right.provider); + if (providerComparison !== 0) { + return providerComparison; + } + const modelComparison = compareStrings(left.model, right.model); + if (modelComparison !== 0) { + return modelComparison; + } + const laneComparison = compareLane(left.lane, right.lane); + if (laneComparison !== 0) { + return laneComparison; + } + const caseComparison = compareStrings(left.caseId, right.caseId); + if (caseComparison !== 0) { + return caseComparison; + } + const conditionComparison = compareCondition(left.condition, right.condition); + if (conditionComparison !== 0) { + return conditionComparison; + } + return compareStrings(left.caseFingerprint, right.caseFingerprint); +} + +function buildRawTokenRecords( + results: readonly EvalResult[], + metadata: RunMetadata, + compiledCasesBySelection: ReadonlyMap, +): RawTokenRecord[] { + const fingerprintBySelection = new Map(); + const rawTokenRecords: RawTokenRecord[] = []; + + for (const result of results) { + const tokenUsage = result.normalizedOutput.tokenUsage; + if (tokenUsage === undefined) { + continue; + } + + const selectionKey = buildSelectedCaseKey(result.lane, result.caseId); + let fingerprint = fingerprintBySelection.get(selectionKey); + if (fingerprint === undefined) { + const compiledCase = compiledCasesBySelection.get(selectionKey); + invariant( + compiledCase !== undefined, + `Missing compiled eval case for token record: ${selectionKey}`, + ); + fingerprint = caseFingerprint(compiledCase); + fingerprintBySelection.set(selectionKey, fingerprint); + } + + rawTokenRecords.push({ + provider: result.providerId, + model: result.modelId ?? metadata.models[0] ?? 'unknown', + lane: result.lane, + caseId: result.caseId, + condition: result.condition, + caseFingerprint: fingerprint, + usage: tokenUsage, + }); + } + + return rawTokenRecords; +} + +function reduceSnapshotTokenRecords( + records: readonly RawTokenRecord[], +): ReducedSnapshotRecord[] { + const groupedRecords = new Map< + string, + ReducedSnapshotRecord & { + cachedTokensTotal: number; + sawMissingCachedTokens: boolean; + } + >(); + + for (const record of records) { + const key = JSON.stringify([ + record.provider, + record.model, + record.lane, + record.caseId, + record.condition, + record.caseFingerprint, + ]); + const existing = groupedRecords.get(key); + if (existing === undefined) { + groupedRecords.set(key, { + provider: record.provider, + model: record.model, + lane: record.lane, + caseId: record.caseId, + condition: record.condition, + caseFingerprint: record.caseFingerprint, + inputTokens: record.usage.inputTokens, + outputTokens: record.usage.outputTokens, + totalTokens: record.usage.totalTokens, + trials: 1, + cachedTokensTotal: record.usage.cachedTokens ?? 0, + sawMissingCachedTokens: record.usage.cachedTokens === undefined, + }); + continue; + } + + existing.inputTokens += record.usage.inputTokens; + existing.outputTokens += record.usage.outputTokens; + existing.totalTokens += record.usage.totalTokens; + existing.trials += 1; + if (record.usage.cachedTokens === undefined) { + existing.sawMissingCachedTokens = true; + } else { + existing.cachedTokensTotal += record.usage.cachedTokens; + } + } + + return [...groupedRecords.values()] + .map(({ cachedTokensTotal, sawMissingCachedTokens, ...record }) => ({ + ...record, + ...(sawMissingCachedTokens ? {} : { cachedTokens: cachedTokensTotal }), + })) + .sort(compareSnapshotIdentity); +} + +function buildValidSnapshotLogicalKeys( + options: Pick< + ResolvedCliOptions, + 'compiledCasesBySelection' | 'selectedCases' + >, +): ReadonlySet { + const validCurrentKeys = new Set(); + + for (const selection of options.selectedCases) { + const selectionKey = buildSelectedCaseKey(selection.lane, selection.caseId); + const compiledCase = options.compiledCasesBySelection.get(selectionKey); + invariant( + compiledCase !== undefined, + `Missing compiled eval case for snapshot selection: ${selectionKey}`, + ); + const fingerprint = caseFingerprint(compiledCase); + for (const condition of selection.conditions) { + validCurrentKeys.add( + buildSnapshotLogicalKey({ + lane: selection.lane, + caseId: selection.caseId, + condition, + caseFingerprint: fingerprint, + }), + ); + } + } + + return validCurrentKeys; +} + +function mergeSnapshotCheckReports( + reports: readonly SnapshotCheckReport[], + regressionThresholdPercent: number, +): SnapshotCheckReport { + const summary: SnapshotCheckSummary = { + total: 0, + new: 0, + orphaned: 0, + unchanged: 0, + improved: 0, + regressed: 0, + }; + const cases: SnapshotCheckCase[] = []; + + for (const report of reports) { + invariant( + report.regressionThresholdPercent === regressionThresholdPercent, + 'snapshot check regression threshold must stay consistent across groups', + ); + summary.total += report.summary.total; + summary.new += report.summary.new; + summary.orphaned += report.summary.orphaned; + summary.unchanged += report.summary.unchanged; + summary.improved += report.summary.improved; + summary.regressed += report.summary.regressed; + cases.push(...report.cases); + } + + cases.sort(compareSnapshotIdentity); + return { + regressionThresholdPercent, + cases, + summary, + }; +} + +async function buildTokenReport( + results: readonly EvalResult[], + metadata: RunMetadata, + options: Pick< + ResolvedCliOptions, + | 'activeLanes' + | 'compiledCasesBySelection' + | 'selectedCases' + | 'snapshotDir' + | 'snapshotMode' + | 'snapshotThresholdPercent' + >, +): Promise { + const rawTokenRecords = buildRawTokenRecords( + results, + metadata, + options.compiledCasesBySelection, + ); + if (rawTokenRecords.length === 0) { + return undefined; + } + + const tokenReport = aggregateTokenRecords( + rawTokenRecords, + options.activeLanes, + ); + invariant( + tokenReport !== undefined, + 'token report must exist for token records', + ); + if (options.snapshotMode === 'off') { + return tokenReport; + } + + const reducedSnapshotRecords = reduceSnapshotTokenRecords(rawTokenRecords); + const groupedSnapshotRecords = new Map< + string, + { + provider: string; + model: string; + records: ReducedSnapshotRecord[]; + } + >(); + for (const record of reducedSnapshotRecords) { + const groupKey = JSON.stringify([record.provider, record.model]); + const existingGroup = groupedSnapshotRecords.get(groupKey); + if (existingGroup === undefined) { + groupedSnapshotRecords.set(groupKey, { + provider: record.provider, + model: record.model, + records: [record], + }); + continue; + } + existingGroup.records.push(record); + } + + const sortedGroups = [...groupedSnapshotRecords.values()].sort( + (left, right) => { + const providerComparison = compareStrings(left.provider, right.provider); + if (providerComparison !== 0) { + return providerComparison; + } + return compareStrings(left.model, right.model); + }, + ); + + if (options.snapshotMode === 'update') { + const createdAtMs = Date.now(); + for (const group of sortedGroups) { + const entries: SnapshotEntry[] = group.records.map((record) => ({ + provider: record.provider, + model: record.model, + lane: record.lane, + caseId: record.caseId, + condition: record.condition, + caseFingerprint: record.caseFingerprint, + inputTokens: record.inputTokens, + outputTokens: record.outputTokens, + totalTokens: record.totalTokens, + createdAtMs, + ...(record.cachedTokens === undefined + ? {} + : { cachedTokens: record.cachedTokens }), + })); + await writeSnapshotFile({ + snapshotDir: options.snapshotDir, + provider: group.provider, + model: group.model, + entries, + }); + } + return tokenReport; + } + + const validCurrentKeys = buildValidSnapshotLogicalKeys(options); + const snapshotReports: SnapshotCheckReport[] = []; + for (const group of sortedGroups) { + const snapshotFile = await loadSnapshotFile({ + snapshotDir: options.snapshotDir, + provider: group.provider, + model: group.model, + validCurrentKeys, + }); + snapshotReports.push( + compareSnapshotRecords({ + currentRecords: group.records.map((record) => ({ + provider: record.provider, + model: record.model, + lane: record.lane, + caseId: record.caseId, + condition: record.condition, + caseFingerprint: record.caseFingerprint, + totalTokens: record.totalTokens, + })), + snapshotRecords: snapshotFile.entries.map((entry) => ({ + provider: entry.provider, + model: entry.model, + lane: entry.lane, + caseId: entry.caseId, + condition: entry.condition, + caseFingerprint: entry.caseFingerprint, + totalTokens: entry.totalTokens, + })), + regressionThresholdPercent: options.snapshotThresholdPercent, + }), + ); + } + + return { + ...tokenReport, + snapshotCheck: mergeSnapshotCheckReports( + snapshotReports, + options.snapshotThresholdPercent, + ), + }; +} + async function runLane( lane: EvalLane, provider: EvalProvider, metadata: RunMetadata, options: ResolvedCliOptions, + reporter: ReporterDispatcher, ): Promise { const caseFilter = options.selectedCases .filter((selection) => selection.lane === lane) @@ -1015,18 +1697,21 @@ async function runLane( conditions: options.activeConditions, caseFilter, concurrency: options.concurrency, + reporter, }); case 'execution': return runExecutionLane(provider, metadata, { conditions: options.activeConditions, caseFilter, concurrency: options.concurrency, + reporter, }); case 'dogfood': return runDogfoodLane(provider, metadata, { conditions: options.activeConditions, caseFilter, concurrency: options.concurrency, + reporter, }); default: return lane satisfies never; @@ -1048,6 +1733,8 @@ function recordLaneFailure( export async function runEvalCli( argumentsList: readonly string[], ): Promise { + registerBuiltinPresets(); + const repoRoot = resolve(fileURLToPath(new URL('..', import.meta.url))); const parsedOptions = parseCliArgs(argumentsList); if (parsedOptions.help) { @@ -1056,6 +1743,10 @@ export async function runEvalCli( } const options = buildResolvedOptions(repoRoot, parsedOptions); + let finalReportInputs: FinalReportInputs | null = null; + const getFinalReportInputs = (): FinalReportInputs | null => + finalReportInputs; + const reporter = createReporterDispatcher(options, getFinalReportInputs); if (options.dryRun) { writeDryRunSummary(options); return 0; @@ -1076,12 +1767,30 @@ export async function runEvalCli( ); const results: EvalResult[] = []; const laneErrors: LaneErrorSummary[] = []; + const runStartedAtMs = Date.parse(metadata.createdAt); + await reporter.dispatch('runStart', { + runId: metadata.runId, + provider: options.providerId, + model: resolveReporterModel(metadata), + lanes: metadata.lanes, + conditions: metadata.conditions, + totalTrials: options.totalTrials, + totalInvocations: options.totalInvocations, + outputDir: options.outputBaseDir, + startedAt: metadata.createdAt, + }); if (options.concurrency > 1) { const lanePromises = options.activeLanes.map(async (lane) => { try { logVerbose(options, `Running ${lane} lane`); - const laneResults = await runLane(lane, provider, metadata, options); + const laneResults = await runLane( + lane, + provider, + metadata, + options, + reporter, + ); logVerbose( options, `Completed ${lane} lane with ${String(laneResults.length)} result(s)`, @@ -1123,7 +1832,13 @@ export async function runEvalCli( for (const lane of options.activeLanes) { logVerbose(options, `Running ${lane} lane`); try { - const laneResults = await runLane(lane, provider, metadata, options); + const laneResults = await runLane( + lane, + provider, + metadata, + options, + reporter, + ); results.push(...laneResults); logVerbose( options, @@ -1156,22 +1871,7 @@ export async function runEvalCli( await loadBaselineReport(options.compareBaselinePath), candidateCoreReport, ); - const jsonReport = - baselineComparison === undefined - ? candidateCoreReport - : generateJsonReport( - results, - metadata, - comparisonMetrics, - baselineComparison, - ); - const markdownReport = generateMarkdownReport( - results, - metadata, - comparisonMetrics, - baselineComparison, - ); - + const tokenReport = await buildTokenReport(results, metadata, options); const artifactStore = new EvalArtifactStore(options.outputBaseDir); await artifactStore.saveRunMetadata(metadata.runId, metadata); const runDir = artifactStore.runDir(metadata.runId); @@ -1179,20 +1879,39 @@ export async function runEvalCli( const jsonReportPath = join(runDir, 'report.json'); const markdownReportPath = join(runDir, 'report.md'); - await writeFile( - jsonReportPath, - `${JSON.stringify(jsonReport, null, 2)}\n`, - 'utf8', - ); - await writeFile(markdownReportPath, markdownReport, 'utf8'); + const finalReporterEnabled = options.reporterNames.includes('final'); + if (finalReporterEnabled) { + finalReportInputs = { + results, + metadata, + comparisonMetrics, + ...(baselineComparison === undefined ? {} : { baselineComparison }), + jsonReportPath, + markdownReportPath, + }; + } + + const runCompletedAt = new Date().toISOString(); + await reporter.dispatch('runFinish', { + runId: metadata.runId, + startedAt: metadata.createdAt, + completedAt: runCompletedAt, + durationMs: Math.max(0, Date.parse(runCompletedAt) - runStartedAtMs), + ...buildRunResultCounts(results), + laneErrors: [...laneErrors], + runDir, + reportJsonPath: finalReporterEnabled ? jsonReportPath : null, + reportMarkdownPath: finalReporterEnabled ? markdownReportPath : null, + ...(tokenReport === undefined ? {} : { tokenReport }), + }); const summary = buildSummary( options, results, laneErrors, metadata, - jsonReportPath, - markdownReportPath, + finalReporterEnabled ? jsonReportPath : undefined, + finalReporterEnabled ? markdownReportPath : undefined, runDir, ); diff --git a/evals/snapshots/compare.ts b/evals/snapshots/compare.ts new file mode 100644 index 00000000..6223ccc7 --- /dev/null +++ b/evals/snapshots/compare.ts @@ -0,0 +1,204 @@ +import { invariant } from '../../src/util/assert.js'; +import { SnapshotEntrySchema } from './schema.js'; +import type { SnapshotEntry } from './schema.js'; +import type { + SnapshotCheckCase, + SnapshotCheckReport, + SnapshotCheckSummary, +} from './schemas/report.js'; +export type { + SnapshotCheckCase, + SnapshotCheckOutcome, + SnapshotCheckReport, + SnapshotCheckSummary, +} from './schemas/report.js'; + +export interface SnapshotComparableRecord extends Pick< + SnapshotEntry, + | 'provider' + | 'model' + | 'lane' + | 'caseId' + | 'condition' + | 'caseFingerprint' + | 'totalTokens' +> {} + +export interface CompareSnapshotRecordsOptions { + currentRecords: readonly SnapshotComparableRecord[]; + snapshotRecords: readonly SnapshotComparableRecord[]; + regressionThresholdPercent: number; +} + +const SnapshotComparableRecordSchema = SnapshotEntrySchema.pick({ + provider: true, + model: true, + lane: true, + caseId: true, + condition: true, + caseFingerprint: true, + totalTokens: true, +}); + +function buildSnapshotComparisonKey(record: SnapshotComparableRecord): string { + return JSON.stringify([ + record.provider, + record.model, + record.lane, + record.caseId, + record.condition, + record.caseFingerprint, + ]); +} + +function createRecordMap( + records: readonly SnapshotComparableRecord[], + label: string, +): Map { + const recordMap = new Map(); + + for (const record of records) { + const parsedRecord = SnapshotComparableRecordSchema.safeParse(record); + if (!parsedRecord.success) { + invariant( + false, + `${label} validation failed: ${parsedRecord.error.message}`, + ); + } + + const key = buildSnapshotComparisonKey(parsedRecord.data); + invariant(!recordMap.has(key), `Duplicate ${label} key: ${key}`); + recordMap.set(key, parsedRecord.data); + } + + return recordMap; +} + +function classifySnapshotCase( + currentRecord: SnapshotComparableRecord | undefined, + snapshotRecord: SnapshotComparableRecord | undefined, + regressionThresholdPercent: number, +): SnapshotCheckCase { + const sourceRecord = currentRecord ?? snapshotRecord; + invariant( + sourceRecord !== undefined, + 'Snapshot comparison source record must exist', + ); + + if (currentRecord === undefined) { + invariant( + snapshotRecord !== undefined, + 'orphaned snapshot comparisons must include a stored record', + ); + return { + ...sourceRecord, + outcome: 'orphaned', + snapshotTotalTokens: snapshotRecord.totalTokens, + }; + } + + if (snapshotRecord === undefined) { + invariant( + currentRecord !== undefined, + 'new snapshot comparisons must include a current record', + ); + return { + ...sourceRecord, + outcome: 'new', + currentTotalTokens: currentRecord.totalTokens, + }; + } + + const currentTotalTokens = currentRecord.totalTokens; + const snapshotTotalTokens = snapshotRecord.totalTokens; + const deltaTokens = currentTotalTokens - snapshotTotalTokens; + const deltaPercent = + snapshotTotalTokens > 0 + ? (deltaTokens / snapshotTotalTokens) * 100 + : undefined; + + if (currentTotalTokens < snapshotTotalTokens) { + return { + ...sourceRecord, + outcome: 'improved', + currentTotalTokens, + snapshotTotalTokens, + deltaTokens, + ...(deltaPercent === undefined ? {} : { deltaPercent }), + }; + } + + const regressionCeiling = + snapshotTotalTokens * (1 + regressionThresholdPercent / 100); + if (currentTotalTokens > regressionCeiling) { + return { + ...sourceRecord, + outcome: 'regressed', + currentTotalTokens, + snapshotTotalTokens, + deltaTokens, + ...(deltaPercent === undefined ? {} : { deltaPercent }), + }; + } + + return { + ...sourceRecord, + outcome: 'unchanged', + currentTotalTokens, + snapshotTotalTokens, + deltaTokens, + ...(deltaPercent === undefined ? {} : { deltaPercent }), + }; +} + +export function compareSnapshotRecords( + options: CompareSnapshotRecordsOptions, +): SnapshotCheckReport { + invariant( + Number.isFinite(options.regressionThresholdPercent) && + options.regressionThresholdPercent >= 0, + 'regressionThresholdPercent must be a finite non-negative number', + ); + + const currentRecordMap = createRecordMap( + options.currentRecords, + 'current snapshot record', + ); + const snapshotRecordMap = createRecordMap( + options.snapshotRecords, + 'stored snapshot record', + ); + + const comparisonKeys = Array.from( + new Set([...currentRecordMap.keys(), ...snapshotRecordMap.keys()]), + ).sort((left, right) => left.localeCompare(right)); + const cases = comparisonKeys.map((key) => + classifySnapshotCase( + currentRecordMap.get(key), + snapshotRecordMap.get(key), + options.regressionThresholdPercent, + ), + ); + + const summary = cases.reduce( + (result, entry) => ({ + ...result, + total: result.total + 1, + [entry.outcome]: result[entry.outcome] + 1, + }), + { + total: 0, + new: 0, + orphaned: 0, + unchanged: 0, + improved: 0, + regressed: 0, + }, + ); + + return { + regressionThresholdPercent: options.regressionThresholdPercent, + cases, + summary, + }; +} diff --git a/evals/snapshots/fingerprint.ts b/evals/snapshots/fingerprint.ts new file mode 100644 index 00000000..37e20a49 --- /dev/null +++ b/evals/snapshots/fingerprint.ts @@ -0,0 +1,129 @@ +import { createHash } from 'node:crypto'; + +import { + AntiPatternRuleSchema, + EvalCaseSchema, + WorkflowCheckSchema, +} from '../lib/schemas.js'; +import { invariant, unreachable } from '../../src/util/assert.js'; +import type { z } from 'zod'; + +import type { AntiPatternRule, EvalCase } from '../lib/types.js'; + +type WorkflowCheckInput = z.infer; +type AntiPatternRuleInput = z.infer; + +interface FingerprintWorkflowCheck { + id: string; + requiredPatterns: string[]; + forbiddenPatterns: string[]; + dependsOn: string[]; +} + +interface FingerprintAntiPattern { + id: string; + patterns: string[]; + severity: AntiPatternRule['severity']; + lanes?: AntiPatternRule['lanes']; +} + +interface FingerprintCaseCommon { + id: string; + lane: EvalCase['lane']; + category: EvalCase['category']; + prompt: string; + workflowChecks: FingerprintWorkflowCheck[]; + antiPatterns: FingerprintAntiPattern[]; +} + +interface FingerprintPromptCase extends FingerprintCaseCommon { + lane: 'prompt'; + expectedPatterns: string[]; + forbiddenPatterns: string[]; +} + +type FingerprintCase = FingerprintPromptCase | FingerprintCaseCommon; + +function projectWorkflowCheck( + check: WorkflowCheckInput, +): FingerprintWorkflowCheck { + return { + id: check.id, + requiredPatterns: [...check.requiredPatterns], + forbiddenPatterns: [...check.forbiddenPatterns], + dependsOn: [...check.dependsOn], + }; +} + +function projectAntiPattern( + rule: AntiPatternRuleInput, +): FingerprintAntiPattern { + return { + id: rule.id, + patterns: [...rule.patterns], + severity: rule.severity, + ...(rule.lanes === undefined ? {} : { lanes: [...rule.lanes] }), + }; +} + +function projectCaseSemanticFields(evalCase: EvalCase): FingerprintCase { + const parsedCase = EvalCaseSchema.safeParse(evalCase); + if (!parsedCase.success) { + invariant(false, `Invalid eval case: ${parsedCase.error.message}`); + } + + const baseCase: FingerprintCaseCommon = { + id: parsedCase.data.id, + lane: parsedCase.data.lane, + category: parsedCase.data.category, + prompt: parsedCase.data.prompt, + workflowChecks: parsedCase.data.workflowChecks.map(projectWorkflowCheck), + antiPatterns: parsedCase.data.antiPatterns.map(projectAntiPattern), + }; + + switch (parsedCase.data.lane) { + case 'prompt': + return { + ...baseCase, + lane: 'prompt', + expectedPatterns: [...parsedCase.data.expectedPatterns], + forbiddenPatterns: [...parsedCase.data.forbiddenPatterns], + }; + case 'execution': + case 'dogfood': + return baseCase; + default: + return unreachable(parsedCase.data, 'Unsupported eval case lane'); + } +} + +function normalizeForStableJson(value: unknown): unknown { + if (Array.isArray(value)) { + return value.map((item) => normalizeForStableJson(item)); + } + + if (value !== null && typeof value === 'object') { + return Object.fromEntries( + Object.keys(value as Record) + .sort() + .map((key) => [ + key, + normalizeForStableJson((value as Record)[key]), + ]), + ); + } + + return value; +} + +export function caseFingerprint(evalCase: EvalCase): string { + const serializedCase = JSON.stringify( + normalizeForStableJson(projectCaseSemanticFields(evalCase)), + ); + invariant( + serializedCase !== undefined, + 'Snapshot fingerprint payload must serialize to JSON', + ); + + return createHash('sha256').update(serializedCase).digest('hex'); +} diff --git a/evals/snapshots/schema.ts b/evals/snapshots/schema.ts new file mode 100644 index 00000000..139e7652 --- /dev/null +++ b/evals/snapshots/schema.ts @@ -0,0 +1,65 @@ +import { z } from 'zod'; + +import { + EvalLaneSchema, + EvalResultSchema, + SkillConditionSchema, + TokenUsageSchema, +} from '../lib/schemas.js'; +import { invariant } from '../../src/util/assert.js'; + +const NonEmptyStringSchema = EvalResultSchema.shape.providerId; +const NonNegativeIntSchema = TokenUsageSchema.shape.totalTokens; +const Sha256HexSchema = z + .string() + .regex( + /^[0-9a-f]{64}$/u, + 'caseFingerprint must be a lowercase SHA-256 hex string', + ); + +const SnapshotLogicalKeySchema = z + .object({ + lane: EvalLaneSchema, + caseId: NonEmptyStringSchema, + condition: SkillConditionSchema, + caseFingerprint: Sha256HexSchema, + }) + .passthrough(); + +export const SnapshotEntrySchema = z + .object({ + provider: NonEmptyStringSchema, + model: NonEmptyStringSchema, + lane: EvalLaneSchema, + caseId: NonEmptyStringSchema, + condition: SkillConditionSchema, + caseFingerprint: Sha256HexSchema, + inputTokens: TokenUsageSchema.shape.inputTokens, + outputTokens: TokenUsageSchema.shape.outputTokens, + totalTokens: TokenUsageSchema.shape.totalTokens, + cachedTokens: TokenUsageSchema.shape.cachedTokens, + createdAtMs: NonNegativeIntSchema, + }) + .strict(); + +export type SnapshotEntry = z.infer; +export type SnapshotLogicalKeyFields = z.infer; + +export function buildSnapshotLogicalKey( + fields: SnapshotLogicalKeyFields, +): string { + const parsedFields = SnapshotLogicalKeySchema.safeParse(fields); + if (!parsedFields.success) { + invariant( + false, + `Invalid snapshot logical key fields: ${parsedFields.error.message}`, + ); + } + + return JSON.stringify([ + parsedFields.data.lane, + parsedFields.data.caseId, + parsedFields.data.condition, + parsedFields.data.caseFingerprint, + ]); +} diff --git a/evals/snapshots/schemas/report.ts b/evals/snapshots/schemas/report.ts new file mode 100644 index 00000000..69f11ce9 --- /dev/null +++ b/evals/snapshots/schemas/report.ts @@ -0,0 +1,66 @@ +import { z } from 'zod'; + +const NonEmptyStringSchema = z.string().min(1); +const NonNegativeIntSchema = z.number().int().nonnegative(); +const FiniteNumberSchema = z.number().finite(); +const Sha256HexSchema = z + .string() + .regex( + /^[0-9a-f]{64}$/u, + 'caseFingerprint must be a lowercase SHA-256 hex string', + ); +const SnapshotLaneSchema = z.enum(['prompt', 'execution', 'dogfood']); +const SnapshotConditionSchema = z.enum([ + 'none', + 'self-load', + 'preloaded', + 'stale', +]); + +export const SnapshotCheckOutcomeSchema = z.enum([ + 'new', + 'orphaned', + 'unchanged', + 'improved', + 'regressed', +]); +export type SnapshotCheckOutcome = z.infer; + +export const SnapshotCheckCaseSchema = z + .object({ + provider: NonEmptyStringSchema, + model: NonEmptyStringSchema, + lane: SnapshotLaneSchema, + caseId: NonEmptyStringSchema, + condition: SnapshotConditionSchema, + caseFingerprint: Sha256HexSchema, + totalTokens: NonNegativeIntSchema, + outcome: SnapshotCheckOutcomeSchema, + currentTotalTokens: NonNegativeIntSchema.optional(), + snapshotTotalTokens: NonNegativeIntSchema.optional(), + deltaTokens: z.number().int().optional(), + deltaPercent: FiniteNumberSchema.optional(), + }) + .strict(); +export type SnapshotCheckCase = z.infer; + +export const SnapshotCheckSummarySchema = z + .object({ + total: NonNegativeIntSchema, + new: NonNegativeIntSchema, + orphaned: NonNegativeIntSchema, + unchanged: NonNegativeIntSchema, + improved: NonNegativeIntSchema, + regressed: NonNegativeIntSchema, + }) + .strict(); +export type SnapshotCheckSummary = z.infer; + +export const SnapshotCheckReportSchema = z + .object({ + regressionThresholdPercent: FiniteNumberSchema.nonnegative(), + cases: z.array(SnapshotCheckCaseSchema), + summary: SnapshotCheckSummarySchema, + }) + .strict(); +export type SnapshotCheckReport = z.infer; diff --git a/evals/snapshots/store.ts b/evals/snapshots/store.ts new file mode 100644 index 00000000..03fa241e --- /dev/null +++ b/evals/snapshots/store.ts @@ -0,0 +1,288 @@ +import { readFile } from 'node:fs/promises'; +import { isAbsolute, relative, resolve } from 'node:path'; + +import { writeTextFileAtomic } from '../../src/storage/manifests.js'; +import { assertString, invariant } from '../../src/util/assert.js'; +import { validatePathSegment } from '../lib/artifacts.js'; +import { buildSnapshotLogicalKey, SnapshotEntrySchema } from './schema.js'; +import type { SnapshotEntry } from './schema.js'; + +interface NodeError { + code?: string; +} + +export interface SnapshotLoadDiagnostic { + kind: 'malformed' | 'stale'; + path: string; + lineNumber: number; + message: string; +} + +export interface LoadSnapshotFileOptions { + snapshotDir: string; + provider: string; + model: string; + validCurrentKeys?: ReadonlySet; + allowMissing?: boolean; +} + +export interface WriteSnapshotFileOptions { + snapshotDir: string; + provider: string; + model: string; + entries: readonly SnapshotEntry[]; +} + +const SnapshotEntryArraySchema = SnapshotEntrySchema.array(); + +function isEnoentError(error: unknown): error is Error & NodeError { + return ( + error instanceof Error && + 'code' in error && + (error as NodeError).code === 'ENOENT' + ); +} + +function resolveSnapshotDir(snapshotDir: string): string { + assertString(snapshotDir, 'snapshotDir must be a string'); + invariant(snapshotDir.length > 0, 'snapshotDir must be a non-empty string'); + + const resolvedSnapshotDir = resolve(snapshotDir); + invariant( + isAbsolute(resolvedSnapshotDir), + 'snapshotDir must resolve to absolute', + ); + + return resolvedSnapshotDir; +} + +function assertPathInsideBase( + baseDir: string, + candidatePath: string, + label: string, +): string { + const normalizedBaseDir = resolve(baseDir); + const normalizedCandidatePath = resolve(candidatePath); + const relativePath = relative(normalizedBaseDir, normalizedCandidatePath); + + invariant( + relativePath === '' || + (!relativePath.startsWith('..') && !isAbsolute(relativePath)), + `${label} must stay within base directory ${normalizedBaseDir}`, + ); + + return normalizedCandidatePath; +} + +function warnSkippedSnapshotEntry( + path: string, + lineNumber: number, + message: string, +): void { + console.warn( + `[evals] Skipping snapshot entry at ${path}:${String(lineNumber)}: ${message}`, + ); +} + +function buildSnapshotEntryKey(entry: SnapshotEntry): string { + return buildSnapshotLogicalKey({ + lane: entry.lane, + caseId: entry.caseId, + condition: entry.condition, + caseFingerprint: entry.caseFingerprint, + }); +} + +function assertEntryMatchesFile( + entry: SnapshotEntry, + provider: string, + model: string, +): void { + invariant( + entry.provider === provider, + `Snapshot entry provider mismatch: expected ${provider}, got ${entry.provider}`, + ); + invariant( + entry.model === model, + `Snapshot entry model mismatch: expected ${model}, got ${entry.model}`, + ); +} + +function serializeEntries(entries: readonly SnapshotEntry[]): string { + if (entries.length === 0) { + return ''; + } + + return `${entries.map((entry) => JSON.stringify(entry)).join('\n')}\n`; +} + +export function snapshotFilePath( + snapshotDir: string, + provider: string, + model: string, +): string { + const resolvedSnapshotDir = resolveSnapshotDir(snapshotDir); + validatePathSegment(provider, 'provider'); + validatePathSegment(model, 'model'); + + return assertPathInsideBase( + resolvedSnapshotDir, + resolve(resolvedSnapshotDir, `${provider}-${model}.jsonl`), + 'snapshot file path', + ); +} + +export async function loadSnapshotFile( + options: LoadSnapshotFileOptions, +): Promise<{ + entries: SnapshotEntry[]; + diagnostics: SnapshotLoadDiagnostic[]; +}> { + const path = snapshotFilePath( + options.snapshotDir, + options.provider, + options.model, + ); + const diagnostics: SnapshotLoadDiagnostic[] = []; + + let rawContents: string; + try { + rawContents = await readFile(path, 'utf8'); + } catch (error) { + if ((options.allowMissing ?? true) && isEnoentError(error)) { + return { entries: [], diagnostics }; + } + + throw error; + } + + const lines = rawContents.split('\n'); + if (lines.at(-1) === '') { + lines.pop(); + } + + const entries: SnapshotEntry[] = []; + for (const [index, line] of lines.entries()) { + const lineNumber = index + 1; + + let parsedJson: unknown; + try { + parsedJson = JSON.parse(line) as unknown; + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + diagnostics.push({ + kind: 'malformed', + path, + lineNumber, + message, + }); + warnSkippedSnapshotEntry(path, lineNumber, message); + continue; + } + + const parsedEntry = SnapshotEntrySchema.safeParse(parsedJson); + if (!parsedEntry.success) { + const message = parsedEntry.error.message; + diagnostics.push({ + kind: 'malformed', + path, + lineNumber, + message, + }); + warnSkippedSnapshotEntry(path, lineNumber, message); + continue; + } + + if (options.validCurrentKeys !== undefined) { + const key = buildSnapshotEntryKey(parsedEntry.data); + if (!options.validCurrentKeys.has(key)) { + const message = `stale snapshot key ${key}`; + diagnostics.push({ + kind: 'stale', + path, + lineNumber, + message, + }); + warnSkippedSnapshotEntry(path, lineNumber, message); + continue; + } + } + + entries.push(parsedEntry.data); + } + + return { entries, diagnostics }; +} + +export async function writeSnapshotFile( + options: WriteSnapshotFileOptions, +): Promise { + const path = snapshotFilePath( + options.snapshotDir, + options.provider, + options.model, + ); + const parsedEntries = SnapshotEntryArraySchema.safeParse(options.entries); + if (!parsedEntries.success) { + invariant( + false, + `Snapshot entries validation failed: ${parsedEntries.error.message}`, + ); + } + + const incomingEntries = parsedEntries.data; + const incomingEntriesByKey = new Map(); + for (const entry of incomingEntries) { + assertEntryMatchesFile(entry, options.provider, options.model); + + const key = buildSnapshotEntryKey(entry); + invariant( + !incomingEntriesByKey.has(key), + `Duplicate snapshot entry key in write request: ${key}`, + ); + incomingEntriesByKey.set(key, entry); + } + + const existingFile = await loadSnapshotFile({ + snapshotDir: options.snapshotDir, + provider: options.provider, + model: options.model, + allowMissing: true, + }); + + const mergedEntries: SnapshotEntry[] = []; + const seenExistingKeys = new Set(); + for (const entry of existingFile.entries) { + const key = buildSnapshotEntryKey(entry); + if (seenExistingKeys.has(key)) { + continue; + } + + seenExistingKeys.add(key); + const replacement = incomingEntriesByKey.get(key); + if (replacement === undefined) { + mergedEntries.push(entry); + continue; + } + + mergedEntries.push(replacement); + incomingEntriesByKey.delete(key); + } + + for (const entry of incomingEntries) { + const key = buildSnapshotEntryKey(entry); + if (!incomingEntriesByKey.has(key)) { + continue; + } + + mergedEntries.push(entry); + incomingEntriesByKey.delete(key); + } + + await writeTextFileAtomic({ + path, + pathLabel: 'snapshot file path', + contents: serializeEntries(mergedEntries), + writeErrorMessage: `Failed to write snapshot file at ${path}.`, + }); +} diff --git a/evals/workspaces/builtins.ts b/evals/workspaces/builtins.ts new file mode 100644 index 00000000..2f92dad5 --- /dev/null +++ b/evals/workspaces/builtins.ts @@ -0,0 +1,28 @@ +import process from 'node:process'; + +import { registerPreset } from './registry.js'; +import type { WorkspacePreset } from './types.js'; + +const AGENT_TTY_SMOKE_PRESET: WorkspacePreset = { + id: 'agent-tty-smoke', + mode: 'isolated', + description: 'Deterministic local smoke preset for agent-tty evals.', + bootstrap: [ + { + command: process.execPath, + args: ['-e', 'process.stdout.write("agent-tty-smoke bootstrap ok\\n")'], + description: 'agent-tty-smoke smoke-probe', + }, + ], +}; + +let registered = false; + +export function registerBuiltinPresets(): void { + if (registered) { + return; + } + + registerPreset(AGENT_TTY_SMOKE_PRESET); + registered = true; +} diff --git a/evals/workspaces/registry.ts b/evals/workspaces/registry.ts new file mode 100644 index 00000000..3add6464 --- /dev/null +++ b/evals/workspaces/registry.ts @@ -0,0 +1,70 @@ +import { WorkspacePresetSchema } from './types.js'; +import type { WorkspacePreset } from './types.js'; + +const presetRegistry = new Map(); + +type SchemaIssue = { + path: readonly PropertyKey[]; + message: string; +}; + +function formatSchemaIssues(issues: readonly SchemaIssue[]): string { + return issues + .map((issue) => { + const path = issue.path.length > 0 ? issue.path.join('.') : ''; + return `${path}: ${issue.message}`; + }) + .join('; '); +} + +function getCandidatePresetId(preset: WorkspacePreset): string { + return preset.id; +} + +export function registerPreset(preset: WorkspacePreset): void { + const parsedPreset = WorkspacePresetSchema.safeParse(preset); + if (!parsedPreset.success) { + throw new Error( + `Invalid workspace preset "${getCandidatePresetId(preset)}": ${formatSchemaIssues(parsedPreset.error.issues)}`, + ); + } + + if (presetRegistry.has(parsedPreset.data.id)) { + throw new Error( + `Workspace preset "${parsedPreset.data.id}" is already registered.`, + ); + } + + presetRegistry.set(parsedPreset.data.id, parsedPreset.data); +} + +export function lookupPreset(id: string): WorkspacePreset { + const preset = presetRegistry.get(id); + if (preset !== undefined) { + return preset; + } + + const availableIds = Array.from(presetRegistry.keys()).sort((left, right) => + left.localeCompare(right), + ); + if (availableIds.length === 0) { + throw new Error( + `Unknown workspace preset "${id}". No workspace presets are registered.`, + ); + } + + throw new Error( + `Unknown workspace preset "${id}". Available: [${availableIds.join(', ')}]`, + ); +} + +// For unit tests only. +export function clearPresetsForTesting(): void { + presetRegistry.clear(); +} + +export function listPresets(): readonly WorkspacePreset[] { + return Array.from(presetRegistry.values()).sort((left, right) => + left.id.localeCompare(right.id), + ); +} diff --git a/evals/workspaces/resolver.ts b/evals/workspaces/resolver.ts new file mode 100644 index 00000000..0d049b65 --- /dev/null +++ b/evals/workspaces/resolver.ts @@ -0,0 +1,144 @@ +/** + * Merge order for workspace presets: + * 1. Preset `env` is applied first; case/request env overrides are layered on top. + * 2. Preset `bootstrap` commands run before any case-level `setup`. + * 3. The `ResolvedWorkspacePlan` returned here is the reportable shape: env values for keys whose suffix (case-insensitive) matches `_TOKEN`, `_KEY`, `_SECRET`, or `_PASSWORD` are replaced with `"[REDACTED]"`. Runtime env must be obtained separately via `deriveEffectiveEnv`. + */ +import { statSync } from 'node:fs'; +import { isAbsolute, resolve } from 'node:path'; + +import { assertString, invariant } from '../../src/util/assert.js'; +import type { ResolvedWorkspacePlan, WorkspacePreset } from './types.js'; + +const REDACTED_ENV_VALUE = '[REDACTED]'; +const SENSITIVE_ENV_KEY_SUFFIX_REGEX = /(?:_TOKEN|_KEY|_SECRET|_PASSWORD)$/i; + +type ResolveWorkspacePresetContext = { + homeDir: string; + repoRoot?: string; +}; + +function assertAbsolutePath(value: string, message: string): void { + assertString(value, message); + invariant(value.length > 0, message); + invariant(isAbsolute(value), message); +} + +function resolveTemplateDir( + ctx: ResolveWorkspacePresetContext, + preset: WorkspacePreset, +): string | undefined { + if (preset.templateDir === undefined) { + return undefined; + } + + const templateDir = isAbsolute(preset.templateDir) + ? preset.templateDir + : (() => { + if (ctx.repoRoot === undefined) { + throw new Error( + `Workspace preset "${preset.id}" uses relative templateDir "${preset.templateDir}" but ctx.repoRoot is required to resolve it.`, + ); + } + + return resolve(ctx.repoRoot, preset.templateDir); + })(); + + try { + const stats = statSync(templateDir); + if (!stats.isDirectory()) { + throw new Error( + `Workspace preset "${preset.id}" templateDir "${templateDir}" is not a directory.`, + ); + } + } catch (error) { + if ( + error instanceof Error && + error.message.startsWith(`Workspace preset "${preset.id}" templateDir`) + ) { + throw error; + } + + const details = error instanceof Error ? `: ${error.message}` : ''; + throw new Error( + `Workspace preset "${preset.id}" templateDir "${templateDir}" does not exist or is not accessible${details}`, + { cause: error }, + ); + } + + return templateDir; +} + +function redactEnv( + env: Record | undefined, +): Readonly> | undefined { + if (env === undefined) { + return undefined; + } + + const redactedEnv: Record = {}; + for (const [key, value] of Object.entries(env)) { + redactedEnv[key] = SENSITIVE_ENV_KEY_SUFFIX_REGEX.test(key) + ? REDACTED_ENV_VALUE + : value; + } + + return redactedEnv; +} + +function normalizeBootstrap( + preset: WorkspacePreset, +): ResolvedWorkspacePlan['bootstrap'] { + return (preset.bootstrap ?? []).map((step) => ({ + command: step.command, + args: step.args === undefined ? [] : [...step.args], + ...(step.description === undefined + ? {} + : { description: step.description }), + })); +} + +export function resolveWorkspacePreset( + ctx: ResolveWorkspacePresetContext, + preset: WorkspacePreset, +): ResolvedWorkspacePlan { + assertAbsolutePath(ctx.homeDir, 'ctx.homeDir must be an absolute path'); + if (ctx.repoRoot !== undefined) { + assertAbsolutePath(ctx.repoRoot, 'ctx.repoRoot must be an absolute path'); + } + + const templateDir = resolveTemplateDir(ctx, preset); + const cwd = + preset.cwd === undefined + ? undefined + : isAbsolute(preset.cwd) + ? preset.cwd + : resolve(ctx.homeDir, preset.cwd); + const bootstrap = normalizeBootstrap(preset); + + const redactedEnv = redactEnv(preset.env); + + return { + presetId: preset.id, + mode: preset.mode, + description: preset.description, + ...(templateDir === undefined ? {} : { templateDir }), + ...(cwd === undefined ? {} : { cwd }), + ...(redactedEnv === undefined ? {} : { env: redactedEnv }), + bootstrap, + bootstrapCount: bootstrap.length, + }; +} + +/** + * Returns raw values (no redaction) for runtime spawn use only. Never serialize or log the result. + */ +export function deriveEffectiveEnv( + preset: WorkspacePreset, + overrides?: Record, +): Record { + return { + ...(preset.env ?? {}), + ...(overrides ?? {}), + }; +} diff --git a/evals/workspaces/types.ts b/evals/workspaces/types.ts new file mode 100644 index 00000000..9d6e583d --- /dev/null +++ b/evals/workspaces/types.ts @@ -0,0 +1,46 @@ +import { z } from 'zod'; + +const NonEmptyStringSchema = z.string().min(1); +const WorkspacePresetIdSchema = z + .string() + .regex(/^[a-z0-9][a-z0-9-_]*$/, 'id must match /^[a-z0-9][a-z0-9-_]*$/'); +const StringRecordSchema = z.record(NonEmptyStringSchema, z.string()); + +const WorkspaceBootstrapStepSchema = z + .object({ + command: NonEmptyStringSchema, + args: z.array(z.string()).optional(), + description: NonEmptyStringSchema.optional(), + }) + .strict(); + +export const WorkspacePresetSchema = z + .object({ + id: WorkspacePresetIdSchema, + mode: z.enum(['isolated', 'shared']), + description: NonEmptyStringSchema, + templateDir: NonEmptyStringSchema.optional(), + bootstrap: z.array(WorkspaceBootstrapStepSchema).optional(), + cwd: NonEmptyStringSchema.optional(), + env: StringRecordSchema.optional(), + }) + .strict(); + +export type WorkspacePreset = z.infer; + +type ResolvedWorkspaceBootstrapStep = { + command: string; + args: readonly string[]; + description?: string; +}; + +export type ResolvedWorkspacePlan = { + presetId: string; + mode: 'isolated' | 'shared'; + description: string; + templateDir?: string; + cwd?: string; + env?: Readonly>; + bootstrap: ReadonlyArray; + bootstrapCount: number; +}; diff --git a/test/integration/evals-reporter-default-parity.test.ts b/test/integration/evals-reporter-default-parity.test.ts new file mode 100644 index 00000000..ac8e2d78 --- /dev/null +++ b/test/integration/evals-reporter-default-parity.test.ts @@ -0,0 +1,216 @@ +import { spawnSync } from 'node:child_process'; +import { + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + realpathSync, + rmSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import process from 'node:process'; + +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; + +const DEFAULT_EVAL_TIMEOUT_MS = 30_000; +const ISO_TIMESTAMP_PATTERN = + /\b\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{3})?Z\b/gu; + +interface EvalRunSummary { + runId?: string; + outputBaseDir: string; + jsonReportPath?: string; + markdownReportPath?: string; +} + +interface NormalizationContext { + outputBaseDir: string; + runId: string; +} + +function expectNonEmptyString(value: unknown, label: string): string { + expect(typeof value).toBe('string'); + if (typeof value !== 'string' || value.length === 0) { + throw new Error(`${label} must be a non-empty string`); + } + return value; +} + +function normalizeStringValue( + value: string, + context: NormalizationContext, +): string { + return value + .replaceAll(context.outputBaseDir, '') + .replaceAll(context.runId, '') + .replace(ISO_TIMESTAMP_PATTERN, ''); +} + +function normalizeJsonReportValue( + value: unknown, + context: NormalizationContext, + key?: string, +): unknown { + if (Array.isArray(value)) { + return value.map((item) => normalizeJsonReportValue(item, context)); + } + + if (value !== null && typeof value === 'object') { + return Object.fromEntries( + Object.entries(value).map(([nestedKey, nestedValue]) => [ + nestedKey, + normalizeJsonReportValue(nestedValue, context, nestedKey), + ]), + ); + } + + if (typeof value === 'string') { + return normalizeStringValue(value, context); + } + + if (typeof value === 'number' && key !== undefined && key.endsWith('Ms')) { + return 0; + } + + return value; +} + +function runEval( + homeDir: string, + outputDir: string, + reporters: readonly string[] = [], +): EvalRunSummary { + mkdirSync(homeDir, { recursive: true }); + + const reporterArgs = reporters.flatMap((reporterName) => [ + '--reporter', + reporterName, + ]); + const result = spawnSync( + process.execPath, + [ + '--import', + 'tsx', + './evals/run.ts', + '--provider', + 'stub', + '--lane', + 'prompt', + '--case', + 'pure-reasoning', + '--condition', + 'none', + '--trials', + '2', + '--concurrency', + '1', + ...reporterArgs, + '--output', + outputDir, + '--json', + ], + { + cwd: process.cwd(), + encoding: 'utf8', + env: { + ...process.env, + AGENT_TTY_HOME: homeDir, + }, + timeout: DEFAULT_EVAL_TIMEOUT_MS, + }, + ); + + expect(result.error).toBeUndefined(); + expect(result.status).not.toBeNull(); + return JSON.parse(result.stdout) as EvalRunSummary; +} + +let testRoot = ''; + +describe( + 'eval reporter final/default parity', + { timeout: DEFAULT_EVAL_TIMEOUT_MS }, + () => { + beforeEach(() => { + // prettier-ignore + testRoot = realpathSync(mkdtempSync(join(tmpdir(), 'agent-tty-evals-reporter-final-'))); + }); + + afterEach(() => { + rmSync(testRoot, { recursive: true, force: true }); + testRoot = ''; + }); + + it('matches the implicit default reporter output with explicit --reporter final', () => { + const implicitSummary = runEval( + join(testRoot, 'implicit-home'), + join(testRoot, 'implicit-output'), + ); + const explicitSummary = runEval( + join(testRoot, 'explicit-home'), + join(testRoot, 'explicit-output'), + ['final'], + ); + + const implicitRunId = expectNonEmptyString( + implicitSummary.runId, + 'implicitSummary.runId', + ); + const explicitRunId = expectNonEmptyString( + explicitSummary.runId, + 'explicitSummary.runId', + ); + const implicitJsonReportPath = expectNonEmptyString( + implicitSummary.jsonReportPath, + 'implicitSummary.jsonReportPath', + ); + const explicitJsonReportPath = expectNonEmptyString( + explicitSummary.jsonReportPath, + 'explicitSummary.jsonReportPath', + ); + const implicitMarkdownReportPath = expectNonEmptyString( + implicitSummary.markdownReportPath, + 'implicitSummary.markdownReportPath', + ); + const explicitMarkdownReportPath = expectNonEmptyString( + explicitSummary.markdownReportPath, + 'explicitSummary.markdownReportPath', + ); + + expect(existsSync(implicitJsonReportPath)).toBe(true); + expect(existsSync(explicitJsonReportPath)).toBe(true); + expect(existsSync(implicitMarkdownReportPath)).toBe(true); + expect(existsSync(explicitMarkdownReportPath)).toBe(true); + + const implicitContext: NormalizationContext = { + outputBaseDir: implicitSummary.outputBaseDir, + runId: implicitRunId, + }; + const explicitContext: NormalizationContext = { + outputBaseDir: explicitSummary.outputBaseDir, + runId: explicitRunId, + }; + + const implicitJsonReport = JSON.parse( + readFileSync(implicitJsonReportPath, 'utf8'), + ) as unknown; + const explicitJsonReport = JSON.parse( + readFileSync(explicitJsonReportPath, 'utf8'), + ) as unknown; + expect( + normalizeJsonReportValue(implicitJsonReport, implicitContext), + ).toEqual(normalizeJsonReportValue(explicitJsonReport, explicitContext)); + + const implicitMarkdownReport = normalizeStringValue( + readFileSync(implicitMarkdownReportPath, 'utf8'), + implicitContext, + ); + const explicitMarkdownReport = normalizeStringValue( + readFileSync(explicitMarkdownReportPath, 'utf8'), + explicitContext, + ); + expect(implicitMarkdownReport).toBe(explicitMarkdownReport); + }); + }, +); diff --git a/test/integration/evals-reporter-jsonl.test.ts b/test/integration/evals-reporter-jsonl.test.ts new file mode 100644 index 00000000..88563de5 --- /dev/null +++ b/test/integration/evals-reporter-jsonl.test.ts @@ -0,0 +1,169 @@ +import { spawnSync } from 'node:child_process'; +import { + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + readdirSync, + realpathSync, + rmSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import process from 'node:process'; + +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; + +const DEFAULT_EVAL_TIMEOUT_MS = 30_000; +const EXPECTED_JSONL_EVENT_TYPES = [ + 'run.start', + 'lane.start', + 'case.start', + 'trial.start', + 'trial.finish', + 'trial.start', + 'trial.finish', + 'case.finish', + 'lane.finish', + 'run.finish', +] as const; + +interface EvalRunSummary { + runId?: string; + jsonReportPath?: string; + markdownReportPath?: string; +} + +interface JsonlEventRecord { + type: string; + payload: { + runId: string; + }; +} + +function expectNonEmptyString(value: unknown, label: string): string { + expect(typeof value).toBe('string'); + if (typeof value !== 'string' || value.length === 0) { + throw new Error(`${label} must be a non-empty string`); + } + return value; +} + +function findFilesNamed(rootDir: string, fileName: string): string[] { + if (!existsSync(rootDir)) { + return []; + } + + const matches: string[] = []; + for (const entry of readdirSync(rootDir, { withFileTypes: true })) { + const entryPath = join(rootDir, entry.name); + if (entry.isDirectory()) { + matches.push(...findFilesNamed(entryPath, fileName)); + continue; + } + if (entry.isFile() && entry.name === fileName) { + matches.push(entryPath); + } + } + + return matches; +} + +function runJsonlEval( + homeDir: string, + outputDir: string, + reporterOutputPath: string, +): EvalRunSummary { + mkdirSync(homeDir, { recursive: true }); + + const result = spawnSync( + process.execPath, + [ + '--import', + 'tsx', + './evals/run.ts', + '--provider', + 'stub', + '--lane', + 'prompt', + '--case', + 'pure-reasoning', + '--condition', + 'none', + '--trials', + '2', + '--concurrency', + '1', + '--reporter', + 'jsonl', + '--reporter-output', + reporterOutputPath, + '--output', + outputDir, + '--json', + ], + { + cwd: process.cwd(), + encoding: 'utf8', + env: { + ...process.env, + AGENT_TTY_HOME: homeDir, + }, + timeout: DEFAULT_EVAL_TIMEOUT_MS, + }, + ); + + expect(result.error).toBeUndefined(); + expect(result.status).not.toBeNull(); + return JSON.parse(result.stdout) as EvalRunSummary; +} + +let testRoot = ''; + +describe( + 'eval reporter jsonl integration', + { timeout: DEFAULT_EVAL_TIMEOUT_MS }, + () => { + beforeEach(() => { + // prettier-ignore + testRoot = realpathSync(mkdtempSync(join(tmpdir(), 'agent-tty-evals-reporter-jsonl-'))); + }); + + afterEach(() => { + rmSync(testRoot, { recursive: true, force: true }); + testRoot = ''; + }); + + it('writes only JSONL events while keeping stdout as a single RunSummary object', () => { + const homeDir = join(testRoot, 'home'); + const outputDir = join(testRoot, 'run'); + const reporterOutputPath = join(testRoot, 'events', 'reporter.jsonl'); + + const summary = runJsonlEval(homeDir, outputDir, reporterOutputPath); + const runId = expectNonEmptyString(summary.runId, 'summary.runId'); + + expect(summary.jsonReportPath).toBeUndefined(); + expect('jsonReportPath' in summary).toBe(false); + expect(summary.markdownReportPath).toBeUndefined(); + expect('markdownReportPath' in summary).toBe(false); + expect(findFilesNamed(outputDir, 'report.json')).toEqual([]); + expect(findFilesNamed(outputDir, 'report.md')).toEqual([]); + + expect(existsSync(reporterOutputPath)).toBe(true); + const jsonlContent = readFileSync(reporterOutputPath, 'utf8'); + const lines = jsonlContent + .trim() + .split('\n') + .filter((line) => line.length > 0); + expect(lines).toHaveLength(10); + + const events = lines.map((line) => JSON.parse(line) as JsonlEventRecord); + expect(events.map((event) => event.type)).toEqual( + EXPECTED_JSONL_EVENT_TYPES, + ); + expect(new Set(events.map((event) => event.payload.runId))).toEqual( + new Set([runId]), + ); + }); + }, +); diff --git a/test/integration/evals-snapshots.test.ts b/test/integration/evals-snapshots.test.ts new file mode 100644 index 00000000..f9ea48e7 --- /dev/null +++ b/test/integration/evals-snapshots.test.ts @@ -0,0 +1,481 @@ +import { spawnSync } from 'node:child_process'; +import { + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + realpathSync, + rmSync, + writeFileSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import process from 'node:process'; + +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; + +import { snapshotFilePath } from '../../evals/snapshots/store.js'; +import type { JsonReport } from '../../evals/lib/types.js'; +import type { SnapshotEntry } from '../../evals/snapshots/schema.js'; + +const DEFAULT_EVAL_TIMEOUT_MS = 30_000; +const FIXTURE_MODEL = 'fixture-model'; +const PASSING_WAIT_RESPONSE = + 'Use agent-tty and run `agent-tty wait --json --pattern "Listening on port 3000"` before starting the tests so you only proceed after the server is ready.'; +const BASELINE_TOKEN_USAGE = { + inputTokens: 80, + outputTokens: 20, + totalTokens: 100, + cachedTokens: 10, +} as const; +const REGRESSED_TOKEN_USAGE = { + inputTokens: 100, + outputTokens: 30, + totalTokens: 130, + cachedTokens: 15, +} as const; + +type FixtureTokenUsage = { + inputTokens: number; + outputTokens: number; + totalTokens: number; + cachedTokens: number; +}; + +interface EvalRunSummary { + ok: boolean; + providerId: string; + outputBaseDir: string; + jsonReportPath: string; +} + +function writeJson(filePath: string, value: unknown): void { + mkdirSync(dirname(filePath), { recursive: true }); + writeFileSync(filePath, `${JSON.stringify(value, null, 2)}\n`, 'utf8'); +} + +function writeFixtureResponse( + fixtureRoot: string, + tokenUsage?: FixtureTokenUsage, +): void { + writeJson(join(fixtureRoot, 'responses', 'wait-for-output.json'), { + response: PASSING_WAIT_RESPONSE, + ...(tokenUsage === undefined ? {} : { tokenUsage }), + }); +} + +function createFixtureProviderDirectory( + fixtureRoot: string, + tokenUsage?: FixtureTokenUsage, +): void { + writeJson(join(fixtureRoot, 'runtime-info.json'), { + providerId: 'fixture', + available: true, + detectedAt: '2026-01-01T00:00:00.000Z', + version: 'fixture', + commandPath: 'fixture', + defaultModelId: FIXTURE_MODEL, + capabilities: { + supportsDetect: true, + supportsPlanMode: true, + supportsAgentMode: true, + supportsStreaming: false, + supportsToolCalls: true, + supportsTranscriptCapture: true, + }, + notes: ['eval snapshot integration fixture'], + }); + writeFixtureResponse(fixtureRoot, tokenUsage); +} + +function runEvalCli( + argumentsList: readonly string[], + env: Record, +) { + const result = spawnSync( + process.execPath, + ['--import', 'tsx', './evals/run.ts', ...argumentsList], + { + cwd: process.cwd(), + encoding: 'utf8', + env: { + ...process.env, + ...env, + }, + timeout: DEFAULT_EVAL_TIMEOUT_MS, + }, + ); + + expect(result.error).toBeUndefined(); + expect(result.status).not.toBeNull(); + return result; +} + +function parseRunSummary(stdout: string): EvalRunSummary { + return JSON.parse(stdout) as EvalRunSummary; +} + +function readJsonReport(summary: EvalRunSummary): JsonReport { + const parsed = JSON.parse( + readFileSync(summary.jsonReportPath, 'utf8'), + ) as JsonReport; + + expect(parsed).toHaveProperty('metadata'); + expect(parsed).toHaveProperty('aggregate'); + expect(Array.isArray(parsed.results)).toBe(true); + return parsed; +} + +function readSnapshotEntries(path: string): SnapshotEntry[] { + return readFileSync(path, 'utf8') + .trim() + .split('\n') + .filter((line) => line.length > 0) + .map((line) => JSON.parse(line) as SnapshotEntry); +} + +function buildFixtureEnv( + testRoot: string, + fixtureRoot: string, + homeName: string, +) { + const homeDir = join(testRoot, homeName); + mkdirSync(homeDir, { recursive: true }); + return { + AGENT_TTY_HOME: homeDir, + EVAL_FIXTURE_DIR: fixtureRoot, + }; +} + +function buildSharedFixtureArgs(outputDir: string): string[] { + return [ + '--provider', + 'fixture', + '--lane', + 'prompt', + '--case', + 'wait-for-output', + '--condition', + 'none', + '--trials', + '2', + '--output', + outputDir, + '--json', + ]; +} + +function projectNonTokenResults(report: JsonReport) { + return report.results.map((result) => ({ + lane: result.lane, + caseId: result.caseId, + condition: result.condition, + trial: result.trial, + ok: result.ok, + score: result.score.total, + errorClass: result.errorClass ?? null, + errorMessage: result.errorMessage ?? null, + })); +} + +let testRoot = ''; + +describe( + 'eval CLI token snapshot orchestration', + { timeout: DEFAULT_EVAL_TIMEOUT_MS }, + () => { + beforeEach(() => { + testRoot = realpathSync( + mkdtempSync(join(tmpdir(), 'agent-tty-evals-snapshots-')), + ); + }); + + afterEach(() => { + rmSync(testRoot, { recursive: true, force: true }); + testRoot = ''; + }); + + it('writes token snapshots, reports unchanged/regressed checks, and preserves non-token aggregates', () => { + const fixtureRoot = join(testRoot, 'fixture'); + const snapshotDir = join(testRoot, 'snapshots'); + createFixtureProviderDirectory(fixtureRoot, BASELINE_TOKEN_USAGE); + + const updateOutputDir = join(testRoot, 'update-output'); + const updateSummary = parseRunSummary( + runEvalCli( + [ + ...buildSharedFixtureArgs(updateOutputDir), + '--snapshot-update', + '--snapshot-dir', + snapshotDir, + ], + buildFixtureEnv(testRoot, fixtureRoot, 'home-update'), + ).stdout, + ); + expect(updateSummary.ok).toBe(true); + + const updateReport = readJsonReport(updateSummary); + expect(updateReport.tokenReport).toMatchObject({ + grandTotal: { + inputTokens: 160, + outputTokens: 40, + totalTokens: 200, + cachedTokens: 20, + trials: 2, + }, + perLane: [ + { + lane: 'prompt', + inputTokens: 160, + outputTokens: 40, + totalTokens: 200, + cachedTokens: 20, + trials: 2, + }, + ], + perCase: [ + { + lane: 'prompt', + caseId: 'wait-for-output', + condition: 'none', + inputTokens: 160, + outputTokens: 40, + totalTokens: 200, + cachedTokens: 20, + trials: 2, + }, + ], + }); + expect(updateReport.tokenReport?.snapshotCheck).toBeUndefined(); + + const writtenSnapshotPath = snapshotFilePath( + snapshotDir, + 'fixture', + FIXTURE_MODEL, + ); + expect(existsSync(writtenSnapshotPath)).toBe(true); + expect(readSnapshotEntries(writtenSnapshotPath)).toEqual([ + expect.objectContaining({ + provider: 'fixture', + model: FIXTURE_MODEL, + lane: 'prompt', + caseId: 'wait-for-output', + condition: 'none', + inputTokens: 160, + outputTokens: 40, + totalTokens: 200, + cachedTokens: 20, + }), + ]); + + const unchangedOutputDir = join(testRoot, 'unchanged-output'); + const unchangedSummary = parseRunSummary( + runEvalCli( + [ + ...buildSharedFixtureArgs(unchangedOutputDir), + '--snapshot-check', + '--snapshot-dir', + snapshotDir, + ], + buildFixtureEnv(testRoot, fixtureRoot, 'home-unchanged'), + ).stdout, + ); + const unchangedReport = readJsonReport(unchangedSummary); + expect(unchangedSummary.ok).toBe(updateSummary.ok); + expect(unchangedReport.aggregate).toEqual(updateReport.aggregate); + expect(projectNonTokenResults(unchangedReport)).toEqual( + projectNonTokenResults(updateReport), + ); + expect(unchangedReport.tokenReport?.snapshotCheck).toMatchObject({ + regressionThresholdPercent: 20, + summary: { + total: 1, + new: 0, + orphaned: 0, + unchanged: 1, + improved: 0, + regressed: 0, + }, + cases: [ + expect.objectContaining({ + provider: 'fixture', + model: FIXTURE_MODEL, + lane: 'prompt', + caseId: 'wait-for-output', + condition: 'none', + outcome: 'unchanged', + currentTotalTokens: 200, + snapshotTotalTokens: 200, + deltaTokens: 0, + deltaPercent: 0, + }), + ], + }); + + writeFixtureResponse(fixtureRoot, REGRESSED_TOKEN_USAGE); + const regressionOutputDir = join(testRoot, 'regression-output'); + const regressionSummary = parseRunSummary( + runEvalCli( + [ + ...buildSharedFixtureArgs(regressionOutputDir), + '--snapshot-check', + '--snapshot-dir', + snapshotDir, + '--snapshot-threshold', + '20', + ], + buildFixtureEnv(testRoot, fixtureRoot, 'home-regression'), + ).stdout, + ); + const regressionReport = readJsonReport(regressionSummary); + expect(regressionSummary.ok).toBe(updateSummary.ok); + expect(regressionReport.aggregate).toEqual(updateReport.aggregate); + expect(projectNonTokenResults(regressionReport)).toEqual( + projectNonTokenResults(updateReport), + ); + expect( + regressionReport.tokenReport?.snapshotCheck?.summary.regressed, + ).toBe(1); + expect(regressionReport.tokenReport?.snapshotCheck?.cases).toEqual([ + expect.objectContaining({ + provider: 'fixture', + model: FIXTURE_MODEL, + lane: 'prompt', + caseId: 'wait-for-output', + condition: 'none', + outcome: 'regressed', + currentTotalTokens: 260, + snapshotTotalTokens: 200, + deltaTokens: 60, + deltaPercent: 30, + }), + ]); + }); + + it('uses the default snapshot directory and default threshold when flags are omitted', () => { + const fixtureRoot = join(testRoot, 'fixture-defaults'); + const outputDir = join(testRoot, 'default-output'); + createFixtureProviderDirectory(fixtureRoot, BASELINE_TOKEN_USAGE); + + const updateSummary = parseRunSummary( + runEvalCli( + [...buildSharedFixtureArgs(outputDir), '--snapshot-update'], + buildFixtureEnv(testRoot, fixtureRoot, 'home-default-update'), + ).stdout, + ); + const defaultSnapshotPath = snapshotFilePath( + join(outputDir, 'snapshots'), + 'fixture', + FIXTURE_MODEL, + ); + expect(existsSync(defaultSnapshotPath)).toBe(true); + expect( + readJsonReport(updateSummary).tokenReport?.snapshotCheck, + ).toBeUndefined(); + + const checkSummary = parseRunSummary( + runEvalCli( + [...buildSharedFixtureArgs(outputDir), '--snapshot-check'], + buildFixtureEnv(testRoot, fixtureRoot, 'home-default-check'), + ).stdout, + ); + expect( + readJsonReport(checkSummary).tokenReport?.snapshotCheck, + ).toMatchObject({ + regressionThresholdPercent: 20, + summary: { + total: 1, + new: 0, + orphaned: 0, + unchanged: 1, + improved: 0, + regressed: 0, + }, + }); + }); + + it('rejects conflicting snapshot modes before creating output or snapshot directories', () => { + const outputDir = join(testRoot, 'conflict-output'); + const snapshotDir = join(testRoot, 'conflict-snapshots'); + const result = runEvalCli( + [ + '--provider', + 'stub', + '--lane', + 'execution', + '--case', + 'hello-prompt', + '--condition', + 'none', + '--output', + outputDir, + '--snapshot-dir', + snapshotDir, + '--snapshot-update', + '--snapshot-check', + '--dry-run', + '--json', + ], + { + AGENT_TTY_HOME: join(testRoot, 'home-conflict'), + }, + ); + + expect(result.status).toBe(1); + expect(JSON.parse(result.stdout) as { error: string }).toMatchObject({ + error: '--snapshot-update and --snapshot-check may not be combined', + }); + expect(existsSync(outputDir)).toBe(false); + expect(existsSync(snapshotDir)).toBe(false); + }); + + it.each(['NaN', '-1', '101'])( + 'rejects invalid snapshot thresholds at parse time: %s', + (threshold) => { + const result = runEvalCli( + [ + '--provider', + 'stub', + '--lane', + 'execution', + '--case', + 'hello-prompt', + '--condition', + 'none', + '--snapshot-check', + '--snapshot-threshold', + threshold, + '--dry-run', + '--json', + ], + { + AGENT_TTY_HOME: join(testRoot, `home-threshold-${threshold}`), + }, + ); + + expect(result.status).toBe(1); + expect( + (JSON.parse(result.stdout) as { error: string }).error, + ).toContain('--snapshot-threshold must be a number between 0 and 100'); + }, + ); + + it('keeps the legacy report shape and writes no snapshots when trials omit token usage', () => { + const fixtureRoot = join(testRoot, 'fixture-no-token'); + const outputDir = join(testRoot, 'no-token-output'); + createFixtureProviderDirectory(fixtureRoot); + + const summary = parseRunSummary( + runEvalCli( + buildSharedFixtureArgs(outputDir), + buildFixtureEnv(testRoot, fixtureRoot, 'home-no-token'), + ).stdout, + ); + expect(summary.ok).toBe(true); + + const report = readJsonReport(summary); + expect(report).not.toHaveProperty('tokenReport'); + expect(existsSync(join(outputDir, 'snapshots'))).toBe(false); + }); + }, +); diff --git a/test/integration/evals-workspace-presets.test.ts b/test/integration/evals-workspace-presets.test.ts new file mode 100644 index 00000000..8cb00d22 --- /dev/null +++ b/test/integration/evals-workspace-presets.test.ts @@ -0,0 +1,154 @@ +import { spawnSync } from 'node:child_process'; +import { + mkdirSync, + mkdtempSync, + readFileSync, + realpathSync, + rmSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import process from 'node:process'; + +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; + +import type { CaseStartEvent } from '../../evals/reporters/types.js'; + +const DEFAULT_EVAL_TIMEOUT_MS = 30_000; + +interface EvalRunSummary { + ok: boolean; + outputBaseDir: string; +} + +interface JsonlRecord> { + type: string; + timestamp: string; + payload: TPayload; +} + +function readJsonlRecords(outputPath: string): JsonlRecord[] { + const content = readFileSync(outputPath, 'utf8').trimEnd(); + expect(content.length).toBeGreaterThan(0); + return content.split('\n').map((line) => JSON.parse(line) as JsonlRecord); +} + +function getRequiredCaseStartPayload( + records: readonly JsonlRecord[], +): CaseStartEvent { + const caseStartRecords = records.filter( + (record) => record.type === 'case.start', + ); + + expect(caseStartRecords).toHaveLength(1); + const caseStartRecord = caseStartRecords[0]; + if (caseStartRecord === undefined) { + throw new Error('Expected exactly one case.start record'); + } + + return caseStartRecord.payload as CaseStartEvent; +} + +function runJsonlExecutionEval( + testRoot: string, + caseId: string, +): { + summary: EvalRunSummary; + records: JsonlRecord[]; +} { + const homeDir = join(testRoot, `${caseId}-home`); + const outputDir = join(testRoot, `${caseId}-output`); + const reporterOutputPath = join(testRoot, `${caseId}.jsonl`); + + mkdirSync(homeDir, { recursive: true }); + mkdirSync(outputDir, { recursive: true }); + + const result = spawnSync( + process.execPath, + [ + '--import', + 'tsx', + './evals/run.ts', + '--provider', + 'stub', + '--lane', + 'execution', + '--case', + caseId, + '--condition', + 'none', + '--trials', + '1', + '--concurrency', + '1', + '--reporter', + 'jsonl', + '--reporter-output', + reporterOutputPath, + '--output', + outputDir, + '--json', + ], + { + cwd: process.cwd(), + encoding: 'utf8', + env: { + ...process.env, + AGENT_TTY_HOME: homeDir, + }, + timeout: DEFAULT_EVAL_TIMEOUT_MS, + }, + ); + + expect(result.error).toBeUndefined(); + expect(result.status).not.toBeNull(); + + return { + summary: JSON.parse(result.stdout) as EvalRunSummary, + records: readJsonlRecords(reporterOutputPath), + }; +} + +let testRoot = ''; + +describe( + 'eval workspace preset reporting', + { timeout: DEFAULT_EVAL_TIMEOUT_MS }, + () => { + beforeEach(() => { + testRoot = realpathSync( + mkdtempSync(join(tmpdir(), 'agent-tty-evals-workspace-presets-')), + ); + }); + + afterEach(() => { + rmSync(testRoot, { recursive: true, force: true }); + testRoot = ''; + }); + + it('includes the builtin workspace plan on case.start for hello-prompt', () => { + const { records, summary } = runJsonlExecutionEval( + testRoot, + 'hello-prompt', + ); + const caseStartPayload = getRequiredCaseStartPayload(records); + + expect(summary.outputBaseDir).toBe(join(testRoot, 'hello-prompt-output')); + expect(caseStartPayload.caseId).toBe('hello-prompt'); + expect(caseStartPayload.workspace).toBeDefined(); + expect(caseStartPayload.workspace?.presetId).toBe('agent-tty-smoke'); + expect(caseStartPayload.workspace?.bootstrapCount).toBe(1); + expect(caseStartPayload.workspace?.cwd).toBeUndefined(); + expect(caseStartPayload.workspace?.env ?? {}).toEqual({}); + }); + + it('omits the workspace block for legacy execution cases without a preset', () => { + const { records } = runJsonlExecutionEval(testRoot, 'color-grid'); + const caseStartPayload = getRequiredCaseStartPayload(records); + + expect(caseStartPayload.caseId).toBe('color-grid'); + expect(caseStartPayload).not.toHaveProperty('workspace'); + expect(caseStartPayload.workspace).toBeUndefined(); + }); + }, +); diff --git a/test/integration/evals/authoring-pilots.test.ts b/test/integration/evals/authoring-pilots.test.ts new file mode 100644 index 00000000..c3193e5f --- /dev/null +++ b/test/integration/evals/authoring-pilots.test.ts @@ -0,0 +1,872 @@ +import { spawnSync } from 'node:child_process'; +import { + mkdtempSync, + mkdirSync, + readFileSync, + realpathSync, + rmSync, + writeFileSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import process from 'node:process'; + +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { z } from 'zod'; + +import exploratoryQaCase from '../../../evals/dogfood/cases/exploratory-qa.js'; +import { doctorGatedCase } from '../../../evals/execution/cases/doctor-gated.js'; +import { helloPromptCase } from '../../../evals/execution/cases/hello-prompt.js'; +import { + JsonReportSchema, + NormalizedProviderOutputSchema, + ProviderAgentRequestSchema, + ProviderAgentResultSchema, + ProviderPromptRequestSchema, + ProviderPromptResultSchema, + ProviderRuntimeInfoSchema, +} from '../../../evals/lib/schemas.js'; +import { createFixtureProvider } from '../../../evals/providers/fixtures.js'; +import { TRIGGER_AGENT_TTY_PROMPT_CASES } from '../../../evals/prompt/cases/trigger-agent-tty.js'; +import type { + DogfoodEvalCase, + ExecutionEvalCase, + PromptEvalCase, + ProviderAgentRequest, + ProviderAgentResult, + ProviderPromptRequest, + ProviderPromptResult, + ProviderRuntimeInfo, +} from '../../../evals/lib/types.js'; + +const DEFAULT_EVAL_TIMEOUT_MS = 45_000; +const FIXTURE_STARTED_AT = '2026-01-01T00:00:00.000Z'; +const FIXTURE_COMPLETED_AT = '2026-01-01T00:00:00.001Z'; +const PLACEHOLDER_HOME_DIR = '/tmp/agent-tty-evals-home'; +const PLACEHOLDER_OUTPUT_DIR = '/tmp/agent-tty-evals-output'; +const PLACEHOLDER_RUN_ID = 'fixture-run'; +const PLACEHOLDER_PROVIDER_ID = 'fixture'; +const PLACEHOLDER_MODEL_ID = 'fixture-model'; + +const LaneSchema = z.enum(['prompt', 'execution', 'dogfood']); +const ConditionSchema = z.enum(['none', 'self-load', 'preloaded', 'stale']); +const ExpectedSkillSchema = z.enum(['none', 'agent-tty', 'dogfood-tui']); +const LaneErrorSchema = z + .object({ + lane: LaneSchema, + message: z.string().min(1), + }) + .strict(); +const SelectedCaseSchema = z + .object({ + lane: LaneSchema, + caseId: z.string().min(1), + category: z.string().min(1), + expectedSkill: ExpectedSkillSchema, + conditions: z.array(ConditionSchema).min(1), + fixture: z.string().min(1).optional(), + target: z.string().min(1).optional(), + }) + .strict(); +const EvalRunSummarySchema = z + .object({ + ok: z.boolean(), + runId: z.string().min(1).optional(), + providerId: z.string().min(1), + modelId: z.string().min(1).optional(), + lanes: z.array(LaneSchema).min(1), + conditions: z.array(ConditionSchema).min(1), + totalInvocations: z.number().int().nonnegative(), + totalResults: z.number().int().nonnegative(), + passed: z.number().int().nonnegative(), + failed: z.number().int().nonnegative(), + outputBaseDir: z.string().min(1), + runDir: z.string().min(1).optional(), + jsonReportPath: z.string().min(1).optional(), + markdownReportPath: z.string().min(1).optional(), + laneErrors: z.array(LaneErrorSchema), + dryRun: z.boolean(), + selectedCases: z.array(SelectedCaseSchema).min(1), + }) + .strict(); + +const JsonReportEnvelopeSchema = z + .object({ + metadata: JsonReportSchema.shape.metadata, + results: JsonReportSchema.shape.results, + }) + .loose(); + +type EvalRunSummary = z.infer; + +type PilotCaseExpectation = { + lane: z.infer; + caseId: string; + expectedSkill: z.infer; + expectedOk: boolean; + fixture?: string; +}; + +const FIXTURE_RUNTIME_INFO = ProviderRuntimeInfoSchema.parse({ + providerId: PLACEHOLDER_PROVIDER_ID, + available: true, + detectedAt: FIXTURE_STARTED_AT, + version: 'fixture', + commandPath: 'fixture', + defaultModelId: PLACEHOLDER_MODEL_ID, + capabilities: { + supportsDetect: true, + supportsPlanMode: true, + supportsAgentMode: true, + supportsStreaming: false, + supportsToolCalls: true, + supportsTranscriptCapture: true, + }, + notes: ['eval authoring pilot integration fixture'], +}) as ProviderRuntimeInfo; + +const VALID_TOKEN_USAGE = { + inputTokens: 120, + outputTokens: 45, + totalTokens: 165, + cachedTokens: 15, +} as const; +const TokenUsageArtifactSchema = z + .object({ + caseId: z.string().min(1), + lane: LaneSchema, + condition: ConditionSchema, + provider: z.string().min(1), + model: z.string().min(1), + trialIndex: z.number().int().nonnegative(), + tokenUsage: z + .object({ + inputTokens: z.number().int().nonnegative(), + outputTokens: z.number().int().nonnegative(), + totalTokens: z.number().int().nonnegative(), + cachedTokens: z.number().int().nonnegative().optional(), + }) + .strict(), + createdAtMs: z.number().int().nonnegative(), + }) + .strict(); +const INVALID_TOKEN_USAGE_CASES = [ + [ + 'partial', + { + inputTokens: 1, + outputTokens: 2, + }, + ], + [ + 'negative', + { + inputTokens: -1, + outputTokens: 2, + totalTokens: 1, + }, + ], + [ + 'fractional', + { + inputTokens: 1.5, + outputTokens: 2, + totalTokens: 3.5, + }, + ], + [ + 'extra key', + { + inputTokens: 1, + outputTokens: 2, + totalTokens: 3, + extra: 4, + }, + ], +] as const; + +const waitForOutputCase = findPromptCaseOrThrow('wait-for-output'); +const PILOT_CASES: readonly PilotCaseExpectation[] = [ + { + lane: 'prompt', + caseId: 'wait-for-output', + expectedSkill: 'agent-tty', + expectedOk: true, + }, + { + lane: 'execution', + caseId: 'hello-prompt', + expectedSkill: 'agent-tty', + expectedOk: true, + fixture: 'hello-prompt', + }, + { + lane: 'execution', + caseId: 'doctor-gated', + expectedSkill: 'agent-tty', + expectedOk: true, + fixture: 'hello-prompt', + }, + { + lane: 'dogfood', + caseId: 'exploratory-qa', + expectedSkill: 'dogfood-tui', + expectedOk: true, + fixture: 'hello-prompt', + }, +] as const; + +function findPromptCaseOrThrow(caseId: string): PromptEvalCase { + const evalCase = TRIGGER_AGENT_TTY_PROMPT_CASES.find( + (candidate) => candidate.id === caseId, + ); + if (evalCase === undefined) { + throw new Error(`Expected prompt case ${caseId} to be registered`); + } + return evalCase; +} + +function requireDefined(value: T | undefined, label: string): T { + if (value === undefined) { + throw new Error(`${label} must be defined`); + } + return value; +} + +function buildPromptRequest(evalCase: PromptEvalCase): ProviderPromptRequest { + return ProviderPromptRequestSchema.parse({ + runId: PLACEHOLDER_RUN_ID, + providerId: PLACEHOLDER_PROVIDER_ID, + condition: 'none', + trial: 1, + modelId: PLACEHOLDER_MODEL_ID, + cwd: process.cwd(), + evalCase, + }) as ProviderPromptRequest; +} + +function buildAgentRequest( + evalCase: ExecutionEvalCase | DogfoodEvalCase, +): ProviderAgentRequest { + return ProviderAgentRequestSchema.parse({ + runId: PLACEHOLDER_RUN_ID, + providerId: PLACEHOLDER_PROVIDER_ID, + condition: 'none', + trial: 1, + modelId: PLACEHOLDER_MODEL_ID, + cwd: process.cwd(), + homeDir: PLACEHOLDER_HOME_DIR, + outputDir: PLACEHOLDER_OUTPUT_DIR, + evalCase, + }) as ProviderAgentRequest; +} + +function buildNormalizedOutput(text: string) { + return NormalizedProviderOutputSchema.parse({ + finalText: text, + messages: [text], + referencedSkills: [], + toolCalls: [], + }); +} + +function buildPromptFixture( + evalCase: PromptEvalCase, + responseText: string, +): ProviderPromptResult { + const request = buildPromptRequest(evalCase); + + return ProviderPromptResultSchema.parse({ + request, + runtime: FIXTURE_RUNTIME_INFO, + ok: true, + exitCode: 0, + signal: null, + startedAt: FIXTURE_STARTED_AT, + completedAt: FIXTURE_COMPLETED_AT, + durationMs: 1, + rawStdout: responseText, + rawStderr: '', + normalized: buildNormalizedOutput(responseText), + }) as ProviderPromptResult; +} + +function buildAgentFixture( + evalCase: ExecutionEvalCase | DogfoodEvalCase, + transcript: string, + options: { + bundlePath?: string; + rawStderr?: string; + } = {}, +): ProviderAgentResult { + const request = buildAgentRequest(evalCase); + + return ProviderAgentResultSchema.parse({ + request, + runtime: FIXTURE_RUNTIME_INFO, + ok: true, + exitCode: 0, + signal: null, + startedAt: FIXTURE_STARTED_AT, + completedAt: FIXTURE_COMPLETED_AT, + durationMs: 1, + rawStdout: transcript, + rawStderr: options.rawStderr ?? '', + normalized: buildNormalizedOutput(transcript), + ...(options.bundlePath === undefined + ? {} + : { bundlePath: options.bundlePath }), + }) as ProviderAgentResult; +} + +function writeJson(filePath: string, value: unknown): void { + mkdirSync(dirname(filePath), { recursive: true }); + writeFileSync(filePath, JSON.stringify(value, null, 2)); +} + +function writeText(filePath: string, value: string): void { + mkdirSync(dirname(filePath), { recursive: true }); + writeFileSync(filePath, value, 'utf8'); +} + +function createFixtureProviderDirectory(fixtureRoot: string): void { + const promptResponse = + 'Use agent-tty and run `agent-tty wait --json --pattern "Listening on port 3000"` before starting the tests so you only proceed after the server is ready.'; + const helloTranscript = [ + 'agent-tty create --json', + 'agent-tty type --json --text "hello world"', + 'agent-tty wait --json --pattern "READY>"', + 'ECHO: hello world', + 'READY>', + 'agent-tty snapshot --json', + 'agent-tty destroy --json', + ].join('\n'); + const doctorTranscript = [ + 'agent-tty create --json', + 'agent-tty doctor --json', + '{"checks":[{"name":"screenshot","status":"pass"}]}', + 'agent-tty screenshot --json doctor-gated-proof.png', + ].join('\n'); + const dogfoodNotes = [ + '# Title', + 'hello-prompt exploratory QA', + '', + '## Reproduction steps', + '1. Run `npx tsx src/cli/main.ts create --json` for the hello-prompt fixture.', + '2. Use `npx tsx src/cli/main.ts snapshot --json` after each scripted input.', + '', + 'Expected: the fixture should echo input and exit cleanly.', + 'Actual: consistent clean shutdown across the scripted inputs.', + '', + '## Findings', + '- Severity: low', + '- Focus / input: blank input was accepted without crashing.', + '', + '## Evidence', + '- screenshot: proof.png', + '- recording: proof.cast', + '- notes: notes.md', + '- manifest: manifest.json', + ].join('\n'); + + const doctorBundleDir = join(fixtureRoot, 'artifacts', 'doctor-gated'); + const dogfoodBundleDir = join(fixtureRoot, 'bundles', 'exploratory-qa'); + mkdirSync(doctorBundleDir, { recursive: true }); + mkdirSync(dogfoodBundleDir, { recursive: true }); + writeText(join(doctorBundleDir, 'doctor-gated-proof.png'), 'png fixture'); + writeText(join(dogfoodBundleDir, 'proof.png'), 'png fixture'); + writeText(join(dogfoodBundleDir, 'proof.cast'), 'cast fixture'); + writeText(join(dogfoodBundleDir, 'notes.md'), dogfoodNotes); + writeJson(join(dogfoodBundleDir, 'manifest.json'), { + generatedBy: 'authoring-pilots.test.ts', + artifacts: ['proof.png', 'proof.cast', 'notes.md', 'manifest.json'], + }); + writeText( + join(dogfoodBundleDir, 'index.html'), + 'fixture', + ); + + writeJson(join(fixtureRoot, 'runtime-info.json'), FIXTURE_RUNTIME_INFO); + writeJson( + join(fixtureRoot, 'responses', 'wait-for-output.json'), + buildPromptFixture(waitForOutputCase, promptResponse), + ); + writeJson( + join(fixtureRoot, 'agent-results', 'hello-prompt.json'), + buildAgentFixture(helloPromptCase, helloTranscript), + ); + writeJson( + join(fixtureRoot, 'agent-results', 'doctor-gated.json'), + buildAgentFixture(doctorGatedCase, doctorTranscript, { + bundlePath: doctorBundleDir, + }), + ); + writeJson( + join(fixtureRoot, 'agent-results', 'exploratory-qa.json'), + buildAgentFixture(exploratoryQaCase, dogfoodNotes, { + bundlePath: dogfoodBundleDir, + }), + ); +} + +function runEvalCli( + argumentsList: readonly string[], + extraEnv: NodeJS.ProcessEnv = {}, +) { + const result = spawnSync( + process.execPath, + ['--import', 'tsx', './evals/run.ts', ...argumentsList], + { + cwd: process.cwd(), + encoding: 'utf8', + timeout: DEFAULT_EVAL_TIMEOUT_MS, + env: { + ...process.env, + ...extraEnv, + }, + }, + ); + + expect(result.error).toBeUndefined(); + expect(result.status).not.toBeNull(); + expect(result.stderr).toBe(''); + return result; +} + +function parseSummary(stdout: string): EvalRunSummary { + return EvalRunSummarySchema.parse(JSON.parse(stdout)); +} + +function expectSelectedCase( + summary: EvalRunSummary, + pilot: PilotCaseExpectation, +): void { + expect(summary.selectedCases).toHaveLength(1); + expect(summary.selectedCases[0]).toMatchObject({ + lane: pilot.lane, + caseId: pilot.caseId, + expectedSkill: pilot.expectedSkill, + conditions: ['none'], + ...(pilot.fixture === undefined ? {} : { fixture: pilot.fixture }), + }); +} + +function readReport(summary: EvalRunSummary) { + if (summary.jsonReportPath === undefined) { + throw new Error('Expected jsonReportPath in non-dry-run eval summary'); + } + const text = readFileSync(summary.jsonReportPath, 'utf8'); + return JsonReportEnvelopeSchema.parse(JSON.parse(text)); +} + +function setPromptFixtureTokenUsage(caseId: string): void { + const fixturePath = join(fixtureRoot, 'responses', `${caseId}.json`); + const fixture = ProviderPromptResultSchema.parse( + JSON.parse(readFileSync(fixturePath, 'utf8')), + ); + writeJson(fixturePath, { + ...fixture, + normalized: { + ...fixture.normalized, + tokenUsage: VALID_TOKEN_USAGE, + }, + }); +} + +function setAgentFixtureTokenUsage(caseId: string): void { + const fixturePath = join(fixtureRoot, 'agent-results', `${caseId}.json`); + const fixture = ProviderAgentResultSchema.parse( + JSON.parse(readFileSync(fixturePath, 'utf8')), + ); + writeJson(fixturePath, { + ...fixture, + normalized: { + ...fixture.normalized, + tokenUsage: VALID_TOKEN_USAGE, + }, + }); +} + +function resolveTokenUsageArtifactPath( + summary: EvalRunSummary, + lane: 'prompt' | 'execution' | 'dogfood', + caseId: string, +): string { + return join( + requireDefined(summary.runDir, 'summary.runDir'), + lane, + caseId, + 'none', + 'token-usage.json', + ); +} + +function readTokenUsageArtifact( + summary: EvalRunSummary, + lane: 'prompt' | 'execution' | 'dogfood', + caseId: string, +) { + return TokenUsageArtifactSchema.parse( + JSON.parse( + readFileSync( + resolveTokenUsageArtifactPath(summary, lane, caseId), + 'utf8', + ), + ), + ); +} + +let testRoot = ''; +let fixtureRoot = ''; + +describe( + 'eval CLI authoring pilot cases', + { timeout: DEFAULT_EVAL_TIMEOUT_MS }, + () => { + beforeEach(() => { + // prettier-ignore + testRoot = realpathSync(mkdtempSync(join(tmpdir(), 'agent-tty-evals-authoring-pilots-'))); + fixtureRoot = join(testRoot, 'fixture-provider'); + createFixtureProviderDirectory(fixtureRoot); + }); + + afterEach(() => { + rmSync(testRoot, { recursive: true, force: true }); + testRoot = ''; + fixtureRoot = ''; + }); + + for (const pilot of PILOT_CASES) { + it(`resolves ${pilot.caseId} in the dry-run matrix`, () => { + const result = runEvalCli([ + '--provider', + 'stub', + '--lane', + pilot.lane, + '--case', + pilot.caseId, + '--condition', + 'none', + '--dry-run', + '--json', + ]); + + expect(result.status).toBe(0); + const summary = parseSummary(result.stdout); + expect(summary).toMatchObject({ + ok: true, + providerId: 'stub', + lanes: [pilot.lane], + conditions: ['none'], + totalInvocations: 1, + totalResults: 0, + passed: 0, + failed: 0, + dryRun: true, + laneErrors: [], + }); + expectSelectedCase(summary, pilot); + }); + + it(`runs ${pilot.caseId} end-to-end through the CLI facade with fixture playback`, () => { + const outputDir = join(testRoot, `run-${pilot.caseId}`); + const result = runEvalCli( + [ + '--provider', + 'fixture', + '--lane', + pilot.lane, + '--case', + pilot.caseId, + '--condition', + 'none', + '--output', + outputDir, + '--json', + ], + { + EVAL_FIXTURE_DIR: fixtureRoot, + }, + ); + + const summary = parseSummary(result.stdout); + expect(result.status).toBe(summary.ok ? 0 : 1); + expect(summary).toMatchObject({ + ok: pilot.expectedOk, + providerId: 'fixture', + lanes: [pilot.lane], + conditions: ['none'], + totalInvocations: 1, + totalResults: 1, + passed: pilot.expectedOk ? 1 : 0, + failed: pilot.expectedOk ? 0 : 1, + dryRun: false, + laneErrors: [], + }); + expectSelectedCase(summary, pilot); + + const report = readReport(summary); + expect(report.metadata.providers).toEqual(['fixture']); + expect(report.metadata.lanes).toEqual([pilot.lane]); + expect(report.metadata.conditions).toEqual(['none']); + expect(report.results).toHaveLength(1); + + const singleResult = requireDefined( + report.results[0], + 'report.results[0]', + ); + expect(singleResult).toMatchObject({ + lane: pilot.lane, + caseId: pilot.caseId, + condition: 'none', + expectedSkill: pilot.expectedSkill, + ok: pilot.expectedOk, + }); + expect(singleResult.errorClass).toBeUndefined(); + expect(singleResult.errorMessage).toBeUndefined(); + }); + } + it('round-trips tokenUsage from full normalized override fixtures', async () => { + const expectedNormalized = NormalizedProviderOutputSchema.parse({ + finalText: 'normalized override text', + messages: ['raw override text', 'normalized override text'], + referencedSkills: ['agent-tty'], + selectedSkill: 'agent-tty', + toolCalls: [{ tool: 'wait', pattern: 'Listening on port 3000' }], + tokenUsage: VALID_TOKEN_USAGE, + }); + writeJson( + join(fixtureRoot, 'normalized', 'wait-for-output.json'), + expectedNormalized, + ); + + const provider = createFixtureProvider(fixtureRoot); + const result = ProviderPromptResultSchema.parse( + await provider.invokePlanMode(buildPromptRequest(waitForOutputCase)), + ); + + expect(NormalizedProviderOutputSchema.parse(result.normalized)).toEqual( + expectedNormalized, + ); + expect(result.normalized.tokenUsage).toEqual(VALID_TOKEN_USAGE); + }); + + it('preserves inline prompt and agent tokenUsage and keeps omissions undefined', async () => { + const promptRequest = buildPromptRequest(waitForOutputCase); + const agentRequest = buildAgentRequest(helloPromptCase); + + const defaultProvider = createFixtureProvider(fixtureRoot); + const defaultPromptResult = ProviderPromptResultSchema.parse( + await defaultProvider.invokePlanMode(promptRequest), + ); + const defaultAgentResult = ProviderAgentResultSchema.parse( + await defaultProvider.invokeAgentMode(agentRequest), + ); + expect(defaultPromptResult.normalized.tokenUsage).toBeUndefined(); + expect(defaultPromptResult.normalized).not.toHaveProperty('tokenUsage'); + expect(defaultAgentResult.normalized.tokenUsage).toBeUndefined(); + expect(defaultAgentResult.normalized).not.toHaveProperty('tokenUsage'); + + const promptText = 'prompt fallback token usage'; + const agentText = 'agent fallback token usage'; + writeJson(join(fixtureRoot, 'responses', 'wait-for-output.json'), { + response: promptText, + tokenUsage: VALID_TOKEN_USAGE, + }); + writeJson(join(fixtureRoot, 'agent-results', 'hello-prompt.json'), { + transcript: agentText, + tokenUsage: VALID_TOKEN_USAGE, + }); + + const provider = createFixtureProvider(fixtureRoot); + const promptResult = ProviderPromptResultSchema.parse( + await provider.invokePlanMode(promptRequest), + ); + const agentResult = ProviderAgentResultSchema.parse( + await provider.invokeAgentMode(agentRequest), + ); + + expect( + NormalizedProviderOutputSchema.parse(promptResult.normalized), + ).toEqual( + NormalizedProviderOutputSchema.parse({ + finalText: promptText, + messages: [promptText], + referencedSkills: [], + toolCalls: [], + tokenUsage: VALID_TOKEN_USAGE, + }), + ); + expect( + NormalizedProviderOutputSchema.parse(agentResult.normalized), + ).toEqual( + NormalizedProviderOutputSchema.parse({ + finalText: agentText, + messages: [agentText], + referencedSkills: [], + toolCalls: [], + tokenUsage: VALID_TOKEN_USAGE, + }), + ); + }); + + it('preserves valid legacy normalized tokenUsage and drops invalid tokenUsage payloads', async () => { + const promptRequest = buildPromptRequest(waitForOutputCase); + const agentRequest = buildAgentRequest(helloPromptCase); + const normalizedFixturePath = join( + fixtureRoot, + 'normalized', + 'hello-prompt.json', + ); + const validLegacyNormalized = { + rawText: 'legacy raw text', + normalizedText: 'legacy normalized text', + skillDetected: 'agent-tty', + toolCalls: [{ tool: 'snapshot' }], + tokenUsage: VALID_TOKEN_USAGE, + }; + writeJson(normalizedFixturePath, validLegacyNormalized); + + let provider = createFixtureProvider(fixtureRoot); + let agentResult = ProviderAgentResultSchema.parse( + await provider.invokeAgentMode(agentRequest), + ); + expect( + NormalizedProviderOutputSchema.parse(agentResult.normalized), + ).toEqual( + NormalizedProviderOutputSchema.parse({ + finalText: validLegacyNormalized.normalizedText, + messages: [ + validLegacyNormalized.rawText, + validLegacyNormalized.normalizedText, + ], + referencedSkills: ['agent-tty'], + selectedSkill: 'agent-tty', + toolCalls: validLegacyNormalized.toolCalls, + tokenUsage: VALID_TOKEN_USAGE, + }), + ); + + for (const [, invalidTokenUsage] of INVALID_TOKEN_USAGE_CASES) { + writeJson(join(fixtureRoot, 'responses', 'wait-for-output.json'), { + response: 'prompt inline invalid token usage', + tokenUsage: invalidTokenUsage, + }); + writeJson(normalizedFixturePath, { + rawText: 'legacy raw invalid token usage', + normalizedText: 'legacy normalized invalid token usage', + tokenUsage: invalidTokenUsage, + }); + + provider = createFixtureProvider(fixtureRoot); + const promptResult = ProviderPromptResultSchema.parse( + await provider.invokePlanMode(promptRequest), + ); + agentResult = ProviderAgentResultSchema.parse( + await provider.invokeAgentMode(agentRequest), + ); + + expect(promptResult.normalized.tokenUsage).toBeUndefined(); + expect(promptResult.normalized).not.toHaveProperty('tokenUsage'); + expect(agentResult.normalized.tokenUsage).toBeUndefined(); + expect(agentResult.normalized).not.toHaveProperty('tokenUsage'); + } + }); + + const tokenUsageArtifactCases = [ + { + lane: 'prompt' as const, + caseId: 'wait-for-output', + enableTokenUsage: () => setPromptFixtureTokenUsage('wait-for-output'), + }, + { + lane: 'execution' as const, + caseId: 'hello-prompt', + enableTokenUsage: () => setAgentFixtureTokenUsage('hello-prompt'), + }, + { + lane: 'dogfood' as const, + caseId: 'exploratory-qa', + enableTokenUsage: () => setAgentFixtureTokenUsage('exploratory-qa'), + }, + ]; + + for (const testCase of tokenUsageArtifactCases) { + it(`writes a token-usage sidecar for ${testCase.lane}:${testCase.caseId} when token usage is present`, () => { + testCase.enableTokenUsage(); + + const outputDir = join( + testRoot, + `token-usage-present-${testCase.lane}-${testCase.caseId}`, + ); + const result = runEvalCli( + [ + '--provider', + 'fixture', + '--lane', + testCase.lane, + '--case', + testCase.caseId, + '--condition', + 'none', + '--output', + outputDir, + '--json', + ], + { EVAL_FIXTURE_DIR: fixtureRoot }, + ); + + expect(result.status).toBe(0); + const summary = parseSummary(result.stdout); + expect( + readTokenUsageArtifact(summary, testCase.lane, testCase.caseId), + ).toEqual({ + caseId: testCase.caseId, + lane: testCase.lane, + condition: 'none', + provider: 'fixture', + model: PLACEHOLDER_MODEL_ID, + trialIndex: 0, + tokenUsage: VALID_TOKEN_USAGE, + createdAtMs: Date.parse(FIXTURE_COMPLETED_AT), + }); + }); + + it(`skips the token-usage sidecar for ${testCase.lane}:${testCase.caseId} when token usage is absent`, () => { + const outputDir = join( + testRoot, + `token-usage-absent-${testCase.lane}-${testCase.caseId}`, + ); + const result = runEvalCli( + [ + '--provider', + 'fixture', + '--lane', + testCase.lane, + '--case', + testCase.caseId, + '--condition', + 'none', + '--output', + outputDir, + '--json', + ], + { EVAL_FIXTURE_DIR: fixtureRoot }, + ); + + expect(result.status).toBe(0); + const summary = parseSummary(result.stdout); + expect(() => + readFileSync( + resolveTokenUsageArtifactPath( + summary, + testCase.lane, + testCase.caseId, + ), + 'utf8', + ), + ).toThrow(); + }); + } + }, +); diff --git a/test/unit/evals/authoring/dogfood.test.ts b/test/unit/evals/authoring/dogfood.test.ts new file mode 100644 index 00000000..60bec0c6 --- /dev/null +++ b/test/unit/evals/authoring/dogfood.test.ts @@ -0,0 +1,413 @@ +import { describe, expect, it } from 'vitest'; + +import { + dogfoodCase, + rawArtifactRequirement, + rawReportRequirement, + rawVerifier, + rawWorkflowCheck, +} from '../../../../evals/authoring/index.js'; +import { DogfoodEvalCaseSchema } from '../../../../evals/lib/schemas.js'; +import type { + ArtifactRequirement, + ReportRequirement, + VerifierSpec, + WorkflowCheck, +} from '../../../../evals/lib/types.js'; + +function escapeRegex(value: string): string { + return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} + +function dogfoodErrorPattern( + caseId: string, + path: string, + message: string, +): RegExp { + return new RegExp( + escapeRegex(`Invalid dogfood case "${caseId}" at ${path}: ${message}`), + 'u', + ); +} + +function createDogfoodBuilder(id = 'dogfood-authoring-happy') { + return dogfoodCase(id) + .category('qa') + .task( + 'Exercise the hello-prompt fixture, capture evidence, and summarize findings.', + ) + .fixture('hello-prompt') + .bundlePath('dogfood/bundles/hello-prompt') + .bundleRequirement('Include screenshots, recordings, and written notes.') + .validationProfile('interactive-renderer') + .proofBundle((bundle) => { + bundle.requiresScreenshot(); + bundle.requiresRecording(); + }) + .report((report) => { + report.title(); + report.reproductionSteps(); + }) + .bundleVerifier() + .workflow((workflow) => { + workflow + .step('launch', 'Launch the workflow.') + .mustMention('agent-tty create') + .weight(2) + .step('evidence', 'Capture reviewer evidence.') + .mustMention('agent-tty screenshot') + .after('launch'); + }) + .budget({ timeoutMs: 120_000, maxAgentSteps: 12, maxWallClockMs: 90_000 }) + .referenceSteps(4); +} + +function createRawDogfoodWorkflowCheck(id = 'raw-workflow'): WorkflowCheck { + return { + id, + description: 'Keep the raw dogfood workflow check unchanged.', + required: false, + requiredPatterns: ['agent-tty screenshot'], + forbiddenPatterns: ['sleep 10'], + dependsOn: ['launch'], + weight: 3, + }; +} + +function createRawDogfoodVerifier(id = 'raw-verifier'): VerifierSpec { + return { + id, + kind: 'bundle', + description: 'Keep the raw dogfood verifier unchanged.', + required: true, + config: { + profile: 'interactive-renderer', + }, + }; +} + +function createRawDogfoodArtifactRequirement(): ArtifactRequirement { + return { + kind: 'notes', + required: true, + description: 'Keep the raw dogfood artifact requirement unchanged.', + minCount: 1, + pathPatterns: ['notes\\.md$'], + }; +} + +function createRawDogfoodReportRequirement( + id = 'raw-report', +): ReportRequirement { + return { + id, + description: 'Keep the raw dogfood report requirement unchanged.', + required: true, + section: 'Evidence', + requiredPatterns: ['evidence'], + forbiddenPatterns: ['TODO'], + }; +} + +describe('dogfoodCase authoring facade', () => { + it('builds a dogfood case that passes schema validation and preserves key fields', () => { + const compiled = createDogfoodBuilder().build(); + + expect(DogfoodEvalCaseSchema.parse(compiled)).toEqual(compiled); + expect(compiled).toMatchObject({ + id: 'dogfood-authoring-happy', + lane: 'dogfood', + category: 'qa', + expectedSkill: 'dogfood-tui', + fixture: 'hello-prompt', + bundlePath: 'dogfood/bundles/hello-prompt', + bundleRequirements: [ + 'Include screenshots, recordings, and written notes.', + ], + validationProfile: 'interactive-renderer', + referenceSteps: 4, + budgets: { + timeoutMs: 120_000, + maxAgentSteps: 12, + maxWallClockMs: 90_000, + }, + }); + expect(compiled.prompt).toContain('Exercise the hello-prompt fixture'); + expect(compiled.prompt).toContain( + 'test/fixtures/apps/hello-prompt/main.ts', + ); + expect(compiled.artifactRequirements).toEqual([ + { + kind: 'screenshot', + required: true, + description: 'Capture at least one screenshot of a noteworthy state.', + minCount: 1, + pathPatterns: ['\\.png$'], + }, + { + kind: 'recording', + required: true, + description: 'Capture at least one terminal recording artifact.', + minCount: 1, + pathPatterns: ['\\.cast$'], + }, + ]); + expect( + compiled.reportRequirements.map((requirement) => requirement.id), + ).toEqual(['title', 'repro-steps']); + expect(compiled.verifiers).toEqual([ + { + id: 'bundle-valid', + kind: 'bundle', + description: + 'Validate the proof bundle with the selected validation profile.', + required: true, + config: { + profile: 'interactive-renderer', + }, + }, + ]); + expect(compiled.workflowChecks).toEqual([ + { + id: 'launch', + description: 'Launch the workflow.', + required: false, + requiredPatterns: ['agent-tty create'], + forbiddenPatterns: [], + dependsOn: [], + weight: 2, + }, + { + id: 'evidence', + description: 'Capture reviewer evidence.', + required: false, + requiredPatterns: ['agent-tty screenshot'], + forbiddenPatterns: [], + dependsOn: ['launch'], + }, + ]); + }); + + it('includes the case id and field path for missing required fields', () => { + expect(() => + dogfoodCase('dogfood-missing-bundle-requirements') + .category('reporting') + .task('Inspect the app and write notes.') + .bundlePath('dogfood/bundles/reporting') + .validationProfile('contract-reporting') + .report((report) => { + report.title(); + }) + .build(), + ).toThrow( + dogfoodErrorPattern( + 'dogfood-missing-bundle-requirements', + 'bundleRequirements', + 'bundleRequirements must include at least one requirement', + ), + ); + + expect(() => + dogfoodCase('dogfood-missing-artifact-or-report-requirements') + .category('qa') + .task('Inspect the app and write notes.') + .bundlePath('dogfood/bundles/reporting') + .bundleRequirement('Produce a complete bundle.') + .validationProfile('interactive-renderer') + .build(), + ).toThrow( + dogfoodErrorPattern( + 'dogfood-missing-artifact-or-report-requirements', + 'artifactRequirements', + 'artifactRequirements or reportRequirements must include at least one requirement', + ), + ); + + expect(() => + dogfoodCase('dogfood-missing-validation-profile') + .category('qa') + .task('Inspect the app and write notes.') + .bundlePath('dogfood/bundles/reporting') + .bundleRequirement('Produce a complete bundle.') + .proofBundle((bundle) => { + bundle.requiresScreenshot(); + }) + .build(), + ).toThrow( + dogfoodErrorPattern( + 'dogfood-missing-validation-profile', + 'validationProfile', + 'validationProfile is required', + ), + ); + }); + + it('fails fast for duplicate workflow step ids', () => { + expect(() => + dogfoodCase('dogfood-duplicate-workflow-step') + .category('qa') + .task('Inspect the app and write notes.') + .bundlePath('dogfood/bundles/reporting') + .bundleRequirement('Produce a complete bundle.') + .validationProfile('interactive-renderer') + .report((report) => { + report.title(); + }) + .workflow((workflow) => { + workflow.step('repeat', 'First step.').mustMention('create'); + workflow.step('repeat', 'Second step.').mustMention('screenshot'); + }), + ).toThrow( + dogfoodErrorPattern( + 'dogfood-duplicate-workflow-step', + 'workflowChecks', + 'Duplicate workflow step id "repeat"', + ), + ); + }); + + it('fails fast for duplicate verifier ids', () => { + expect(() => + dogfoodCase('dogfood-duplicate-verifier') + .category('qa') + .task('Inspect the app and write notes.') + .bundlePath('dogfood/bundles/reporting') + .bundleRequirement('Produce a complete bundle.') + .validationProfile('interactive-renderer') + .report((report) => { + report.title(); + }) + .bundleVerifier() + .bundleVerifier(), + ).toThrow( + dogfoodErrorPattern( + 'dogfood-duplicate-verifier', + 'verifiers', + 'Duplicate verifier id "bundle-valid"', + ), + ); + }); + + it('fails fast for duplicate report requirement ids', () => { + expect(() => + dogfoodCase('dogfood-duplicate-report-requirement') + .category('reporting') + .task('Inspect the app and write notes.') + .bundlePath('dogfood/bundles/reporting') + .bundleRequirement('Produce a complete bundle.') + .validationProfile('contract-reporting') + .report((report) => { + report.section( + 'summary', + 'Summary', + 'Include a short summary section.', + ['summary'], + ); + report.section( + 'summary', + 'Summary', + 'Duplicate the same summary section id.', + ['summary'], + ); + }), + ).toThrow( + dogfoodErrorPattern( + 'dogfood-duplicate-report-requirement', + 'reportRequirements', + 'Duplicate report requirement id "summary"', + ), + ); + }); + + it('fails fast when a workflow step both requires and forbids the same literal', () => { + expect(() => + dogfoodCase('dogfood-contradictory-step') + .category('qa') + .task('Inspect the app and write notes.') + .bundlePath('dogfood/bundles/reporting') + .bundleRequirement('Produce a complete bundle.') + .validationProfile('interactive-renderer') + .report((report) => { + report.title(); + }) + .workflow((workflow) => { + workflow + .step('conflict', 'Contradictory workflow rule.') + .mustMention('agent-tty screenshot') + .mustNotMention('agent-tty screenshot'); + }), + ).toThrow( + dogfoodErrorPattern( + 'dogfood-contradictory-step', + 'workflowChecks', + 'Workflow step "conflict" cannot require and forbid the same literal pattern "agent-tty screenshot"', + ), + ); + }); + + it('fails when workflow() is used but no checks are added', () => { + expect(() => + dogfoodCase('dogfood-empty-workflow') + .category('qa') + .task('Inspect the app and write notes.') + .bundlePath('dogfood/bundles/reporting') + .bundleRequirement('Produce a complete bundle.') + .validationProfile('interactive-renderer') + .report((report) => { + report.title(); + }) + .workflow(() => undefined) + .build(), + ).toThrow( + dogfoodErrorPattern( + 'dogfood-empty-workflow', + 'workflowChecks', + 'workflow() must add at least one workflow check', + ), + ); + }); + + it('returns deep-equal fresh objects on repeated build() calls', () => { + const builder = createDogfoodBuilder('dogfood-build-idempotence'); + + const first = builder.build(); + const second = builder.build(); + + expect(first).toEqual(second); + expect(first).not.toBe(second); + expect(first.artifactRequirements).not.toBe(second.artifactRequirements); + expect(first.artifactRequirements[0]).not.toBe( + second.artifactRequirements[0], + ); + expect(first.reportRequirements).not.toBe(second.reportRequirements); + expect(first.reportRequirements[0]).not.toBe(second.reportRequirements[0]); + expect(first.verifiers).not.toBe(second.verifiers); + expect(first.workflowChecks).not.toBe(second.workflowChecks); + }); + + it('preserves raw workflow, verifier, artifact, and report fragments unchanged through compilation', () => { + const rawCheck = rawWorkflowCheck(createRawDogfoodWorkflowCheck()); + const rawSpec = rawVerifier(createRawDogfoodVerifier()); + const rawArtifact = rawArtifactRequirement( + createRawDogfoodArtifactRequirement(), + ); + const rawReport = rawReportRequirement(createRawDogfoodReportRequirement()); + const compiled = dogfoodCase('dogfood-raw-fragments') + .category('reporting') + .task('Compile the raw fragments unchanged.') + .bundlePath('dogfood/bundles/raw') + .bundleRequirement('Produce a complete bundle.') + .validationProfile('interactive-renderer') + .rawWorkflowCheck(rawCheck) + .rawVerifier(rawSpec) + .rawArtifactRequirement(rawArtifact) + .rawReportRequirement(rawReport) + .build(); + + expect(DogfoodEvalCaseSchema.parse(compiled)).toEqual(compiled); + expect(compiled.workflowChecks).toEqual([rawCheck]); + expect(compiled.verifiers).toEqual([rawSpec]); + expect(compiled.artifactRequirements).toEqual([rawArtifact]); + expect(compiled.reportRequirements).toEqual([rawReport]); + }); +}); diff --git a/test/unit/evals/authoring/execution.test.ts b/test/unit/evals/authoring/execution.test.ts new file mode 100644 index 00000000..333f28e7 --- /dev/null +++ b/test/unit/evals/authoring/execution.test.ts @@ -0,0 +1,352 @@ +import { describe, expect, it } from 'vitest'; + +import { + executionCase, + rawArtifactRequirement, + rawVerifier, + rawWorkflowCheck, +} from '../../../../evals/authoring/index.js'; +import { ExecutionEvalCaseSchema } from '../../../../evals/lib/schemas.js'; +import type { + ArtifactRequirement, + VerifierSpec, + WorkflowCheck, +} from '../../../../evals/lib/types.js'; + +function escapeRegex(value: string): string { + return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} + +function executionErrorPattern( + caseId: string, + path: string, + message: string, +): RegExp { + return new RegExp( + escapeRegex(`Invalid execution case "${caseId}" at ${path}: ${message}`), + 'u', + ); +} + +function createExecutionBuilder(id = 'execution-authoring-happy') { + return executionCase(id) + .category('session') + .task( + 'Launch the hello-prompt fixture, submit input, and capture proof output.', + ) + .fixture('hello-prompt') + .target('test/fixtures/apps/hello-prompt/main.ts') + .workflow((workflow) => { + workflow + .createSession({ id: 'create' }) + .input('hello', { id: 'input' }) + .waitFor('world', { id: 'wait' }) + .snapshot({ id: 'snapshot' }) + .destroy({ id: 'destroy' }); + }) + .assertions((assertions) => { + assertions.snapshotContains('hello', 'world'); + }) + .artifact( + 'screenshot', + 'Capture at least one screenshot artifact.', + /\.png$/u, + ) + .budget({ timeoutMs: 90_000, maxAgentSteps: 8, maxWallClockMs: 60_000 }) + .referenceSteps(5); +} + +function createRawExecutionWorkflowCheck(id = 'raw-workflow'): WorkflowCheck { + return { + id, + description: 'Keep the raw execution workflow check unchanged.', + required: true, + requiredPatterns: ['agent-tty wait'], + forbiddenPatterns: ['sleep 5'], + dependsOn: ['create'], + weight: 4, + }; +} + +function createRawExecutionVerifier(id = 'raw-verifier'): VerifierSpec { + return { + id, + kind: 'command', + description: 'Keep the raw verifier unchanged.', + required: true, + config: { + command: 'node', + argv: ['scripts/check.js'], + }, + }; +} + +function createRawExecutionArtifactRequirement( + kind: ArtifactRequirement['kind'] = 'json', +): ArtifactRequirement { + return { + kind, + required: true, + description: 'Keep the raw artifact requirement unchanged.', + minCount: 1, + pathPatterns: ['output\\.json$'], + }; +} + +function requireDefined(value: T | undefined, label: string): T { + if (value === undefined) { + throw new Error(`${label} must be defined`); + } + return value; +} + +describe('executionCase authoring facade', () => { + it('builds an execution case that passes schema validation and preserves key fields', () => { + const compiled = createExecutionBuilder().build(); + + expect(ExecutionEvalCaseSchema.parse(compiled)).toEqual(compiled); + expect(compiled).toMatchObject({ + id: 'execution-authoring-happy', + lane: 'execution', + category: 'session', + expectedSkill: 'agent-tty', + fixture: 'hello-prompt', + target: 'test/fixtures/apps/hello-prompt/main.ts', + referenceSteps: 5, + budgets: { + timeoutMs: 90_000, + maxAgentSteps: 8, + maxWallClockMs: 60_000, + }, + }); + expect(compiled.prompt).toContain('Launch the hello-prompt fixture'); + expect(compiled.prompt).toContain( + 'test/fixtures/apps/hello-prompt/main.ts', + ); + expect(compiled.setup).toHaveLength(1); + expect(compiled.setup[0]).toMatchObject({ + id: 'launch-hello-prompt', + description: + 'Create an agent-tty session that runs the hello-prompt fixture.', + }); + expect(compiled.workflowChecks.map((check) => check.id)).toEqual([ + 'create', + 'input', + 'wait', + 'snapshot', + 'destroy', + ]); + const inputCheck = requireDefined( + compiled.workflowChecks[1], + 'workflowChecks[1]', + ); + const waitCheck = requireDefined( + compiled.workflowChecks[2], + 'workflowChecks[2]', + ); + expect(inputCheck.dependsOn).toEqual(['create']); + expect(waitCheck.dependsOn).toEqual(['input']); + expect(compiled.verifiers).toEqual([ + { + id: 'snapshot-contains', + kind: 'snapshot', + description: + 'The snapshot should contain the required content patterns.', + required: true, + config: { + patterns: ['hello', 'world'], + }, + }, + ]); + expect(compiled.artifactRequirements).toEqual([ + { + kind: 'screenshot', + required: true, + description: 'Capture at least one screenshot artifact.', + minCount: 1, + pathPatterns: ['/\\.png$/u'], + }, + ]); + }); + + it('includes the case id and field path for missing required fields', () => { + expect(() => + executionCase('execution-missing-task') + .category('session') + .target('README.md') + .verifier( + 'snapshot-check', + 'snapshot', + 'Require a snapshot verifier.', + { + patterns: ['ready'], + }, + ) + .build(), + ).toThrow( + executionErrorPattern( + 'execution-missing-task', + 'prompt', + 'task is required', + ), + ); + + expect(() => + executionCase('execution-missing-fixture-or-target') + .category('session') + .task('Drive the session.') + .verifier( + 'snapshot-check', + 'snapshot', + 'Require a snapshot verifier.', + { + patterns: ['ready'], + }, + ) + .build(), + ).toThrow( + executionErrorPattern( + 'execution-missing-fixture-or-target', + 'fixture', + 'fixture or target is required', + ), + ); + + expect(() => + executionCase('execution-missing-verifiers') + .category('session') + .task('Drive the session.') + .target('README.md') + .build(), + ).toThrow( + executionErrorPattern( + 'execution-missing-verifiers', + 'verifiers', + 'verifiers must include at least one verifier', + ), + ); + }); + + it('fails fast for duplicate workflow step ids', () => { + expect(() => + executionCase('execution-duplicate-workflow-step') + .category('session') + .task('Drive the session.') + .target('README.md') + .workflow((workflow) => { + workflow.createSession({ id: 'repeat' }); + workflow.waitFor('ready', { id: 'repeat' }); + }), + ).toThrow( + executionErrorPattern( + 'execution-duplicate-workflow-step', + 'workflowChecks', + 'Duplicate workflow step id "repeat"', + ), + ); + }); + + it('fails fast for duplicate verifier ids', () => { + expect(() => + executionCase('execution-duplicate-verifier') + .category('artifact') + .task('Check the captured output.') + .target('README.md') + .assertions((assertions) => { + assertions.snapshotContains('ready'); + assertions.snapshotContains('done'); + }), + ).toThrow( + executionErrorPattern( + 'execution-duplicate-verifier', + 'verifiers', + 'Duplicate verifier id "snapshot-contains"', + ), + ); + }); + + it('fails fast when a workflow check both requires and forbids the same literal', () => { + expect(() => + executionCase('execution-contradictory-step') + .category('session') + .task('Drive the session.') + .target('README.md') + .workflow((workflow) => { + workflow.createSession({ + id: 'conflict', + pattern: 'agent-tty create', + forbiddenPattern: 'agent-tty create', + }); + }), + ).toThrow( + executionErrorPattern( + 'execution-contradictory-step', + 'workflowChecks', + 'Workflow step "conflict" cannot require and forbid the same literal pattern "agent-tty create"', + ), + ); + }); + + it('fails when workflow() is used but no checks are added', () => { + expect(() => + executionCase('execution-empty-workflow') + .category('session') + .task('Drive the session.') + .target('README.md') + .verifier( + 'snapshot-check', + 'snapshot', + 'Require a snapshot verifier.', + { + patterns: ['ready'], + }, + ) + .workflow(() => undefined) + .build(), + ).toThrow( + executionErrorPattern( + 'execution-empty-workflow', + 'workflowChecks', + 'workflow() must add at least one workflow check', + ), + ); + }); + + it('returns deep-equal fresh objects on repeated build() calls', () => { + const builder = createExecutionBuilder('execution-build-idempotence'); + + const first = builder.build(); + const second = builder.build(); + + expect(first).toEqual(second); + expect(first).not.toBe(second); + expect(first.setup).not.toBe(second.setup); + expect(first.setup[0]).not.toBe(second.setup[0]); + expect(first.verifiers).not.toBe(second.verifiers); + expect(first.verifiers[0]).not.toBe(second.verifiers[0]); + expect(first.workflowChecks).not.toBe(second.workflowChecks); + expect(first.artifactRequirements).not.toBe(second.artifactRequirements); + }); + + it('preserves raw workflow, verifier, and artifact fragments unchanged through compilation', () => { + const rawCheck = rawWorkflowCheck(createRawExecutionWorkflowCheck()); + const rawSpec = rawVerifier(createRawExecutionVerifier()); + const rawArtifact = rawArtifactRequirement( + createRawExecutionArtifactRequirement(), + ); + const compiled = executionCase('execution-raw-fragments') + .category('artifact') + .task('Compile the raw fragments unchanged.') + .target('README.md') + .rawWorkflowCheck(rawCheck) + .rawVerifier(rawSpec) + .rawArtifactRequirement(rawArtifact) + .build(); + + expect(ExecutionEvalCaseSchema.parse(compiled)).toEqual(compiled); + expect(compiled.setup).toEqual([]); + expect(compiled.workflowChecks).toEqual([rawCheck]); + expect(compiled.verifiers).toEqual([rawSpec]); + expect(compiled.artifactRequirements).toEqual([rawArtifact]); + }); +}); diff --git a/test/unit/evals/authoring/parity.test.ts b/test/unit/evals/authoring/parity.test.ts new file mode 100644 index 00000000..7092b90a --- /dev/null +++ b/test/unit/evals/authoring/parity.test.ts @@ -0,0 +1,500 @@ +import { describe, expect, it } from 'vitest'; + +import exploratoryQaCase from '../../../../evals/dogfood/cases/exploratory-qa.js'; +import { doctorGatedCase } from '../../../../evals/execution/cases/doctor-gated.js'; +import { helloPromptCase } from '../../../../evals/execution/cases/hello-prompt.js'; +import { + DogfoodEvalCaseSchema, + ExecutionEvalCaseSchema, + PromptEvalCaseSchema, +} from '../../../../evals/lib/schemas.js'; +import { TRIGGER_AGENT_TTY_PROMPT_CASES } from '../../../../evals/prompt/cases/trigger-agent-tty.js'; +import type { + AntiPatternRule, + DogfoodEvalCase, + ExecutionEvalCase, + PromptEvalCase, +} from '../../../../evals/lib/types.js'; + +const PROCESS_EXEC_PATH_SENTINEL = ''; +const ALL_SKILL_CONDITIONS = [ + 'none', + 'self-load', + 'preloaded', + 'stale', +] as const; +const HELLO_PROMPT_FIXTURE_ARGV = [ + '--import', + 'tsx', + 'test/fixtures/apps/hello-prompt/main.ts', +] as const; + +const LEGACY_TERMINAL_ANTI_PATTERNS: AntiPatternRule[] = [ + { + id: 'blind-sleep', + severity: 'error', + description: + 'Detected a blind sleep instead of waiting on terminal state or a specific condition.', + patterns: [ + '(?:^|[;&|]\\s*)(sleep\\s+\\d+(?:\\.\\d+)?)\\b', + '\\b(time\\.sleep\\s*\\(\\s*\\d+(?:\\.\\d+)?\\s*\\))', + ], + suggestedFix: + 'Replace blind sleeps with agent-tty wait, snapshot polling, or an explicit loop that checks for a real condition.', + }, + { + id: 'tmux-usage', + severity: 'error', + description: + 'Detected tmux usage instead of the supported agent-tty session workflow.', + patterns: [ + '\\btmux\\b(?:\\s+(?:new(?:-session)?|attach(?:-session)?|kill-session|ls|list-sessions|new-window|split-window|send-keys)\\b)?', + ], + suggestedFix: + 'Use agent-tty run/create plus wait/snapshot/screenshot instead of tmux for long-lived terminal automation.', + }, + { + id: 'screen-usage', + severity: 'error', + description: + 'Detected screen usage instead of the supported agent-tty session workflow.', + patterns: ['(?:^|[;&|]\\s*)(screen\\b(?:\\s+\\S+)?)'], + suggestedFix: + 'Use agent-tty sessions and artifacts instead of screen for detached terminal execution.', + }, + { + id: 'adhoc-screenshot', + severity: 'error', + description: + 'Detected an ad hoc screenshot or desktop automation tool instead of agent-tty screenshot artifacts.', + patterns: [ + '\\b(import\\s+-window)\\b', + '\\b(scrot)\\b', + '\\b(gnome-screenshot)\\b', + '\\b(screencapture)\\b', + '\\b(xdotool)\\b', + '\\b(xwd)\\b', + ], + suggestedFix: + 'Capture reviewable terminal visuals with agent-tty screenshot or record export instead of ad hoc desktop tools.', + }, + { + id: 'missing-json-flag', + severity: 'warning', + description: + 'Detected an agent-tty invocation without --json, which makes automation less reliable.', + patterns: ['\\bagent-tty\\b[^;&|\\n]*'], + suggestedFix: + 'Add --json to agent-tty commands used in transcripts, evals, or automation so downstream parsing is stable.', + }, + { + id: 'orphaned-session', + severity: 'warning', + description: + 'Detected session creation evidence without matching agent-tty destroy/kill cleanup.', + patterns: [ + '\\bagent-tty\\b[^\\n]*\\b(?:run|create)\\b', + '\\bagent-tty\\b[^\\n]*\\b(?:destroy|kill)\\b', + '\\bsession(?:_id|Id)\\b', + ], + suggestedFix: + 'Track created session IDs and destroy or kill them in teardown/finally blocks so eval runs do not leak sessions.', + }, + { + id: 'direct-manifest-write', + severity: 'info', + description: + 'Detected a direct manifest write instead of going through the validated storage helpers.', + patterns: [ + '\\b((?:fs(?:\\.promises)?\\.)?writeFile(?:Sync)?\\s*\\([^)]*\\bmanifest[A-Za-z0-9_]*\\b[^)]*\\))', + '\\b(writeFile(?:Sync)?\\s*\\([^)]*\\bmanifest[A-Za-z0-9_]*\\b[^)]*\\))', + ], + suggestedFix: + 'Write manifests through the storage helpers so path validation and schema guarantees stay centralized.', + }, +]; + +const LEGACY_WAIT_FOR_OUTPUT_CASE = PromptEvalCaseSchema.parse({ + id: 'wait-for-output', + lane: 'prompt', + category: 'trigger', + prompt: + "I need to wait until my server prints 'Listening on port 3000' before running tests", + expectedSkill: 'agent-tty', + context: + 'The answer should prefer waiting on observable terminal text over fixed delays before starting the next step.', + expectedPatterns: ['/agent-tty/i', '/\\bwait\\b/i'], + forbiddenPatterns: [ + '/(?:^|\\n)\\s*sleep\\s+\\d+(?:\\.\\d+)?\\b|(?:(?<=\\buse\\s)|(?<=\\brun\\s)|(?<=\\badd\\s)|(?<=\\binsert\\s))sleep\\s+\\d+(?:\\.\\d+)?\\b/i', + '/setTimeout/i', + ], + rubric: [ + 'Chooses agent-tty for terminal readiness coordination.', + 'Uses wait against concrete terminal output instead of fixed timing guesses.', + ], + workflowChecks: [ + { + id: 'wait-for-output.select-agent-tty', + description: 'Explicitly selects agent-tty.', + required: true, + requiredPatterns: ['/agent-tty/i'], + forbiddenPatterns: [], + dependsOn: [], + }, + { + id: 'wait-for-output.observe-readiness', + description: 'Waits for the listening message before running tests.', + required: true, + requiredPatterns: ['/\\bwait\\b/i', '/Listening on port 3000/i'], + forbiddenPatterns: [ + '/(?:^|\\n)\\s*sleep\\s+\\d+(?:\\.\\d+)?\\b|(?:(?<=\\buse\\s)|(?<=\\brun\\s)|(?<=\\badd\\s)|(?<=\\binsert\\s))sleep\\s+\\d+(?:\\.\\d+)?\\b/i', + '/setTimeout/i', + ], + dependsOn: [], + }, + ], + antiPatterns: [], + budgets: { + timeoutMs: 30_000, + }, +}) as PromptEvalCase; + +const LEGACY_HELLO_PROMPT_CASE = ExecutionEvalCaseSchema.parse({ + id: 'hello-prompt', + lane: 'execution', + category: 'session', + prompt: + "ACTUALLY PERFORM this task by running agent-tty CLI commands via `npx tsx src/cli/main.ts`; do not just describe the steps. Use the isolated `AGENT_TTY_HOME` provided for this eval so session state stays contained. Use the repository fixture app `hello-prompt` from `test/fixtures/apps/hello-prompt/main.ts` via the provided setup command. Launch the hello-prompt fixture, send 'hello world' as input, wait for the READY> prompt to reappear, take a snapshot to verify the echo, then destroy the session.", + expectedSkill: 'agent-tty', + conditions: [...ALL_SKILL_CONDITIONS], + setup: [ + { + id: 'launch-hello-prompt', + description: + 'Create an agent-tty session that runs the hello-prompt fixture.', + command: PROCESS_EXEC_PATH_SENTINEL, + argv: [...HELLO_PROMPT_FIXTURE_ARGV], + timeoutMs: 30_000, + }, + ], + verifiers: [ + { + id: 'hello-prompt-snapshot', + kind: 'snapshot', + description: + 'The transcript snapshot should include the echoed text and the READY prompt.', + required: true, + config: { + patterns: ['ECHO:\\s*hello world', 'READY>'], + }, + }, + ], + workflowChecks: [ + { + id: 'create', + description: 'Create the fixture session.', + required: false, + requiredPatterns: [ + '(?:\\bagent-tty\\b[^\\n]*\\bcreate\\b|\\bcreate(?:d|ing)?\\b[^\\n]*\\bsession\\b)', + ], + forbiddenPatterns: [], + dependsOn: [], + }, + { + id: 'input', + description: 'Send hello world with run or type.', + required: false, + requiredPatterns: [ + '(?:\\bagent-tty\\b[^\\n]*\\b(?:run|type)\\b[^\\n]*hello world|\\b(?:run|type)(?:ning|s|ned)?\\b[^\\n]*hello world\\b|ECHO:\\s*hello world)', + ], + forbiddenPatterns: [], + dependsOn: ['create'], + }, + { + id: 'wait', + description: 'Wait for the READY prompt to reappear after the echo.', + required: false, + requiredPatterns: [ + '(?:(?:\\bagent-tty\\b[^\\n]*\\bwait\\b|\\bwait(?:ed|ing)?\\b)|ECHO:\\s*hello world[\\s\\S]*READY>)', + ], + forbiddenPatterns: [], + dependsOn: ['input'], + }, + { + id: 'snapshot', + description: 'Capture a snapshot for verification.', + required: false, + requiredPatterns: [ + '(?:\\bagent-tty\\b[^\\n]*\\bsnapshot\\b|\\bsnapshot(?:ed|ting)?\\b)', + ], + forbiddenPatterns: [], + dependsOn: ['wait'], + }, + { + id: 'destroy', + description: 'Destroy the session after verification.', + required: false, + requiredPatterns: [ + '(?:\\bagent-tty\\b[^\\n]*\\b(?:destroy|kill)\\b|\\b(?:destroy|kill|cleanup)(?:ed|ing)?\\b[^\\n]*\\bsession\\b)', + ], + forbiddenPatterns: [], + dependsOn: ['snapshot'], + }, + ], + antiPatterns: LEGACY_TERMINAL_ANTI_PATTERNS, + artifactRequirements: [], + budgets: { + timeoutMs: 120_000, + maxAgentSteps: 12, + maxWallClockMs: 60_000, + }, + fixture: 'hello-prompt', + referenceSteps: 5, + workspace: 'agent-tty-smoke', +}) as ExecutionEvalCase; + +const LEGACY_DOCTOR_GATED_CASE = ExecutionEvalCaseSchema.parse({ + id: 'doctor-gated', + lane: 'execution', + category: 'artifact', + prompt: + 'ACTUALLY PERFORM this task by running agent-tty CLI commands via `npx tsx src/cli/main.ts`; do not just describe the steps. Use the isolated `AGENT_TTY_HOME` provided for this eval so session state stays contained. Use the repository fixture app `hello-prompt` from `test/fixtures/apps/hello-prompt/main.ts` via the provided setup command. Before capturing a screenshot, run doctor --json to verify renderer prerequisites and then capture a screenshot of hello-prompt.', + expectedSkill: 'agent-tty', + conditions: [...ALL_SKILL_CONDITIONS], + setup: [ + { + id: 'launch-doctor-gated', + description: + 'Create an agent-tty session that runs the hello-prompt fixture.', + command: PROCESS_EXEC_PATH_SENTINEL, + argv: [...HELLO_PROMPT_FIXTURE_ARGV], + timeoutMs: 30_000, + }, + ], + verifiers: [ + { + id: 'doctor-gated-screenshot', + kind: 'screenshot', + description: + 'A screenshot artifact should be produced after the doctor check passes.', + required: true, + config: { + kind: 'screenshot', + }, + }, + ], + workflowChecks: [ + { + id: 'create', + description: 'Create the fixture session.', + required: false, + requiredPatterns: [ + '(?:\\bagent-tty\\b[^\\n]*\\bcreate\\b|\\bcreate(?:d|ing)?\\b[^\\n]*\\bsession\\b)', + ], + forbiddenPatterns: [], + dependsOn: [], + }, + { + id: 'doctor', + description: 'Run doctor --json before any renderer-dependent capture.', + required: false, + requiredPatterns: [ + '(?:\\bagent-tty\\b[^\\n]*\\bdoctor\\b[^\\n]*--json\\b|\\bdoctor\\b[^\\n]*--json\\b)', + ], + forbiddenPatterns: [], + dependsOn: ['create'], + }, + { + id: 'screenshot', + description: 'Capture the screenshot only after the doctor gate.', + required: false, + requiredPatterns: [ + '(?:(?:\\bagent-tty\\b[^\\n]*\\bdoctor\\b[^\\n]*--json\\b|\\bdoctor\\b[^\\n]*--json\\b))[\\s\\S]*?(?:(?:\\bagent-tty\\b[^\\n]*\\bscreenshot\\b|\\bscreenshot(?:ed|ting)?\\b))', + ], + forbiddenPatterns: [], + dependsOn: ['doctor'], + }, + ], + antiPatterns: LEGACY_TERMINAL_ANTI_PATTERNS, + artifactRequirements: [ + { + kind: 'screenshot', + required: true, + description: 'A PNG screenshot should be saved after the doctor check.', + minCount: 1, + pathPatterns: ['\\.png$'], + }, + ], + budgets: { + timeoutMs: 180_000, + maxAgentSteps: 14, + maxWallClockMs: 75_000, + }, + fixture: 'hello-prompt', + referenceSteps: 5, +}) as ExecutionEvalCase; + +const LEGACY_EXPLORATORY_QA_CASE = DogfoodEvalCaseSchema.parse({ + id: 'exploratory-qa', + lane: 'dogfood', + category: 'qa', + prompt: + 'ACTUALLY PERFORM this dogfood task by running agent-tty CLI commands via `npx tsx src/cli/main.ts`; do not only describe what you would test. Use the isolated `AGENT_TTY_HOME` provided for this eval so session state and artifacts stay contained. Target the repository fixture app `hello-prompt` from `test/fixtures/apps/hello-prompt/main.ts` for this investigation. Capture the requested evidence bundle artifacts in the provided proof-bundle directory, including screenshots, recordings, snapshots, WebM exports, and structured notes whenever the case requires them. Launch the hello-prompt fixture, test exactly three inputs (`hello world`, a blank line, and `symbols-!@#$%^&*`), capture a snapshot after each input, then send `exit` to verify clean shutdown. Save at least one screenshot and one recording, and write a brief findings report with severity and evidence references.', + expectedSkill: 'dogfood-tui', + bundlePath: 'proof-bundle', + bundleRequirements: [ + 'Produce a reviewable proof bundle for an exploratory QA investigation.', + 'Capture renderer-backed evidence for the tested interactions and edge cases.', + 'Write structured notes that summarize findings, severity, and evidence references.', + ], + conditions: [...ALL_SKILL_CONDITIONS], + validationProfile: 'interactive-renderer', + artifactRequirements: [ + { + kind: 'screenshot', + required: true, + description: 'Capture at least one screenshot of a noteworthy state.', + minCount: 1, + pathPatterns: ['\\.png$'], + }, + { + kind: 'recording', + required: true, + description: 'Capture at least one terminal recording artifact.', + minCount: 1, + pathPatterns: ['\\.cast$'], + }, + { + kind: 'notes', + required: true, + description: 'Write exploratory QA notes in a markdown report.', + minCount: 1, + pathPatterns: ['(?:^|/)(?:README|NOTES|index|notes)\\.md$'], + }, + ], + reportRequirements: [ + { + id: 'title', + description: 'Report must have a descriptive title.', + required: true, + section: 'Title', + requiredPatterns: [ + '/(?:^|\\n)\\s*(?:#{1,3}\\s*Title\\b|\\*\\*Title:?\\*\\*)/im', + ], + forbiddenPatterns: [], + }, + { + id: 'repro-steps', + description: 'Include step-by-step reproduction commands.', + required: true, + section: 'Reproduction steps', + requiredPatterns: [ + '/(?:^|\\n)\\s*(?:#{1,3}\\s*(?:Reproduction steps|Repro(?:duction)? steps|Steps)\\b|\\*\\*(?:Reproduction steps|Repro(?:duction)? steps|Steps):?\\*\\*)/im', + '/\\b(?:agent-tty|npx\\s+tsx\\s+src\\/cli\\/main\\.ts)\\b/i', + ], + forbiddenPatterns: [], + }, + { + id: 'findings', + description: 'List findings with severity classification.', + required: true, + section: 'Findings', + requiredPatterns: [ + '/(?:^|\\n)\\s*(?:#{1,3}\\s*(?:Findings|Issues)\\b|\\*\\*(?:Findings|Issues):?\\*\\*)/im', + '/\\b(?:severity|critical|high|medium|low|info)\\b/i', + ], + forbiddenPatterns: [], + }, + { + id: 'evidence', + description: + 'Reference captured artifacts such as screenshots and recordings.', + required: true, + section: 'Evidence', + requiredPatterns: [ + '/(?:^|\\n)\\s*(?:#{1,3}\\s*Evidence\\b|\\*\\*Evidence:?\\*\\*)/im', + '/\\.(?:png|cast|webm|json|md)\\b/i', + ], + forbiddenPatterns: [], + }, + ], + verifiers: [ + { + id: 'bundle-valid', + kind: 'bundle', + description: + 'Validate the exploratory QA proof bundle with the interactive renderer profile.', + required: true, + config: { + profile: 'interactive-renderer', + }, + }, + ], + workflowChecks: [], + antiPatterns: LEGACY_TERMINAL_ANTI_PATTERNS, + budgets: { + timeoutMs: 600_000, + maxAgentSteps: 30, + maxWallClockMs: 600_000, + }, + fixture: 'hello-prompt', + workspace: 'agent-tty-smoke', +}) as DogfoodEvalCase; + +function findPromptCaseOrThrow(caseId: string): PromptEvalCase { + const evalCase = TRIGGER_AGENT_TTY_PROMPT_CASES.find( + (candidate) => candidate.id === caseId, + ); + if (evalCase === undefined) { + throw new Error(`Expected prompt case ${caseId} to be registered`); + } + return evalCase; +} + +function normalizeRuntimeShape< + T extends PromptEvalCase | ExecutionEvalCase | DogfoodEvalCase, +>(evalCase: T): T { + const normalized = structuredClone(evalCase); + if ('setup' in normalized) { + normalized.setup = normalized.setup.map((step) => ({ + ...step, + // fixtureSetupStep() uses process.execPath, which varies across dev and CI Node installations. + command: PROCESS_EXEC_PATH_SENTINEL, + })); + } + return normalized; +} + +describe('authoring facade legacy parity', () => { + it('preserves the legacy wait-for-output prompt case runtime shape', () => { + const current = PromptEvalCaseSchema.parse( + findPromptCaseOrThrow('wait-for-output'), + ) as PromptEvalCase; + + expect(normalizeRuntimeShape(current)).toEqual(LEGACY_WAIT_FOR_OUTPUT_CASE); + }); + + it('preserves the legacy hello-prompt execution case runtime shape', () => { + const current = ExecutionEvalCaseSchema.parse( + helloPromptCase, + ) as ExecutionEvalCase; + + expect(normalizeRuntimeShape(current)).toEqual(LEGACY_HELLO_PROMPT_CASE); + }); + + it('preserves the legacy doctor-gated execution case runtime shape', () => { + const current = ExecutionEvalCaseSchema.parse( + doctorGatedCase, + ) as ExecutionEvalCase; + + expect(normalizeRuntimeShape(current)).toEqual(LEGACY_DOCTOR_GATED_CASE); + }); + + it('preserves the legacy exploratory-qa dogfood case runtime shape', () => { + const current = DogfoodEvalCaseSchema.parse( + exploratoryQaCase, + ) as DogfoodEvalCase; + + expect(normalizeRuntimeShape(current)).toEqual(LEGACY_EXPLORATORY_QA_CASE); + }); +}); diff --git a/test/unit/evals/authoring/prompt.test.ts b/test/unit/evals/authoring/prompt.test.ts new file mode 100644 index 00000000..c2156a52 --- /dev/null +++ b/test/unit/evals/authoring/prompt.test.ts @@ -0,0 +1,229 @@ +import { describe, expect, it } from 'vitest'; + +import { + promptCase, + rawWorkflowCheck, +} from '../../../../evals/authoring/index.js'; +import { PromptEvalCaseSchema } from '../../../../evals/lib/schemas.js'; +import type { WorkflowCheck } from '../../../../evals/lib/types.js'; + +function escapeRegex(value: string): string { + return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} + +function promptErrorPattern( + caseId: string, + path: string, + message: string, +): RegExp { + return new RegExp( + escapeRegex(`Invalid prompt case "${caseId}" at ${path}: ${message}`), + 'u', + ); +} + +function createPromptBuilder(id = 'prompt-authoring-happy') { + return promptCase(id) + .category('workflow') + .prompt('Use agent-tty to create a session and capture a snapshot.') + .expectSkill('agent-tty') + .context('Use the isolated AGENT_TTY_HOME for this eval.') + .expectedPattern(/agent-tty\b/u, 'snapshot') + .mustNotMention('tmux') + .rubric('Uses agent-tty commands instead of only describing the workflow.'); +} + +function createRawWorkflowStep(id = 'raw-check'): WorkflowCheck { + return { + id, + description: 'Use the raw workflow check unchanged.', + required: true, + requiredPatterns: ['raw required pattern'], + forbiddenPatterns: ['raw forbidden pattern'], + dependsOn: ['setup'], + weight: 2, + }; +} + +describe('promptCase authoring facade', () => { + it('builds a prompt case that passes schema validation and preserves key fields', () => { + const compiled = createPromptBuilder() + .workflow((workflow) => { + workflow + .step('create', 'Create the session first.') + .mustMention('agent-tty create') + .weight(2) + .step('snapshot', 'Capture a verification snapshot.') + .mustMention(/snapshot\b/u) + .after('create'); + }) + .budget({ timeoutMs: 45_000 }) + .build(); + + expect(PromptEvalCaseSchema.parse(compiled)).toEqual(compiled); + expect(compiled).toMatchObject({ + id: 'prompt-authoring-happy', + lane: 'prompt', + category: 'workflow', + expectedSkill: 'agent-tty', + budgets: { timeoutMs: 45_000 }, + expectedPatterns: ['/agent-tty\\b/u', 'snapshot'], + forbiddenPatterns: ['tmux'], + rubric: [ + 'Uses agent-tty commands instead of only describing the workflow.', + ], + }); + expect(compiled.context).toContain('AGENT_TTY_HOME'); + expect(compiled.workflowChecks).toEqual([ + { + id: 'create', + description: 'Create the session first.', + required: true, + requiredPatterns: ['agent-tty create'], + forbiddenPatterns: [], + dependsOn: [], + weight: 2, + }, + { + id: 'snapshot', + description: 'Capture a verification snapshot.', + required: true, + requiredPatterns: ['/snapshot\\b/u'], + forbiddenPatterns: [], + dependsOn: ['create'], + }, + ]); + }); + + it('includes the case id and field path for missing required fields', () => { + expect(() => + promptCase('prompt-missing-category') + .prompt('Use agent-tty.') + .expectSkill('agent-tty') + .expectedPattern('agent-tty') + .build(), + ).toThrow( + promptErrorPattern( + 'prompt-missing-category', + 'category', + 'category is required', + ), + ); + + expect(() => + promptCase('prompt-missing-prompt') + .category('trigger') + .expectSkill('agent-tty') + .expectedPattern('agent-tty') + .build(), + ).toThrow( + promptErrorPattern( + 'prompt-missing-prompt', + 'prompt', + 'prompt is required', + ), + ); + + expect(() => + promptCase('prompt-missing-expected-patterns') + .category('selection') + .prompt('Use agent-tty.') + .expectSkill('agent-tty') + .build(), + ).toThrow( + promptErrorPattern( + 'prompt-missing-expected-patterns', + 'expectedPatterns', + 'expectedPatterns must include at least one pattern', + ), + ); + }); + + it('fails fast for duplicate workflow step ids', () => { + expect(() => + createPromptBuilder('prompt-duplicate-workflow-step').workflow( + (workflow) => { + workflow + .step('collect', 'Collect the first signal.') + .mustMention('create'); + workflow + .step('collect', 'Collect the second signal.') + .mustMention('wait'); + }, + ), + ).toThrow( + promptErrorPattern( + 'prompt-duplicate-workflow-step', + 'workflowChecks', + 'Duplicate workflow step id "collect"', + ), + ); + }); + + it('fails fast when a workflow step both requires and forbids the same literal', () => { + expect(() => + createPromptBuilder('prompt-contradictory-step').workflow((workflow) => { + workflow + .step('conflict', 'Contradictory workflow rule.') + .mustMention('agent-tty wait') + .mustNotMention('agent-tty wait'); + }), + ).toThrow( + promptErrorPattern( + 'prompt-contradictory-step', + 'workflowChecks', + 'Workflow step "conflict" cannot require and forbid the same literal pattern "agent-tty wait"', + ), + ); + }); + + it('fails when workflow() is used but no checks are added', () => { + expect(() => + createPromptBuilder('prompt-empty-workflow') + .workflow(() => undefined) + .build(), + ).toThrow( + promptErrorPattern( + 'prompt-empty-workflow', + 'workflowChecks', + 'workflow() must add at least one workflow check', + ), + ); + }); + + it('returns deep-equal fresh objects on repeated build() calls', () => { + const builder = createPromptBuilder('prompt-build-idempotence').workflow( + (workflow) => { + workflow + .step('create', 'Create the session first.') + .mustMention('agent-tty create') + .step('snapshot', 'Capture the snapshot second.') + .mustMention('agent-tty snapshot') + .after('create'); + }, + ); + + const first = builder.build(); + const second = builder.build(); + + expect(first).toEqual(second); + expect(first).not.toBe(second); + expect(first.workflowChecks).not.toBe(second.workflowChecks); + expect(first.workflowChecks[0]).not.toBe(second.workflowChecks[0]); + expect(first.budgets).not.toBe(second.budgets); + }); + + it('preserves rawWorkflowCheck fragments unchanged through compilation', () => { + const rawCheck = rawWorkflowCheck(createRawWorkflowStep()); + const compiled = promptCase('prompt-raw-fragments') + .category('selection') + .prompt('Reach for the raw workflow check helper when needed.') + .expectSkill('agent-tty') + .expectedPattern('raw workflow') + .rawWorkflowCheck(rawCheck) + .build(); + + expect(PromptEvalCaseSchema.parse(compiled)).toEqual(compiled); + expect(compiled.workflowChecks).toEqual([rawCheck]); + }); +}); diff --git a/test/unit/evals/claude.test.ts b/test/unit/evals/claude.test.ts new file mode 100644 index 00000000..8dd5a635 --- /dev/null +++ b/test/unit/evals/claude.test.ts @@ -0,0 +1,99 @@ +import { describe, expect, it } from 'vitest'; + +import { ClaudeProvider } from '../../../evals/providers/claude.js'; + +function toJsonLines(records: readonly unknown[]): string { + return records.map((record) => JSON.stringify(record)).join('\n'); +} + +describe('ClaudeProvider.parse', () => { + it('normalizes complete JSON usage objects and prefers the last valid record', () => { + const provider = new ClaudeProvider(); + const raw = toJsonLines([ + { + type: 'assistant', + usage: { + input_tokens: 20, + output_tokens: 5, + total_tokens: 25, + cache_read_input_tokens: 4, + }, + message: { + role: 'assistant', + content: [{ type: 'text', text: 'working' }], + }, + }, + { + type: 'result', + result: 'done', + usage: { + input_tokens: 120, + output_tokens: 45, + total_tokens: 165, + cache_read_input_tokens: 60, + service_tier: 'standard', + }, + }, + ]); + + const normalized = provider.parse(raw); + + expect(normalized.finalText).toBe('done'); + expect(normalized.tokenUsage).toStrictEqual({ + inputTokens: 120, + outputTokens: 45, + totalTokens: 165, + cachedTokens: 60, + }); + }); + + it.each([ + { + name: 'missing total_tokens', + usage: { + input_tokens: 120, + output_tokens: 45, + cache_read_input_tokens: 60, + }, + }, + { + name: 'fractional totals', + usage: { + input_tokens: 120, + output_tokens: 45, + total_tokens: 165.5, + cache_read_input_tokens: 60, + }, + }, + { + name: 'negative cached tokens', + usage: { + input_tokens: 120, + output_tokens: 45, + total_tokens: 165, + cache_read_input_tokens: -1, + }, + }, + ])('omits tokenUsage for $name', ({ usage }) => { + const provider = new ClaudeProvider(); + const raw = toJsonLines([ + { + type: 'result', + result: 'done', + usage, + }, + ]); + + const normalized = provider.parse(raw); + + expect(normalized.tokenUsage).toBeUndefined(); + }); + + it('does not emit tokenUsage for plain-text output', () => { + const provider = new ClaudeProvider(); + + const normalized = provider.parse('Assistant: done'); + + expect(normalized.tokenUsage).toBeUndefined(); + }); +}); diff --git a/test/unit/evals/codex.test.ts b/test/unit/evals/codex.test.ts index 608bc0c0..7fd6400a 100644 --- a/test/unit/evals/codex.test.ts +++ b/test/unit/evals/codex.test.ts @@ -6,10 +6,14 @@ import { } from '../../../evals/lib/antiPatterns.js'; import { CodexProvider } from '../../../evals/providers/codex.js'; +function toJsonLines(records: readonly unknown[]): string { + return records.map((record) => JSON.stringify(record)).join('\n'); +} + describe('CodexProvider.parse', () => { it('normalizes command_execution items into shell-style tool calls', () => { const provider = new CodexProvider(); - const raw = [ + const raw = toJsonLines([ { type: 'thread.started', thread_id: 'thread_123' }, { type: 'item.completed', @@ -30,9 +34,7 @@ describe('CodexProvider.parse', () => { text: 'done', }, }, - ] - .map((record) => JSON.stringify(record)) - .join('\n'); + ]); const normalized = provider.parse(raw); @@ -61,7 +63,7 @@ describe('CodexProvider.parse', () => { it('normalizes function_call records with parsed arguments and outputs', () => { const provider = new CodexProvider(); - const raw = [ + const raw = toJsonLines([ { type: 'function_call', name: 'shell', @@ -87,9 +89,7 @@ describe('CodexProvider.parse', () => { type: 'agent_message', text: 'done', }, - ] - .map((record) => JSON.stringify(record)) - .join('\n'); + ]); const normalized = provider.parse(raw); @@ -110,4 +110,97 @@ describe('CodexProvider.parse', () => { ); expect(countAgentTtyCalls(normalized)).toBe(1); }); + + it('normalizes complete JSON usage objects and prefers the last valid record', () => { + const provider = new CodexProvider(); + const raw = toJsonLines([ + { + type: 'response.created', + usage: { + input_tokens: 20, + output_tokens: 5, + total_tokens: 25, + input_tokens_details: { + cached_tokens: 4, + }, + ignored: 'field', + }, + }, + { + type: 'item.completed', + item: { + id: 'item_1', + type: 'agent_message', + text: 'done', + }, + }, + { + type: 'turn.completed', + usage: { + input_tokens: 30, + output_tokens: 9, + total_tokens: 39, + cached_input_tokens: 12, + }, + }, + ]); + + const normalized = provider.parse(raw); + + expect(normalized.finalText).toBe('done'); + expect(normalized.tokenUsage).toStrictEqual({ + inputTokens: 30, + outputTokens: 9, + totalTokens: 39, + cachedTokens: 12, + }); + }); + + it.each([ + { + name: 'missing total_tokens', + records: [ + { + type: 'turn.completed', + usage: { + input_tokens: 30, + output_tokens: 9, + cached_input_tokens: 12, + }, + }, + ], + }, + { + name: 'conflicting cached-token aliases', + records: [ + { + type: 'turn.completed', + usage: { + input_tokens: 30, + output_tokens: 9, + total_tokens: 39, + cached_input_tokens: 12, + input_tokens_details: { + cached_tokens: 13, + }, + }, + }, + ], + }, + ])('omits tokenUsage for $name', ({ records }) => { + const provider = new CodexProvider(); + const raw = toJsonLines(records); + + const normalized = provider.parse(raw); + + expect(normalized.tokenUsage).toBeUndefined(); + }); + + it('does not emit tokenUsage for plain-text output', () => { + const provider = new CodexProvider(); + + const normalized = provider.parse('done'); + + expect(normalized.tokenUsage).toBeUndefined(); + }); }); diff --git a/test/unit/evals/lib/artifacts.test.ts b/test/unit/evals/lib/artifacts.test.ts new file mode 100644 index 00000000..a500ae3e --- /dev/null +++ b/test/unit/evals/lib/artifacts.test.ts @@ -0,0 +1,107 @@ +import { mkdtemp, readFile, readdir, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { afterEach, describe, expect, it } from 'vitest'; + +import { writeTokenUsageArtifact } from '../../../../evals/lib/artifacts.js'; + +const VALID_TOKEN_USAGE = { + inputTokens: 12, + outputTokens: 8, + totalTokens: 20, + cachedTokens: 4, +} as const; + +let tempRoot: string | undefined; + +async function createTempRoot(): Promise { + tempRoot = await mkdtemp(join(tmpdir(), 'agent-tty-token-usage-artifact-')); + return tempRoot; +} + +afterEach(async () => { + if (tempRoot !== undefined) { + await rm(tempRoot, { recursive: true, force: true }); + tempRoot = undefined; + } +}); + +describe('writeTokenUsageArtifact', () => { + it('writes token-usage artifacts at the expected path', async () => { + const artifactsDir = await createTempRoot(); + + const artifactPath = await writeTokenUsageArtifact({ + artifactsDir, + caseId: 'wait-for-output', + lane: 'prompt', + condition: 'none', + provider: 'fixture', + model: 'fixture-model', + trialIndex: 0, + tokenUsage: VALID_TOKEN_USAGE, + createdAtMs: 123, + }); + + expect(artifactPath).toBe( + join( + artifactsDir, + 'prompt', + 'wait-for-output', + 'none', + 'token-usage.json', + ), + ); + expect(JSON.parse(await readFile(artifactPath, 'utf8'))).toEqual({ + caseId: 'wait-for-output', + lane: 'prompt', + condition: 'none', + provider: 'fixture', + model: 'fixture-model', + trialIndex: 0, + tokenUsage: VALID_TOKEN_USAGE, + createdAtMs: 123, + }); + }); + + it('rejects invalid payloads without leaving partial files behind', async () => { + const artifactsDir = await createTempRoot(); + const artifactPath = await writeTokenUsageArtifact({ + artifactsDir, + caseId: 'hello-prompt', + lane: 'execution', + condition: 'none', + provider: 'fixture', + model: 'fixture-model', + trialIndex: 0, + tokenUsage: VALID_TOKEN_USAGE, + createdAtMs: 456, + }); + const initialContents = await readFile(artifactPath, 'utf8'); + + const invalidArtifact = { + artifactsDir, + caseId: 'hello-prompt', + lane: 'execution', + condition: 'none', + provider: 'fixture', + model: 'fixture-model', + trialIndex: 0, + tokenUsage: { + inputTokens: 12, + outputTokens: 8, + totalTokens: 20, + unexpected: 99, + }, + createdAtMs: 789, + } as unknown as Parameters[0]; + + await expect(writeTokenUsageArtifact(invalidArtifact)).rejects.toThrow( + /Token usage artifact validation failed/, + ); + expect(await readFile(artifactPath, 'utf8')).toBe(initialContents); + expect( + await readdir(join(artifactsDir, 'execution', 'hello-prompt', 'none')), + ).toEqual(['token-usage.json']); + }); +}); diff --git a/test/unit/evals/reporters/console.test.ts b/test/unit/evals/reporters/console.test.ts new file mode 100644 index 00000000..132b6220 --- /dev/null +++ b/test/unit/evals/reporters/console.test.ts @@ -0,0 +1,87 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; + +import { ConsoleReporter } from '../../../../evals/reporters/console.js'; + +import { + createCaseFinishEvent, + createCaseStartEvent, + createLaneFinishEvent, + createLaneStartEvent, + createRunFinishEvent, + createRunStartEvent, + createTrialFinishEvent, + createTrialStartEvent, +} from './fixtures.js'; + +function emitRepresentativeStream(reporter: ConsoleReporter): void { + reporter.onRunStart(createRunStartEvent()); + reporter.onLaneStart(createLaneStartEvent()); + reporter.onCaseStart(createCaseStartEvent()); + reporter.onTrialStart(createTrialStartEvent()); + reporter.onTrialFinish(createTrialFinishEvent()); + reporter.onCaseFinish(createCaseFinishEvent()); + reporter.onLaneFinish(createLaneFinishEvent()); + reporter.onRunFinish(createRunFinishEvent()); +} + +afterEach(() => { + vi.restoreAllMocks(); +}); + +describe('ConsoleReporter', () => { + it('emits concise boundary lines by default and omits trial lines', () => { + const lines: string[] = []; + const reporter = new ConsoleReporter({ + writeLine: (line) => { + lines.push(line); + }, + }); + + emitRepresentativeStream(reporter); + + expect(lines).toEqual([ + 'run run-123 started: provider=stub model=stub-model lanes=prompt,execution conditions=none,self-load trials=2 invocations=8', + 'lane prompt started: cases=1 conditions=1 concurrency=2 planned=3', + 'case prompt/case-1 [none] started: trials=2', + 'case prompt/case-1 [none] finished: passed=1 failed=0 errored=0 meanScore=0.5 durationMs=2000', + 'lane prompt finished: total=1 passed=1 failed=0 errored=0 durationMs=3000', + 'run run-123 finished: total=1 passed=1 failed=0 errored=0 durationMs=4000', + ]); + }); + + it('adds trial-level lines in verbose mode without dropping boundary lines', () => { + const lines: string[] = []; + const reporter = new ConsoleReporter({ + verbose: true, + writeLine: (line) => { + lines.push(line); + }, + }); + + emitRepresentativeStream(reporter); + + expect(lines).toEqual([ + 'run run-123 started: provider=stub model=stub-model lanes=prompt,execution conditions=none,self-load trials=2 invocations=8', + 'lane prompt started: cases=1 conditions=1 concurrency=2 planned=3', + 'case prompt/case-1 [none] started: trials=2', + 'trial prompt/case-1[none]#1 started', + 'trial prompt/case-1[none]#1 passed ok=true durationMs=1234 score=0.5', + 'case prompt/case-1 [none] finished: passed=1 failed=0 errored=0 meanScore=0.5 durationMs=2000', + 'lane prompt finished: total=1 passed=1 failed=0 errored=0 durationMs=3000', + 'run run-123 finished: total=1 passed=1 failed=0 errored=0 durationMs=4000', + ]); + }); + + it('writes to stderr by default when writeLine is not provided', () => { + const stderrWriteSpy = vi + .spyOn(process.stderr, 'write') + .mockReturnValue(true); + const reporter = new ConsoleReporter(); + + reporter.onRunStart(createRunStartEvent()); + + expect(stderrWriteSpy).toHaveBeenCalledWith( + 'run run-123 started: provider=stub model=stub-model lanes=prompt,execution conditions=none,self-load trials=2 invocations=8\n', + ); + }); +}); diff --git a/test/unit/evals/reporters/dispatch.test.ts b/test/unit/evals/reporters/dispatch.test.ts new file mode 100644 index 00000000..587c092a --- /dev/null +++ b/test/unit/evals/reporters/dispatch.test.ts @@ -0,0 +1,354 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; + +import { + ReporterDispatcher, + redactSecretLikeValues, +} from '../../../../evals/reporters/dispatch.js'; +import type { TokenReportSummary } from '../../../../evals/lib/types.js'; +import type { + RunFinishEvent, + RunStartEvent, +} from '../../../../evals/reporters/types.js'; + +function createRunStartEvent( + overrides: Partial = {}, +): RunStartEvent { + return { + runId: 'run-123', + provider: 'stub', + model: 'stub-model', + lanes: ['prompt', 'execution'], + conditions: ['none', 'self-load'], + totalTrials: 2, + totalInvocations: 8, + outputDir: '/tmp/evals/run-123', + startedAt: '2026-01-01T00:00:00.000Z', + ...overrides, + }; +} + +function createRunFinishEvent( + overrides: Partial = {}, +): RunFinishEvent { + return { + runId: 'run-123', + startedAt: '2026-01-01T00:00:00.000Z', + completedAt: '2026-01-01T00:00:05.000Z', + durationMs: 5000, + total: 8, + passed: 6, + failed: 1, + errored: 1, + laneErrors: [], + runDir: '/tmp/evals/run-123', + reportJsonPath: '/tmp/evals/run-123/report.json', + reportMarkdownPath: '/tmp/evals/run-123/report.md', + ...overrides, + }; +} + +function delay(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +function createTokenReportSummary(): TokenReportSummary { + return { + grandTotal: { + inputTokens: 12, + outputTokens: 8, + totalTokens: 20, + cachedTokens: 4, + trials: 1, + }, + perLane: [ + { + lane: 'prompt', + inputTokens: 12, + outputTokens: 8, + totalTokens: 20, + cachedTokens: 4, + trials: 1, + }, + ], + perCase: [ + { + lane: 'prompt', + caseId: 'case-1', + condition: 'none', + inputTokens: 12, + outputTokens: 8, + totalTokens: 20, + cachedTokens: 4, + trials: 1, + }, + ], + }; +} + +afterEach(() => { + vi.restoreAllMocks(); +}); + +describe('ReporterDispatcher', () => { + it('preserves reporter order across multi-reporter fan-out', async () => { + const seen: string[] = []; + const dispatcher = new ReporterDispatcher([ + { + name: 'first', + onRunStart: () => { + seen.push('first:runStart'); + }, + onRunFinish: () => { + seen.push('first:runFinish'); + }, + }, + { + name: 'second', + onRunStart: () => { + seen.push('second:runStart'); + }, + onRunFinish: () => { + seen.push('second:runFinish'); + }, + }, + { + name: 'third', + onRunStart: () => { + seen.push('third:runStart'); + }, + onRunFinish: () => { + seen.push('third:runFinish'); + }, + }, + ]); + + await dispatcher.dispatch('runStart', createRunStartEvent()); + await dispatcher.dispatch('runFinish', createRunFinishEvent()); + + expect(seen).toEqual([ + 'first:runStart', + 'second:runStart', + 'third:runStart', + 'first:runFinish', + 'second:runFinish', + 'third:runFinish', + ]); + }); + + it('isolates reporter failures and logs them to stderr', async () => { + const seen: string[] = []; + const stderrWriteSpy = vi + .spyOn(process.stderr, 'write') + .mockReturnValue(true); + const dispatcher = new ReporterDispatcher([ + { + name: 'first', + onRunStart: () => { + seen.push('first'); + }, + }, + { + name: 'broken', + onRunStart: () => { + throw new Error('boom'); + }, + }, + { + name: 'third', + onRunStart: () => { + seen.push('third'); + }, + }, + ]); + + await dispatcher.dispatch('runStart', createRunStartEvent()); + + expect(seen).toEqual(['first', 'third']); + expect(stderrWriteSpy).toHaveBeenCalledWith( + 'reporter "broken" failed on runStart: boom\n', + ); + }); + + it('throws descriptive validation errors for missing required fields and wrong types', async () => { + const dispatcher = new ReporterDispatcher(); + const missingProvider = { + ...createRunStartEvent(), + } as Partial; + delete missingProvider.provider; + + await expect( + dispatcher.dispatch( + 'runStart', + missingProvider as unknown as RunStartEvent, + ), + ).rejects.toThrow( + /Invalid reporter payload for event "runStart": provider:/, + ); + + await expect( + dispatcher.dispatch('runStart', { + ...createRunStartEvent(), + totalTrials: 'two', + } as unknown as RunStartEvent), + ).rejects.toThrow( + /Invalid reporter payload for event "runStart": totalTrials:/, + ); + }); + + it('validates runFinish payloads with and without tokenReport', async () => { + const onRunFinish = vi.fn(); + const dispatcher = new ReporterDispatcher([ + { + name: 'capture', + onRunFinish, + }, + ]); + const bareRunFinish = createRunFinishEvent(); + const tokenizedRunFinish = createRunFinishEvent({ + tokenReport: createTokenReportSummary(), + }); + + await expect( + dispatcher.dispatch('runFinish', bareRunFinish), + ).resolves.toBeUndefined(); + await expect( + dispatcher.dispatch('runFinish', tokenizedRunFinish), + ).resolves.toBeUndefined(); + + expect(onRunFinish).toHaveBeenNthCalledWith(1, bareRunFinish); + expect(onRunFinish).toHaveBeenNthCalledWith(2, tokenizedRunFinish); + }); + + it('rejects malformed nested tokenReport payloads on runFinish', async () => { + const dispatcher = new ReporterDispatcher(); + const tokenReport = createTokenReportSummary(); + + await expect( + dispatcher.dispatch('runFinish', { + ...createRunFinishEvent(), + tokenReport: { + ...tokenReport, + perLane: [ + { + ...tokenReport.perLane[0], + unexpected: 1, + }, + ], + }, + } as unknown as RunFinishEvent), + ).rejects.toThrow(/tokenReport\.perLane\.0: Unrecognized key/); + + await expect( + dispatcher.dispatch('runFinish', { + ...createRunFinishEvent(), + tokenReport: { + ...tokenReport, + grandTotal: { + ...tokenReport.grandTotal, + totalTokens: 20.5, + }, + }, + } as unknown as RunFinishEvent), + ).rejects.toThrow(/tokenReport\.grandTotal\.totalTokens:/); + }); + + it('rejects duplicate reporter names at construction', () => { + expect( + () => + new ReporterDispatcher([{ name: 'duplicate' }, { name: 'duplicate' }]), + ).toThrow('Duplicate reporter name: duplicate'); + }); + + it('awaits async reporters sequentially in reporter order', async () => { + const events: string[] = []; + const dispatcher = new ReporterDispatcher([ + { + name: 'slow', + onRunStart: async () => { + events.push('slow:start'); + await delay(25); + events.push('slow:end'); + }, + }, + { + name: 'fast', + onRunStart: async () => { + events.push('fast:start'); + await delay(5); + events.push('fast:end'); + }, + }, + ]); + + await dispatcher.dispatch('runStart', createRunStartEvent()); + + expect(events).toEqual([ + 'slow:start', + 'slow:end', + 'fast:start', + 'fast:end', + ]); + }); +}); + +describe('redactSecretLikeValues', () => { + it('redacts secret-like keys recursively while preserving ordinary keys', () => { + const input = { + FOO_TOKEN: 'abc123', + BAR_KEY: 42, + ordinary: 'keep-me', + creds: { + SESSION_SECRET: 'shh', + nestedValue: 'keep-nested', + }, + items: [ + { + PASSWORD: 'hunter2', + label: 'first', + }, + { + label: 'second', + }, + ], + }; + + const redacted = redactSecretLikeValues(input); + + expect(redacted).toEqual({ + FOO_TOKEN: '[REDACTED]', + BAR_KEY: '[REDACTED]', + ordinary: 'keep-me', + creds: { + SESSION_SECRET: '[REDACTED]', + nestedValue: 'keep-nested', + }, + items: [ + { + PASSWORD: '[REDACTED]', + label: 'first', + }, + { + label: 'second', + }, + ], + }); + expect(input).toEqual({ + FOO_TOKEN: 'abc123', + BAR_KEY: 42, + ordinary: 'keep-me', + creds: { + SESSION_SECRET: 'shh', + nestedValue: 'keep-nested', + }, + items: [ + { + PASSWORD: 'hunter2', + label: 'first', + }, + { + label: 'second', + }, + ], + }); + }); +}); diff --git a/test/unit/evals/reporters/final-report.test.ts b/test/unit/evals/reporters/final-report.test.ts new file mode 100644 index 00000000..623e4b79 --- /dev/null +++ b/test/unit/evals/reporters/final-report.test.ts @@ -0,0 +1,245 @@ +import * as fs from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { afterEach, describe, expect, it } from 'vitest'; + +import { + generateJsonReport, + generateMarkdownReport, +} from '../../../../evals/lib/reporting.js'; +import type { + EvalResult, + RunMetadata, + TokenReportSummary, +} from '../../../../evals/lib/types.js'; +import { FinalReportReporter } from '../../../../evals/reporters/final-report.js'; + +import { createRunFinishEvent } from './fixtures.js'; + +const tempDirs: string[] = []; + +function createRunMetadata(overrides: Partial = {}): RunMetadata { + return { + runId: 'report-run', + createdAt: '2026-01-01T00:00:00.000Z', + repoRoot: '/tmp/report-repo', + providers: ['stub'], + models: ['stub-model'], + lanes: ['prompt'], + conditions: ['none'], + totalTrials: 1, + notes: [], + ...overrides, + }; +} + +function createEvalResult(overrides: Partial = {}): EvalResult { + return { + runId: 'report-run', + providerId: 'stub', + modelId: 'stub-model', + lane: 'prompt', + caseId: 'case-1', + category: 'trigger', + condition: 'none', + expectedSkill: 'agent-tty', + trial: 1, + ok: true, + score: { total: 1, maxPossible: 1, items: [] }, + workflowChecks: [], + antiPatternFindings: [], + normalizedOutput: { + finalText: '', + messages: [], + referencedSkills: [], + toolCalls: [], + }, + startedAt: '2026-01-01T00:00:00.000Z', + completedAt: '2026-01-01T00:00:01.000Z', + durationMs: 1000, + ...overrides, + }; +} + +function createTokenReportSummary(): TokenReportSummary { + return { + grandTotal: { + inputTokens: 90, + outputTokens: 30, + totalTokens: 120, + cachedTokens: 12, + trials: 1, + }, + perLane: [ + { + lane: 'prompt', + inputTokens: 90, + outputTokens: 30, + totalTokens: 120, + cachedTokens: 12, + trials: 1, + }, + ], + perCase: [ + { + lane: 'prompt', + caseId: 'case-1', + condition: 'none', + inputTokens: 90, + outputTokens: 30, + totalTokens: 120, + cachedTokens: 12, + trials: 1, + }, + ], + snapshotCheck: { + regressionThresholdPercent: 10, + cases: [ + { + provider: 'stub', + model: 'stub-model', + lane: 'prompt', + caseId: 'case-1', + condition: 'none', + caseFingerprint: 'b'.repeat(64), + totalTokens: 120, + outcome: 'unchanged', + currentTotalTokens: 120, + snapshotTotalTokens: 118, + deltaTokens: 2, + deltaPercent: 1.7, + }, + ], + summary: { + total: 1, + new: 0, + orphaned: 0, + unchanged: 1, + improved: 0, + regressed: 0, + }, + }, + }; +} + +afterEach(async () => { + await Promise.all( + tempDirs + .splice(0, tempDirs.length) + .map((directory) => fs.rm(directory, { recursive: true, force: true })), + ); +}); + +describe('FinalReportReporter', () => { + it('writes report.json and report.md with byte-identical legacy serialization', async () => { + const tempDir = await fs.mkdtemp(join(tmpdir(), 'final-report-reporter-')); + tempDirs.push(tempDir); + const results = [createEvalResult()]; + const metadata = createRunMetadata(); + const jsonReportPath = join(tempDir, 'report.json'); + const markdownReportPath = join(tempDir, 'report.md'); + const reporter = new FinalReportReporter({ + getFinalReportInputs: () => ({ + results, + metadata, + comparisonMetrics: [], + jsonReportPath, + markdownReportPath, + }), + }); + + await reporter.onRunFinish( + createRunFinishEvent({ + runId: metadata.runId, + runDir: tempDir, + reportJsonPath: jsonReportPath, + reportMarkdownPath: markdownReportPath, + }), + ); + + const expectedJson = `${JSON.stringify(generateJsonReport(results, metadata, [], undefined), null, 2)}\n`; + const expectedMarkdown = generateMarkdownReport( + results, + metadata, + [], + undefined, + ); + + expect(await fs.readFile(jsonReportPath, 'utf8')).toBe(expectedJson); + expect(await fs.readFile(markdownReportPath, 'utf8')).toBe( + expectedMarkdown, + ); + }); + + it('threads tokenReport into the written JSON and Markdown reports', async () => { + const tempDir = await fs.mkdtemp(join(tmpdir(), 'final-report-token-')); + tempDirs.push(tempDir); + const results = [createEvalResult()]; + const metadata = createRunMetadata(); + const tokenReport = createTokenReportSummary(); + const jsonReportPath = join(tempDir, 'report.json'); + const markdownReportPath = join(tempDir, 'report.md'); + const reporter = new FinalReportReporter({ + getFinalReportInputs: () => ({ + results, + metadata, + comparisonMetrics: [], + jsonReportPath, + markdownReportPath, + }), + }); + + await reporter.onRunFinish( + createRunFinishEvent({ + runId: metadata.runId, + runDir: tempDir, + reportJsonPath: jsonReportPath, + reportMarkdownPath: markdownReportPath, + tokenReport, + }), + ); + + const expectedJson = `${JSON.stringify(generateJsonReport(results, metadata, [], undefined, tokenReport), null, 2)}\n`; + const expectedMarkdown = generateMarkdownReport( + results, + metadata, + [], + undefined, + tokenReport, + ); + const writtenJson = await fs.readFile(jsonReportPath, 'utf8'); + const writtenMarkdown = await fs.readFile(markdownReportPath, 'utf8'); + + expect(writtenJson).toBe(expectedJson); + expect(writtenMarkdown).toBe(expectedMarkdown); + expect(JSON.parse(writtenJson)).toMatchObject({ tokenReport }); + expect(writtenMarkdown).toContain('## Token usage'); + expect(writtenMarkdown.indexOf('## Token usage')).toBeGreaterThan( + writtenMarkdown.indexOf('## Anti-pattern summary'), + ); + }); + + it('is a no-op when final report inputs are unavailable', async () => { + const tempDir = await fs.mkdtemp(join(tmpdir(), 'final-report-skip-')); + tempDirs.push(tempDir); + const jsonReportPath = join(tempDir, 'report.json'); + const markdownReportPath = join(tempDir, 'report.md'); + const reporter = new FinalReportReporter({ + getFinalReportInputs: () => null, + }); + + await expect( + reporter.onRunFinish( + createRunFinishEvent({ + runDir: tempDir, + reportJsonPath: jsonReportPath, + reportMarkdownPath: markdownReportPath, + }), + ), + ).resolves.toBeUndefined(); + + await expect(fs.access(jsonReportPath)).rejects.toThrow(); + await expect(fs.access(markdownReportPath)).rejects.toThrow(); + }); +}); diff --git a/test/unit/evals/reporters/fixtures.ts b/test/unit/evals/reporters/fixtures.ts new file mode 100644 index 00000000..4b7d3e5d --- /dev/null +++ b/test/unit/evals/reporters/fixtures.ts @@ -0,0 +1,160 @@ +import type { + CaseFinishEvent, + CaseStartEvent, + LaneFinishEvent, + LaneStartEvent, + RunFinishEvent, + RunStartEvent, + TrialFinishEvent, + TrialStartEvent, +} from '../../../../evals/reporters/types.js'; + +const STARTED_AT = '2026-01-01T00:00:00.000Z'; +const COMPLETED_AT = '2026-01-01T00:00:05.000Z'; + +export function createRunStartEvent( + overrides: Partial = {}, +): RunStartEvent { + return { + runId: 'run-123', + provider: 'stub', + model: 'stub-model', + lanes: ['prompt', 'execution'], + conditions: ['none', 'self-load'], + totalTrials: 2, + totalInvocations: 8, + outputDir: '/tmp/evals/run-123', + startedAt: STARTED_AT, + ...overrides, + }; +} + +export function createLaneStartEvent( + overrides: Partial = {}, +): LaneStartEvent { + return { + runId: 'run-123', + lane: 'prompt', + caseIds: ['case-1'], + conditions: ['none'], + concurrency: 2, + plannedItems: 3, + startedAt: STARTED_AT, + ...overrides, + }; +} + +export function createCaseStartEvent( + overrides: Partial = {}, +): CaseStartEvent { + return { + runId: 'run-123', + lane: 'prompt', + caseId: 'case-1', + condition: 'none', + plannedTrials: 2, + startedAt: STARTED_AT, + ...overrides, + }; +} + +export function createTrialStartEvent( + overrides: Partial = {}, +): TrialStartEvent { + return { + runId: 'run-123', + lane: 'prompt', + caseId: 'case-1', + condition: 'none', + trial: 1, + startedAt: STARTED_AT, + requestedOutputPath: null, + requestedArtifactPath: null, + ...overrides, + }; +} + +export function createTrialFinishEvent( + overrides: Partial = {}, +): TrialFinishEvent { + return { + runId: 'run-123', + lane: 'prompt', + caseId: 'case-1', + condition: 'none', + trial: 1, + startedAt: STARTED_AT, + completedAt: COMPLETED_AT, + durationMs: 1234, + status: 'passed', + ok: true, + errorClass: null, + errorMessage: null, + score: 0.5, + transcriptPath: null, + stdoutPath: null, + stderrPath: null, + eventLogPath: null, + bundlePath: null, + artifactManifestPath: null, + ...overrides, + }; +} + +export function createCaseFinishEvent( + overrides: Partial = {}, +): CaseFinishEvent { + return { + runId: 'run-123', + lane: 'prompt', + caseId: 'case-1', + condition: 'none', + startedAt: STARTED_AT, + completedAt: COMPLETED_AT, + durationMs: 2000, + passed: 1, + failed: 0, + errored: 0, + meanScore: 0.5, + artifactPath: null, + reportPath: null, + ...overrides, + }; +} + +export function createLaneFinishEvent( + overrides: Partial = {}, +): LaneFinishEvent { + return { + runId: 'run-123', + lane: 'prompt', + startedAt: STARTED_AT, + completedAt: COMPLETED_AT, + durationMs: 3000, + total: 1, + passed: 1, + failed: 0, + errored: 0, + ...overrides, + }; +} + +export function createRunFinishEvent( + overrides: Partial = {}, +): RunFinishEvent { + return { + runId: 'run-123', + startedAt: STARTED_AT, + completedAt: COMPLETED_AT, + durationMs: 4000, + total: 1, + passed: 1, + failed: 0, + errored: 0, + laneErrors: [], + runDir: '/tmp/evals/run-123', + reportJsonPath: '/tmp/evals/run-123/report.json', + reportMarkdownPath: '/tmp/evals/run-123/report.md', + ...overrides, + }; +} diff --git a/test/unit/evals/reporters/jsonl.test.ts b/test/unit/evals/reporters/jsonl.test.ts new file mode 100644 index 00000000..8cac099f --- /dev/null +++ b/test/unit/evals/reporters/jsonl.test.ts @@ -0,0 +1,137 @@ +import * as fs from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { afterEach, describe, expect, it, vi } from 'vitest'; + +import { JsonlReporter } from '../../../../evals/reporters/jsonl.js'; + +import { + createCaseFinishEvent, + createCaseStartEvent, + createLaneFinishEvent, + createLaneStartEvent, + createRunFinishEvent, + createRunStartEvent, + createTrialFinishEvent, + createTrialStartEvent, +} from './fixtures.js'; + +const ISO_TIMESTAMP_PATTERN = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/; + +const tempDirs: string[] = []; + +async function readJsonlRecords( + outputPath: string, +): Promise< + Array<{ type: string; timestamp: string; payload: Record }> +> { + const content = await fs.readFile(outputPath, 'utf8'); + const lines = content.trimEnd().split('\n'); + return lines.map( + (line) => JSON.parse(line) as Record, + ) as Array<{ + type: string; + timestamp: string; + payload: Record; + }>; +} + +afterEach(async () => { + vi.restoreAllMocks(); + await Promise.all( + tempDirs + .splice(0, tempDirs.length) + .map((directory) => fs.rm(directory, { recursive: true, force: true })), + ); +}); + +describe('JsonlReporter', () => { + it('appends ordered JSONL event records and creates parent directories once', async () => { + const tempDir = await fs.mkdtemp(join(tmpdir(), 'jsonl-reporter-')); + tempDirs.push(tempDir); + const outputPath = join(tempDir, 'nested', 'events.jsonl'); + const reporter = new JsonlReporter({ outputPath }); + + await reporter.onRunStart(createRunStartEvent({ runId: 'run-jsonl' })); + await reporter.onLaneStart( + createLaneStartEvent({ runId: 'run-jsonl', lane: 'execution' }), + ); + await reporter.onCaseStart( + createCaseStartEvent({ runId: 'run-jsonl', lane: 'execution' }), + ); + await reporter.onTrialStart( + createTrialStartEvent({ runId: 'run-jsonl', lane: 'execution' }), + ); + await reporter.onTrialFinish( + createTrialFinishEvent({ runId: 'run-jsonl', lane: 'execution' }), + ); + await reporter.onCaseFinish( + createCaseFinishEvent({ runId: 'run-jsonl', lane: 'execution' }), + ); + await reporter.onLaneFinish( + createLaneFinishEvent({ runId: 'run-jsonl', lane: 'execution' }), + ); + await reporter.onRunFinish(createRunFinishEvent({ runId: 'run-jsonl' })); + + const records = await readJsonlRecords(outputPath); + + expect(records).toHaveLength(8); + expect(records.map((record) => record.type)).toEqual([ + 'run.start', + 'lane.start', + 'case.start', + 'trial.start', + 'trial.finish', + 'case.finish', + 'lane.finish', + 'run.finish', + ]); + expect(records[0]?.payload).toMatchObject({ + runId: 'run-jsonl', + provider: 'stub', + }); + expect(records[1]?.payload).toMatchObject({ + runId: 'run-jsonl', + lane: 'execution', + }); + expect(records[2]?.payload).toMatchObject({ + caseId: 'case-1', + condition: 'none', + }); + expect(records[3]?.payload).toMatchObject({ + trial: 1, + requestedOutputPath: null, + }); + expect(records[4]?.payload).toMatchObject({ + status: 'passed', + ok: true, + score: 0.5, + }); + expect(records[5]?.payload).toMatchObject({ + meanScore: 0.5, + durationMs: 2000, + }); + expect(records[6]?.payload).toMatchObject({ + total: 1, + passed: 1, + }); + expect(records[7]?.payload).toMatchObject({ + reportJsonPath: '/tmp/evals/run-123/report.json', + reportMarkdownPath: '/tmp/evals/run-123/report.md', + }); + for (const record of records) { + expect(record.timestamp).toMatch(ISO_TIMESTAMP_PATTERN); + } + expect(await fs.readFile(outputPath, 'utf8')).toContain('\n'); + }); + + it('rejects empty or non-string output paths', () => { + expect(() => new JsonlReporter({ outputPath: '' })).toThrow( + 'jsonl reporter outputPath must not be empty', + ); + expect( + () => new JsonlReporter({ outputPath: 123 as unknown as string }), + ).toThrow('jsonl reporter outputPath must be a string'); + }); +}); diff --git a/test/unit/evals/reporters/runtime.test.ts b/test/unit/evals/reporters/runtime.test.ts new file mode 100644 index 00000000..aec73a23 --- /dev/null +++ b/test/unit/evals/reporters/runtime.test.ts @@ -0,0 +1,402 @@ +import { describe, expect, it } from 'vitest'; + +import type { EvalWorkItemIdentity } from '../../../../evals/lib/types.js'; +import { + CaseProgressTracker, + computePlannedCases, + type CaseProgressDispatcher, +} from '../../../../evals/reporters/runtime.js'; +import type { + CaseFinishEvent, + CaseStartEvent, +} from '../../../../evals/reporters/types.js'; + +interface RuntimeItem extends Pick< + EvalWorkItemIdentity, + 'caseId' | 'condition' +> { + trial: number; +} + +interface RuntimeResult { + ok: boolean; + score?: number | null; +} + +type RecordedCaseEvent = + | { eventName: 'caseStart'; payload: CaseStartEvent } + | { eventName: 'caseFinish'; payload: CaseFinishEvent }; + +function createItem( + caseId: string, + condition: EvalWorkItemIdentity['condition'], + trial: number, +): RuntimeItem { + return { caseId, condition, trial }; +} + +function createDispatcher(events: RecordedCaseEvent[]): CaseProgressDispatcher { + return { + dispatch(eventName, payload) { + if (eventName === 'caseStart') { + events.push({ eventName, payload: payload as CaseStartEvent }); + return; + } + + events.push({ eventName, payload: payload as CaseFinishEvent }); + }, + }; +} + +function createNowQueue(...timestamps: string[]): () => string { + let index = 0; + return () => { + const timestamp = timestamps[index]; + if (timestamp === undefined) { + throw new Error(`Missing timestamp at index ${String(index)}`); + } + index += 1; + return timestamp; + }; +} + +function getRequiredItem(items: readonly T[], index: number): T { + const item = items[index]; + if (item === undefined) { + throw new Error(`Missing item at index ${String(index)}`); + } + return item; +} + +function getRequiredEvent( + events: readonly RecordedCaseEvent[], + index: number, +): RecordedCaseEvent { + const event = events[index]; + if (event === undefined) { + throw new Error(`Missing event at index ${String(index)}`); + } + return event; +} + +function getCaseFinishPayload(event: RecordedCaseEvent): CaseFinishEvent { + expect(event.eventName).toBe('caseFinish'); + if (event.eventName !== 'caseFinish') { + throw new Error('Expected a caseFinish event'); + } + return event.payload; +} + +describe('computePlannedCases', () => { + it('counts planned trials per case and condition pair', () => { + const items: RuntimeItem[] = [ + createItem('alpha', 'none', 1), + createItem('alpha', 'none', 2), + createItem('alpha', 'self-load', 1), + createItem('beta', 'none', 1), + ]; + + const plannedCases = computePlannedCases(items); + + expect(plannedCases.size).toBe(3); + expect(plannedCases.get('alpha\u0000none')).toEqual({ + caseId: 'alpha', + condition: 'none', + plannedTrials: 2, + }); + expect(plannedCases.get('alpha\u0000self-load')).toEqual({ + caseId: 'alpha', + condition: 'self-load', + plannedTrials: 1, + }); + expect(plannedCases.get('beta\u0000none')).toEqual({ + caseId: 'beta', + condition: 'none', + plannedTrials: 1, + }); + }); +}); + +describe('CaseProgressTracker', () => { + it('emits caseStart on the first trial start and caseFinish on the last trial finish', async () => { + const items: RuntimeItem[] = [ + createItem('alpha', 'none', 1), + createItem('alpha', 'none', 2), + ]; + const firstItem = getRequiredItem(items, 0); + const secondItem = getRequiredItem(items, 1); + const events: RecordedCaseEvent[] = []; + const tracker = new CaseProgressTracker({ + runId: 'run-123', + lane: 'prompt', + plannedCases: computePlannedCases(items), + dispatcher: createDispatcher(events), + now: createNowQueue( + '2026-01-01T00:00:00.000Z', + '2026-01-01T00:00:05.000Z', + ), + }); + + await tracker.onTrialStart(firstItem); + await tracker.onTrialStart(secondItem); + + expect(events).toEqual([ + { + eventName: 'caseStart', + payload: { + runId: 'run-123', + lane: 'prompt', + caseId: 'alpha', + condition: 'none', + plannedTrials: 2, + startedAt: '2026-01-01T00:00:00.000Z', + }, + }, + ]); + + await tracker.onTrialFinish(secondItem, { + status: 'fulfilled', + value: { ok: true, score: 0.25 }, + }); + expect(events).toHaveLength(1); + + await tracker.onTrialFinish(firstItem, { + status: 'fulfilled', + value: { ok: true, score: 0.75 }, + }); + + expect(events).toEqual([ + { + eventName: 'caseStart', + payload: { + runId: 'run-123', + lane: 'prompt', + caseId: 'alpha', + condition: 'none', + plannedTrials: 2, + startedAt: '2026-01-01T00:00:00.000Z', + }, + }, + { + eventName: 'caseFinish', + payload: { + runId: 'run-123', + lane: 'prompt', + caseId: 'alpha', + condition: 'none', + startedAt: '2026-01-01T00:00:00.000Z', + completedAt: '2026-01-01T00:00:05.000Z', + durationMs: 5000, + passed: 2, + failed: 0, + errored: 0, + meanScore: 0.5, + artifactPath: null, + reportPath: null, + }, + }, + ]); + }); + + it('tracks mixed passed, failed, and errored trials', async () => { + const items: RuntimeItem[] = [ + createItem('alpha', 'none', 1), + createItem('alpha', 'none', 2), + createItem('alpha', 'none', 3), + ]; + const firstItem = getRequiredItem(items, 0); + const secondItem = getRequiredItem(items, 1); + const thirdItem = getRequiredItem(items, 2); + const events: RecordedCaseEvent[] = []; + const tracker = new CaseProgressTracker({ + runId: 'run-456', + lane: 'execution', + plannedCases: computePlannedCases(items), + dispatcher: createDispatcher(events), + now: createNowQueue( + '2026-01-02T00:00:00.000Z', + '2026-01-02T00:00:02.000Z', + ), + }); + + await tracker.onTrialStart(firstItem); + await tracker.onTrialStart(secondItem); + await tracker.onTrialStart(thirdItem); + await tracker.onTrialFinish(firstItem, { + status: 'fulfilled', + value: { ok: true, score: 1 }, + }); + await tracker.onTrialFinish(secondItem, { + status: 'fulfilled', + value: { ok: false, score: 0.25 }, + }); + await tracker.onTrialFinish(thirdItem, { + status: 'rejected', + reason: new Error('worker crashed'), + }); + + expect(getRequiredEvent(events, 1)).toEqual({ + eventName: 'caseFinish', + payload: { + runId: 'run-456', + lane: 'execution', + caseId: 'alpha', + condition: 'none', + startedAt: '2026-01-02T00:00:00.000Z', + completedAt: '2026-01-02T00:00:02.000Z', + durationMs: 2000, + passed: 1, + failed: 1, + errored: 1, + meanScore: 0.625, + artifactPath: null, + reportPath: null, + }, + }); + }); + + it('reports a null meanScore when no numeric scores are present', async () => { + const items: RuntimeItem[] = [ + createItem('beta', 'self-load', 1), + createItem('beta', 'self-load', 2), + ]; + const firstItem = getRequiredItem(items, 0); + const secondItem = getRequiredItem(items, 1); + const events: RecordedCaseEvent[] = []; + const tracker = new CaseProgressTracker({ + runId: 'run-789', + lane: 'dogfood', + plannedCases: computePlannedCases(items), + dispatcher: createDispatcher(events), + now: createNowQueue( + '2026-01-03T00:00:00.000Z', + '2026-01-03T00:00:01.000Z', + ), + }); + + await tracker.onTrialStart(firstItem); + await tracker.onTrialStart(secondItem); + await tracker.onTrialFinish(firstItem, { + status: 'fulfilled', + value: { ok: false, score: null }, + }); + await tracker.onTrialFinish(secondItem, { + status: 'rejected', + reason: new Error('missing output'), + }); + + const finishEvent = getCaseFinishPayload(getRequiredEvent(events, 1)); + expect(finishEvent.runId).toBe('run-789'); + expect(finishEvent.lane).toBe('dogfood'); + expect(finishEvent.caseId).toBe('beta'); + expect(finishEvent.condition).toBe('self-load'); + expect(finishEvent.passed).toBe(0); + expect(finishEvent.failed).toBe(1); + expect(finishEvent.errored).toBe(1); + expect(finishEvent.meanScore).toBeNull(); + }); + + it('emits exactly one start and finish pair per case and condition even when settlements are out of order', async () => { + const items: RuntimeItem[] = [ + createItem('alpha', 'none', 1), + createItem('alpha', 'none', 2), + createItem('alpha', 'self-load', 1), + createItem('beta', 'none', 1), + ]; + const alphaNoneFirst = getRequiredItem(items, 0); + const alphaNoneSecond = getRequiredItem(items, 1); + const alphaSelfLoad = getRequiredItem(items, 2); + const betaNone = getRequiredItem(items, 3); + const events: RecordedCaseEvent[] = []; + const tracker = new CaseProgressTracker({ + runId: 'run-999', + lane: 'prompt', + plannedCases: computePlannedCases(items), + dispatcher: createDispatcher(events), + now: createNowQueue( + '2026-01-04T00:00:00.000Z', + '2026-01-04T00:00:01.000Z', + '2026-01-04T00:00:02.000Z', + '2026-01-04T00:00:03.000Z', + '2026-01-04T00:00:04.000Z', + '2026-01-04T00:00:05.000Z', + ), + }); + + await tracker.onTrialStart(alphaNoneFirst); + await tracker.onTrialStart(alphaSelfLoad); + await tracker.onTrialStart(betaNone); + await tracker.onTrialStart(alphaNoneSecond); + + await tracker.onTrialFinish(betaNone, { + status: 'fulfilled', + value: { ok: true, score: 1 }, + }); + await tracker.onTrialFinish(alphaNoneSecond, { + status: 'fulfilled', + value: { ok: false, score: 0.2 }, + }); + await tracker.onTrialFinish(alphaSelfLoad, { + status: 'fulfilled', + value: { ok: true, score: 0.8 }, + }); + await tracker.onTrialFinish(alphaNoneFirst, { + status: 'fulfilled', + value: { ok: true, score: 0.6 }, + }); + + const caseStartEvents = events.filter( + ( + event, + ): event is Extract => + event.eventName === 'caseStart', + ); + const caseFinishEvents = events.filter( + ( + event, + ): event is Extract => + event.eventName === 'caseFinish', + ); + const countByKey = ( + caseEvents: Array<{ + payload: { + caseId: string; + condition: string; + }; + }>, + ): Map => { + const counts = new Map(); + for (const event of caseEvents) { + const key = `${event.payload.caseId}\u0000${event.payload.condition}`; + counts.set(key, (counts.get(key) ?? 0) + 1); + } + return counts; + }; + + expect(events).toHaveLength(6); + expect(countByKey(caseStartEvents)).toEqual( + new Map([ + ['alpha\u0000none', 1], + ['alpha\u0000self-load', 1], + ['beta\u0000none', 1], + ]), + ); + expect(countByKey(caseFinishEvents)).toEqual( + new Map([ + ['alpha\u0000none', 1], + ['alpha\u0000self-load', 1], + ['beta\u0000none', 1], + ]), + ); + expect(caseFinishEvents.map((event) => event.payload.caseId)).toEqual([ + 'beta', + 'alpha', + 'alpha', + ]); + expect(caseFinishEvents.map((event) => event.payload.condition)).toEqual([ + 'none', + 'self-load', + 'none', + ]); + }); +}); diff --git a/test/unit/evals/reporting.test.ts b/test/unit/evals/reporting.test.ts index eff13b17..8bb23493 100644 --- a/test/unit/evals/reporting.test.ts +++ b/test/unit/evals/reporting.test.ts @@ -12,7 +12,11 @@ import { computePassRate, computeStdDev, } from '../../../evals/lib/statistics.js'; -import type { EvalResult, RunMetadata } from '../../../evals/lib/types.js'; +import type { + EvalResult, + RunMetadata, + TokenReportSummary, +} from '../../../evals/lib/types.js'; function createRunMetadata(overrides: Partial = {}): RunMetadata { return { @@ -162,6 +166,274 @@ function formatConfidenceInterval( return `[${formatter(interval.lower)}, ${formatter(interval.upper)}]`; } +function createSnapshotCheckReport(): NonNullable< + TokenReportSummary['snapshotCheck'] +> { + return { + regressionThresholdPercent: 10, + cases: [ + { + provider: 'stub', + model: 'stub-model', + lane: 'prompt', + caseId: 'case-1', + condition: 'none', + caseFingerprint: 'a'.repeat(64), + totalTokens: 130, + outcome: 'regressed', + currentTotalTokens: 130, + snapshotTotalTokens: 100, + deltaTokens: 30, + deltaPercent: 30, + }, + ], + summary: { + total: 1, + new: 0, + orphaned: 0, + unchanged: 0, + improved: 0, + regressed: 1, + }, + }; +} + +function createTokenReportSummary( + overrides: Omit, 'grandTotal'> & { + grandTotal?: Partial; + } = {}, +): TokenReportSummary { + const base: TokenReportSummary = { + grandTotal: { + inputTokens: 120, + outputTokens: 45, + totalTokens: 165, + cachedTokens: 15, + trials: 3, + }, + perLane: [ + { + lane: 'prompt', + inputTokens: 80, + outputTokens: 25, + totalTokens: 105, + cachedTokens: 10, + trials: 2, + }, + { + lane: 'execution', + inputTokens: 40, + outputTokens: 20, + totalTokens: 60, + cachedTokens: 5, + trials: 1, + }, + ], + perCase: [ + { + lane: 'prompt', + caseId: 'case-1', + condition: 'none', + inputTokens: 80, + outputTokens: 25, + totalTokens: 105, + cachedTokens: 10, + trials: 2, + }, + { + lane: 'execution', + caseId: 'case-2', + condition: 'self-load', + inputTokens: 40, + outputTokens: 20, + totalTokens: 60, + cachedTokens: 5, + trials: 1, + }, + ], + }; + + return { + ...base, + ...overrides, + grandTotal: { + ...base.grandTotal, + ...overrides.grandTotal, + }, + perLane: overrides.perLane ?? base.perLane, + perCase: overrides.perCase ?? base.perCase, + }; +} + +const LEGACY_EXPECTED_JSON = `{ + "metadata": { + "runId": "report-run", + "createdAt": "2026-01-01T00:00:00.000Z", + "repoRoot": "/tmp/report-repo", + "providers": [ + "stub" + ], + "models": [], + "lanes": [ + "prompt", + "execution" + ], + "conditions": [ + "none", + "self-load" + ], + "totalTrials": 1, + "notes": [] + }, + "aggregate": { + "totalCases": 2, + "passed": 1, + "failed": 1, + "passRate": 0.5, + "averageScore": 0.6, + "workflowComplianceRate": 0, + "antiPatternIncidenceRate": 0 + }, + "comparisons": [], + "results": [ + { + "runId": "report-run", + "providerId": "stub", + "lane": "prompt", + "caseId": "case-1", + "category": "trigger", + "condition": "none", + "expectedSkill": "agent-tty", + "trial": 1, + "ok": true, + "score": { + "total": 10, + "maxPossible": 10, + "items": [] + }, + "workflowChecks": [], + "antiPatternFindings": [], + "normalizedOutput": { + "finalText": "", + "messages": [], + "referencedSkills": [], + "toolCalls": [] + }, + "startedAt": "2026-01-01T00:00:00.000Z", + "completedAt": "2026-01-01T00:00:01.000Z", + "durationMs": 1000 + }, + { + "runId": "report-run", + "providerId": "stub", + "lane": "execution", + "caseId": "case-2", + "category": "session", + "condition": "self-load", + "expectedSkill": "agent-tty", + "trial": 1, + "ok": false, + "score": { + "total": 2, + "maxPossible": 10, + "items": [] + }, + "workflowChecks": [], + "antiPatternFindings": [], + "normalizedOutput": { + "finalText": "", + "messages": [], + "referencedSkills": [], + "toolCalls": [] + }, + "startedAt": "2026-01-01T00:00:00.000Z", + "completedAt": "2026-01-01T00:00:01.000Z", + "durationMs": 1000 + } + ], + "aggregateMetrics": { + "totalCases": 2, + "passed": 1, + "failed": 1, + "passRate": 0.5, + "averageScore": 0.6, + "medianScore": 0.6, + "minScore": 0.2, + "maxScore": 1, + "workflowComplianceRate": 0, + "antiPatternIncidenceRate": 0 + }, + "laneSummaries": { + "prompt": { + "totalCases": 1, + "passed": 1, + "failed": 0, + "passRate": 1, + "averageScore": 1, + "medianScore": 1, + "minScore": 1, + "maxScore": 1, + "workflowComplianceRate": 0, + "antiPatternIncidenceRate": 0 + }, + "execution": { + "totalCases": 1, + "passed": 0, + "failed": 1, + "passRate": 0, + "averageScore": 0.2, + "medianScore": 0.2, + "minScore": 0.2, + "maxScore": 0.2, + "workflowComplianceRate": 0, + "antiPatternIncidenceRate": 0 + } + }, + "providerComparisons": [], + "resultRefs": [ + "case-1", + "case-2" + ] +} +`; + +const LEGACY_EXPECTED_MARKDOWN = `# Eval Report + +## Executive summary + +- Run ID: \`report-run\` +- Created: \`2026-01-01T00:00:00.000Z\` +- Repo root: \`/tmp/report-repo\` +- Providers: \`stub\` +- Models: — +- Lanes: \`prompt\`, \`execution\` +- Conditions: \`none\`, \`self-load\` +- Trials: 1 +- Total / Passed / Failed: 2 / 1 / 1 +- Pass rate / Mean score: 50.0% / 0.600 + +| Total | Passed | Failed | Pass Rate | Mean | Median | Min | Max | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 2 | 1 | 1 | 50.0% | 0.600 | 0.600 | 0.200 | 1.000 | + +## Lane breakdown + +| Lane | Total | Passed | Failed | Pass Rate | Mean | +| :--- | ---: | ---: | ---: | ---: | ---: | +| \`prompt\` | 1 | 1 | 0 | 100.0% | 1.000 | +| \`execution\` | 1 | 0 | 1 | 0.0% | 0.200 | + +## Failed cases + +| Case | Lane | Provider | Condition | Score | Error | +| :--- | :--- | :--- | :--- | ---: | :--- | +| \`case-2\` | \`execution\` | \`stub\` | \`self-load\` | 0.200 | — | + +## Anti-pattern summary + +- None. +`; + describe('generateJsonReport trial aggregation', () => { it('includes aggregated data only when totalTrials is greater than one', () => { const report = generateJsonReport( @@ -310,6 +582,138 @@ describe('generateMarkdownReport trial aggregation', () => { }); }); +describe('eval reporting tokenReport', () => { + it('includes tokenReport in JSON and markdown without snapshotCheck', () => { + const results = createSingleTrialResults(); + const metadata = createRunMetadata({ totalTrials: 1 }); + const tokenReport = createTokenReportSummary(); + + const report = generateJsonReport( + results, + metadata, + [], + undefined, + tokenReport, + ); + const markdown = generateMarkdownReport( + results, + metadata, + [], + undefined, + tokenReport, + ); + + expect(report.tokenReport).toEqual(tokenReport); + expect(markdown).toContain('## Token usage'); + expect(markdown).toContain( + '| Lane | Input | Output | Total | Cached | Trials |', + ); + expect(markdown).toContain('| `prompt` | 80 | 25 | 105 | 10 | 2 |'); + expect(markdown).toContain( + '| `execution` | `case-2` | `self-load` | 40 | 20 | 60 | 5 | 1 |', + ); + expect(markdown).not.toContain('### Snapshot check'); + }); + + it('renders snapshotCheck details inside the appended token usage section', () => { + const results = createSingleTrialResults(); + const metadata = createRunMetadata({ totalTrials: 1 }); + const tokenReport = createTokenReportSummary({ + snapshotCheck: createSnapshotCheckReport(), + }); + + const report = generateJsonReport( + results, + metadata, + [], + undefined, + tokenReport, + ); + const markdown = generateMarkdownReport( + results, + metadata, + [], + undefined, + tokenReport, + ); + + expect(report.tokenReport?.snapshotCheck).toEqual( + tokenReport.snapshotCheck, + ); + expect(markdown).toContain('### Snapshot check'); + expect(markdown).toContain( + '- Warning: 1 regressed snapshot case(s) exceeded the threshold.', + ); + expect(markdown).toContain( + '| `stub` | `stub-model` | `prompt` | `case-1` | `none` | `regressed` | 130 | 100 | 30 | 30.0% |', + ); + expect(markdown.indexOf('## Token usage')).toBeGreaterThan( + markdown.indexOf('## Anti-pattern summary'), + ); + }); + + it('keeps legacy JSON and Markdown byte-identical when no tokenReport is supplied', () => { + const results = createSingleTrialResults(); + const metadata = createRunMetadata({ totalTrials: 1 }); + + const json = `${JSON.stringify(generateJsonReport(results, metadata), null, 2)}\n`; + const markdown = generateMarkdownReport(results, metadata); + + expect(json).toBe(LEGACY_EXPECTED_JSON); + expect(markdown).toBe(LEGACY_EXPECTED_MARKDOWN); + }); + + it('omits tokenReport and the markdown section when grandTotal.trials is zero', () => { + const results = createSingleTrialResults(); + const metadata = createRunMetadata({ totalTrials: 1 }); + const tokenReport = createTokenReportSummary({ + grandTotal: { trials: 0 }, + perLane: [], + perCase: [], + }); + + const json = `${JSON.stringify(generateJsonReport(results, metadata, [], undefined, tokenReport), null, 2)}\n`; + const markdown = generateMarkdownReport( + results, + metadata, + [], + undefined, + tokenReport, + ); + + expect(json).toBe(LEGACY_EXPECTED_JSON); + expect(markdown).toBe(LEGACY_EXPECTED_MARKDOWN); + }); + + it('rejects invalid tokenReport payloads with unknown keys and non-integer counts', () => { + const results = createSingleTrialResults(); + const metadata = createRunMetadata({ totalTrials: 1 }); + const tokenReport = createTokenReportSummary(); + + expect(() => + generateJsonReport(results, metadata, [], undefined, { + ...tokenReport, + perLane: [ + { + ...tokenReport.perLane[0], + unexpected: 1, + }, + ], + } as unknown as TokenReportSummary), + ).toThrow(/Unrecognized key/); + + expect(() => + generateJsonReport(results, metadata, [], undefined, { + ...tokenReport, + grandTotal: { + ...tokenReport.grandTotal, + totalTokens: 165.5, + }, + } as unknown as TokenReportSummary), + ).toThrow(/totalTokens/); + }); +}); + describe('eval reporting condition comparison summary', () => { function createConditionComparisonMetadata(): RunMetadata { return { diff --git a/test/unit/evals/run.test.ts b/test/unit/evals/run.test.ts index 5141efa3..1ee50347 100644 --- a/test/unit/evals/run.test.ts +++ b/test/unit/evals/run.test.ts @@ -1,4 +1,6 @@ -import { resolve } from 'node:path'; +import { access, mkdtemp, readFile, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join, resolve } from 'node:path'; import { fileURLToPath } from 'node:url'; import { afterEach, describe, expect, it, vi } from 'vitest'; @@ -6,6 +8,7 @@ import { afterEach, describe, expect, it, vi } from 'vitest'; import { SKILL_CONDITIONS } from '../../../evals/lib/matrix.js'; import { parseCliArgs, + resolveReporterSelection, resolveRequestedConditions, runEvalCli, } from '../../../evals/run.js'; @@ -23,6 +26,15 @@ function getWrittenStdout(calls: readonly unknown[][]): string { .join(''); } +async function pathExists(path: string): Promise { + try { + await access(path); + return true; + } catch { + return false; + } +} + function resolveRepoPath(...segments: string[]): string { return resolve( fileURLToPath(new URL('../../..', import.meta.url)), @@ -63,11 +75,57 @@ describe('parseCliArgs', () => { expect(options.conditions).toEqual(['none', 'preloaded']); }); + it('collects repeated --reporter values in CLI order', () => { + const options = parseCliArgs([ + '--provider', + 'stub', + '--lane', + 'prompt', + '--reporter', + 'jsonl', + '--reporter', + 'console', + ]); + + expect(options.reporters).toEqual(['jsonl', 'console']); + }); + it('defaults to no explicit condition filters when --condition is omitted', () => { const options = parseCliArgs(['--provider', 'stub', '--lane', 'prompt']); expect(options.conditions).toEqual([]); }); + it('parses snapshot update flags and option values', () => { + const options = parseCliArgs([ + '--provider', + 'stub', + '--lane', + 'prompt', + '--snapshot-update', + '--snapshot-threshold', + '12.5', + '--snapshot-dir', + 'tmp/snapshots', + ]); + + expect(options.snapshotUpdate).toBe(true); + expect(options.snapshotCheck).toBe(false); + expect(options.snapshotThreshold).toBe('12.5'); + expect(options.snapshotDir).toBe('tmp/snapshots'); + }); + + it('parses --snapshot-check independently from --snapshot-update', () => { + const options = parseCliArgs([ + '--provider', + 'stub', + '--lane', + 'prompt', + '--snapshot-check', + ]); + + expect(options.snapshotUpdate).toBe(false); + expect(options.snapshotCheck).toBe(true); + }); }); describe('resolveRequestedConditions', () => { @@ -108,6 +166,127 @@ describe('resolveRequestedConditions', () => { }); }); +describe('resolveReporterSelection', () => { + const repoRoot = resolveRepoPath(); + + it('defaults to final when no reporters are provided', () => { + expect( + resolveReporterSelection(repoRoot, { + reporters: [], + progress: false, + }), + ).toEqual({ reporterNames: ['final'] }); + }); + + it('adds console for --progress without duplicating an explicit console reporter', () => { + expect( + resolveReporterSelection(repoRoot, { + reporters: ['jsonl'], + reporterOutput: 'tmp/events.jsonl', + progress: true, + }), + ).toEqual({ + reporterNames: ['jsonl', 'console'], + reporterOutputPath: resolve(repoRoot, 'tmp/events.jsonl'), + }); + + expect( + resolveReporterSelection(repoRoot, { + reporters: ['console'], + progress: true, + }), + ).toEqual({ reporterNames: ['console'] }); + }); + + it('rejects unknown reporters', () => { + expect(() => + resolveReporterSelection(repoRoot, { + reporters: ['unknown'], + progress: false, + }), + ).toThrow( + 'Unsupported reporter: unknown. Expected one of final, console, jsonl', + ); + }); + + it('rejects jsonl without --reporter-output', () => { + expect(() => + resolveReporterSelection(repoRoot, { + reporters: ['jsonl'], + progress: false, + }), + ).toThrow('--reporter jsonl requires --reporter-output'); + }); +}); + +describe('runEvalCli snapshot option validation', () => { + it('rejects mutually exclusive snapshot update and check modes', async () => { + await expect( + runEvalCli([ + '--provider', + 'stub', + '--lane', + 'execution', + '--case', + 'hello-prompt', + '--condition', + 'none', + '--snapshot-update', + '--snapshot-check', + '--dry-run', + ]), + ).rejects.toThrow( + '--snapshot-update and --snapshot-check may not be combined', + ); + }); + + it.each(['NaN', '-1', '101'])( + 'rejects invalid snapshot thresholds: %s', + async (threshold) => { + await expect( + runEvalCli([ + '--provider', + 'stub', + '--lane', + 'execution', + '--case', + 'hello-prompt', + '--condition', + 'none', + '--snapshot-check', + '--snapshot-threshold', + threshold, + '--dry-run', + ]), + ).rejects.toThrow( + '--snapshot-threshold must be a number between 0 and 100', + ); + }, + ); + + it('includes the snapshot options in --help output', async () => { + const stdoutWriteSpy = vi + .spyOn(process.stdout, 'write') + .mockReturnValue(true); + + const exitCode = await runEvalCli(['--help']); + + expect(exitCode).toBe(0); + expect( + getWrittenStdout(stdoutWriteSpy.mock.calls as unknown[][]), + ).toContain('--snapshot-update'); + expect( + getWrittenStdout(stdoutWriteSpy.mock.calls as unknown[][]), + ).toContain('--snapshot-check'); + expect( + getWrittenStdout(stdoutWriteSpy.mock.calls as unknown[][]), + ).toContain('--snapshot-threshold '); + expect( + getWrittenStdout(stdoutWriteSpy.mock.calls as unknown[][]), + ).toContain('--snapshot-dir '); + }); +}); + describe('runEvalCli dry-run output', () => { it('prints the resolved output directory in human-readable output', async () => { const stdoutWriteSpy = vi @@ -168,3 +347,67 @@ describe('runEvalCli dry-run output', () => { ).toMatchObject({ outputBaseDir: outputDir }); }); }); + +describe('runEvalCli reporter output', () => { + it('keeps stdout to one JSON object when non-final reporters are active', async () => { + const stdoutWriteSpy = vi + .spyOn(process.stdout, 'write') + .mockReturnValue(true); + const stderrWriteSpy = vi + .spyOn(process.stderr, 'write') + .mockReturnValue(true); + const tempRoot = await mkdtemp(join(tmpdir(), 'agent-tty-evals-run-')); + const outputDir = join(tempRoot, 'reports'); + const reporterOutputPath = join(tempRoot, 'events.jsonl'); + + try { + await runEvalCli([ + '--provider', + 'stub', + '--lane', + 'prompt', + '--case', + 'pure-reasoning', + '--condition', + 'none', + '--output', + outputDir, + '--reporter', + 'jsonl', + '--reporter-output', + reporterOutputPath, + '--progress', + '--json', + ]); + + expect(stdoutWriteSpy.mock.calls).toHaveLength(1); + expect(stderrWriteSpy.mock.calls.length).toBeGreaterThan(0); + + const summary = JSON.parse( + getWrittenStdout(stdoutWriteSpy.mock.calls as unknown[][]), + ) as { + outputBaseDir: string; + runDir?: string; + jsonReportPath?: string; + markdownReportPath?: string; + }; + + expect(summary).toMatchObject({ outputBaseDir: outputDir }); + expect(summary).not.toHaveProperty('jsonReportPath'); + expect(summary).not.toHaveProperty('markdownReportPath'); + expect(typeof summary.runDir).toBe('string'); + if (typeof summary.runDir !== 'string') { + throw new Error('expected runDir in JSON summary'); + } + + expect(await pathExists(join(summary.runDir, 'report.json'))).toBe(false); + expect(await pathExists(join(summary.runDir, 'report.md'))).toBe(false); + + const jsonlOutput = await readFile(reporterOutputPath, 'utf8'); + expect(jsonlOutput).toContain('"type":"run.start"'); + expect(jsonlOutput).toContain('"type":"run.finish"'); + } finally { + await rm(tempRoot, { force: true, recursive: true }); + } + }); +}); diff --git a/test/unit/evals/scheduler.test.ts b/test/unit/evals/scheduler.test.ts index f2731bc6..a7dbd624 100644 --- a/test/unit/evals/scheduler.test.ts +++ b/test/unit/evals/scheduler.test.ts @@ -173,6 +173,132 @@ describe('runScheduled', () => { ).toEqual(items.map((item) => item.key)); }); + it('runs onItemStart immediately before executor and awaits it inline', async () => { + const items: TestItem[] = [{ key: 'alpha' }]; + const steps: string[] = []; + let hookCompleted = false; + + const results = await runScheduled( + items, + (item) => { + steps.push(`executor:${item.key}:${String(hookCompleted)}`); + return Promise.resolve(`${item.key} ok`); + }, + { + concurrency: 1, + onItemStart: async (item) => { + steps.push(`start:${item.key}`); + await delay(5); + hookCompleted = true; + }, + }, + ); + + expect(steps).toEqual(['start:alpha', 'executor:alpha:true']); + expect(results).toMatchObject([ + { item: items[0], status: 'fulfilled', value: 'alpha ok' }, + ]); + }); + + it('passes fulfilled settlements to onItemFinish', async () => { + const items: TestItem[] = [{ key: 'alpha' }]; + const seen: Array<{ key: string; status: string; value?: string }> = []; + + await runScheduled(items, (item) => Promise.resolve(`${item.key} ok`), { + concurrency: 1, + onItemFinish: async (item, settled) => { + await delay(5); + if (settled.status === 'fulfilled') { + seen.push({ + key: item.key, + status: settled.status, + value: settled.value, + }); + } + }, + }); + + expect(seen).toEqual([ + { key: 'alpha', status: 'fulfilled', value: 'alpha ok' }, + ]); + }); + + it('passes rejected settlements to onItemFinish', async () => { + const items: TestItem[] = [{ key: 'beta' }]; + const seen: Array<{ key: string; status: string; message: string }> = []; + + await runScheduled( + items, + async () => Promise.reject(new Error('beta boom')), + { + concurrency: 1, + onItemFinish: async (item, settled) => { + await delay(5); + if (settled.status === 'rejected') { + seen.push({ + key: item.key, + status: settled.status, + message: + settled.reason instanceof Error + ? settled.reason.message + : String(settled.reason), + }); + } + }, + }, + ); + + expect(seen).toEqual([ + { key: 'beta', status: 'rejected', message: 'beta boom' }, + ]); + }); + + it('preserves input-order settlement and hook coverage with concurrency greater than one', async () => { + const items: TestItem[] = [ + { key: 'item-0', delayMs: 30 }, + { key: 'item-1', delayMs: 0 }, + { key: 'item-2', delayMs: 5 }, + ]; + const startKeys: string[] = []; + const finishKeys: string[] = []; + + const results = await runScheduled( + items, + async (item) => { + await delay(item.delayMs ?? 0); + return item.key; + }, + { + concurrency: 2, + onItemStart: async (item) => { + startKeys.push(item.key); + await delay(1); + }, + onItemFinish: async (item, settled) => { + finishKeys.push(`${item.key}:${settled.status}`); + await delay(1); + }, + }, + ); + + expect(startKeys).toEqual(['item-0', 'item-1', 'item-2']); + expect(finishKeys).toEqual([ + 'item-1:fulfilled', + 'item-2:fulfilled', + 'item-0:fulfilled', + ]); + expect(results.map((settlement) => settlement.item.key)).toEqual([ + 'item-0', + 'item-1', + 'item-2', + ]); + expect( + results.map((settlement) => + settlement.status === 'fulfilled' ? settlement.value : 'rejected', + ), + ).toEqual(['item-0', 'item-1', 'item-2']); + }); + it('logs lifecycle lines with the work-item key prefix', async () => { const items: TestItem[] = [{ key: 'alpha' }, { key: 'beta' }]; const lines: string[] = []; @@ -202,6 +328,45 @@ describe('runScheduled', () => { } }); + it('keeps logLine output unchanged when hooks are provided', async () => { + const items: TestItem[] = [{ key: 'alpha' }, { key: 'beta' }]; + const lines: string[] = []; + const hookEvents: string[] = []; + + await runScheduled( + items, + (item) => { + if (item.key === 'beta') { + throw new Error('beta failed'); + } + return Promise.resolve(`${item.key} ok`); + }, + { + concurrency: 1, + logLine: (line) => lines.push(line), + onItemStart: (item) => { + hookEvents.push(`start:${item.key}`); + }, + onItemFinish: (item, settled) => { + hookEvents.push(`finish:${item.key}:${settled.status}`); + }, + }, + ); + + expect(lines).toEqual([ + '[alpha] start', + '[alpha] ok', + '[beta] start', + '[beta] failed: Error: beta failed', + ]); + expect(hookEvents).toEqual([ + 'start:alpha', + 'finish:alpha:fulfilled', + 'start:beta', + 'finish:beta:rejected', + ]); + }); + it('asserts on invalid concurrency values', async () => { const items: TestItem[] = [{ key: 'item-0' }]; diff --git a/test/unit/evals/snapshots/compare.test.ts b/test/unit/evals/snapshots/compare.test.ts new file mode 100644 index 00000000..d65c2c7c --- /dev/null +++ b/test/unit/evals/snapshots/compare.test.ts @@ -0,0 +1,124 @@ +import { describe, expect, it } from 'vitest'; + +import { compareSnapshotRecords } from '../../../../evals/snapshots/compare.js'; +import { SnapshotCheckReportSchema } from '../../../../evals/snapshots/schemas/report.js'; +import type { SnapshotComparableRecord } from '../../../../evals/snapshots/compare.js'; + +function createComparableRecord( + overrides: Partial = {}, +): SnapshotComparableRecord { + return { + provider: 'openai', + model: 'gpt-4.1', + lane: 'prompt', + caseId: 'case-1', + condition: 'preloaded', + caseFingerprint: 'a'.repeat(64), + totalTokens: 100, + ...overrides, + }; +} + +describe('compareSnapshotRecords', () => { + it('treats values at or below the threshold boundary as unchanged and sorts cases deterministically', () => { + const report = compareSnapshotRecords({ + regressionThresholdPercent: 10, + currentRecords: [ + createComparableRecord({ caseId: 'case-c', totalTokens: 111 }), + createComparableRecord({ caseId: 'case-a', totalTokens: 110 }), + createComparableRecord({ caseId: 'case-b', totalTokens: 109 }), + ], + snapshotRecords: [ + createComparableRecord({ caseId: 'case-b', totalTokens: 100 }), + createComparableRecord({ caseId: 'case-c', totalTokens: 100 }), + createComparableRecord({ caseId: 'case-a', totalTokens: 100 }), + ], + }); + + expect(report.cases.map((entry) => entry.caseId)).toEqual([ + 'case-a', + 'case-b', + 'case-c', + ]); + expect(report.cases.map((entry) => entry.outcome)).toEqual([ + 'unchanged', + 'unchanged', + 'regressed', + ]); + expect(report.summary).toEqual({ + total: 3, + new: 0, + orphaned: 0, + unchanged: 2, + improved: 0, + regressed: 1, + }); + }); + + it('classifies new, orphaned, equal, and improved cases', () => { + const report = compareSnapshotRecords({ + regressionThresholdPercent: 10, + currentRecords: [ + createComparableRecord({ caseId: 'equal', totalTokens: 100 }), + createComparableRecord({ caseId: 'improved', totalTokens: 80 }), + createComparableRecord({ caseId: 'new', totalTokens: 90 }), + ], + snapshotRecords: [ + createComparableRecord({ caseId: 'equal', totalTokens: 100 }), + createComparableRecord({ caseId: 'improved', totalTokens: 100 }), + createComparableRecord({ caseId: 'orphaned', totalTokens: 130 }), + ], + }); + + expect( + Object.fromEntries( + report.cases.map((entry) => [entry.caseId, entry.outcome]), + ), + ).toEqual({ + equal: 'unchanged', + improved: 'improved', + new: 'new', + orphaned: 'orphaned', + }); + expect(report.summary).toEqual({ + total: 4, + new: 1, + orphaned: 1, + unchanged: 1, + improved: 1, + regressed: 0, + }); + }); + + it('parses compare-produced reports with the shared strict snapshot schema', () => { + const report = compareSnapshotRecords({ + regressionThresholdPercent: 10, + currentRecords: [createComparableRecord({ totalTokens: 105 })], + snapshotRecords: [createComparableRecord({ totalTokens: 100 })], + }); + + expect(() => SnapshotCheckReportSchema.parse(report)).not.toThrow(); + }); + + it('omits deltaPercent for zero-token baselines while still flagging regressions', () => { + const report = compareSnapshotRecords({ + regressionThresholdPercent: 10, + currentRecords: [ + createComparableRecord({ caseId: 'zero-baseline', totalTokens: 1 }), + ], + snapshotRecords: [ + createComparableRecord({ caseId: 'zero-baseline', totalTokens: 0 }), + ], + }); + + expect(report.cases).toEqual([ + expect.objectContaining({ + caseId: 'zero-baseline', + outcome: 'regressed', + currentTotalTokens: 1, + snapshotTotalTokens: 0, + }), + ]); + expect(report.cases[0]).not.toHaveProperty('deltaPercent'); + }); +}); diff --git a/test/unit/evals/snapshots/fingerprint.test.ts b/test/unit/evals/snapshots/fingerprint.test.ts new file mode 100644 index 00000000..29487f86 --- /dev/null +++ b/test/unit/evals/snapshots/fingerprint.test.ts @@ -0,0 +1,152 @@ +import { describe, expect, it } from 'vitest'; + +import { promptCase } from '../../../../evals/authoring/prompt.js'; +import { caseFingerprint } from '../../../../evals/snapshots/fingerprint.js'; +import type { + AntiPatternRule, + PromptEvalCase, + WorkflowCheck, +} from '../../../../evals/lib/types.js'; + +const CREATE_WORKFLOW_CHECK: WorkflowCheck = { + id: 'create', + description: 'Create the session.', + required: true, + requiredPatterns: ['create session'], + forbiddenPatterns: [], + dependsOn: [], +}; + +const WAIT_WORKFLOW_CHECK: WorkflowCheck = { + id: 'wait', + description: 'Wait for the expected output.', + required: true, + requiredPatterns: ['wait for output'], + forbiddenPatterns: [], + dependsOn: ['create'], +}; + +const DEFAULT_WORKFLOW_CHECKS: WorkflowCheck[] = [ + CREATE_WORKFLOW_CHECK, + WAIT_WORKFLOW_CHECK, +]; + +const DEFAULT_ANTI_PATTERN: AntiPatternRule = { + id: 'sleep', + description: 'Avoid fixed sleeps.', + severity: 'warning', + patterns: ['\\bsleep\\b'], + suggestedFix: 'Use wait instead.', + lanes: ['prompt'], +}; + +const DEFAULT_ANTI_PATTERNS: AntiPatternRule[] = [DEFAULT_ANTI_PATTERN]; + +function reorderObjectKeysDeep(value: unknown): unknown { + if (Array.isArray(value)) { + return value.map((item) => reorderObjectKeysDeep(item)); + } + + if (value !== null && typeof value === 'object') { + return Object.fromEntries( + Object.entries(value as Record) + .reverse() + .map(([key, entryValue]) => [key, reorderObjectKeysDeep(entryValue)]), + ); + } + + return value; +} + +function createPromptEvalCase( + overrides: { + prompt?: string; + expectedPatterns?: string[]; + forbiddenPatterns?: string[]; + workflowChecks?: WorkflowCheck[]; + antiPatterns?: AntiPatternRule[]; + budgetMs?: number; + } = {}, +): PromptEvalCase { + const builder = promptCase('snapshot-case') + .category('workflow') + .prompt( + overrides.prompt ?? 'Use agent-tty to collect a snapshot of the app.', + ) + .expectSkill('agent-tty') + .budget(overrides.budgetMs ?? 30_000) + .antiPatterns(...(overrides.antiPatterns ?? DEFAULT_ANTI_PATTERNS)); + + for (const pattern of overrides.expectedPatterns ?? [ + 'agent-tty', + 'snapshot', + ]) { + builder.expectedPattern(pattern); + } + for (const pattern of overrides.forbiddenPatterns ?? ['tmux']) { + builder.forbiddenPattern(pattern); + } + for (const check of overrides.workflowChecks ?? DEFAULT_WORKFLOW_CHECKS) { + builder.rawWorkflowCheck(check); + } + + return builder.build(); +} + +describe('caseFingerprint', () => { + it('is stable across repeated builds and ignores excluded compiled fields', () => { + const defaultCase = createPromptEvalCase(); + const rebuiltCase = createPromptEvalCase(); + const differentBudgetCase = createPromptEvalCase({ budgetMs: 60_000 }); + + expect(caseFingerprint(defaultCase)).toBe(caseFingerprint(rebuiltCase)); + expect(caseFingerprint(defaultCase)).toBe( + caseFingerprint(differentBudgetCase), + ); + }); + + it('ignores object key insertion order when serializing semantic fields', () => { + const compiledCase = createPromptEvalCase(); + const reorderedCase = reorderObjectKeysDeep(compiledCase) as PromptEvalCase; + + expect(caseFingerprint(compiledCase)).toBe(caseFingerprint(reorderedCase)); + }); + + it('changes when the prompt text changes', () => { + const baselineCase = createPromptEvalCase(); + const changedPromptCase = createPromptEvalCase({ + prompt: 'Use agent-tty to capture a screenshot of the app instead.', + }); + + expect(caseFingerprint(changedPromptCase)).not.toBe( + caseFingerprint(baselineCase), + ); + }); + + it('changes when anti-pattern semantics change', () => { + const baselineCase = createPromptEvalCase(); + const changedAntiPatternCase = createPromptEvalCase({ + antiPatterns: [ + { + ...DEFAULT_ANTI_PATTERN, + patterns: ['\\bsleep\\b', '\\bsetTimeout\\b'], + }, + ], + }); + + expect(caseFingerprint(changedAntiPatternCase)).not.toBe( + caseFingerprint(baselineCase), + ); + }); + + it('changes when workflow step order changes', () => { + const baselineCase = createPromptEvalCase(); + const reorderedWorkflowCase = createPromptEvalCase({ + workflowChecks: [WAIT_WORKFLOW_CHECK, CREATE_WORKFLOW_CHECK], + }); + + expect(caseFingerprint(reorderedWorkflowCase)).not.toBe( + caseFingerprint(baselineCase), + ); + }); +}); diff --git a/test/unit/evals/snapshots/store.test.ts b/test/unit/evals/snapshots/store.test.ts new file mode 100644 index 00000000..1964f24f --- /dev/null +++ b/test/unit/evals/snapshots/store.test.ts @@ -0,0 +1,219 @@ +import { + mkdtemp, + readdir, + readFile, + realpath, + rm, + writeFile, +} from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { afterEach, describe, expect, it, vi } from 'vitest'; + +import { buildSnapshotLogicalKey } from '../../../../evals/snapshots/schema.js'; +import { + loadSnapshotFile, + snapshotFilePath, + writeSnapshotFile, +} from '../../../../evals/snapshots/store.js'; +import type { SnapshotEntry } from '../../../../evals/snapshots/schema.js'; + +const temporaryDirectories: string[] = []; + +afterEach(async () => { + vi.restoreAllMocks(); + await Promise.all( + temporaryDirectories + .splice(0) + .map((directory) => rm(directory, { recursive: true, force: true })), + ); +}); + +async function createSnapshotDir(): Promise { + const directory = await realpath( + await mkdtemp(join(tmpdir(), 'eval-snapshots-')), + ); + temporaryDirectories.push(directory); + return directory; +} + +function createSnapshotEntry( + overrides: Partial = {}, +): SnapshotEntry { + return { + provider: 'openai', + model: 'gpt-4.1', + lane: 'prompt', + caseId: 'case-1', + condition: 'preloaded', + caseFingerprint: 'a'.repeat(64), + inputTokens: 120, + outputTokens: 40, + totalTokens: 160, + cachedTokens: 8, + createdAtMs: 1_713_456_789_000, + ...overrides, + }; +} + +describe('snapshot store', () => { + it('round-trips JSONL snapshot entries', async () => { + const snapshotDir = await createSnapshotDir(); + const entry = createSnapshotEntry(); + + await writeSnapshotFile({ + snapshotDir, + provider: entry.provider, + model: entry.model, + entries: [entry], + }); + + const loaded = await loadSnapshotFile({ + snapshotDir, + provider: entry.provider, + model: entry.model, + }); + + expect(loaded.diagnostics).toEqual([]); + expect(loaded.entries).toEqual([entry]); + await expect( + readFile( + snapshotFilePath(snapshotDir, entry.provider, entry.model), + 'utf8', + ), + ).resolves.toBe(`${JSON.stringify(entry)}\n`); + }); + + it('leaves the original file untouched when validation fails before commit', async () => { + const snapshotDir = await createSnapshotDir(); + const entry = createSnapshotEntry(); + const path = snapshotFilePath(snapshotDir, entry.provider, entry.model); + + await writeSnapshotFile({ + snapshotDir, + provider: entry.provider, + model: entry.model, + entries: [entry], + }); + const originalContents = await readFile(path, 'utf8'); + + await expect( + writeSnapshotFile({ + snapshotDir, + provider: entry.provider, + model: entry.model, + entries: [ + { + ...entry, + totalTokens: -1, + } as SnapshotEntry, + ], + }), + ).rejects.toThrow(/Snapshot entries validation failed/u); + + await expect(readFile(path, 'utf8')).resolves.toBe(originalContents); + await expect(readdir(snapshotDir)).resolves.not.toContainEqual( + expect.stringContaining('.tmp-'), + ); + }); + + it('warns and skips malformed or stale entries while loading', async () => { + const snapshotDir = await createSnapshotDir(); + const currentEntry = createSnapshotEntry(); + const staleEntry = createSnapshotEntry({ + caseId: 'case-2', + caseFingerprint: 'b'.repeat(64), + totalTokens: 190, + }); + const path = snapshotFilePath( + snapshotDir, + currentEntry.provider, + currentEntry.model, + ); + const warnSpy = vi + .spyOn(console, 'warn') + .mockImplementation(() => undefined); + + await writeFile( + path, + [ + 'not valid json', + JSON.stringify(currentEntry), + JSON.stringify(staleEntry), + ].join('\n') + '\n', + 'utf8', + ); + + const loaded = await loadSnapshotFile({ + snapshotDir, + provider: currentEntry.provider, + model: currentEntry.model, + validCurrentKeys: new Set([buildSnapshotLogicalKey(currentEntry)]), + }); + + expect(loaded.entries).toEqual([currentEntry]); + expect(loaded.diagnostics).toEqual([ + expect.objectContaining({ kind: 'malformed', lineNumber: 1, path }), + expect.objectContaining({ kind: 'stale', lineNumber: 3, path }), + ]); + expect(warnSpy).toHaveBeenCalledTimes(2); + expect(warnSpy).toHaveBeenNthCalledWith( + 1, + expect.stringContaining(`[evals] Skipping snapshot entry at ${path}:1:`), + ); + expect(warnSpy).toHaveBeenNthCalledWith( + 2, + expect.stringContaining(`[evals] Skipping snapshot entry at ${path}:3:`), + ); + }); + + it('merges rewrites by logical key instead of appending duplicates', async () => { + const snapshotDir = await createSnapshotDir(); + const originalEntry = createSnapshotEntry(); + const preservedEntry = createSnapshotEntry({ + caseId: 'case-2', + caseFingerprint: 'b'.repeat(64), + totalTokens: 210, + }); + const updatedEntry = createSnapshotEntry({ + totalTokens: 175, + createdAtMs: originalEntry.createdAtMs + 1_000, + }); + const appendedEntry = createSnapshotEntry({ + caseId: 'case-3', + caseFingerprint: 'c'.repeat(64), + totalTokens: 220, + }); + const path = snapshotFilePath( + snapshotDir, + originalEntry.provider, + originalEntry.model, + ); + + await writeSnapshotFile({ + snapshotDir, + provider: originalEntry.provider, + model: originalEntry.model, + entries: [originalEntry, preservedEntry], + }); + await writeSnapshotFile({ + snapshotDir, + provider: originalEntry.provider, + model: originalEntry.model, + entries: [updatedEntry, appendedEntry], + }); + + const fileContents = await readFile(path, 'utf8'); + const parsedEntries = fileContents + .trim() + .split('\n') + .map((line) => JSON.parse(line) as SnapshotEntry); + + expect(parsedEntries).toEqual([ + updatedEntry, + preservedEntry, + appendedEntry, + ]); + }); +}); diff --git a/test/unit/evals/tokenAggregation.test.ts b/test/unit/evals/tokenAggregation.test.ts new file mode 100644 index 00000000..adcf18cf --- /dev/null +++ b/test/unit/evals/tokenAggregation.test.ts @@ -0,0 +1,266 @@ +import { describe, expect, it } from 'vitest'; + +import { + aggregateTokenRecords, + type RawTokenRecord, +} from '../../../evals/lib/tokenAggregation.js'; + +function createRawTokenRecord( + overrides: Partial = {}, +): RawTokenRecord { + return { + provider: 'fixture', + model: 'fixture-model', + lane: 'prompt', + caseId: 'case-1', + condition: 'none', + caseFingerprint: 'a'.repeat(64), + usage: { + inputTokens: 80, + outputTokens: 20, + totalTokens: 100, + cachedTokens: 10, + }, + ...overrides, + }; +} + +describe('aggregateTokenRecords', () => { + it('returns undefined for an empty record list', () => { + expect(aggregateTokenRecords([], ['prompt', 'execution'])).toBeUndefined(); + }); + + it('aggregates a single token record into grand totals, lanes, and cases', () => { + expect( + aggregateTokenRecords([createRawTokenRecord()], ['prompt', 'execution']), + ).toEqual({ + grandTotal: { + inputTokens: 80, + outputTokens: 20, + totalTokens: 100, + cachedTokens: 10, + trials: 1, + }, + perLane: [ + { + lane: 'prompt', + inputTokens: 80, + outputTokens: 20, + totalTokens: 100, + cachedTokens: 10, + trials: 1, + }, + ], + perCase: [ + { + lane: 'prompt', + caseId: 'case-1', + condition: 'none', + inputTokens: 80, + outputTokens: 20, + totalTokens: 100, + cachedTokens: 10, + trials: 1, + }, + ], + }); + }); + + it('sums multiple trials for the same case and condition', () => { + const report = aggregateTokenRecords( + [ + createRawTokenRecord(), + createRawTokenRecord({ + usage: { + inputTokens: 60, + outputTokens: 30, + totalTokens: 90, + cachedTokens: 5, + }, + }), + ], + ['prompt'], + ); + + expect(report).toEqual({ + grandTotal: { + inputTokens: 140, + outputTokens: 50, + totalTokens: 190, + cachedTokens: 15, + trials: 2, + }, + perLane: [ + { + lane: 'prompt', + inputTokens: 140, + outputTokens: 50, + totalTokens: 190, + cachedTokens: 15, + trials: 2, + }, + ], + perCase: [ + { + lane: 'prompt', + caseId: 'case-1', + condition: 'none', + inputTokens: 140, + outputTokens: 50, + totalTokens: 190, + cachedTokens: 15, + trials: 2, + }, + ], + }); + }); + + it('groups lanes using the provided lane order and sorts cases deterministically', () => { + const report = aggregateTokenRecords( + [ + createRawTokenRecord({ + lane: 'execution', + caseId: 'case-b', + condition: 'preloaded', + usage: { + inputTokens: 30, + outputTokens: 10, + totalTokens: 40, + cachedTokens: 4, + }, + }), + createRawTokenRecord({ + lane: 'prompt', + caseId: 'case-c', + condition: 'none', + usage: { + inputTokens: 20, + outputTokens: 5, + totalTokens: 25, + cachedTokens: 2, + }, + }), + createRawTokenRecord({ + lane: 'execution', + caseId: 'case-a', + condition: 'none', + usage: { + inputTokens: 10, + outputTokens: 5, + totalTokens: 15, + cachedTokens: 1, + }, + }), + ], + ['execution', 'prompt', 'dogfood'], + ); + + expect(report?.perLane.map((entry) => entry.lane)).toEqual([ + 'execution', + 'prompt', + ]); + expect( + report?.perCase.map((entry) => [ + entry.lane, + entry.caseId, + entry.condition, + ]), + ).toEqual([ + ['execution', 'case-a', 'none'], + ['execution', 'case-b', 'preloaded'], + ['prompt', 'case-c', 'none'], + ]); + }); + + it('omits cachedTokens for aggregate levels that mix cached and uncached records', () => { + const report = aggregateTokenRecords( + [ + createRawTokenRecord({ + lane: 'prompt', + caseId: 'case-1', + usage: { + inputTokens: 80, + outputTokens: 20, + totalTokens: 100, + cachedTokens: 10, + }, + }), + createRawTokenRecord({ + lane: 'prompt', + caseId: 'case-2', + usage: { + inputTokens: 40, + outputTokens: 10, + totalTokens: 50, + }, + }), + createRawTokenRecord({ + lane: 'execution', + caseId: 'case-3', + usage: { + inputTokens: 20, + outputTokens: 5, + totalTokens: 25, + cachedTokens: 3, + }, + }), + ], + ['prompt', 'execution'], + ); + + expect(report?.grandTotal).toEqual({ + inputTokens: 140, + outputTokens: 35, + totalTokens: 175, + trials: 3, + }); + expect(report?.perLane).toEqual([ + { + lane: 'prompt', + inputTokens: 120, + outputTokens: 30, + totalTokens: 150, + trials: 2, + }, + { + lane: 'execution', + inputTokens: 20, + outputTokens: 5, + totalTokens: 25, + cachedTokens: 3, + trials: 1, + }, + ]); + expect(report?.perCase).toEqual([ + { + lane: 'prompt', + caseId: 'case-1', + condition: 'none', + inputTokens: 80, + outputTokens: 20, + totalTokens: 100, + cachedTokens: 10, + trials: 1, + }, + { + lane: 'prompt', + caseId: 'case-2', + condition: 'none', + inputTokens: 40, + outputTokens: 10, + totalTokens: 50, + trials: 1, + }, + { + lane: 'execution', + caseId: 'case-3', + condition: 'none', + inputTokens: 20, + outputTokens: 5, + totalTokens: 25, + cachedTokens: 3, + trials: 1, + }, + ]); + }); +}); diff --git a/test/unit/evals/workspaces/builtins.test.ts b/test/unit/evals/workspaces/builtins.test.ts new file mode 100644 index 00000000..20fe5299 --- /dev/null +++ b/test/unit/evals/workspaces/builtins.test.ts @@ -0,0 +1,77 @@ +import * as fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import process from 'node:process'; + +import { afterEach, describe, expect, it, vi } from 'vitest'; + +const tempDirs: string[] = []; + +afterEach(() => { + for (const tempDir of tempDirs.splice(0, tempDirs.length)) { + fs.rmSync(tempDir, { recursive: true, force: true }); + } +}); + +function createTempDir(): string { + const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'atty-ws-builtins-')); + tempDirs.push(tempDir); + return tempDir; +} + +async function loadFreshWorkspaceModules() { + vi.resetModules(); + + const builtinsModule = + await import('../../../../evals/workspaces/builtins.js'); + const registryModule = + await import('../../../../evals/workspaces/registry.js'); + const resolverModule = + await import('../../../../evals/workspaces/resolver.js'); + + registryModule.clearPresetsForTesting(); + + return { + ...builtinsModule, + ...registryModule, + ...resolverModule, + }; +} + +describe('registerBuiltinPresets', () => { + it('registers agent-tty-smoke and makes repeated registration a no-op', async () => { + const { lookupPreset, registerBuiltinPresets } = + await loadFreshWorkspaceModules(); + + expect(() => registerBuiltinPresets()).not.toThrow(); + const firstLookup = lookupPreset('agent-tty-smoke'); + + expect(() => registerBuiltinPresets()).not.toThrow(); + const secondLookup = lookupPreset('agent-tty-smoke'); + + expect(firstLookup).toEqual(secondLookup); + expect(firstLookup.mode).toBe('isolated'); + expect(firstLookup.bootstrap).toHaveLength(1); + const bootstrap = firstLookup.bootstrap; + if (bootstrap === undefined) { + throw new Error('Expected agent-tty-smoke bootstrap to be defined'); + } + expect(bootstrap[0]?.command).toBe(process.execPath); + expect(bootstrap[0]?.args?.[0]).toBe('-e'); + }); + + it('resolves the builtin preset with one bootstrap step and no env', async () => { + const { lookupPreset, registerBuiltinPresets, resolveWorkspacePreset } = + await loadFreshWorkspaceModules(); + const homeDir = createTempDir(); + + registerBuiltinPresets(); + const plan = resolveWorkspacePreset( + { homeDir }, + lookupPreset('agent-tty-smoke'), + ); + + expect(plan.bootstrapCount).toBe(1); + expect(plan).not.toHaveProperty('env'); + }); +}); diff --git a/test/unit/evals/workspaces/registry.test.ts b/test/unit/evals/workspaces/registry.test.ts new file mode 100644 index 00000000..5dff0a0b --- /dev/null +++ b/test/unit/evals/workspaces/registry.test.ts @@ -0,0 +1,79 @@ +import { beforeEach, describe, expect, it } from 'vitest'; + +import { + clearPresetsForTesting, + listPresets, + lookupPreset, + registerPreset, +} from '../../../../evals/workspaces/registry.js'; +import type { WorkspacePreset } from '../../../../evals/workspaces/types.js'; + +function createPreset(id: string): WorkspacePreset { + return { + id, + mode: 'isolated', + description: `${id || 'workspace'} preset description`, + }; +} + +beforeEach(() => { + clearPresetsForTesting(); +}); + +describe('workspace preset registry', () => { + it('registers a minimal valid preset and looks it up by id', () => { + const preset = createPreset('alpha'); + + registerPreset(preset); + + expect(lookupPreset('alpha')).toEqual(preset); + }); + + it('rejects duplicate preset ids', () => { + registerPreset(createPreset('duplicate-id')); + + expect(() => registerPreset(createPreset('duplicate-id'))).toThrow( + 'Workspace preset "duplicate-id" is already registered.', + ); + }); + + it.each(['Bad-ID', '', '_leading-underscore'])( + 'rejects invalid preset id %j', + (id) => { + const registerInvalidPreset = () => registerPreset(createPreset(id)); + + expect(registerInvalidPreset).toThrow(`Invalid workspace preset "${id}"`); + expect(registerInvalidPreset).toThrow( + 'id must match /^[a-z0-9][a-z0-9-_]*$/', + ); + }, + ); + + it('lists sorted available ids when lookup misses', () => { + registerPreset(createPreset('zeta')); + registerPreset(createPreset('alpha')); + registerPreset(createPreset('beta')); + + expect(() => lookupPreset('missing-preset')).toThrow( + 'Unknown workspace preset "missing-preset". Available: [alpha, beta, zeta]', + ); + }); + + it('reports that no presets are registered when lookup misses on an empty registry', () => { + expect(() => lookupPreset('missing-preset')).toThrow( + 'Unknown workspace preset "missing-preset". No workspace presets are registered.', + ); + }); + + it('returns a sorted snapshot from listPresets', () => { + const gamma = createPreset('gamma'); + const alpha = createPreset('alpha'); + const beta = createPreset('beta'); + + registerPreset(gamma); + registerPreset(alpha); + registerPreset(beta); + + expect(listPresets()).toEqual([alpha, beta, gamma]); + }); +}); diff --git a/test/unit/evals/workspaces/resolver.test.ts b/test/unit/evals/workspaces/resolver.test.ts new file mode 100644 index 00000000..87e721e5 --- /dev/null +++ b/test/unit/evals/workspaces/resolver.test.ts @@ -0,0 +1,169 @@ +import * as fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; + +import { afterEach, describe, expect, it } from 'vitest'; + +import { + deriveEffectiveEnv, + resolveWorkspacePreset, +} from '../../../../evals/workspaces/resolver.js'; +import type { WorkspacePreset } from '../../../../evals/workspaces/types.js'; + +const tempDirs: string[] = []; + +afterEach(() => { + for (const tempDir of tempDirs.splice(0, tempDirs.length)) { + fs.rmSync(tempDir, { recursive: true, force: true }); + } +}); + +function createTempDir(): string { + const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'atty-ws-resolver-')); + tempDirs.push(tempDir); + return tempDir; +} + +function createPreset( + overrides: Partial = {}, +): WorkspacePreset { + return { + id: 'resolver-preset', + mode: 'isolated', + description: 'Resolver test preset', + ...overrides, + }; +} + +describe('resolveWorkspacePreset', () => { + it('throws when templateDir does not exist and names the preset id and path', () => { + const homeDir = createTempDir(); + const missingTemplateDir = path.join(homeDir, 'missing-template'); + const preset = createPreset({ + id: 'missing-template', + templateDir: missingTemplateDir, + }); + + const resolvePreset = () => resolveWorkspacePreset({ homeDir }, preset); + + expect(resolvePreset).toThrow(`Workspace preset "${preset.id}"`); + expect(resolvePreset).toThrow(missingTemplateDir); + }); + + it('throws when templateDir points at a file instead of a directory', () => { + const homeDir = createTempDir(); + const templateFile = path.join(homeDir, 'template.txt'); + fs.writeFileSync(templateFile, 'not a directory'); + const preset = createPreset({ + id: 'file-template', + templateDir: templateFile, + }); + + expect(() => resolveWorkspacePreset({ homeDir }, preset)).toThrow( + `Workspace preset "${preset.id}" templateDir "${templateFile}" is not a directory.`, + ); + }); + + it('redacts secret-like env keys with a case-insensitive suffix match', () => { + const homeDir = createTempDir(); + const plan = resolveWorkspacePreset( + { homeDir }, + createPreset({ + id: 'redacted-env', + env: { + API_TOKEN: 's1', + api_key: 's2', + Database_Password: 's3', + MY_SECRET: 's4', + FOO_BAR: 'ok', + }, + }), + ); + + expect(plan.env).toEqual({ + API_TOKEN: '[REDACTED]', + api_key: '[REDACTED]', + Database_Password: '[REDACTED]', + MY_SECRET: '[REDACTED]', + FOO_BAR: 'ok', + }); + }); + + it('omits cwd when absent and preserves absolute or home-relative cwd values', () => { + const homeDir = createTempDir(); + const absoluteCwd = path.join(homeDir, 'absolute-cwd'); + + const omittedCwdPlan = resolveWorkspacePreset( + { homeDir }, + createPreset({ id: 'cwd-omitted' }), + ); + const absoluteCwdPlan = resolveWorkspacePreset( + { homeDir }, + createPreset({ id: 'cwd-absolute', cwd: absoluteCwd }), + ); + const relativeCwdPlan = resolveWorkspacePreset( + { homeDir }, + createPreset({ id: 'cwd-relative', cwd: './work' }), + ); + + expect(omittedCwdPlan).not.toHaveProperty('cwd'); + expect(absoluteCwdPlan.cwd).toBe(absoluteCwd); + expect(relativeCwdPlan.cwd).toBe(path.resolve(homeDir, './work')); + }); + + it('normalizes bootstrap args and counts bootstrap steps', () => { + const homeDir = createTempDir(); + const plan = resolveWorkspacePreset( + { homeDir }, + createPreset({ + id: 'bootstrap-normalization', + bootstrap: [{ command: 'node' }], + }), + ); + + expect(plan.bootstrap).toEqual([{ command: 'node', args: [] }]); + expect(plan.bootstrapCount).toBe(1); + }); + + it('never serializes raw secret env values into the resolved plan', () => { + const homeDir = createTempDir(); + const secretValue = 'super-secret-value'; + const planJson = JSON.stringify( + resolveWorkspacePreset( + { homeDir }, + createPreset({ + id: 'secret-invariant', + env: { API_TOKEN: secretValue }, + }), + ), + ); + + expect(planJson).not.toContain(secretValue); + expect(planJson).toContain('[REDACTED]'); + }); +}); + +describe('deriveEffectiveEnv', () => { + it('merges preset env first, applies overrides on top, and preserves raw secret values', () => { + const preset = createPreset({ + id: 'effective-env', + env: { + API_TOKEN: 'super-secret-value', + SHARED: 'preset-value', + KEEP: 'keep-me', + }, + }); + + expect( + deriveEffectiveEnv(preset, { + SHARED: 'override-value', + EXTRA: 'extra-value', + }), + ).toEqual({ + API_TOKEN: 'super-secret-value', + SHARED: 'override-value', + KEEP: 'keep-me', + EXTRA: 'extra-value', + }); + }); +});