Skip to content

feat(cli): add reproducible task-fixture replay for deterministic unattended-run evaluation (#224)#225

Merged
qwen-code-dev-bot merged 1 commit into
mainfrom
issue/224-reproducible-task-fixtures
Jul 22, 2026
Merged

feat(cli): add reproducible task-fixture replay for deterministic unattended-run evaluation (#224)#225
qwen-code-dev-bot merged 1 commit into
mainfrom
issue/224-reproducible-task-fixtures

Conversation

@qwen-code-dev-bot

Copy link
Copy Markdown
Owner

Summary

Adds a versioned task-fixture format and an opt-in (--replay-fixture) deterministic replay mode for unattended (-p) runs, decomposed from roadmap #38 (observable autonomous execution).

Before/after evaluation of a model or configuration change was noisy because there was no shared, repeatable definition of the task being measured — re-running an ad hoc prompt is not reproducible. A fixture declares a bounded prompt plus a deterministic provider/tool script; replaying it drives one unattended run whose run-summary (#40) fields are reproducible, so scorecards (#44) compare like-for-like inputs.

Design

  • New src/task-fixture.ts: a schema-validated fixture format (prompt + script of text/tool-call steps), fail-closed parsing that rejects raw-credential fields, and fixtureStreamProvider — a deterministic in-process provider that replays the script one step per round (fixed usage; a terminal empty-text response when the script is exhausted so the run terminates).
  • src/provider.ts: a StreamProvider type (the network streamChat is the default implementation).
  • src/agent.ts: an optional streamProvider seam on AgentOptions; runAgent uses it instead of streamChat when present.
  • src/index.ts: opt-in --replay-fixture <file> wired into both headless paths — loads the fixture (fail closed, exit 2), runs from the fixture's prompt, and injects the deterministic provider.

The replay never touches the network provider; the fixture format has no raw-credential fields and loading fails closed.

Testing

  • tests/unit/task-fixture.test.ts (15 tests): schema validation, version negotiation, fail-closed on malformed/credential fields, file reading, deterministic replay sequencing, cross-replay determinism, and script-exhaustion termination.
  • tests/integration/task-fixture.test.ts (4 tests): two CLI replays of one fixture produce byte-identical deterministic summary fields with zero network chat calls, the scripted tool call is reflected, and malformed/credential/missing fixtures exit 2.

Gates: typecheck, build, unit (1623), integration (442 + desktop), smoke (51) all pass.

Closes #224

…ttended-run evaluation (#224)

Before/after evaluation of a model or configuration change was noisy because there was no shared, repeatable definition of the task being measured: re-running an ad hoc prompt is not reproducible. Add a versioned task-fixture format (bounded prompt + deterministic provider/tool script) and an opt-in --replay-fixture mode that drives one unattended run from the fixture via a deterministic in-process provider, so the same fixture always yields the same tool-call sequence, rounds, and token totals and feeds reproducible inputs to the run summary (#40) and scorecard (#44). Fixture loading fails closed and rejects raw-credential fields; the replay never touches the network provider.
@qwen-code-dev-bot

Copy link
Copy Markdown
Owner Author

E2E verification

  • Repository: qwen-code-dev-bot/oh-my-cli
  • Pull request: #225
  • Commit: fa051e627f0569a1e4c6805face9f4a54a69bb47
  • Evidence mode: terminal
  • Result: PASS
Scenario Command Exit Duration
integration npm run test:integration 0 28.566s
smoke npm run smoke 0 3.017s

E2E evidence

Full immutable report

@qwen-code-dev-bot
qwen-code-dev-bot merged commit 0b7870c into main Jul 22, 2026
2 checks passed
@qwen-code-dev-bot
qwen-code-dev-bot deleted the issue/224-reproducible-task-fixtures branch July 22, 2026 20:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evaluation: define and replay one deterministic task fixture for reproducible unattended-run measurement

1 participant