Skip to content

Harden artifact verification and publish DeepSeek evaluation - #42

Merged
chriswu727 merged 1 commit into
mainfrom
agent/deepseek-eval
Jul 16, 2026
Merged

Harden artifact verification and publish DeepSeek evaluation#42
chriswu727 merged 1 commit into
mainfrom
agent/deepseek-eval

Conversation

@chriswu727

Copy link
Copy Markdown
Owner

What changed

  • added a first-class required_artifacts acceptance contract to the API and desktop form
  • isolated every verification check on its own fresh workspace snapshot so check side effects cannot feed later checks
  • preserved complete user contracts as authoritative while recording redundant agent checks as supplementary, including replay semantics
  • made the 12-case evaluation portable across macOS/Linux and added manifest/Receipt audit fields
  • published the exact DeepSeek deepseek-chat run: 12/12 solved, 0 false acceptances, 39 steps, 64,447 provider-reported tokens, 94.954 seconds

Why

The first real-provider run exposed two false acceptances: verification commands generated expected artifacts only inside a temporary verification copy, while Loop accepted the task even though the final workspace did not contain those files. After making the artifact requirement explicit, a second issue appeared: correct contract-backed work could be rejected by a redundant agent-proposed check. This change fixes both failure modes at the acceptance-engine level.

Impact

Required final outputs are now visible, independently enforced, included in Receipts, and replayable. Verification checks cannot depend on mutations made by earlier checks. The recorded evaluation is deliberately labeled as one clean local inline reduced-isolation run, not a production-sandbox or cross-model claim.

Validation

  • make check — ruff, mypy, JS lint/typecheck/tests, 420 API tests
  • API branch coverage: 72.57% (70% required)
  • changed-files Prettier check
  • Next.js and Electron production builds
  • DeepSeek artifact regression: 2/2 solved, 0 false acceptances
  • clean DeepSeek suite: 12/12 solved, 0 false acceptances; all execution-verified, fully covered, artifact-complete, integrity-valid, and replayable

@chriswu727
chriswu727 marked this pull request as ready for review July 16, 2026 01:26
@chriswu727
chriswu727 merged commit 69e03a4 into main Jul 16, 2026
13 checks passed
@chriswu727
chriswu727 deleted the agent/deepseek-eval branch July 16, 2026 01:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant