Skip to content

Harden verified completion loop - #43

Merged
chriswu727 merged 1 commit into
mainfrom
agent/harden-verified-completion
Jul 16, 2026
Merged

Harden verified completion loop#43
chriswu727 merged 1 commit into
mainfrom
agent/harden-verified-completion

Conversation

@chriswu727

Copy link
Copy Markdown
Owner

What changed

  • make complete user contracts authoritative before running redundant agent checks
  • auto-submit verified work after successful commands, not only direct file edits
  • reject empty test suites deterministically and return actionable discovery guidance
  • preserve the latest local failure diagnostic while keeping older history bounded
  • distinguish truncated previews from complete file writes and source files
  • replace the lexical HTML benchmark oracle with parsed structure and row assertions
  • require discoverable test artifacts plus visible external behavior assertions
  • publish a fresh DeepSeek report and rewrite the README around Loop's core product idea

Why

The previous benchmark exposed three avoidable failure modes: a substring-based HTML
oracle rewarded removing <thead>, an exit-zero test command could execute zero tests,
and bounded observations hid the useful end of a traceback. Data-producing tasks also
spent extra turns re-reading artifacts or proposing checks already covered by the user
contract.

Impact

The final clean DeepSeek deepseek-chat run solves 12/12 cases with zero false
acceptances in 30 steps, 42,403 provider-reported tokens, and 65.795 seconds. Every case
is execution-verified, artifact-complete, integrity-valid, and replayable.

Validation

  • make check — 428 API tests, 16 web tests, 14 desktop tests; lint, formatting,
    TypeScript, and mypy passed
  • real-provider focused regressions for HTML, Python unittest discovery, data artifacts,
    and string utilities
  • clean full 12-case DeepSeek run with matching manifest SHA-256
  • Prettier and report/README consistency checks

@chriswu727
chriswu727 marked this pull request as ready for review July 16, 2026 03:28
@chriswu727
chriswu727 merged commit c4aeafb into main Jul 16, 2026
13 checks passed
@chriswu727
chriswu727 deleted the agent/harden-verified-completion branch July 16, 2026 03:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant