Recover empty contract criteria safely#55
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d554379ff8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| )[:12] | ||
| recovered = False | ||
| if not criteria: | ||
| criteria = _bounded_fallback_criteria(fallback_criteria or []) |
There was a problem hiding this comment.
Do not treat test-pass goals as substantiated criteria
When the compiler returns criteria: [], this falls back to the entire user goal as one criterion; for common goals such as Implement CSV export and make all tests pass, _test_previews_cover_contract() skips marker matching for any criterion containing test and pass, so _effective_checks() can promote an unrelated discovered test command as contract evidence even when the tests do not mention the requested behavior. If the critic accepts, the recovered contract can lock on placeholder/stale tests and later verify by passing them, creating a false acceptance for the actual task.
Useful? React with 👍 / 👎.
Summary
Root cause
An otherwise usable compiler response with
criteria: []failed schema validation before repository evidence or the independent critic could participate, causing an unnecessary step-zero clarification in one accessible-dialog evaluation run.Safety behavior
Recovery does not auto-accept the instruction. The accessible-dialog fixture locks only because its discovered test source substantiates the recovered criterion. An unrelated no-evidence goal still fails closed. Recovery is bounded to 4,000 characters and malformed non-criteria fields remain invalid.
Verification