fix(reviewer): run the unfalsifiable-criteria check on the live path - #48
Conversation
The installed-executor productive trial re-ran with #47 installed and the vacuous criterion was still approved: the check lived only in _heuristic_review, which runs solely when the model call fails. In normal operation the model reviewer approved grep -Fxq 'Hello from JustAi' greeting.py && echo 'success' || echo 'failure' so the task verified vacuously and the receipt again read 'done: failure'. Same shape as #40 -- a deterministic backstop is worthless unless it runs on the path production actually takes. Focused 24/24; full suite 607 + 14 subtests.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
Pull request overview
This PR ensures the “unfalsifiable success criteria” deterministic check runs on the live review() path (not only the _heuristic_review fallback), so an LLM approval can’t bypass it.
Changes:
- Run
_unfalsifiable_criteria(plan)as a deterministic backstop inreview()and use it to override an LLM approval. - Add a regression test that forces the model reviewer to approve and asserts
review()still rejects vacuous criteria.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
justai/reviewer.py |
Executes the unfalsifiable-criteria check on the live LLM-first review path and applies a deterministic override. |
tests/test_reviewer_vacuous_criteria.py |
Adds a regression test to ensure deterministic rejection happens even when the model approves. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| # the model call fails -- i.e. never in normal operation. The productive | ||
| # trial caught exactly that: the model approved a criterion that exits 0 | ||
| # whichever branch runs, so the task verified vacuously. | ||
| fabricated = _unfalsifiable_criteria(plan) + _fabricated_preconditions(plan) |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0da3a0a8e5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # the model call fails -- i.e. never in normal operation. The productive | ||
| # trial caught exactly that: the model approved a criterion that exits 0 | ||
| # whichever branch runs, so the task verified vacuously. | ||
| fabricated = _unfalsifiable_criteria(plan) + _fabricated_preconditions(plan) |
There was a problem hiding this comment.
Adding _unfalsifiable_criteria to this unconditional override makes its existing broad pattern reject valid plans even when the model approves them. For example, echo checking && test -f output.txt can exit nonzero when the file is absent, but _ALWAYS_ZERO_RE treats every single-line command beginning with echo or printf as always successful; similarly, the if heuristic assumes ordinary branch commands cannot fail. Such criteria now cause review() to return approved=False on the normal model-backed path, potentially trapping valid plans in replanning, so the detector needs to distinguish genuinely falsifiable trailing commands before it is used as a hard override.
Useful? React with 👍 / 👎.
Re-running the productive trial with #47 installed showed the vacuous criterion still approved — the check lived only in
_heuristic_review, which runs only when the model call fails. The model reviewer approved it, so the task verified vacuously and the receipt again readdone: failure.Same shape as #40: a deterministic backstop is worthless unless it runs on the path production takes.
Focused 24/24; full suite 607 + 14.