fix: teach the planner to write falsifiable criteria; stop mislabelling the override - #49
Conversation
…ng the override
From the productive trial. With the live-path check in place the run correctly
ended 'blocked', but two things were wrong:
- The override line claimed the 'precondition-fabrication check' fired when it
was the unfalsifiable-criteria check. Both feed one list, so the label was
hardcoded and wrong; it now names a deterministic check without guessing
which.
- Nothing told the planner up front, so it emitted the same vacuous shape
through both replans and burned the budget. The planner prompt now states
that success_criteria must exit non-zero on failure, names the forbidden
tails ('|| echo', '|| true', '|| :', '; true', bare 'echo'), and gives
good/bad examples.
Rejecting after the fact is the backstop; the prompt is where the fix belongs.
Full suite 607 + 14 subtests.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ff713a9393
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "depends_on": [<list of 0-based indices of tasks this depends on>] | ||
| } | ||
|
|
||
| success_criteria MUST be falsifiable: it has to exit non-zero when the work did |
There was a problem hiding this comment.
The new prose is inserted between the first task object and the closing ], where the model would expect another array item or the array terminator. Because the same prompt demands JSON-only output and _call_litellm passes the response directly to json.loads, a model that follows or imitates this malformed structure will fail parsing on both attempts and force decompose onto its heuristic fallback instead of producing the improved live plan. Close the example schema before presenting the falsifiability guidance.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
This PR updates the planner prompt to require falsifiable success_criteria (commands that can fail) and fixes an incorrect deterministic-override label shown when deterministic checks reject an otherwise LLM-approved plan.
Changes:
- Strengthens the planner’s system prompt with explicit rules/examples for falsifiable
success_criteria(non-zero on failure; forbid always-zero tails). - Fixes the deterministic override message so it no longer claims the precondition-fabrication check fired when other deterministic checks may have.
- (Suggested) Renames the internal “deterministic issues” accumulator variable to reduce future mislabeling risk.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
justai/scope_planner.py |
Tightens planner instructions for falsifiable success criteria, but currently breaks the “JSON-only” example structure in the prompt. |
justai/reviewer.py |
Corrects deterministic override messaging to avoid misattributing which deterministic check triggered. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| "[deterministic override] The model reviewer APPROVED this plan; a " | ||
| "deterministic check overrode that approval for the reason(s) below." |
| "success_criteria": "<bash command that exits 0 on success and NON-ZERO on failure>", | ||
| "depends_on": [<list of 0-based indices of tasks this depends on>] | ||
| } | ||
|
|
||
| success_criteria MUST be falsifiable: it has to exit non-zero when the work did | ||
| not happen. Never end it with "|| echo ...", "|| true", "|| :" or "; true", and | ||
| never make it a bare "echo ..." -- those exit 0 whichever branch runs, so the | ||
| task would be marked done without verifying anything. Write the check itself: | ||
| good: grep -Fxq 'Hello' greeting.py | ||
| good: test -s out.json && python3 -c "import json;json.load(open('out.json'))" | ||
| bad: grep -Fxq 'Hello' greeting.py && echo 'success' || echo 'failure' | ||
|
|
||
| ] | ||
| } |
From the productive trial. With the live-path check in place the run correctly ended
blocked, but the override line claimed the precondition-fabrication check fired when it was the unfalsifiable-criteria check (both feed one list, label was hardcoded), and nothing told the planner up front — so it re-emitted the same vacuous shape through both replans.Planner prompt now requires a criterion that exits non-zero on failure, names the forbidden tails, and gives good/bad examples. Rejecting afterwards is the backstop; the prompt is where the fix belongs.
Full suite 607 + 14.