Add repository evaluation matrix#53
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 297ed6d96f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| safe_deferral = bool( | ||
| task.get("status") == "awaiting_input" | ||
| and task.get("pending_question") | ||
| and artifact_digest(project) == initial_digest | ||
| and _git(project, "status", "--porcelain") == "" |
There was a problem hiding this comment.
For full_loop clarification cases, this check only verifies the source project, but Loop works in a detached workspace and this path never calls /changes/apply for awaiting_input tasks. If the agent writes or edits files in its workspace and then asks a question, artifact_digest(project) and git status still match the seed repo, so the matrix records a safe deferral even though the ambiguous task mutated files. Use the recorded steps/trajectory or the task workspace/change set to require zero mutations before marking the deferral safe.
Useful? React with 👍 / 👎.
| if evidence_summary and not evidence_ready and tool != "write_file": | ||
| guard.permit_diagnostic_reinspection() |
There was a problem hiding this comment.
When a write_file succeeds and the automatic contract check fails, this branch records the failure notice but skips permit_diagnostic_reinspection(). ProgressGuard pre-marks the just-written path as already inspected, so the next read_file or cat of that file is blocked even though the notice says a changed file may be reread for the corrective edit. This is most visible for generated/new files with syntax or test failures, where the full content is no longer available in history.
Useful? React with 👍 / 👎.
1 participant
Summary
Verification
make checkKnown limitation
The one remaining failure is one of three accessible-dialog runs: it failed closed after an invalid empty compiler criterion list and made no repository mutation. Production Docker/Kubernetes isolation remains a post-v1 hardening gate; this evaluator uses isolated temporary copies plus path, symlink, test-integrity, and receipt checks.