Skip to content

Enforce isolated repository evaluation#56

Merged
chriswu727 merged 1 commit into
mainfrom
agent/enforce-isolated-evaluation
Jul 22, 2026
Merged

Enforce isolated repository evaluation#56
chriswu727 merged 1 commit into
mainfrom
agent/enforce-isolated-evaluation

Conversation

@chriswu727

Copy link
Copy Markdown
Owner

What changed

  • add --require-isolation container|kubernetes to the repository matrix and bind it into checkpoints
  • fail the primary evaluation gate if any full-Loop cell reports a different or missing isolation backend
  • fail the gate for missing, unreported, or mixed model identity
  • add a disposable Docker-required full-Loop matrix launcher and Make target
  • document the exact Gate 4 workflow without changing the frozen DeepSeek result

Why

The existing reports recorded isolation but did not require it. A misconfigured run could silently downgrade to inline and still pass the release gate. Model fallback could also mix identities while the headline gate remained green.

Validation

  • Python: ruff, format, mypy, 547 tests
  • Web: lint, typecheck, 19 tests, production build
  • Desktop: lint, typecheck, 14 tests, production build
  • shellcheck and bash -n for the launcher
  • live Docker sandbox command: exit 0 with expected container output
  • launcher preflight: refuses before model invocation when no provider is configured
  • pnpm and Python production dependency audits: no known vulnerabilities

A paid real-provider Docker matrix was not run because no provider credential is present in the environment; the frozen 95.24% report remains unchanged.

@chriswu727
chriswu727 marked this pull request as ready for review July 22, 2026 04:04
@chriswu727
chriswu727 merged commit 38ea5b3 into main Jul 22, 2026
13 checks passed
@chriswu727
chriswu727 deleted the agent/enforce-isolated-evaluation branch July 22, 2026 04:04

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2fbb6db3e1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +642 to +643
primary_gate["model_identity_passed"] = same_model_across_modes
primary_gate["passed"] = bool(primary_gate["passed"] and same_model_across_modes)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Gate on every executor model, not just the final one

When this new gate is meant to reject mixed-model fallbacks, same_model_across_modes only reflects each result’s collapsed model value. Full-loop receipts already carry provenance.executor_models, but _full_loop reduces them to provenance.model; if an early successful call uses a fallback provider and a later call succeeds on the primary again, the cell can report the primary as its final model and this gate still passes despite a mixed-model run. The gate should validate all executor identities recorded for each full-loop receipt, not just the final/collapsed identity.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant