feat: add governed sol route evaluation scaffold

[hummbl-dev]

Summary

• add a preregistered GPT-5.6 Sol xhigh/Max/Ultra exploratory pilot packet
• add portable manifest and receipt schemas with stdlib-only validator and fixtures
• add an isolated route-evaluation runner with strict manifest preflight, offline dry-run support, and guarded telemetry handling
• add focused tests covering manifest contracts, receipt honesty, runner behavior, and the frozen pilot packet

Validation

• python3 tools/validate_route_evaluation.py experiments/gpt-5.6-sol-route-pilot/v0.1/pilot-manifest.json
• python3 -m pytest tests/test_route_evaluation_manifest.py tests/test_route_evaluation_receipt.py tests/test_route_evaluation_runner.py tests/test_route_pilot_packet.py tests/test_ultra_publication_receipt.py -q
• python3 tools/run_route_evaluation.py --manifest experiments/gpt-5.6-sol-route-pilot/v0.1/pilot-manifest.json --task-id route-attestation --route ultra --mode dry-run --state-root /tmp/route-pilot-dryrun.TKfjPV/state --temp-root /tmp/route-pilot-dryrun.TKfjPV/tmp
• python3 tools/run_route_evaluation.py --manifest experiments/gpt-5.6-sol-route-pilot/v0.1/pilot-manifest.json --task-id decomposable-greenfield --route ultra --mode dry-run --state-root /tmp/route-pilot-dryrun.TKfjPV/state2 --temp-root /tmp/route-pilot-dryrun.TKfjPV/tmp2

Limitations

• no attestation or scored inference cells have been run yet
• aggregate token and dollar telemetry remain unavailable until the runtime exposes authoritative semantics
• fixture-only write scope is enforced; fixture-only read scope cannot be cryptographically proven under the current Codex sandbox model
• unrelated local edits remain unstaged in README.md, tests/test_ultra_publication_receipt.py, and receipts/2026-07-11-gpt-5.6-sol-ultra-merge.json

Review focus

• receipt honesty around delegation, concurrency, and token aggregation
• runner preflight and invalidation behavior before any live inference
• pilot task-shape design and frozen decision thresholds

[coderabbitai]

Warning

Review limit reached

@hummbl-dev, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 3 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 375dc255-b7f9-4f8e-b90c-40ac617b2ea3

📥 Commits

Reviewing files that changed from the base of the PR and between 4e9580f and 7b56c6a.

📒 Files selected for processing (54)

• docs/evaluations/gpt-5.6-sol-route-pilot-v0.1.md
• experiments/gpt-5.6-sol-route-pilot/v0.1/PREREGISTRATION.md
• experiments/gpt-5.6-sol-route-pilot/v0.1/pilot-manifest.json
• fixtures/invalid/route_evaluation_manifest.v0.1.arm.invalid.json
• fixtures/invalid/route_evaluation_manifest.v0.1.budget.invalid.json
• fixtures/invalid/route_evaluation_manifest.v0.1.hash.invalid.json
• fixtures/invalid/route_evaluation_manifest.v0.1.invalidation.invalid.json
• fixtures/invalid/route_evaluation_manifest.v0.1.sequence.invalid.json
• fixtures/invalid/route_evaluation_receipt.v0.2.delegation.invalid.json
• fixtures/invalid/route_evaluation_receipt.v0.2.deviation.invalid.json
• fixtures/invalid/route_evaluation_receipt.v0.2.numeric-overflow.invalid.json
• fixtures/invalid/route_evaluation_receipt.v0.2.reroute.invalid.json
• fixtures/invalid/route_evaluation_receipt.v0.2.route-verification.invalid.json
• fixtures/invalid/route_evaluation_receipt.v0.2.timestamp.invalid.json
• fixtures/invalid/route_evaluation_receipt.v0.2.token-sum.invalid.json
• fixtures/invalid/route_evaluation_receipt.v0.2.unavailable-value.invalid.json
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/attestation/prompt.md
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/attestation/rubric.json
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/attestation/task.json
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/attestation/validate.py
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/attestation/workspace/numbers.json
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/attestation/workspace/slug.py
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/attestation/workspace/status.txt
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/decomposable-greenfield/prompt.md
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/decomposable-greenfield/rubric.json
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/decomposable-greenfield/task.json
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/decomposable-greenfield/validate.py
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/decomposable-greenfield/workspace/SPEC.md
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/decomposable-greenfield/workspace/sample-events.jsonl
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/maintenance-negative-control/prompt.md
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/maintenance-negative-control/rubric.json
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/maintenance-negative-control/task.json
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/maintenance-negative-control/validate.py
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/maintenance-negative-control/workspace/README.md
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/maintenance-negative-control/workspace/reference-notes.txt
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/maintenance-negative-control/workspace/service-config.json
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/tightly-coupled-debugging/prompt.md
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/tightly-coupled-debugging/rubric.json
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/tightly-coupled-debugging/task.json
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/tightly-coupled-debugging/validate.py
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/tightly-coupled-debugging/workspace/README.md
• fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/tightly-coupled-debugging/workspace/scheduler.py
• fixtures/valid/route_evaluation_manifest.v0.1.valid.json
• fixtures/valid/route_evaluation_receipt.v0.2.max.valid.json
• fixtures/valid/route_evaluation_receipt.v0.2.ultra.valid.json
• fixtures/valid/route_evaluation_receipt.v0.2.xhigh.valid.json
• schemas/route_evaluation_manifest.v0.1.schema.json
• schemas/route_evaluation_receipt.v0.2.schema.json
• tests/test_route_evaluation_manifest.py
• tests/test_route_evaluation_receipt.py
• tests/test_route_evaluation_runner.py
• tests/test_route_pilot_packet.py
• tools/run_route_evaluation.py
• tools/validate_route_evaluation.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/codex/sol-route-pilot-v0-1

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[hummbl-dev]

Peer Review — independent non-author review

This PR adds a governed route-evaluation scaffold for GPT-5.6 Sol (xhigh/Max/Ultra): a preregistered pilot manifest, portable manifest/receipt JSON schemas with a stdlib-only validator, an isolated execution runner with strict preflight and dry-run support, four fixture tasks with behavioral validators, and focused tests. The design emphasizes receipt honesty (explicit unavailable telemetry, no fabricated values), frozen manifests with hash reconciliation, and one-attempt-per-cell enforcement. No inference has been run yet — this is scaffold only.

Verdict: NEEDS_DISCUSSION

The architecture is well-considered and the safety posture is stronger than typical ML eval harnesses. However, there are code-execution safety gaps in the fixture validators that should be discussed before live runs, and a few missing test coverage areas for critical preflight paths. None of these block merging the scaffold, but they should be resolved before any attestation or scored cell execution.

Findings

HIGH — Attestation validator execs candidate code without import scanning
fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/attestation/validate.py:48-50

The attestation validator calls exec(compile(source, str(slug_path), "exec"), namespace) on candidate slug.py without any AST import validation. The other two scored validators (decomposable-greenfield/validate.py:88-103, tightly-coupled-debugging/validate.py:87-104) both walk the AST and reject imports outside sys.stdlib_module_names. This inconsistency means the attestation task's candidate code could import os; os.system(...) or import socket with no guard. While the runner sanitizes the environment and copies the workspace to an isolated temp dir, the validator process itself has full filesystem and network access. Recommend adding the same _validate_imports AST check before the exec() call, or document why the attestation task is exempt.

HIGH — exec() runs candidate code in-process with no isolation or timeout
All three fixture validators (attestation/validate.py:50, decomposable-greenfield/validate.py:111, tightly-coupled-debugging/validate.py:100)

exec(compile(...)) runs candidate code in the validator's own Python process. There is no process isolation, no resource limit, and no timeout. A malicious or buggy candidate could: modify the validator's own namespace to fake a pass, access files outside the workspace (the validator process has full FS access), enter an infinite loop (blocking the cell deadline), or crash the validator process. The decomposable validator partially mitigates this by also running cli.py via subprocess.run (line 300), but the library code (incident_digest.py) is still exec'd in-process. Consider running candidate code in a subprocess with timeout= and a sanitized env, consistent with how the runner invokes the validators themselves.

MEDIUM — Environment leakage in decomposable validator subprocess calls
fixtures/pilots/gpt-5.6-sol-route-pilot/v0.1/decomposable-greenfield/validate.py:311, 351

The subprocess.run calls use env={**os.environ, "PYTHONDONTWRITEBYTECODE": "1"}, which passes the full parent environment to the candidate CLI process. This contradicts the runner's _sanitized_env() allowlist approach (run_route_evaluation.py:460-467). When invoked via the runner, the environment is already sanitized, but when run standalone (e.g., in tests or CI), any secrets in env vars are inherited by candidate code. Recommend using a minimal env or the same allowlist pattern.

MEDIUM — No test coverage for clean-worktree enforcement
tests/test_route_evaluation_runner.py

_require_clean_worktree (run_route_evaluation.py:481-495) is a critical safety property — it prevents running cells against a dirty source repository. There is no test covering it. All runner tests use dry-run mode (which skips the check) or mocked subprocess. A test that creates a dirty worktree and asserts RunnerError for attestation/scored mode would close this gap.

MEDIUM — No test coverage for manifest hash reconciliation
tests/test_route_evaluation_runner.py

_reconcile_manifest_harness and _require_manifest_file_hash (run_route_evaluation.py:939-1017) verify that manifest-declared hashes match actual file bytes for the runner, validator, receipt schema, Codex binary, prompts, and rubrics. No test covers hash-mismatch failures. This is the primary tamper-detection mechanism and should have at least one test that tampers with a tracked file and asserts rejection.

LOW — relativePath schema definition is permissive
schemas/route_evaluation_manifest.v0.1.schema.json:281

relativePath is defined as {"type": "string", "minLength": 1} with no pattern constraint. It doesn't prevent absolute paths or .. traversal at the schema level. The Python validator's _require_safe_path (validate_route_evaluation.py:271-275) handles this correctly, but defense-in-depth would add a pattern like ^[^/\\] or exclude .. segments in the schema itself.

LOW — invalidation_conditions schema minItems is 1 but 8 conditions are required
schemas/route_evaluation_manifest.v0.1.schema.json:268

The schema enforces minItems: 1, while the validator checks for 8 specific required conditions (validate_route_evaluation.py:43-52, 547-553). A manifest with a single bogus condition would pass the schema but fail the validator. This is a schema/validator semantic gap — consider encoding the required conditions in the schema's enum/contains or accepting that the validator is the authoritative layer.

LOW — _kill_process_group fallback may orphan children
tools/run_route_evaluation.py:470-478

If os.killpg fails (e.g., ProcessLookupError because the process already exited), the code falls back to process.kill(), which only kills the parent PID. For the ultra route (multi-agent), this could leave orphaned child processes. Consider logging the killpg failure or documenting that the fallback is best-effort.

LOW — Receipt stringList allows empty arrays for all uses
schemas/route_evaluation_receipt.v0.2.schema.json:78

stringList has no minItems, allowing empty arrays. This is intentional for invalidation_reasons (valid runs have none), but it's also used for findings, artifact_refs, critical_failures, etc. where empty may not be semantically meaningful. Consider a separate nonEmptyStringList (which already exists at line 82) for fields where at least one entry is expected.

INFO — Hardcoded catalog/binary hashes
tools/run_route_evaluation.py:34-37, tools/validate_route_evaluation.py:53-61

CATALOG_SHA256, MODEL_RECORD_SHA256, CATALOG_COMP_HASH, and EXPECTED_CODEX_SHA256 are hardcoded. These are attestation anchors, not secrets, and are appropriate for a frozen pilot. The runner reconciles them against the manifest and local bytes. No action needed, but these will need updating when the Codex CLI or model catalog changes.

INFO — --state-db defaults to ~/.codex/state_5.sqlite
tools/run_route_evaluation.py:1173

The default state database path is Path.home() / ".codex" / "state_5.sqlite". Access is read-only via SQLite URI ?mode=ro (line 279), and only approved columns are selected (lines 294-298). The --no-state-db-attestation flag disables it entirely. This is safe but worth noting for environments where the home directory is shared.

What's done well

• Receipt honesty is excellent. Unavailable telemetry is explicit with reasons; no values are fabricated. Token arithmetic is validated (total = input + output). The token_aggregation_method field prevents silent aggregation ambiguity.
• Runner isolation is thorough. Symlink detection in workspace copy and tree hash, shell=False everywhere, argv validation (no shell strings), git/gh/codex command blocking, env allowlist, temp-root-outside-repo and AGENTS-ancestor checks, one-attempt policy via bundle-exists check.
• Manifest hash reconciliation is strong. The runner verifies hashes of the runner script, validator, receipt schema, Codex binary, prompts, rubrics, and fixture tree against the frozen manifest before any execution. This makes tampering detectable.
• Strict JSON parsing (validate_route_evaluation.py:1005-1014): rejects NaN/Infinity via parse_constant, rejects duplicate object keys via object_pairs_hook, rejects non-finite floats. This prevents subtle JSON-level manipulation.
• Read-only SQLite access via URI ?mode=ro with schema compatibility checks before querying. Only approved columns are selected; the test at line 249-274 verifies that extra columns (like title, preview) are not leaked.
• Execution order enforcement: attestation cells must pass before scored cells; within attestation, routes must run in preregistered order; scored cells must run in sequence. This prevents cherry-picking favorable runs.
• Invalid fixture set is comprehensive (12 invalid fixtures covering arm drift, budget mismatch, hash mismatch, invalidation gaps, sequence gaps, delegation/reroute/token-sum/timestamp/route-verification/numeric-overflow/unavailable-value violations).
• Tests verify both directions: initial workspaces don't already pass validators (line 342-347), and known-good edits do pass (line 349-391). This prevents trivially-passing fixtures.
• No external dependencies — the validator implements a JSON Schema subset in pure stdlib, consistent with the offline/airgapped design goal.
• Route-neutral prompts — tests verify prompts don't contain route names (xhigh/max/ultra) and include required safety instructions.

CI Status

Workflow	Status
CodeRabbit	pass

Only CodeRabbit ran on this PR. No GitHub Actions CI workflows (test, lint, build) are configured for this repository or were triggered. The PR body includes local validation commands but these are not gated by CI. Recommend adding a GitHub Actions workflow that runs the pytest suite on PRs touching tools/, schemas/, tests/, or fixtures/.

[hummbl-dev]

Initial review pass — pinned head 7b56c6aa1245 against main.

Evidence:

• Scope: 54 files, +7195/-0. Notable paths: docs/evaluations/gpt-5.6-sol-route-pilot-v0.1.md, experiments/gpt-5.6-sol-route-pilot/v0.1/PREREGISTRATION.md, experiments/gpt-5.6-sol-route-pilot/v0.1/pilot-manifest.json, fixtures/invalid/route_evaluation_manifest.v0.1.arm.invalid.json, fixtures/invalid/route_evaluation_manifest.v0.1.budget.invalid.json, +49 more
• Mergeability: MERGEABLE
• Head checks: PASS — 2 reported checks have no failing or pending result

Findings:

• P2: protected/governance-sensitive paths are in scope: schemas/route_evaluation_manifest.v0.1.schema.json, schemas/route_evaluation_receipt.v0.2.schema.json.
• P2: 2 added TODO/TBD/FIXME/placeholder marker(s) need disposition.
• P2: large review surface requires decomposition or an explicit large-change rationale.
• P3: 191 added normative MUST/SHALL/REQUIRED line(s) require authority-source validation.

Next review focus: Check routing defaults, fallback/fail-closed behavior, schema compatibility, and negative-path tests before treating the policy as executable.

Gate: CONTINUE. This is an evidence-backed triage comment, not final merge approval; promotion still requires content validation against this exact head and the repository's review rules.

[hummbl-agent]

Blocking findings posted in PR comment.

[hummbl-agent]

Findings: