feat: add governed sol route evaluation scaffold

docs/evaluations/gpt-5.6-sol-route-pilot-v0.1.md

@@ -0,0 +1,178 @@
+# GPT-5.6 Sol xhigh–Max–Ultra exploratory pilot v0.1
+
+## Status
+
+Preregistered design. This document fixes the hypothesis, treatment, tasks,
+execution order, resource envelope, invalidation rules, and decision thresholds
+before any attestation or scored inference run. It is not evidence that any route
+is superior.
+
+All validation is local. GitHub Actions must not be invoked while organizational
+Actions minutes remain exhausted.
+
+## Frozen environment
+
+- Source repository: `hummbl-dev/model-routing-as-code`
+- Source commit: `4e9580f816d0d7982e14149e82c40fe7b67670f2`
+- Codex CLI: `0.144.1`
+- Base model: `gpt-5.6-sol`
+- Refreshed model-catalog SHA-256:
+  `3ade3703c3a258c2a7780c3e8314e52489baaa2c8634df2de795029f9364f3eb`
+- GPT-5.6 Sol catalog-record SHA-256:
+  `ad2e4fc51d9e8c355ec5a3c94020525f01e3afc3476a6578f304bffad3363c04`
+- Catalog configuration hash: `3000`
+- Catalog multi-agent version: `v2`
+- Supported route values observed in the catalog: `xhigh`, `max`, and `ultra`
+
+Catalog presence proves configuration availability, not successful execution.
+The three-arm attestation gate below must pass before scored work.
+
+## Falsifiable hypothesis
+
+For a consequential decomposable task, Ultra will either discover a validated
+critical defect or satisfy a critical rubric criterion missed by both xhigh and
+Max, or reduce time to a no-worse validated deliverable by at least 20 percent.
+It will not produce a stable material advantage on the maintenance negative
+control, and Max will match or exceed it on the tightly coupled task unless
+parallel discovery changes the validated outcome.
+
+Failure to observe these task-shape differences rejects the proposed routing
+doctrine for this pilot. One exploratory pass cannot canonize a policy even when
+the directional hypothesis is supported.
+
+## Experimental design
+
+| Element | Preregistered value |
+| --- | --- |
+| Independent variable | `model_reasoning_effort`: xhigh, max, or ultra; Ultra alone has multi-agent enabled |
+| Primary outcomes | Frozen-rubric task success, critical failures, evidence completeness, and monotonic wall-clock time to validated output |
+| Secondary outcomes | Repair cycles, unique defects, duplicate work, synthesis loss, interventions, runtime token counters when qualified, and cost only if authoritative telemetry exists |
+| Control posture | xhigh is the maintenance baseline; Max is the tightly coupled counterfactual |
+| Exploratory sample | Three tasks × three routes × one repetition = nine scored cells |
+| Attestation sample | One unscored fixture × three routes = three cells |
+| Resource regime | Route-native capacity within the common operational ceilings below |
+| Policy effect | None; the pilot may only authorize, reject, or redesign a repeated confirmatory block |
+
+Each cell starts from an identical copied fixture under an isolated temporary
+workspace. Raw JSONL, final workspaces, command output, and runtime metadata stay
+outside Git. Published receipts contain only reviewed values and hashes.
+
+## Frozen tasks and order
+
+The task fixture packet contains one unscored attestation task and three scored
+task shapes:
+
+1. `maintenance_negative_control`: a bounded deterministic maintenance defect
+   that should not need escalation beyond xhigh.
+2. `tightly_coupled_debugging`: a correctness problem whose constraints interact
+   across the implementation.
+3. `decomposable_greenfield`: a governed multi-artifact build with independent
+   evidence, implementation, validation, and synthesis boundaries.
+
+Scored execution order is a fixed Latin square:
+
+| Ordinal | Task | Route |
+| ---: | --- | --- |
+| 1 | maintenance negative control | xhigh |
+| 2 | maintenance negative control | max |
+| 3 | maintenance negative control | ultra |
+| 4 | tightly coupled debugging | max |
+| 5 | tightly coupled debugging | ultra |
+| 6 | tightly coupled debugging | xhigh |
+| 7 | decomposable greenfield | ultra |
+| 8 | decomposable greenfield | xhigh |
+| 9 | decomposable greenfield | max |
+
+No route may inspect another route's workspace, final response, receipt, or
+validation output before all scored cells finish.
+
+## Hard resource envelope
+
+- Maximum inference invocations: 12 total—three attestation and nine scored.
+- Attempts per cell: one. A retry requires invalidating the affected block and a
+  new preregistration revision.
+- Maximum elapsed time per invocation: 600 seconds.
+- Maximum aggregate harness elapsed time: 7,200 seconds.
+- Network, sandbox, approvals, tools, prompt bytes, validation commands, and
+  source fixtures are identical across routes. Only the registered treatment
+  changes.
+- Any usage-limit or rate-limit response stops the block. It is a confound, not a
+  model failure.
+
+The local subscription CLI exposes no enforceable USD or aggregate-token ceiling
+and no authoritative cost field. Those budget values are explicitly unavailable.
+The invocation and elapsed-time ceilings are enforceable operational limits; they
+must not be relabeled as equal-dollar or equal-token budgets.
+
+## Attestation gate
+
+Before scored execution, every route must complete the same unscored fixture and
+produce a valid private run bundle.
+
+Required evidence for each arm:
+
+- the requested model and effort;
+- a runtime-observed effective model and effort from structured events or the
+  read-only Codex state record;
+- no reroute;
+- matching CLI and model-catalog identity;
+- deterministic workspace validation;
+- monotonic wall-clock duration;
+- no usage, rate, timeout, sandbox, or protected-file confound.
+
+xhigh and Max must have multi-agent disabled and no child thread. Ultra must have
+multi-agent enabled and expose at least one actual child execution or equivalent
+structured delegation event on the legitimately decomposable attestation task.
+Configured agent count alone is insufficient.
+
+The runtime `tokens_used` counter is retained with its source and semantics marked
+unqualified until it reconciles with structured token events. It is not converted
+to billed tokens or cost. If any arm fails attestation, no scored cell runs.
+
+## Validation and adjudication
+
+Automated task validators execute from frozen argv arrays without a shell. Their
+results measure only the behaviors they cover. After all cells complete:
+
+1. Copy sanitized deliverables to route-blinded labels derived before review.
+2. Two reviewers independently apply the frozen rubric without access to the
+   route-label mapping.
+3. Resolve disagreements through a recorded consensus rule; preserve both initial
+   scores.
+4. Unblind only after automated and human scores are immutable.
+5. Record unique useful findings, duplicated investigation, rejected findings,
+   synthesis omissions, and human interventions without estimating unavailable
+   quantities.
+
+## Pilot decision rules
+
+The pilot is invalid if route attestation, task hashes, validation commands,
+catalog identity, effective model/effort, common environment, or the one-attempt
+rule drifts.
+
+Ultra has directional support on the decomposable task only if it:
+
+- uniquely changes a critical rubric outcome missed by both single-agent routes;
+  or
+- produces a no-worse validated score at least 20 percent faster than the faster
+  valid single-agent route;
+
+and introduces no additional critical failure or material synthesis loss.
+
+A tie favors the cheaper route, but no economic winner can be declared when cost
+or qualified token telemetry is unavailable. Max has directional support for the
+tightly coupled class if it meets or exceeds the other routes without material
+time, safety, or evidence disadvantage. xhigh remains the maintenance default
+when escalation does not change the validated result.
+
+The pilot advances to confirmation only when all nine cells are valid, at least
+one task meaningfully discriminates routes, and the observed task-shape pattern is
+consistent with the hypothesis. Otherwise the decision is `KEEP_CURRENT`,
+`REDESIGN`, or `INCONCLUSIVE`.
+
+## Confirmatory boundary
+
+Confirmation is a separate authorized block: at least two decomposable tasks, two
+tightly coupled tasks, one negative control, three repetitions per route/task,
+counterbalanced order, and a new spending authorization. This pilot cannot change
+the route policy by itself.

experiments/gpt-5.6-sol-route-pilot/v0.1/PREREGISTRATION.md

@@ -0,0 +1,140 @@
+# GPT-5.6 Sol xhigh–Max–Ultra route pilot v0.1
+
+## Status
+
+This packet was preregistered on 2026-07-11 before execution. No inference has
+been run from this packet. The pilot is exploratory, uses one run per cell, and
+cannot authorize a routing-policy change. A later confirmation experiment would
+be required before any task-class-specific policy update; no policy update may be
+made from this pilot alone.
+
+## Research question and hypothesis
+
+For frozen, consequential local tasks, does four-agent orchestration discover a
+validated outcome improvement or reduce time to a validated deliverable enough to
+justify its coordination and aggregate-use burden relative to the two single-agent
+arms?
+
+The preregistered hypothesis is conditional: parallel orchestration may help the
+decomposable greenfield task, should not materially help the maintenance negative
+control, and may incur duplication or synthesis burden on tightly coupled debugging.
+No arm is assumed superior before observation.
+
+## Frozen source and runtime
+
+- Repository: `https://github.com/hummbl-dev/model-routing-as-code.git`
+- Merge commit: `4e9580f816d0d7982e14149e82c40fe7b67670f2`
+- Source worktree: must be clean; any task, rubric, validator, runner, schema, or
+  catalog drift invalidates the affected block as specified by the manifest.
+- Base model: `gpt-5.6-sol`
+- Codex CLI: `/usr/local/bin/codex`, version `0.144.1`, SHA-256
+  `c6eb747e4145ecb3bed2647dbd0f8464b190a5ccba964666ef7c98d4681a4a4c`
+- Validator interpreter: `/usr/local/bin/python3`, Python `3.13.7`
+- Catalog canonical SHA-256:
+  `3ade3703c3a258c2a7780c3e8314e52489baaa2c8634df2de795029f9364f3eb`
+- `gpt-5.6-sol` catalog record SHA-256:
+  `ad2e4fc51d9e8c355ec5a3c94020525f01e3afc3476a6578f304bffad3363c04`
+- Catalog metadata: `comp_hash=3000`, context window `372000`, multi-agent
+  version `v2`, API-supported settings `xhigh`, `max`, and `ultra`.
+
+The runner must ignore user configuration and rules, use copied fixture workspaces
+under a temporary directory, keep route outputs isolated, prohibit GitHub Actions,
+and expose the same offline tool and workspace-write envelope to every arm.
+
+## Attestation before scored cells
+
+One identical, route-neutral attestation is run once for each arm before any scored
+cell. It contains three small independent checks: a text-state edit, a JSON
+calculation, and a Python normalization repair. The prompt permits delegation only
+when the configured route provides it. These three invocations are unscored and
+exist only to attest effective route metadata, workspace writing, tool execution,
+and local validation.
+
+An attestation failure stops the pilot before scored work. It is not retried or
+silently converted into a scored invocation.
+
+## Scored tasks and frozen order
+
+The three task fixtures are synthetic and contain no credentials, private data, or
+network dependency:
+
+1. `maintenance-negative-control`: a two-file support-address update with exact
+   scope preservation.
+2. `tightly-coupled-debugging`: repair an event-driven non-preemptive scheduler
+   whose timing, priority, tie-break, and validation rules interact.
+3. `decomposable-greenfield`: build a library, CLI, schema, and documentation for a
+   deterministic incident digest.
+
+The Latin-square-style order is frozen:
+
+| Sequence | Task | Arm |
+| ---: | --- | --- |
+| 1 | maintenance-negative-control | xhigh |
+| 2 | maintenance-negative-control | max |
+| 3 | maintenance-negative-control | ultra |
+| 4 | tightly-coupled-debugging | max |
+| 5 | tightly-coupled-debugging | ultra |
+| 6 | tightly-coupled-debugging | xhigh |
+| 7 | decomposable-greenfield | ultra |
+| 8 | decomposable-greenfield | xhigh |
+| 9 | decomposable-greenfield | max |
+
+There is one attempt and one repetition per cell. A failed, timed-out, rerouted, or
+invalid cell remains failed or invalid; it is not retried under this protocol.
+
+## Resource ceiling and stopping rule
+
+- Total inference invocations: at most 12.
+- Unscored route attestations: exactly 3 if all attestations are reached.
+- Scored invocations: at most 9.
+- Attempts per scored cell: 1.
+- Scored-cell deadline: 600 seconds.
+- Whole-pilot deadline: 7,200 seconds, including attestation and validation.
+- Resource-budget regime: route-native capacity; aggregate and per-agent use are
+  recorded separately when exposed.
+
+Enforceable aggregate and per-agent USD and token ceilings are unavailable because
+this Codex CLI surface exposes no enforceable interface for pre-binding any of those
+values. No numeric estimate is substituted. The runner must stop immediately on a
+usage limit or rate limit, record the remaining cells as not run, and report the
+economic comparison as inconclusive. The same stop rule applies if any configured
+ceiling unexpectedly binds or the effective route cannot be attested.
+
+## Validation and adjudication
+
+Each route receives identical prompt and workspace bytes for a task. Task-local
+instructions, validators, rubrics, and frozen inputs are protected; modifying them
+invalidates the cell even if a candidate changes tests to produce a passing result.
+Only the explicit fixture-relative `workspace/...` editable paths may change.
+
+Automated local validation establishes the behavioral hard gate. The preregistered
+rubric preserves component scores rather than collapsing correctness, scope,
+documentation, duplication, or synthesis into an unexplained composite. Route
+outputs remain hidden from later arms until all nine cells finish or the pilot
+stops. Any human review must be blinded to route identity where practical.
+
+Primary observations are validated task success, critical correctness failures,
+and elapsed time to the first validated deliverable. Secondary observations include
+actual cost and token telemetry only if exposed, validation repair cycles, unique
+useful findings, duplicate work, synthesis loss, human intervention, and scope or
+safety failures. Unavailable measurements remain unavailable.
+
+## Raw-output and publication boundary
+
+Raw prompts, event streams, transcripts, diffs, and validation logs are written to
+the external root selected through `ROUTE_PILOT_OUTPUT_ROOT`; raw outputs remain
+outside the repository and are never published directly. The tracked repository may
+receive only sanitized, content-reviewed receipts and hashes. Raw artifacts must be
+content-reviewed before any later publication decision.
+
+## Invalidation and decision boundary
+
+The manifest lists machine-enforced invalidation conditions, including unverified
+effective route, model rerouting, catalog or fixture drift, usage/rate limits,
+budget binding, and validation-contract changes. A changed task, prompt, rubric,
+validator, success criterion, runtime envelope, or evidence standard requires a new
+protocol revision and rerunning the complete comparison block.
+
+This pilot can identify promising task-class hypotheses. It cannot establish that
+one arm is universally faster, cheaper, safer, or more capable, and it cannot
+support a policy update without repeated confirmation.