feat: whole-codebase and corpus model routing policy v0.1

[hummbl-dev]

Summary

• docs/routing-policy-v0.1.md: Machine-readable routing policy for whole-codebase and corpus workloads
• 8 fixtures: valid, invalid, stale, privacy-conflict, and risk scenarios

Closes #8
Refs hummbl-dev/hummbl-dev#155, hummbl-dev/autoresearch-pipeline#30, hummbl-dev/hummbl-bibliography#78, hummbl-dev/hummbl-dev#153

Required artifacts

1. Routing-policy specification

• 13 workload classes
• 18 route dimensions
• 9 route dispositions
• Governing doctrine: cheapest correct model wins only after all gates pass

2. Machine-readable workload schema

• YAML schema with source, capabilities, hard_gates, soft_rank, constraints

3. Model/scaffold capability registry

• Initial candidate lanes (7+ models)

4. Benchmark-result import schema

• benchmark_evidence_date field in route schema

5. Route decision schema and receipt

• YAML schema with selected_route, hard_gate_results, soft_rank_results, rejected_routes, receipt_id

6. Hard-gate and soft-ranking logic

• 12 hard gates (correctness, safety, privacy, authority, latency, reliability, reproducibility, governance, model_version_verified, budget_metadata, authority_metadata, privacy_metadata)
• Soft ranking: cost > latency > reliability > evidence recency
• Fail closed on missing metadata

7. Fixtures

• valid-minimal-route.json
• invalid-missing-hard-gate.json
• stale-evidence.json
• privacy-conflict.json
• missing-model-version.json
• missing-budget-metadata.json
• cheaper-route-rejected.json
• preview-model-risk.json

8. Example route cards

• Example route card for REPO_ARCHITECTURE_RECONSTRUCTION
• Example route cards for each fixture workload class

9. Migration path

• Documented as non-goal: no migration into Founder Mode without approval

Required policy behaviors

• Select lowest-cost route meeting all hard gates
• Prefer tool navigation when source exceeds effective context
• Require different-provider reviewer for high-stakes claims
• Keep private sources within approved boundaries
• Escalate on missing/stale/mismatched benchmark coverage
• Fail closed on missing metadata
• Record why cheaper routes were rejected
• Include reevaluation and expiry dates

Validation

• Documentation-only — no runtime to validate

Rollback

Revert the commit. No runtime impact.

Review

Requesting independent review. The originating agent must not self-approve or merge.

[coderabbitai]

Warning

Review limit reached

@hummbl-dev, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 3 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 043d6208-f244-419a-baf4-a50b0213141a

📥 Commits

Reviewing files that changed from the base of the PR and between 4e9580f and e312e06.

📒 Files selected for processing (39)

• .gitignore
• README.md
• data/gpt-5.6-sol-ultra/archive_manifest.json
• data/gpt-5.6-sol-ultra/archived_chart_records.json
• data/gpt-5.6-sol-ultra/benchmark_dataset.v0.1.json
• docs/evaluations/2026-07-11-gpt-5.6-sol-ultra-evidence-packet.md
• docs/evaluations/ultra-evaluation-protocol.v0.1.md
• docs/route-policy/reasoning-effort-route-policy.v0.1.md
• docs/routing-policy-v0.1.md
• docs/schemas/routing-policy/v0.1/fixtures/cheaper-route-rejected.json
• docs/schemas/routing-policy/v0.1/fixtures/invalid-missing-hard-gate.json
• docs/schemas/routing-policy/v0.1/fixtures/missing-budget-metadata.json
• docs/schemas/routing-policy/v0.1/fixtures/missing-model-version.json
• docs/schemas/routing-policy/v0.1/fixtures/preview-model-risk.json
• docs/schemas/routing-policy/v0.1/fixtures/privacy-conflict.json
• docs/schemas/routing-policy/v0.1/fixtures/stale-evidence.json
• docs/schemas/routing-policy/v0.1/fixtures/valid-minimal-route.json
• fixtures/invalid/ultra_evaluation_receipt.v0.1.duplicate-lane.invalid.json
• fixtures/invalid/ultra_evaluation_receipt.v0.1.empirical-policy.invalid.json
• fixtures/invalid/ultra_evaluation_receipt.v0.1.evidence-class.invalid.json
• fixtures/invalid/ultra_evaluation_receipt.v0.1.lane-state.invalid.json
• fixtures/invalid/ultra_evaluation_receipt.v0.1.none-observed-details.invalid.json
• fixtures/invalid/ultra_evaluation_receipt.v0.1.numeric-overflow.invalid.json
• fixtures/invalid/ultra_evaluation_receipt.v0.1.route-mismatch.invalid.json
• fixtures/invalid/ultra_evaluation_receipt.v0.1.schema-identity.invalid.json
• fixtures/invalid/ultra_evaluation_receipt.v0.1.unavailable-value.invalid.json
• fixtures/valid/ultra_evaluation_receipt.v0.1.valid.json
• receipts/2026-07-11-gpt-5.6-sol-ultra-claim-verification.md
• receipts/2026-07-11-gpt-5.6-sol-ultra-content-review.md
• receipts/2026-07-11-gpt-5.6-sol-ultra-run.json
• schemas/ultra_evaluation_receipt.v0.1.schema.json
• tests/fixtures/gpt56-ultra-archive-sample.html
• tests/test_extract_gpt56_ultra_archive.py
• tests/test_ultra_benchmark_dataset.py
• tests/test_ultra_evaluation_packet.py
• tests/test_ultra_evaluation_receipt.py
• tools/extract_gpt56_ultra_archive.py
• tools/validate_ultra_benchmark_dataset.py
• tools/validate_ultra_evaluation_receipt.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch spec/whole-codebase-routing-policy-v0.1

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[hummbl-dev]

Peer Review — independent non-author review

This PR introduces a comprehensive model routing policy v0.1 with 39 files: a routing-policy spec, JSON schema for evaluation receipts, three Python validation/extraction tools (stdlib-only, no external dependencies), 8 valid/invalid fixtures, a benchmark dataset with cross-artifact validation, and 4 test modules. The code quality is generally high with thoughtful design choices (deterministic JSON serialization, hand-rolled JSON Schema subset validator, explicit unavailable-telemetry modeling, cross-artifact hash reconciliation).

Verdict: NEEDS_DISCUSSION

No blocking security issues found. The findings below are robustness, test-coverage, and operational gaps that should be discussed before merge, particularly the lack of CI for Python tests and unhandled exceptions in two of the three CLI tools.

Findings

HIGH

1. No CI workflow runs the Python tests
There is no .github/workflows/ directory on either the PR branch or main. The only check is CodeRabbit. The PR adds 4 test modules (~850 lines of tests) and 3 Python tools, but nothing runs them in CI. A regression could merge silently. Recommend adding a minimal pytest/unittest workflow before or shortly after merge.

2. Unhandled exceptions in benchmark validator and extract tool CLIs
tools/validate_ultra_evaluation_receipt.py catches OSError, ValueError on file load and returns clean exit codes (0/1/2). The other two tools do not:

• tools/validate_ultra_benchmark_dataset.py main() (line 450): validate_paths() raises DatasetValidationError which propagates as an unhandled traceback (exit code 1, but noisy).
• tools/extract_gpt56_ultra_archive.py main() (line 610): ValueError from extraction/validation propagates unhandled.

Recommend wrapping both main() bodies in try/except matching the receipt validator's pattern for consistent CLI behavior.

MEDIUM

3. Missing test coverage for several semantic validation checks
tools/validate_ultra_evaluation_receipt.py implements semantic checks that have no corresponding test:

• completed_at.value preceding started_at (lines 356–359)
• completed_at.value later than recorded_at (lines 360–367)
• outstanding final_state with available completed_at (lines 344–348)
• outcome_improvement benefited with empty observed_improvements (lines 374–378)
• outcome_improvement unavailable with non-empty observed_improvements (lines 379–384)
• outcome_improvement unavailable with empty unavailable_measurements (lines 385–389)
• aggregate_tokens.value being a float instead of integer (lines 288–291)
• Wrong telemetry unit (e.g., aggregate_cost with unit "tokens" instead of "USD", lines 285–286)

These are cross-field constraints the schema can't express, so they're only enforced in Python. Each should have at least one negative test.

4. Division-by-zero risk in benchmark validator derived-economics computation
tools/validate_ultra_benchmark_dataset.py _validate_comparisons() (lines 312–323) computes:

round(ultra["cost"] / single["cost"], 2)
round(ultra["output_tokens"] / single["output_tokens"], 2)
round((ultra["latency"] - single["latency"]) / single["latency"] * 100, 1)

If any denominator is 0.0, this raises ZeroDivisionError (not DatasetValidationError). The data is provider-reported and unlikely to be zero, but a malformed dataset would produce an unhandled crash instead of a clean validation error.

LOW

5. UnicodeDecodeError not caught in extract tool
tools/extract_gpt56_ultra_archive.py calls archive_bytes.decode("utf-8") in multiple places (lines 372, 558, 623) without catching UnicodeDecodeError. A non-UTF-8 archive would produce an unhandled crash. Consider errors="strict" with a try/except or an explicit encoding check.

6. Regex-based HTML parsing assumes single-line JSON objects
_ESCAPED_RECORD = re.compile(r'\{\\"eval\\":.*?\}') (line 68) uses .*? without re.DOTALL, so it only matches JSON objects on a single line. If the Wayback capture reformats the HTML with line breaks inside the embedded JSON, the regex silently fails. The extract_selected_records guard (raises if no records found) is a safety net, but the failure mode would be opaque. Document this assumption or add re.DOTALL with a more specific terminator.

7. Internal Slack URLs preserved in manifest
tools/extract_gpt56_ultra_archive.py _internal_slack_source_urls() (line 397) collects and preserves https://openai-corpws.slack.com/ URLs in the manifest's provenance_privacy section and limitations. This is a deliberate provenance/privacy tradeoff documented in the code, and the manifest notes credentials_or_tokens_found: False. Flagging for awareness — these internal workspace URLs are now in a public repo. Verify no sensitive thread IDs or tokens are embedded in the URL paths.

8. No maxLength on string fields in schema
schemas/ultra_evaluation_receipt.v0.1.schema.json defines nonEmptyString with minLength: 1 but no maxLength. Fields like receipt_id, task_id, lane_id, scope, rationale, etc. could be pathologically large. The hand-rolled validator also doesn't implement maxLength (it's not used in the schema). Consider adding maxLength bounds for operational safety.

INFO

9. Fixture tests check fragment presence, not exact error sets
test_all_invalid_fixtures_fail_with_clear_paths asserts that specific string fragments appear in the error output, but does not assert the complete error set. A fixture that triggers additional unexpected errors would pass silently. This is a reasonable fragility tradeoff, but consider at least asserting that the error count is within an expected range.

10. Hand-rolled JSON Schema validator supports a limited keyword subset
_validate_schema_subset implements type, const, enum, required, properties, additionalProperties, minItems, maxItems, uniqueItems, items, minLength, pattern, format: date-time, minimum, maximum, oneOf, $ref. It does NOT implement maxLength, exclusiveMinimum, exclusiveMaximum, prefixItems, contains, allOf, anyOf, not, or dependentRequired. This is fine for the current schema (which doesn't use these), but the limitation should be documented in the module docstring to prevent future schema authors from using unsupported keywords silently.

11. parse_float / parse_constant hooks not used in extract tool
tools/extract_gpt56_ultra_archive.py uses json.loads() without parse_constant or parse_float hooks (lines 117–118), so NaN/Infinity in the source HTML would be accepted as Python floats. The receipt validator correctly rejects these. This is an extraction tool (not a validator), so it's not a bug, but the extracted values could contain non-finite numbers that the downstream validator would catch. Consider adding the same hooks for defense-in-depth.

12. _agent_count in extract tool lacks error handling
tools/extract_gpt56_ultra_archive.py line 98: return int(record["eval_variant"].split(" ", 1)[0]) — no try/except, unlike the benchmark validator's version which catches KeyError, AttributeError, ValueError. In practice, _is_target_record filters records before _agent_count is called, but the companion evidence path (_latency_companion_evidence) calls _decode_flat_records directly (though it doesn't call _agent_count on those records).

What's done well

• No external dependencies: All three tools are stdlib-only Python. The hand-rolled JSON Schema subset validator is a deliberate, well-documented choice.
• Deterministic JSON serialization: canonical_json_bytes with sort_keys=True and fixed indentation enables reproducible SHA-256 hashing across artifacts.
• Cross-artifact reconciliation: The benchmark validator checks hash consistency across raw records, manifest, and dataset — tampering with any one artifact is detected.
• Explicit unavailable modeling: The schema's oneOf pattern for available/unavailable telemetry is excellent. It forces honesty about what was measured vs. what was not.
• additionalProperties: false everywhere: Every object in the schema has this set. No silent extra fields allowed.
• Non-finite number rejection: The receipt validator rejects NaN, Infinity, and overflowed floats at both parse time (parse_constant, parse_float hooks) and validation time (math.isfinite).
• Fixture completeness check: test_all_invalid_fixtures_fail_with_clear_paths asserts set(expectations) == fixture_names, preventing stale or orphaned fixtures.
• Provenance chain: The extract tool preserves Wayback capture timestamps, content hashes, and source URLs throughout the pipeline.
• policy_status: "provisional" as a const: Prevents overclaiming empirical validation in v0.1. Smart governance-by-schema.
• Score reconciliation: The extract tool explicitly compares archived max scores against current headline scores and flags whether they agree after rounding.

CI Status

Workflow	Status
CodeRabbit	✅ pass

No GitHub Actions workflows exist in the repository. The Python test suite is not executed in CI.

[hummbl-dev]

Initial review pass — pinned head e312e063f2b9 against main.

Evidence:

• Scope: 39 files, +8464/-0. Notable paths: .gitignore, README.md, data/gpt-5.6-sol-ultra/archive_manifest.json, data/gpt-5.6-sol-ultra/archived_chart_records.json, data/gpt-5.6-sol-ultra/benchmark_dataset.v0.1.json, +34 more
• Mergeability: CONFLICTING
• Head checks: PASS — 2 reported checks have no failing or pending result; merge conflict reported

Findings:

• P2: protected/governance-sensitive paths are in scope: docs/schemas/routing-policy/v0.1/fixtures/cheaper-route-rejected.json, docs/schemas/routing-policy/v0.1/fixtures/invalid-missing-hard-gate.json, docs/schemas/routing-policy/v0.1/fixtures/missing-budget-metadata.json.
• P2: large review surface requires decomposition or an explicit large-change rationale.
• P3: 82 added normative MUST/SHALL/REQUIRED line(s) require authority-source validation.

Next review focus: Check routing defaults, fallback/fail-closed behavior, schema compatibility, and negative-path tests before treating the policy as executable.

Gate: HOLD. This is an evidence-backed triage comment, not final merge approval; promotion still requires content validation against this exact head and the repository's review rules.