Whole-codebase and corpus model routing policy v0.1

[hummbl-dev]

Parent benchmark: hummbl-dev/hummbl-dev#155
Harness: https://github.com/hummbl-dev/autoresearch-pipeline/issues/30
Corpus pack: hummbl-dev/hummbl-bibliography#78
Master index: hummbl-dev/hummbl-dev#153

Status

CANDIDATE ROUTING POLICY — EVIDENCE REQUIRED — NO DEFAULT ROUTE CHANGE AUTHORIZED

Objective

Define a machine-readable routing policy for whole-repository understanding and massive-source intelligence workloads, grounded in HUMMBL benchmark evidence rather than advertised context length, vendor reputation, or one aggregate leaderboard.

Governing doctrine

Cheapest correct model wins only after correctness, safety, privacy, authority, latency, reliability, reproducibility, and governance gates pass.

A larger context window does not itself justify routing.

Workload classes

At minimum:

REPO_INVENTORY
REPO_ARCHITECTURE_RECONSTRUCTION
CROSS_FILE_TRACE
CLAIM_AND_REFERENCE_AUDIT
ISSUE_ROOT_CAUSE
BOUNDED_PATCH_IMPLEMENTATION
INDEPENDENT_CODE_REVIEW
BULK_SOURCE_INGESTION
MULTI_DOCUMENT_SYNTHESIS
CONTRADICTION_AND_VERSION_ANALYSIS
OCR_HISTORICAL_CORPUS
MULTIMODAL_SOURCE_PACKET
FINAL_SCHOLARLY_CLAIM_REVIEW

Candidate route dimensions

Each route must consider:

source size and estimated tokens
source type and modality
private/public classification
required tool surface
single-context feasibility
retrieval/memory requirement
reasoning depth
citation precision requirement
implementation authority
latency target
cost ceiling
reliability target
independent-review requirement
provider and regional constraints
model availability / preview status
benchmark evidence date

Route dispositions

Models/scaffolds may receive one or more scoped dispositions:

REJECTED
DISCOVERY_ONLY
BULK_EXTRACTOR
BOUNDED_READER
PRIMARY_READER_WITH_REVIEW
IMPLEMENTER_WITH_GATES
INDEPENDENT_REVIEWER
FINAL_SYNTHESIS_WITH_HUMAN_REVIEW
PROMOTED_WITH_SCOPE

No model should be labeled universally best.

Initial candidate lanes

Subject to exact model/version verification at run time:

• GPT-5.6 Sol / Codex-compatible lane;
• Claude Fable 5;
• Claude Opus 4.8;
• Claude Sonnet 5;
• Gemini 3.1 Pro Preview custom-tools lane;
• Gemini 3.5 Flash;
• Grok 4.5;
• admitted open/local models with declared serving and quantization configuration.

Preview models must carry stability and deprecation risk. Aliases must carry resolution risk unless pinned or receipted.

Required policy behavior

1. Select the lowest-cost route meeting all hard gates.
2. Prefer tool navigation or structured memory when source size exceeds effective context or globally distributed reasoning is required.
3. Require a different-provider reviewer for high-stakes architecture, scientific, legal, health, security, or publication claims where feasible.
4. Keep private repositories and sensitive corpora within approved provider/environment boundaries.
5. Escalate when benchmark coverage is missing, stale, or materially mismatched to the workload.
6. Fail closed on missing model version, budget, privacy, or authority metadata.
7. Record why a cheaper route was rejected.
8. Include reevaluation and expiry dates for every promoted route.

Required artifacts

1. routing-policy specification;
2. machine-readable workload schema;
3. model/scaffold capability registry;
4. benchmark-result import schema;
5. route decision schema and receipt;
6. hard-gate and soft-ranking logic;
7. valid, invalid, stale-evidence, and privacy-conflict fixtures;
8. deterministic evaluator or validator;
9. example route cards for each workload class;
10. migration path into Founder Mode / HUMMBL runtime routing only after approval.

Example route structure

workload: REPO_ARCHITECTURE_RECONSTRUCTION
source_classification: private
source_estimated_tokens: 640000
required_capabilities:
  - file_search
  - cross_file_reasoning
  - exact_citations
hard_gates:
  privacy: pass
  authority: pass
  benchmark_freshness: pass
  minimum_accuracy: pass
candidates:
  - model: claude-fable-5
    scaffold: TOOL_NAVIGATED
  - model: gpt-5.6-sol
    scaffold: MEMORY_AUGMENTED
selected_route: null
review_route: null
expires_at: null

Acceptance criteria

• Policy imports benchmark evidence rather than duplicating scores manually.
• Effective context and scaffold condition are separate fields from advertised context.
• Privacy, authority, and cost are hard gates where appropriate.
• Every route has evidence date, scope, expiry, and fallback.
• Missing evidence yields escalation, not fabricated ranking.
• A cheaper model can win when it meets the same correctness gates.
• High-stakes final claims require human and independent review.
• No production/default model route changes occur without Reuben approval and transition receipt.

Immediate next action

Return the schema design, benchmark-ingestion contract, and five example route decisions using placeholder outcomes. Do not encode unverified model rankings.

[hummbl-dev]

Company-level coordination and implementation plan created: hummbl-dev/hummbl-dev#156

This issue remains the canonical model-routing policy implementation lane. The plan adds sequencing for scope closure, schemas/validator, HUMMBL task classes, benchmark imports, cost-per-accepted-outcome receipts, observability, adapters, and later product consumption. No production route changes or greenfield repo are authorized.

[hummbl-dev]

GPT-5.6 GA routing signal — no duplicate issue

OpenAI’s official 2026-07-09 GPT-5.6 announcement introduces the Sol, Terra, and Luna tiers; max and ultra effort/execution modes; Programmatic Tool Calling; and a multi-agent beta in the Responses API:

• https://openai.com/index/gpt-5-6/

This issue remains the canonical policy home for evaluating those model/tier candidates under cheapest-correct-model doctrine. Do not create a separate GPT-5.6 routing policy.

Required downstream distinction:

• This repo: route eligibility, model/tier/effort representation, cost/capability/privacy/safety/reliability gates, fallback, and promotion evidence.
• agent-runtime-governance#12: generated orchestration-program and parallel-subagent authority, evidence, sandbox, cancellation, fan-out, and receipt controls.
• autoresearch-pipeline#31: comparable cost-per-accepted-outcome runs.
• hummbl-dev#158: governed integration slice.

Treat OpenAI benchmark and efficiency claims as dated vendor evidence until independently reproduced on HUMMBL tasks. ultra must not be treated as a generally preferred route merely because it uses parallel agents; require task fit, accepted outcome, resource cost, latency, and governance compliance.

[hummbl-dev]

Operator-approved benchmark-import delta — preserve execution identity and accepted-outcome economics

Reuben approved the 2026-07-11 recommendation to make tier, effort, topology, scaffold, and accepted-outcome cost first-class routing evidence.

This issue remains the canonical routing-policy implementation lane. No new routing-policy issue is needed.

Required import-contract additions

The benchmark import schema must preserve, as separate fields:

requested_model_id: string
resolved_model_id: string|null
model_snapshot_posture: pinned|dated|moving_alias|unknown
model_tier: string|null
effort_mode: string|null
execution_topology:
  mode: single_agent|multi_agent
  subagent_count: integer
  orchestration_mode: none|provider_native|harness_managed|other
tool_calling_mode: none|native|programmatic|harness_adapter
context_strategy: single_context|tool_navigated|memory_augmented|multi_agent
scaffold_id: string
scaffold_version: string
price_snapshot_ref: string|null
outcome_disposition: accepted|accepted_with_corrections|partial|rejected|not_reviewed
total_cost_usd: number
human_correction_minutes: number
human_review_minutes: number
rollback_or_remediation_minutes: number
cost_per_accepted_outcome_usd: number|null

Do not encode model + tier + effort + topology as one opaque route label. In particular, do not treat a provider-native multi-agent mode such as ultra as equivalent to a distinct base model or automatically superior route.

Required policy behavior

• Fail closed when exact or resolved model identity, alias posture, tier, effort, topology, scaffold, price posture, privacy, authority, or benchmark date is missing.
• Use cost_per_accepted_outcome_usd only when the benchmark outcome is accepted; rejected/failed runs must remain visible with null accepted-outcome economics.
• Preserve the full cost stack: model, tools, reviewer, correction, and remediation.
• Require explicit reason when a cheaper candidate is rejected.
• Prevent throughput proxies such as PR count, patch count, or raw completion rate from satisfying correctness gates.
• Keep provider-reported benchmark results distinct from independently reproduced HUMMBL evidence.
• Require evidence expiry and reevaluation dates for any promoted route.
• Import benchmark evidence from autoresearch-pipeline#31; do not manually duplicate scores in policy files.

Placeholder route-card fixtures

Add or update placeholder-only examples that demonstrate:

1. same base model, different effort modes;
2. same model, single-agent versus multi-agent topology;
3. same model and topology, different scaffold/context strategy;
4. cheaper route accepted after all hard gates;
5. expensive route rejected because correction/remediation destroys accepted-outcome economics;
6. missing-evidence route that escalates rather than fabricating a ranking.

No placeholder may encode an unverified winner.

Acceptance criteria for this delta

• Benchmark import contract accepts the schema produced by autoresearch-pipeline#31.
• Tier, effort, topology, tool-calling mode, and scaffold remain independently queryable.
• Rejected and failed runs remain available to the route evaluator.
• Invalid fixtures reject conflated identity and invalid cost-per-accepted-outcome values.
• Route receipts explain why cheaper candidates failed hard gates.
• No production/default route change occurs without separate operator approval and transition receipt.

This approval authorizes schema, validator, fixtures, placeholder route cards, and PR preparation only.

[hummbl-dev]

Linked accounting dependency — 2026-07-11

Rebound Accounting v0.1 is now tracked in #21: #21

The whole-codebase and corpus routing policy should import or reference the resulting receipt fields rather than duplicating them. Route comparison should include induced volume, retries, required review, downstream burden, meaningful stratification, and total cost per accepted governed outcome. No default route change is authorized by this linkage.

[hummbl-dev]

Operator-approved METR delta — GPT-5.6 Governed Autonomy Profile v0.1

Reuben approved this policy direction on 2026-07-11 after review of METR’s GPT-5.6 Sol predeployment evaluation:

• https://metr.org/blog/2026-06-26-gpt-5-6-sol/
• https://deploymentsafety.openai.com/gpt-5-6

This issue remains the canonical routing-policy lane. Do not create a separate GPT-5.6 governance program.

Governing principle

Maximize cognitive freedom inside the task; minimize ambient authority outside the task.

The approved posture is more autonomy within stronger externally enforced boundaries, not global capability suppression.

Required routing invariants

model capability increase does not imply authority increase
subagent count increase does not imply authority increase
task persistence increase does not imply scope increase
apparent success does not imply valid completion
correct artifact + invalid method does not equal accepted outcome

Every consequential route must resolve distinct objects for:

1. task/objective;
2. runtime;
3. provider/model/tier/effort/topology/scaffold;
4. authority envelope;
5. acceptance contract.

Privacy, runtime, authority, and consequence gates occur before model/cost ranking.

Required schema delta

Add independently queryable fields equivalent to:

authority_profile_id: string
work_permit_hash: string
protected_surfaces: []
authorization_default: deny_protected

integrity:
  disposition: valid|boundary_violation|evaluation_compromised|other
  evaluator_access_attempted: boolean
  credential_scope_violation: boolean
  authority_escalation_attempted: boolean
  evidence_tamper_attempted: boolean
  prohibited_strategy_attempted: boolean
  completion_claim_verified: boolean
  quarantine_reason: string|null

monitoring:
  tool_trace_complete: boolean
  filesystem_trace_complete: boolean
  network_trace_complete: boolean
  subagent_graph_complete: boolean
  generated_program_ref: string|null
  reasoning_telemetry_posture: unavailable|summary|full_authorized
  blind_validation_completed: boolean

reversibility:
  pre_state_snapshot_ref: string|null
  rollback_tested: boolean
  irreversible_actions: []

acceptance:
  independent_review_required: boolean
  independent_review_completed: boolean
  human_gate_required: boolean
  human_gate_completed: boolean

These extend, and must not conflate, the already approved model identity, tier, effort, topology, scaffold, outcome, and accepted-outcome economics fields.

Policy behavior

• Unspecified authority for protected surfaces means denied.
• Model capability may increase task difficulty, but may not silently increase tool, credential, network, mutation, publication, payment, identity, security, or production authority.
• Integrity and task outcome are orthogonal dispositions.
• A technically correct artifact obtained through an invalid method is non-promotable.
• ultra is parallel cognition, not parallel authority.
• Same-model parallel agents do not satisfy independent-review requirements.
• Raw chain-of-thought availability is optional telemetry, not a governance dependency.
• Missing work permit, authority profile, integrity disposition, evaluator-isolation posture, rollback posture, or required review must fail closed or escalate.

Promotion rule

PROMOTABLE =
  accepted outcome
  AND valid integrity disposition
  AND complete required evidence
  AND no protected-surface violation
  AND successful blind validation
  AND required independent review
  AND demonstrated rollback

Acceptance criteria for this delta

• Route schema separates intelligence, runtime, authority, and acceptance.
• Integrity disposition is independently represented from task outcome.
• Protected-surface authority defaults to deny.
• Parallel topology cannot expand authority or satisfy independent review by itself.
• Blind-validation and rollback postures are first-class route evidence.
• Invalid method prevents promotion even when the artifact is correct.
• No production/default route change occurs without a separate transition receipt and explicit operator approval.

This approval authorizes policy/schema design, fixtures, validation, and draft-PR preparation only.