Evaluate Colibrì as a governed Anvil local-inference route

[hummbl-dev]

Parent evidence

• research-source-packets#23 — bounded public source packet for JustVugg/colibri
• hummbl-research#56 — standing External Claims Ledger
• hummbl-dev#155 — benchmark/evaluation coordination surface
• model-routing-as-code#19 — governed reasoning-budget policy
• model-routing-as-code#21 — total-cost and rebound accounting

Decision posture

INTAKE_APPROVED / EXECUTION_GATED

This issue authorizes a governed evaluation design and low-cost preflight work. It does not authorize downloading the approximately 370 GB converted model, converting the approximately 756 GB FP8 checkpoint, exposing an API beyond loopback, or promoting Colibrì into a production route.

Objective

Determine whether JustVugg/colibri should be classified as:

1. RESEARCH_ONLY
2. BOUNDED_LOCAL_ROUTE
3. PROMOTION_CANDIDATE_WITH_SCOPE
4. REJECTED_FOR_CURRENT_FLEET

The candidate route is unusual: a very large sparse MoE model is backed by local NVMe, with dense tensors resident in RAM, a learned/pinned expert cache, and an optional RTX 3080 Ti VRAM hot tier. Evaluation must therefore govern storage, RAM, CPU, VRAM, quality, latency, persistence, and operational burden together rather than treating model choice as the only route variable.

Target host

Primary feasibility target: Anvil, the Windows desktop with RTX 3080 Ti and WSL2/Linux capability.

The evaluator must record current hardware and storage facts rather than relying on remembered inventory. No claim about free space, RAM, filesystem placement, thermals, or achievable throughput is valid until measured on the target host.

Phase 0 — source and trust-boundary audit

Complete before local installation:

• inspect the pinned upstream commit and any newer delta
• map the C engine, CLI subprocess protocol, converter, resource planner, doctor, persistence files, API gateway, web client, and desktop shell
• identify model-weight, converted-artifact, code-license, and telemetry boundaries
• determine whether standard checks execute all present tests
• record missing or stale documentation, including any referenced-but-absent plan documents
• identify any path where malformed model metadata could affect allocation, file access, persistence, or process execution

Phase 1 — converter audit

Audit convert_fp8_to_int4.py without downloading the full model:

• FP8 dequantization semantics
• block-scale handling
• int4/int8 packing and per-row scales
• tensor-name and shape preservation
• MTP-head int8 override
• shard-at-a-time download/delete behavior
• resumability and partial-state detection
• integrity checks and failure recovery
• free-space projection
• authoritative source and artifact hashes

Outcome: CONVERTER_ACCEPTABLE_FOR_BOUNDED_TRIAL, REQUIRES_PATCH, or REJECT.

Phase 2 — Anvil preflight without full model download

Produce a measured host receipt covering:

• OS, WSL2/native route, kernel, compiler, OpenMP, Python, and optional CUDA toolchain
• CPU model, core/thread count, instruction-set support
• installed and available RAM
• RTX 3080 Ti detected VRAM and current free VRAM
• local NVMe device, filesystem, mount path, free capacity, and whether the path avoids network/9p constraints
• representative buffered and direct/random read throughput using a bounded synthetic or existing local test file
• sustained-read temperature observations if a bounded thermal test is run
• make check and CUDA fixture status where supported
• tiny-oracle or deterministic fixture validation
• coli plan / coli doctor behavior against a safe fixture or header-only test model

Do not represent synthetic I/O results as full-checkpoint inference throughput.

Phase 3 — explicit download/conversion gate

Before acquiring hundreds of gigabytes, require a human approval comment containing:

• selected artifact source and exact version/hash
• license disposition for weights and converted artifacts
• required download bytes and final disk footprint
• free-space margin after installation
• expected download/conversion duration
• expected cold and warm throughput range with assumptions
• rollback and deletion plan
• security posture
• unresolved quality risks

No silent model download or automatic conversion is permitted.

Phase 4 — bounded full-model trial, only after approval

If Phase 3 is approved:

1. verify artifact integrity
2. run coli doctor
3. preserve the exact coli plan JSON
4. start loopback-only with an API key
5. run deterministic smoke prompts
6. measure cold, warming, and learned-cache behavior separately
7. record token latency, tokens/second, cache-hit rate, RSS, disk throughput, CPU utilization, GPU utilization, thermals, and errors
8. run bounded quality subsets from HellaSwag, ARC Challenge, and MMLU
9. compare against at least one existing local route and one managed frontier route on the same task packet
10. stop and roll back on unsafe memory pressure, swap activity, persistent thermal limits, corrupted state, or unacceptable quality

Required receipt

Define or reuse a compatible receipt under the authority of this repository. Proposed semantic shape:

schema: colibri_route_evaluation.v0.1
candidate:
  repository:
  commit:
  model_artifact:
  artifact_hashes: []
host:
  identity:
  os_route:
  cpu:
  ram:
  storage:
  gpu:
preflight:
  source_audit:
  converter_audit:
  build_checks:
  fixture_checks:
  doctor_report:
  resource_plan:
execution:
  cold_runs: []
  warm_runs: []
  learned_cache_runs: []
quality:
  benchmark_subsets: []
  comparison_routes: []
security:
  bind_address:
  api_key_required:
  persistence_files: []
  network_exposure:
cost:
  download_bytes:
  disk_footprint:
  elapsed_operator_time:
  machine_time:
  energy_or_thermal_notes:
findings: []
blockers: []
disposition:

Do not canonize this schema name until repository review confirms whether an existing route-trial or execution-receipt contract should be extended instead.

Promotion criteria

A scoped promotion requires all of the following:

• Source packet is complete enough to support the execution claims.
• Code and artifact licenses are acceptable for the intended use.
• Standard and fixture checks pass or deviations are explicitly accepted.
• Full-model output quality is measured rather than inferred from tiny-model equivalence.
• Useful-response latency is acceptable for at least one named workload.
• Total operational cost is competitive for that workload after storage, machine time, maintenance, and human review are included.
• No silent cloud fallback or undeclared telemetry exists.
• API remains loopback-only unless a separately governed mesh exposure is approved.
• Persistence and reset semantics are understood and tested.
• The route has explicit workload exclusions and escalation behavior.
• Promotion is narrower than “744B local model works” and names the exact task class it improves.

Rejection criteria

Reject or defer when any of the following holds:

• model quality is materially degraded or remains unmeasured
• useful-response latency is operationally unacceptable
• storage or thermal burden is disproportionate
• converter integrity cannot be established
• license terms are unclear or incompatible
• the route creates fragile maintenance liability without unique value
• a smaller local model or managed route produces better accepted outcomes at lower total cost

Non-goals

• Replacing the existing local model fleet
• Making Colibrì the default route
• Publicly endorsing upstream benchmark claims
• Exposing the service directly to the Internet
• Contributing upstream without a separate bounded approval
• Treating the RTX 3080 Ti as sufficient evidence that the route will be fast
• Downloading the model during static audit or preflight

Definition of done

Close when a reproducible evaluation receipt and decision are preserved with one of the four explicit dispositions. A source audit alone is insufficient for promotion; a failed or deferred Phase 3 gate is a valid completed outcome when the reasoning is documented.