feat: add measured machine autotuning

[ZacharyZcR]

Summary

• add coli tune, a bounded topology-aware sweep of quality-preserving execution knobs
• generate one calibration continuation and use fixed-token replay for every candidate
• persist profiles by CPU/GPU/model/engine fingerprint and apply them through --auto-tier
• preserve explicit environment overrides and provide --no-tune-profile
• reject candidates that regress expert hit rate or p99 latency, and require a reverse-order confirmation run before activation
• ship the new module in release archives, make install, Nix, and the slim container

The sweep intentionally excludes weights, quantization, router policy, TOPK, TOPP, sampling, and other quality-affecting settings.

Validation

• make -C c test-c: passed (26 C test binaries)
• make -C c test-python: passed (227 tests, 13 skipped)
• make -C c colibri -j2: passed
• fake-engine integration covers fixed replay, profile persistence, explicit override precedence, winner selection, and warmup-drift rejection
• CUDA build passed on the dual-socket 6x RTX 5090 host

Full-model safety-gate check

Ran the command against /data/models/GLM-5.2-colibri-int4 on the 6x RTX 5090 host with 8 fixed replay tokens.

The exploratory sweep initially made COLI_CUDA_ASYNC=0 look faster (1.82 vs 1.74 tok/s, identical 97.3% hit rate). The required reverse-order confirmation measured the candidate at 1.39 tok/s and the later baseline at 1.51 tok/s, so the profile was correctly rejected and no override became active. This is the startup/cache drift case the confirmation gate is designed to catch.

[JustVugg]

Heads-up: this now conflicts on dev after #649/#674 landed (shared c/coli / README). Could you rebase / Update branch? CI was green, just needs the merge with current dev. Cleared to merge once it's clean.

[monotophic]

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic (We/our = the joint analysis; me/my = @monotophic's own hardware and commitments).

Strong design — and the safety-gate anecdote in the PR body is the part we want to underline, because we run the same discipline from the other side and it catches exactly this failure mode.

1. The reverse-order confirmation gate is the right call (independent corroboration)

Our measurement protocol on the M5 Max campaign converged on the same rule you've built into coli tune: candidate wins from a sweep are drift suspects until re-measured under reversed ordering (we run interleaved same-config controls as drift canaries; escalation only on >3% anomaly). Warm-cache/startup drift crowning a false winner — your COLI_CUDA_ASYNC=0 case — is the textbook instance; we've discarded sweep "winners" for the same reason. A tuner that requires the confirmation run before activating a profile is, in our experience, the difference between a tuner and a superstition generator. Same for the quality-knob exclusion (no TOPK/TOPP/sampling): we treat identical-output as a hard contract and sweep only execution knobs, so the boundary you drew matches what we'd draw.

2. An experiment offer: profile transfer on a verified identical-twin pair

Your fingerprint design (CPU/GPU/model/engine) implicitly asks: are tuned profiles machine-specific, or do they transfer within a hardware class? We can answer that with a controlled experiment: my two GMKtec Strix Halo boxes are a verified twin pair — measured STREAM triad peaks of 114.1 vs 112.7 GB/s (1.2% symmetry, best-of-10, 4.8 GB working set), symmetric fio storage baselines, identical CPUs/RAM.

Protocol we'd run, once the branch is rebased and runnable on our side:

1. Run coli tune independently on both boxes, same model, same settings.
2. Diff the persisted profiles. Twin symmetry gives the null hypothesis: if profiles differ materially between 1.2%-identical machines, the sweep is fitting noise; if they agree, that's direct evidence a class-level fingerprint tier is viable.
3. Optional third arm: apply box A's profile on box B and measure the delta vs B's own — the transfer-loss number itself.

3. Why we care: the predictive complement

We maintain a digital twin of Colibri decode on our hardware — a serial-sum model (disk + expert-matmul + attention + lm_head + other) with every parameter carrying measured provenance and an uncertainty class, validated by hindcasting against a 34-run measurement corpus. Its components are substitutable, so it prices candidate configs and candidate hardware before anything runs. coli tune is the empirical half of that same space — it searches what the twin predicts. The convergence of interest here is practical: our cross-box data (two SoC families measured so far) shows some parameters are class-constants — e.g. STREAM efficiency lands at 44% of the spec-sheet number on both families (114/256 GB/s Strix Halo, 120/273 GB/s GB10) — which suggests tuned profiles may carry further than a strict per-machine fingerprint assumes. The transfer experiment in §2 is the cheap, decisive test.

Happy to coordinate timing once the rebase lands — the twin pair is otherwise idle for this.