Skip to content

docs: define colibri as an experimental inference engine - #674

Merged
JustVugg merged 6 commits into
JustVugg:devfrom
ZacharyZcR:docs/experimental-inference-engine
Jul 28, 2026
Merged

docs: define colibri as an experimental inference engine#674
JustVugg merged 6 commits into
JustVugg:devfrom
ZacharyZcR:docs/experimental-inference-engine

Conversation

@ZacharyZcR

@ZacharyZcR ZacharyZcR commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

  • define colibri as an experimental inference engine and research platform
  • state the objective: optimize the complete inference path to reduce hardware dependency and inference cost
  • distinguish measured findings, conditional wins, correctness properties, and open hypotheses instead of presenting every technique as guaranteed acceleration
  • document the core techniques: hierarchical placement, learned weight staging, I/O overlap, heterogeneous execution, compressed state, and measured speculation
  • publish an open research matrix covering routing history, multi-SSD I/O, automatic hardware planning, representation/compression, speculation, and CPU/GPU overlap
  • give contributors a reproducible experiment protocol and explicitly welcome controlled negative results
  • update the roadmap and correct the absolute GLM-5.2 int4 think mode: stochastic reasoning loops / EOS starvation on stock v1.0.0; #307's TEMP=0.9 NUCLEUS=0.95 stopgap makes it severeΒ #455 wording
  • align the English, Simplified Chinese, Traditional Chinese, and Italian READMEs

Scope

README positioning and research documentation only. No runtime, build, API, or configuration behavior changes.

Validation

  • git diff --check
  • verified the positioning, measured-findings, and open-experiments sections in all four language files
  • reviewed modified Markdown links, tables, and structure

@ZacharyZcR

Copy link
Copy Markdown
Contributor Author

Added the Discord community link to the top navigation and contribution section in all four README languages: https://discord.gg/fpQxKnRb

@JustVugg
JustVugg merged commit b95fc97 into JustVugg:dev Jul 28, 2026
10 checks passed
@monotophic

Copy link
Copy Markdown
Contributor

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic (We/our = the joint analysis; me/my = @monotophic's own hardware and commitments).

Glad to see this merged β€” the measured / conditional / correctness / hypothesis taxonomy is the repo formalizing what its best threads already do, and we want to put two concrete datapoints behind it plus a flag of interest on one research-matrix row.

The taxonomy does real work β€” two cases from our own record

A conditional win, correctly filed: COLI_METAL_RESSET=1 on my M5 Max took decode from 0.89 to a flat 2.28–2.39 tok/s (measured both ways, #379) β€” a >2.5Γ— win that is platform-conditional by mechanism, not a guaranteed acceleration. Under the old framing that number invites over-claiming; under this taxonomy it files cleanly as a conditional win with a stated condition. That's the framework earning its keep.

A controlled negative result, welcomed: we built and instrumented three generations of prefetch on the storage tier and refuted all three with the instrument (full record on #387; occupancy-vs-saturation trap documented on #537). Under "every technique is an acceleration," that work looks like failure; under this document it's exactly the contribution the protocol asks for. A repo that welcomes scored refutations gets honest numbers.

The "automatic hardware planning" row: we're active in this lane

The research matrix names automatic hardware planning as an open direction. We've been building the measurement-first end of it: a digital twin of Colibri decode β€” a serial-sum model (disk + expert-matmul + attention + lm_head + other) whose every parameter carries measured provenance and an uncertainty class, validated by hindcast against a 34-run corpus on my M5 Max. Components are atomic and substitutable, so the same model prices a candidate config β€” or candidate hardware β€” before anything runs.

What makes planning tractable is that some parameters turn out to be class-constants rather than machine-specifics. Two measured examples from my fleet: sustained STREAM lands at 44% of the spec-sheet bandwidth on two different SoC families (114 of 256 GB/s theoretical on Strix Halo, 120 of 273 on GB10 β€” best-of-10 triad, working set ≫ LLC), and the ~74% expert cache-hit ceiling on my M5 Max behaves as a routing-concentration constant of model+workload rather than a cache-size effect (#537). Constants like these are what let a planner generalize beyond the box it was calibrated on.

This lane also composes with #673: coli tune searches the config space empirically on one machine; a twin predicts across machines; profiles and predictions cross-validate each other. We've offered a twin-pair profile-transfer experiment on that thread that tests exactly this seam.

Flagging interest, not claiming the row: our methodology and fleet dataset (three machine classes measured across five boxes, more standing up) are available to whoever works this direction, and we expect to bring a concrete proposal here once our current transport-layer work (#671's follow-on rungs) clears the queue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants