Inkling (Thinking Machines 975B MoE): new engine, o200k tokenizer, bf16->int4 converter

[nbeerbower]

What

Support for Thinking Machines Inkling (975B total / 41B active MoE, Apache 2.0) — a second full architecture for colibri, following the olmoe.c standalone-engine precedent:

• c/inkling.c — Stage-A engine: GQA with interleaved sliding-window (512, 16 KV heads) / global (8 KV heads) attention at 5:1, learned relative-position bias banks (no RoPE), log-length tau scaling past 128k, four depthwise causal short-convs per layer (fp32, residual inside, cached conv state for decode), dense + MoE layers, sigmoid router with loss-free bias and jointly-normalized routed+shared weights, mup logit scaling. Routed experts stream per-expert from the fused 3D tensors (LRU cache, colibri int4/int8 container or f32). -p "prompt" streaming greedy mode; ref-oracle harness mode.
• c/tok.h — o200k pre-tokenizer support (case-aware Split regex), auto-detected from tokenizer.json so the GLM/cl100k path is byte-identical to before. New tok_unicode_o200k.h range tables.
• c/tools/convert_inkling_int4.py — bf16 checkpoint → colibri container (int4 experts + .qs per-row scales, bf16 residents, f32 norms/convs/router). Threaded quantization, shard-resumable, --watch converts while the download is still running. Note for reviewers: TML's fused w13 tensors store gate/up rows interleaved; the converter de-interleaves exactly per transformers' conversion mapping.
• c/tools/make_tiny_inkling.py — tiny random-init oracle generator (HF transformers ≥ 5.14).
• Zen-4-friendly kernels in inkling.c: bf16 residents dotted via AVX512-BF16 vdpbf16ps (halves resident RAM vs f32 — required: f32 residents would be ~172 GB), int8 expert dot via AVX-512 VNNI (IDOT family; IDOT=0 scalar path preserved). Scalar fallbacks keep the dependency-free default CPU path intact.

Validation

• Token-exact oracle (the colibri standard): tiny random-init Inkling via HF transformers — 36/36 teacher-forced argmax, 24/24 greedy, in f32, int4-container+VNNI, IDOT=0 scalar, and bf16-resident modes.

• Tokenizer: 357/357 strings encode identically to HF tokenizers (case transitions, contractions, CJK, combining marks, emoji+modifiers, 300 mixed-charset fuzz cases).

• Converter e2e (--selftest-e2e): fabricates a TML-named checkpoint from the tiny oracle (inverse mapping, re-fused, re-interleaved), converts back, engine reproduces the oracle exactly in both passthrough and int4 modes.

• Real checkpoint (975B, int4 experts + bf16 residents, 469 GiB snapshot):

  $ SNAP=~/Models/inkling_i4 ./inkling -p "The capital of France is" -n 24
  The capital of France is Paris.
  The capital of the UK is London.
  The capital of Russia is Moscow.
  The capital of Italy
  

  Converted weights published at nbeerbower/Inkling-colibri-int4.

Perf status (honest numbers)

Ryzen 9 7900 (24t), 187 GB RAM, NVMe snapshot, commit b60f881, make CUDA=0, single run, cold cache: RSS 90.5 GB, prefill 150 s (5 tok), decode 0.06 tok/s at cache 16/layer — expert streaming bound, as expected for Stage A with a plain LRU. The architecture is unusually kind to long context (55/66 layers have a fixed 512-token window; 1M-token KV ≈ 46 GB bf16, RAM-resident), so the usual roadmap applies and should pay off more than it did for GLM: hot-store/pinning + prefetch (tier.h), then a GQA CUDA tier — the attention here needs no MLA-style custom kernels.

make check passes (62 tests), -Wall -Wextra clean, GLM oracle path untouched (tok.h dispatch is additive).

Happy to split this into smaller PRs if you prefer (tokenizer / engine / converter).

🤖 Generated with Claude Code

https://claude.ai/code/session_016KLDCYJSyHxd39ChTr4T27

[nbeerbower]

Pushed a Stage-B performance commit (3ef6775) — same architecture, the expert path grew the glm.c playbook:

• cache slots keep container rows packed (int4 stays 4-bit in RAM, ~28 MB/expert → double the cache per GB); new matmul_q4 unpacks nibbles in-register, numerically identical to the int8 path
• moe() routes everything first, then fills all missing experts in one parallel burst (prefill batches the whole sequence's misses, so the NVMe finally sees queue depth), then computes
• usage-history pinning (SNAP/.coli_usage, top PIN_N/layer non-evictable) + RAM-sized auto cap

Measured on the 975B checkpoint (Ryzen 9 7900, NVMe, cap auto=64/layer, 1024 pinned):

	Stage A	Stage B cold	Stage B pinned+warm
prefill (5 tok)	150.2 s	31.5 s	21.7 s
decode	0.06 tok/s	0.21 tok/s	0.25 tok/s
cache hit	~0%	72.0%	81.5%

All four oracle modes (f32, int4-container VNNI, IDOT=0 scalar, bf16 residents) remain token-exact, make check 68/68, 0 warnings.

🤖 Generated with Claude Code

https://claude.ai/code/session_016KLDCYJSyHxd39ChTr4T27

[nbeerbower]

Final push for this PR — it now includes the CUDA resident tier and cache warming, bringing inkling to feature parity with what glm already has (no concepts new to colibri):

• CUDA tier (backend_cuda_ink.cu, CUDA=1): bf16 residents (~36 GB) live in VRAM on a warp-per-row kernel with the same numeric contract as the CPU vdpbf16ps path; freed RAM goes to the expert cache. Pure-CPU default build unchanged and dependency-free.
• Cache warming (glm's .coli_usage idea): per-expert usage counts persist and accumulate; top PIN_N/layer pinned at startup. Fully toggleable (PIN=off, PIN_N, USAGE_SAVE=0). The -f promptfile mode exists to build the history for a binary with no serve mode yet — happy to drop it if it reads as scope.
• docs/inkling.md: quickstart, env reference, benchmark methodology.

Measured on the 975B (Ryzen 9 7900 / A6000 / NVMe, commit 87981c1, greedy):

Configuration	Prefill	Decode	Hit
Stage A	150.2 s	0.06 tok/s	~0%
CPU perf pass	21.1 s	0.25 tok/s	81.5%
+ CUDA residents	18.4 s	0.32 tok/s	83.6%
trained-prompt pins (best case)	1.9 s	2.51 tok/s	100%
steady state (novel prompt, diverse history)	35.4 s	0.25 tok/s	82.2%

The honest number is the last row; it grows toward the best case as usage history accumulates (counts persist across runs). Phase timers show the remaining time is ~90% CPU expert matmul at high hit rates — GPU expert compute is the natural follow-up PR, not included here.

All modes remain token-exact vs the transformers oracle (f32, int4-container VNNI + IDOT=0, bf16 CPU, bf16 CUDA); make check 68/68, 0 warnings on both builds.

🤖 Generated with Claude Code

https://claude.ai/code/session_016KLDCYJSyHxd39ChTr4T27

[JustVugg]

Thank you for this — real work, and I want to be straight with you about the bar rather than let it sit open for another week.

The maintainer's rule, and it's the right one: a new engine lands only if he can run it on his own machine first. Not "the CI compiles it" — run it. Right now neither of us can: there's no checkpoint here, so the only thing that could be verified is that gcc doesn't complain. That's not verification, that's a spell-check.

Why the bar is that high, concretely. This week, on the one engine we can run:

• EXPERT_BUDGET shipped on numbers nobody checked and had to be quarantined (EXPERT_BUDGET (merged in #254, now on main): measured empty operating window — needs a quality gate and an MTP-contract decision #303) — measured 0.13 tok/s against a 0.30 baseline while claiming a speedup, and loading more experts per token than not using it at all;
• three separate users are getting incoherent output and the root cause is still unconfirmed ([Bug]: python coli chat doesn't end #307, Root cause of incoherent output: per-row int4 scales — fix with group-scaled quantization (fmt=4) #225);
• a CI check went in that could never fail;
• two tests I wrote myself "found" bugs that didn't exist.

That's the state of the engine that gets daily attention. A second engine that nobody here can execute doesn't get daily attention — it gets zero, until the month it breaks and the person holding it is the maintainer, who never had the weights.

What would change the answer, and I think it's reachable:

1. A model he can actually run, or a tiny one the CI can execute end-to-end. This repo already has the pattern: make_glm_oracle.py builds a glm_tiny/ and the engine is scored token-exact against it. An engine with an oracle test is an engine that can't silently rot.
2. A named maintainer — you, on the record, for that file.
3. Isolation from glm.c's path. This is the part I'd check hardest, and it's specific per PR (see below).

Until then the honest home is a fork, linked from the README. That's not a brush-off: it means people can find and use your work, and it doesn't hand a maintainer 10,000 lines he can't test. If it gets traction and someone can run it in CI, it comes back.

I'd rather tell you this now than leave it open for a month and close it then.

[JustVugg]

Specific to this one — and it's a more favourable read than #165.

You have a track record here: #232 landed today after I merged and tested it (PROF=1 works, the instrumentation costs ~48µs/token against tokens that take 3-30 seconds, so it's free in practice). That matters. This isn't a first PR from someone we've never seen.

inkling.c (+1,255) is isolated and that's the right shape — a separate file, like olmoe.c. backend_cuda_ink.{cu,h} likewise. If the whole thing were that, my only objection would be "nobody can run it".

The part I'd check hardest is c/tok.h (+115/-1) — 118 changed lines in the tokenizer glm.c depends on. That's the one place this PR can break the flagship engine, and it's the reason I'd want a real review rather than a wave-through. If the o200k support is purely additive (new tables, new branches, glm's path untouched byte-for-byte), it's defensible. If it refactors anything shared, that needs to be visible.

Concretely, that's testable without an Inkling checkpoint: if the tokenizer change is additive, the GLM oracle still scores token-exact against glm_tiny/, and tests/test_stops.c (which loads a synthetic tokenizer and checks the special flags) still passes. If you can show those green with your tok.h, you've retired the only risk I can actually name.

The remaining one is the maintainer's rule: he needs to run Inkling here before it lands. 975B is not a small ask, so if there's a smaller Thinking Machines checkpoint — or a way to build a tiny one with convert_inkling_int4.py, the way make_glm_oracle.py builds glm_tiny/ — that's the path, and it would also give the engine an oracle test so it can't rot silently.

[nbeerbower]

Totally get wanting to try it yourself before merging. Just wanted to get pen on paper mostly.

Made a separate PR #330 for just the tokenizer bit.

[JustVugg]

The model is interesting and there are many things we can try to bring it into the hummingbird catalog. Now with kimi 3 it would also be interesting. The open world is going wild!