Nearly double the hit-rate

[withinboredom]

Summary

I saw a video on YouTube about this project and realized the problems you're trying to solve are very similar to ours over on Swytch (A leaderless distributed cache with serializable guarantees). The caching algorithm that makes it possible is called "cloxcache" and we're currently writing a paper on it. The caching family would be CLOCK-LRU-K -- and it adapts LFU/LRU on-the-fly. I've adapted it from Go to C, and your cache structures. To our knowledge, it's the most performant cache-eviction algorithm currently in existence on large caches, outperforming other eviction algorithms and ~99.998% as good as a perfect cache can ever be.

Anyway, enough with the marketing speak, here's the data:

Reproduction:

export SNAP=$MODEL PROMPT="Explain how MoE offloading works in detail." NGEN=256
export TEMP=0 DRAFT=8 AUTOPIN=0 DIRECT=1
export RAM_GB=32
./glm 8

Finals:

Run	hit rate	decode time	disk io	rss
baseline	9.4%	3,865s	1,176s	32.9gb
cloxcache	15.1%	3,628s	1,103s	31.67gb
delta	+60%	-237s	-73s	--

Hardware:

Linux beta 6.8.0-134-generic #134-Ubuntu SMP PREEMPT_DYNAMIC Fri Jun 26 18:43:11 UTC 2026 x86_64 x86_64 x86_64 GNU/Linu
Architecture:                x86_64
  CPU op-mode(s):            32-bit, 64-bit
  Address sizes:             43 bits physical, 48 bits virtual
  Byte Order:                Little Endian
CPU(s):                      12
  On-line CPU(s) list:       0-11
Vendor ID:                   AuthenticAMD
  BIOS Vendor ID:            Advanced Micro Devices, Inc.
  Model name:                AMD Ryzen 5 3600 6-Core Processor
    BIOS Model name:         AMD Ryzen 5 3600 6-Core Processor               Unknown CPU @ 3.6GHz
    BIOS CPU family:         107
    CPU family:              23
    Model:                   113
    Thread(s) per core:      2
    Core(s) per socket:      6
    Socket(s):               1
    Stepping:                0
    Frequency boost:         enabled
    CPU(s) scaling MHz:      59%
    CPU max MHz:             4208.2031
    CPU min MHz:             2200.0000
    BogoMIPS:                7199.68
    Flags:                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt
                             pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16
                             sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefet
                             ch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibpb st
                             ibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_oc
                             cup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clea
                             n flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca
                             sev sev_es ibpb_exit_to_user
Virtualization features:
  Virtualization:            AMD-V
Caches (sum of all):
  L1d:                       192 KiB (6 instances)
  L1i:                       192 KiB (6 instances)
  L2:                        3 MiB (6 instances)
  L3:                        32 MiB (2 instances)
NUMA:
  NUMA node(s):              1
  NUMA node0 CPU(s):         0-11
Vulnerabilities:
  Gather data sampling:      Not affected
  Indirect target selection: Not affected
  Itlb multihit:             Not affected
  L1tf:                      Not affected
  Mds:                       Not affected
  Meltdown:                  Not affected
  Mmio stale data:           Not affected
  Reg file data sampling:    Not affected
  Retbleed:                  Mitigation; untrained return thunk; SMT enabled with STIBP protection
  Spec rstack overflow:      Mitigation; Safe RET
  Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:                Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
  Srbds:                     Not affected
  Tsa:                       Not affected
  Tsx async abort:           Not affected
  Vmscape:                   Mitigation; IBPB before exit to userspace
free -h
               total        used        free      shared  buff/cache   available
Mem:            62Gi        12Gi        31Gi       198Mi        19Gi        49Gi
Swap:             0B          0B          0B

Validation

• make -C c check
• CUDA changes were tested with make -C c cuda-test (if applicable)
• Performance claims include hardware, commands, and repeatable measurements

Compatibility

• The default CPU build remains dependency-free
• No model files, generated binaries, or benchmark artifacts are included

[JustVugg]

Thank you for this — I ran it on our reference box and I owe you the numbers, because they're interesting in a way that isn't "yes" or "no".

What I measured

ABA (dev → dev+223 → dev), same prompt, same env, DIRECT=1 URING=1 PIPE=1 AUTOPIN=0, RAM_GB=22, NGEN=16, machine idle and verified idle between runs:

run	hit rate	experts loaded/token	decode
A1 — dev	3.8%	876.5 (11.69/layer)	251s
B — dev + #223	3.7%	876.5 (11.69/layer)	393s
A2 — dev	3.7%	876.5 (11.69/layer)	785s

Identical to the decimal. The SLRU changes nothing here — not "a little", exactly nothing.

Why, and why it isn't your bug

[RAM_GB=22.0] resident 10.6 GB + reserve 6.2 GB -> cap lowered 8->3

Three slots per layer, eight experts wanted per token. With three seats and eight arrivals, every eviction policy loses identically — there's no room left to be clever. Your algorithm isn't failing; it has nothing to work with.

You ran RAM_GB=32 and reported baseline hit 9.4% → 15.1%. That's a cap around 8, where choosing the victim well actually changes who's resident next step. I think your result is real and our box simply cannot reach the regime where it applies: to get cap=8 here we'd need ~28 GB of budget on a 25.4 GB machine. So this is not a refutation — it's a boundary condition your PR doesn't state.

The ask: what's the lowest cap at which cloxcache starts to win? If it's ~6+, this is a good PR for large-memory hosts and should say so — the README line would be "on hosts with cache headroom" rather than a general claim. That's a much stronger PR than one that quietly does nothing for the low-RAM users this engine targets.

The other half, which is the useful part for both of us

Look at the decode column: 251s, 393s, 785s. A1 and A2 are the same binary, same prompt, same machine, twenty minutes apart — a 3.1× spread.

Which means: had I run only A1→B, I'd have posted "#223 is 56% slower, regression." Had I run only A2→B, I'd have posted "#223 is 2× faster, merging." Both would have been garbage, and B's 393s sits neatly between two runs of the same code.

On a disk-bound box wall-clock is noise. The only stable signals are experts loaded/token and hit rate — and those are what I'd suggest you quote, because your own table leads with decode time (3,865s → 3,628s, -6%), which on our hardware is well inside the noise floor. Your hit-rate number is the honest one and it's the one doing the work.

One small thing while I'm here: the title says "nearly double the hit-rate", but 9.4% → 15.1% is +60%, not double. I'd retitle — the real result doesn't need the help, and this repo has had a rough week with numbers that promised more than they delivered (see #303).

Where this leaves it

I'm not merging it yet, and I want to be straight about why: it adds real surface (tier.h, TierAdapt, per-layer efreq allocations) for a benefit measured at zero on the machine that represents this project's thesis. That's not a verdict on the work — it merges cleanly onto dev (only a README conflict, on an expert count: 75 layers × 256 + the MTP head = 19,456, so I kept ours and took your SLRU wording), builds with zero warnings, and passes the full suite including your test_tier.

It's a judgment call about scope, not correctness, and it's an easy yes the moment we know the cap threshold. If you can give me that number — or a run at a cap this class of hardware can actually reach — I'll take it.

Genuinely appreciate the rigour, and the fact that this is the safe kind of change: it touches eviction only, never which expert gets selected, so it cannot alter output. That distinction matters a lot around here lately.

[KingIcyCreamProjects]

Following the ABA results — two things from a code-level read of the eviction path on dev that bear directly on the cap-threshold question:

1. The engine currently can't attribute hits to the tier this PR changes. m->hits counts a pin-tier hit and an ecache hit identically (both lookup branches bump the same counter; the PROF harness reports only the lumped value). At any given config the ecache could be serving ~0% of hits or a large share — which is exactly the ambiguity behind "what's the lowest cap at which cloxcache starts to win." I've split the counters on a branch (hit_pin / hit_ecache, surfaced in the serve stats line, REPLAY summary, and [PROF] report): https://github.com/KingIcyCreamProjects/colibri/tree/telemetry/pin-vs-ecache-hits — happy to send it as a PR. It would let the cap sweep show why a policy is flat (pin tier absorbing everything vs a genuine floor effect) before anyone spends another A/B on it.

2. There's a third eviction site, Linux-only. This PR converts the moe() promotion and pilot_realload scans, but pilot_uring_batch (c/glm.c:3472 on dev, #ifdef __linux__) still runs the old min-used scan — so on a URING=1 Linux host the uring pilot loader would evict under a different policy than the two converted sites. No-op on non-uring builds, but worth converting for consistency since the claim is about the array's global eviction order.

[withinboredom]

The ask: what's the lowest cap at which cloxcache starts to win?

The honest answer is a ratio, not a cap: it starts winning when cap exceeds roughly a third of the per-token unique expert demand per layer, peaks around 0.4–0.8×, and thins out again above 1× (where plain recency already holds the working set). Your box makes the arithmetic concrete: 876.5/75 ≈ 11.7 loads/layer/token at 3.8% hit ⇒ ~12 unique experts demanded per layer per token, against 3 slots. That's a 4× full flush every token — no resident survives to the next token under any policy, so "identical to the decimal" is the expected outcome, not a near-miss. You're right that it isn't my bug, and right that it isn't your refutation: cap 3 against demand 12 is ratio 0.25, below the floor. My RAM_GB=32 number sits at cap 8 against the same demand — ratio 0.66, mid-sweet-spot. Both measurements are the same curve.

And you don't have to take the curve from me, because:

On a disk-bound box wall-clock is noise.

— is correct, and I won't argue with it. Since the PR I built an offline replayer that removes the hardware from the measurement entirely, and it's on the branch: capture a routing trace during any normal run (one env var, one line per routing decision, negligible overhead), then replay the identical request stream through plain LRU, this PR's SLRU, and Belady's clairvoyant OPT:

ROUTE_TRACE=run.trace ./glm
make tools/route_sim
for c in 3 4 5 6 8 12; do tools/route_sim -c $c -p 0 run.trace; done

Run any cap, including the ones your RAM can't reach. It also scores each eviction decision against hindsight (was the victim re-requested before another resident would have been?), which is the metric hit rate can't see. For calibration, my traced workload (256 experts/layer, demand ≈ 8/layer/token, -p 0 matching your AUTOPIN=0):

┌─────┬────────────┬──────────┬───────────┬─────────────┐
│ cap │ cap/demand │ LRU hit% │ SLRU hit% │ OPT ceiling │
├─────┼────────────┼──────────┼───────────┼─────────────┤
│ 3   │ 0.38       │ 8.0      │ 9.4       │ 25.6        │
├─────┼────────────┼──────────┼───────────┼─────────────┤
│ 4   │ 0.50       │ 11.8     │ 13.6      │ 31.8        │
├─────┼────────────┼──────────┼───────────┼─────────────┤
│ 5   │ 0.63       │ 16.3     │ 18.5      │ 36.8        │
├─────┼────────────┼──────────┼───────────┼─────────────┤
│ 6   │ 0.75       │ 20.8     │ 22.8      │ 41.0        │
├─────┼────────────┼──────────┼───────────┼─────────────┤
│ 8   │ 1.00       │ 31.5     │ 31.7      │ 47.1        │
└─────┴────────────┴──────────┴───────────┴─────────────┘

By your scaling (~2 GB/cap between RAM_GB=22→cap 3 and 32→cap 8), cap 4–5 is RAM_GB ≈ 24–26, so cap 4 may even be physically reachable on the 25.4 GB box if you want confirmation — but the simulator will tell you the answer for your routing distribution either way. The OPT column is worth a look independently of this PR: it prices exactly what each additional GB of cache budget buys on your workload before anyone tunes another heuristic.

the README line would be "on hosts with cache headroom" rather than a general claim

Agreed, and I'd go further — proposed wording: "SLRU eviction improves expert-cache hit rate when per-layer cache capacity exceeds ~⅓ of per-token expert demand (measured +60% hit rate at cap/demand ≈ 0.65); below that threshold the cache fully flushes each token and the policy is provably inert — identical residency to LRU, zero cost, zero risk." The inertness your A/B demonstrated is worth stating as the safety property it is.

I'd retitle — the real result doesn't need the help

Also agreed, no defense offered. "+60% hit rate" it is.

One relevant note: the branch has moved since you tested. Building the simulator caught two real defects in my own implementation (ghost-eviction ordering that silently destroyed ghost memory under exactly the LRU-victim pattern; adaptation windows mis-paced so the hit-rate feedback learned from noise) and added eviction-driven frequency decay, which the replayer shows matters most at small caps — the regime you care about. All verified against OPT on trace replay; test_tier covers each. Worth retesting from head rather than the revision you benchmarked — though at cap 3, I expect and now predict on the record that you'll still measure exactly nothing.

Happy to have the merge gate on the documented threshold; it's the right bar. Run the replay on your trace and the number in the README can be yours rather than mine.