A ~28M-parameter transformer that runs fully offline, paired with a kNN retrieval memory it can lean on while writing. The tokenizer, model, training loop, sampler, memory, and inference server are all written from scratch; the only runtime dependencies are PyTorch and NumPy. Everything trains and runs on a laptop, on CPU or an Apple-Silicon GPU via MPS.
Measuring the memory carefully changed what this project ships. Almost everything a retrieval memory is worth here is a single tuned number — about 84% of it. The standard way of trying to do better, kNN-LM's gate on the model's own uncertainty, adds +0.10% on top of that constant on one of the two held-out books (interval reaching +0.13%), and on the other its confidence interval includes zero. Gating on the retrieved neighbours instead is worth +0.52% and +1.89%, significant on both.
New to any of this? Lookaside, explained is a from-scratch walkthrough of the whole system and the experiment, assuming no machine-learning background: tokenizers, transformers, retrieval, the finding, the method, and everything wrong with it.
Compare mode races the same prompt and the same random seed with the memory off and
on. The two runs sample identical tokens until the memory first changes a sampled
token; the orange mark is that fork. Words the memory supplied are highlighted, the
side panel shows the corpus passage each one was recalled from, and the gauges track
the model's entropy H and the blend weight λ as it writes.
kNN-LM interpolates the model's distribution with a distribution over retrieved
neighbours under a weight λ. Its adaptive variant sets λ from the model's own
uncertainty, λ = λ_max · σ(a·H + b), with H the next-token entropy and a, b
fitted to held-out likelihood. Lookaside implements that, and fitting it kept
producing a gate that barely moved.
Two never-trained Doyle books make it possible to check why. Every gate below is fitted on one of them and scored on the other, so nothing is measured on text it was tuned on:
| the gate conditions on | The Valley of Fear | His Last Bow |
|---|---|---|
| nothing — no memory at all | 42.53 | 38.94 |
| a constant λ, fitted | 38.59 | 37.54 |
entropy H — kNN-LM's signal |
38.59 | 37.50 |
| the gate this project used to ship | 38.49 | 37.51 |
d1 + d_spread — the neighbours |
37.86 | 37.34 |
| a per-token oracle (cheating) | 31.31 | 31.14 |
Perplexity on the complete held-out halves of two books the model never trained on.
d1 is the squared L2 distance to the nearest stored key; d_spread is how much
closer that key is than a typical one among the 64 retrieved.
The first thing to take from that table is not the gate at all. A single fitted number — one λ, applied to every token — captures 9.26% of the 10.97% total, about 84% of everything the memory is worth here. Every gating scheme, including this project's own, is competing over the remaining sliver. So the honest way to compare them is by what they add on top of that constant, with error bars, block-bootstrapped over 256-token blocks because neighbouring tokens are nowhere near independent:
| improvement over a fitted constant λ | The Valley of Fear | His Last Bow |
|---|---|---|
entropy H — kNN-LM's signal |
+0.009% [−0.061, +0.080] |
+0.104% [+0.078, +0.130] |
d1 alone |
+0.362% [+0.282, +0.444] |
+0.055% [−0.035, +0.141] |
d_spread alone |
+1.703% [+1.496, +1.905] |
+0.426% [+0.250, +0.593] |
d1 + d_spread |
+1.889% [+1.659, +2.115] |
+0.519% [+0.337, +0.696] |
| all five neighbour signals | +1.897% [+1.661, +2.130] |
+0.522% [+0.336, +0.699] |
The entropy gate's interval includes zero on one book and reaches +0.13% on the other. Neighbour distance and margin are worth 0.52% and 1.89%, significant in both directions, and two signals get essentially all of what five do. The effect is small in absolute terms and varies more than three-fold between two books by one author, so treat the size as unestablished; the ordering survives resampling comfortably.
A gate exists to predict the blend weight a token wants, so measure that directly: split the 89,541 held-out tokens into fifths by each signal and find the single λ that is optimal inside each fifth.
By neighbour distance the optimal λ falls monotonically, 0.163 down to 0.049, a 3.3×
spread and exactly the shape σ(w·d + b) can track. By entropy, four of the five
fifths want the same weight, 0.066 to 0.084. Only the most confident fifth departs
from it, at 0.157, and those tokens sit at perplexity 3.7 — already predicted so well
that almost no total likelihood rides on their λ.
The reason is a distinction the entropy gate quietly gets wrong. Whether the memory wins a token and how much that win is worth are different quantities, and the two signals predict opposite ones:
| signal | correlation with the memory winning the token | correlation with nats actually saved |
|---|---|---|
entropy H |
−0.358 | +0.031 |
p_LM top-1 |
+0.375 | −0.019 |
d1 |
−0.186 | −0.025 |
d_spread |
+0.069 | +0.176 |
Entropy is the best available predictor of which tokens the memory will win, and
almost useless for how many nats those wins are worth. The margin is the reverse. A
gate fitted to likelihood is optimising the second column, so it is reading the one
thing entropy cannot tell it: not how bad p_LM is, but whether p_kNN is better.
Two controls, because "your fit just failed" is the obvious objection:
| control | perplexity recovered |
|---|---|
| a constant λ | 9.259% |
| entropy gate, fitted by Adam | 9.267% |
entropy gate, best of a 19,481-point grid over (a, b) |
9.267% |
entropy gate, allowed a quadratic in H — non-monotone |
9.263% |
An exhaustive grid finds what the optimiser found to three decimals, so this is not an
optimisation failure; and letting λ be a non-monotone function of H does not help
either, so it is not the sigmoid's shape. On the trained-text set the grid's own optimum
is a = +0.00 — a literal constant, and the entropy gate's interval there is
[−0.000, +0.000]. On the reverse book split the grid does beat the optimiser, by 0.015
points, which puts the entropy gate's honest value there at 0.115 points over a
constant rather than 0.100.
A gate that opens further can pay for it on text the weights already handle. The gate fitted on a never-trained book was applied, unchanged, to withheld slices of books the model did train on (30,200 tokens from five of the six; the sixth was the fit set):
| gate | perplexity | vs no memory |
|---|---|---|
| no memory | 21.93 | — |
| the gate this project used to ship | 21.88 | 0.24% |
entropy H, refitted here |
21.71 | 1.04% |
| a constant λ | 21.71 | 1.04% |
d1 + d_spread, fitted on a never-trained book |
21.68 | 1.15% |
It pays no measurable penalty. The +0.109% it gains over the constant here has interval
[−0.017, +0.241] and should be read as "no worse", not as a win. The refitted entropy
gate ties the constant to four decimals, which is the same finding again on different
text.
That is enough to make it safe to ship, and checkpoints/lookaside-demo/ now defaults
to gate_mode="confidence". Generating with it, λ ranges over 0.028 to 0.279
(mean 0.065) against the shipped 750k-entry memory, where the entropy gate spans only
0.082 to 0.114 (runs/demo_receipt.log), so the selectivity survives outside the
fitting set. The full write-up, with the caveats, is in
What should the gate watch?.
The experiment holds two Doyle books (The Valley of Fear, His Last Bow) out of training entirely and puts them in the memory instead, alongside the six Doyle books the model did train on. Each held-out book is split in half: one half goes into the datastore, the other becomes an evaluation set. Withheld tail slices of the six trained books form a second evaluation set. Nothing evaluated is in the datastore — guaranteed by the split itself and by a test that pins the self-retrieval invariant, not by inspection.
On books the model never read, the memory recovers 4.1% and 11.0% of perplexity depending on which book is measured, knowledge added without touching a single weight. That two-books spread is wider than every gate difference above, which is why both are always reported.
A 500k-entry datastore spread over the full 5.7M-token classics corpus once produced no gain, while 600k entries over the 725,788-token Doyle corpus produced a clear one. Those runs differ in density and domain, so they cannot separate the causes. Holding the model, corpus and evaluation positions fixed and varying only the entry cap does:
Below roughly 11% coverage a λ pinned at 0.1 is worse than having no memory at all — at 1% coverage it costs 4.6% perplexity, because the nearest stored key is not actually a near match and its vote is noise. Fitting λ at each level avoids that hole by shrinking itself: 0.006 at 1% coverage, rising to 0.079 at 83%. Both climb to 7.2–7.3% and are still rising there. Note that both curves are plain constants, not gates — this sweep is about density, and its single fitted parameter is the one place a random position split is used rather than a book-level one.
Bucketing held-out tokens by how often the model saw them in the six Doyle books it trained on:
| times seen in training | tokens | gain from the memory | best λ |
|---|---|---|---|
| never | 116 | −1.5% | 0.011 |
| 1–10 | 2,752 | 8.5% | 0.054 |
| 11–100 | 17,389 | 15.0% | 0.086 |
| 101–1,000 | 24,647 | 9.2% | 0.092 |
| more than 1,000 | 44,637 | 1.3% | 0.069 |
The payoff concentrates in the middle of the frequency range: words the model has met but not often enough to pin down. Common words gain almost nothing. Tokens it never saw are the honest failure case — only 116 of them, the memory makes them slightly worse, and the λ they want is 0.011, essentially closed, because a token missing from the training corpus is usually missing from the datastore too.
Same prompt and seed, memory off then on, from runs/demo_receipt.log. Words the
memory drove are bold:
Prompt: The tragedy at Birlstone came about in this way.
LM-only: …One of the adventurers and the other are the work of the hospitals. The address was committed by Sir Henry, who has not been sent to Baker Street by the death of Stapletons. "Anything that you know?" asked Holmes…
+ memory: …One of the adventurers and the other are the facts of the cabman. The address is a second man, but he has not known the death of his murdered friend and of any one who could find it. "When I met him I found myself at once upon my own wits' gossip…
The memory drove 5 of 120 tokens here, and the log records where each came from:
facts and of from The Memoirs of Sherlock Holmes, the and fellow from The
Hound of the Baskervilles, plenty from The Return of Sherlock Holmes. All five come
from books the model trained on, not the held-out ones, so a single 120-token sample
demonstrates the provenance machinery rather than evidencing held-out knowledge. The perplexity tables above are that
evidence. Prose quality is what ~28M parameters and a few laptop-hours buy: fluent
Doyle-flavoured pastiche, not literature.
Earlier experiments — including a regime where the fitted gate correctly closes the memory entirely, and one where a weaker 14M model does learn to open on uncertainty — are in When does retrieval actually help?.
flowchart LR
P[prompt] --> T[byte-level BPE]
T --> M[decoder transformer<br/>RoPE · RMSNorm · SwiGLU · GQA · KV-cache]
M -->|logits| L[p_LM]
M -->|hidden state h| D[(kNN datastore<br/>keys=h, values=next token)]
D -->|k nearest| K[p_kNN + distances]
K -->|distance, margin| G{"gate<br/>λ = λmax·σ(w·z+b)"}
K --> B[(blend)]
L --> B
G -->|λ| B
B --> S[sample: temp · top-p · rep-penalty]
S --> O[token + provenance]
The gate sits downstream of the search, because what it watches is the search result. A gate reading only the model's own output could in principle skip retrieval before paying for it, and this one cannot — but in practice that costs nothing, because a fitted sigmoid never returns exactly zero, so the entropy gate searches at every step too. Measured against the 750k-entry memory: 44.2 tok/s for the confidence gate and 44.4 for the entropy gate, within run-to-run variance, against 110.0 tok/s with the memory off. The search is what retrieval costs, not the gate that reads it.
The shipped model is 27.8M parameters: d_model 512, 8 layers, 8 heads with 4 KV
heads (grouped-query attention), context 256, byte-level BPE vocab of 8192.
lookaside/model.py is a decoder-only transformer: rotary position embeddings,
RMSNorm pre-norm, SwiGLU feed-forwards, grouped-query attention, tied input/output
embeddings, and an incremental KV-cache with a sliding window so generation runs past
the training context.
lookaside/tokenizer.py is a byte-level BPE: a reversible byte↔unicode mapping
(nothing falls out of vocabulary), a regex pre-tokenizer, and a heap-backed
incremental merge trainer.
lookaside/retrieval.py is the memory. For every corpus position it stores the hidden
state the LM head consumes (the key) and the true next token (the value). At inference
it finds the nearest keys to the current hidden state, converts their distances into a
distribution, and blends that in under the gate. Search is pure-torch brute force by
default, with an optional faiss HNSW index. Every retrieved token keeps a pointer back
to the corpus position it came from, so the UI can show its source.
pip install -r requirements.txt
# Option A: run the shipped demo checkpoint — fully offline, no download needed.
# The first run builds the retrieval memory from the bundled Doyle corpus
# (about a minute, one time) and caches it; startup is instant after that.
python -m lookaside serve # open http://127.0.0.1:8000
# Option B: retrain everything from scratch, into a directory of your own.
# (--out matters: the default is the committed checkpoint, and these steps
# overwrite model.pt and retrieval_config.json in place.)
python scripts/download_corpus.py --set classics # 32 books (needs internet once)
python scripts/train_demo.py --out checkpoints/my-run \
--exclude the_valley_of_fear his_last_bow \
--d-model 512 --layers 8 --heads 8 --kv-heads 4 \
--steps 2600 # ~2.5 h on Apple-Silicon MPS
python scripts/finalize_demo.py --ckpt checkpoints/my-run # memory, entropy gate, ablation
python scripts/gate_study.py --eval-set holdout \
--write-config checkpoints/my-run/retrieval_config.json # the gate the demo shipslookaside generate --prompt "Sherlock Holmes turned to me and said, " --show-sources
lookaside generate --no-knn # parametric model only (the A/B baseline)
lookaside generate --gate-mode entropy # the kNN-LM-style gate, for comparison
lookaside generate --gate-mode fixed --lam 0.3
lookaside serve --port 8000 # streaming web UI
lookaside build-datastore # (re)build the retrieval memory
lookaside fit-gate # fit the *entropy* gate; overwrites the config
lookaside ablation # perplexity across this checkpoint's gate modes
lookaside train --steps 3000 # train end to endWith no --gate-mode, --lam or --top-k-knn, the checkpoint's own fitted gate is
used exactly as fitted — a confidence gate's gate_mu/gate_sd are statistics of
neighbour distances at a particular k, so changing --top-k-knn invalidates it.
fit-gate and ablation take their held-out tail from the same files the memory
indexes, so their numbers include self-retrieval; both say so when they run. The
book-level protocol behind the tables above is scripts/gate_study.py.
lookaside is installed via pip install -e ., or use python -m lookaside.
# the gate tables and the fitted gate the demo ships
python3 scripts/gate_study.py --eval-set holdout --write-config runs/confidence_gate.json
python3 scripts/gate_study.py --eval-set holdout --fit-book the_valley_of_fear
# what that gate costs on text the model did train on
python3 scripts/gate_study.py --eval-set trained \
--gate-from checkpoints/lookaside-demo/retrieval_config.json
python3 scripts/coverage_sweep.py # the density curve
python3 scripts/demo_receipt.py # the sample, lambda ranges and throughput
python3 scripts/trace_token.py # one prediction through every stage, with numbers
python3 scripts/make_figures.py # redraw the three study figures from that JSONThe first run builds a datastore and caches per-token features, so the rest are quick.
runs/gate_study.log, runs/coverage_sweep.log and runs/demo_receipt.log are the
captured output behind the numbers above, and runs/*.json are the payloads
make_figures.py draws from — committed so the figures can be re-derived from a clone,
not just trusted. The older runs/exp*.log and runs/*classics*.log back the earlier
experiments in docs/when-does-retrieval-help.md;
those predate the Mnemosyne → Lookaside rename and still name checkpoints/mnemosyne-*
paths, which is left as-is because a dated record should not be edited after the fact.
lookaside/
config.py dataclass configs (model / train / sampling / retrieval)
tokenizer.py byte-level BPE tokenizer
model.py the decoder-only transformer
data.py Gutenberg cleaning, tokenisation, batching
train.py AdamW + cosine schedule, grad-accum, best-on-val checkpointing
sample.py temperature / top-k / top-p / repetition / no-repeat-ngram
generate.py streaming generation with a sliding-window KV-cache
retrieval.py datastore, kNN search, the gates, provenance, ablation
bundle.py load model + tokenizer + memory (builds datastore on first run)
server.py dependency-free http.server + Server-Sent Events
cli.py the `lookaside` command
web/index.html self-contained streaming UI (no CDNs, works offline)
scripts/
download_corpus · train_demo · finalize_demo the model
gate_study · coverage_sweep · demo_receipt the experiments
trace_token one prediction, stage by stage
make_figures · svgchart the figures
tests/ 139 tests
pip install pytest && pytestThe suite (139 tests, ~4 s) covers the parts most likely to fail silently: lossless
tokenizer round-trips, KV-cache output matching a full forward pass, causal masking,
the datastore key/value off-by-one, the self-retrieval invariant, and — because a
fitted gate would otherwise mean something different at inference than it did at fit
time — the per-step gate signals checked against a batched reference. CI runs it on
pushes to main and on pull requests, on Linux (Python 3.9 and 3.12) and macOS
(Python 3.12).
The transformer components are standard building blocks. The retrieval core is kNN-LM (Khandelwal, Levy, Jurafsky, Zettlemoyer & Lewis, ICLR 2020), and its adaptive-gate variant — λ from the model's entropy — is the baseline measured against here.
Conditioning retrieval on properties of the search itself is not new either. Zheng et
al., Adaptive Nearest Neighbor Machine Translation (ACL 2021), learn to weight the
k neighbours from their distances; He, Neubig & Berg-Kirkpatrick, Efficient Nearest
Neighbor Language Models (EMNLP 2021), and Drozdov et al., You can't pick your
neighbors, or can you? (EMNLP Findings 2022), both adapt retrieval on learned
features of it. Anyone weighing the result above should read those first.
What I have not found reported is the controlled head-to-head: the same gate form, the same fitting objective, the same held-out data, entropy against neighbour distance — together with the finding that entropy's contribution is indistinguishable from zero on one held-out book and worth +0.10% on the other, with an interval reaching +0.13%, and the reason why (it predicts whether the memory wins a token, not how much that win is worth). The token-level provenance UI and packaging the whole stack to train and run offline on a laptop are also mine.
MIT © 2026 Sef