feat(contextual): pipeline part 1 — config, frozen encoder, benchmark

[scheuclu]

Contextual word-drift pipeline — Part 1/3: foundation + benchmark

First slice of the contextual word-drift pipeline (#34). Goal: upgrade WordDrift from static Word2Vec/TWEC drift (300-d, distributional) to contextual drift — run the historical corpus through a frozen transformer encoder and aggregate per-word contextual centroids per year, so we capture sense shifts Word2Vec can't (zoom lens→video-call, delve becoming LLM-ese). A frozen shared encoder puts every year in one coordinate system, so no Procrustes/TWEC compass is needed.

This is a stacked PR series (each builds on the previous):

• Part 1 — this PR: config + frozen-encoder wrapper + throughput benchmark
• Part 2: accumulator + streaming + finalize + orchestrator (--smoke verified)
• Part 3: drift analysis + local eval (anisotropy, stable words, drifters, dispersion)

What's here

File	What
config.py	CONTEXTUAL_* knobs (model, layer, pooling, window/batch, min-count, centering, PCA-removal, stopwords, output dirs). Append-only — nothing existing touched.
pipeline/contextual_model.py	ContextualEncoder: frozen bert-base-uncased (bf16, SDPA, inference_mode) + fast tokenizer. encode_windows() returns hidden states + attention mask + a word_ids tensor mapping every subword → source word (uses word_ids(), not "##"-stripping). Width read from config.hidden_size.
scripts/contextual_benchmark.py	Throughput probe over real 2018 windows (writes nothing). Sweeps batch size, prints words/sec + an hours-per-year ETA table.
pyproject.toml / uv.lock	Declare transformers (was installed but undeclared).

Benchmark result (GB10)

 batch     win/s     words/s     subwd/s   peakGB
--------------------------------------------------
    64     476.8       54176       71141     0.76
    96     586.1       66595       87449     1.01
   128     706.3       80262      105396     1.26   <- optimum
   192     642.1       72962       95810     1.76
   256     605.5       68811       90358     2.25
   512     558.5       63464       83338     4.25

batch=128 is the optimum at ~80k words/s. With dynamic padding, smaller batches waste less compute on padding; we're nowhere near memory-bound (1.26 GB of 128 GB). CONTEXTUAL_BATCH_SIZE set to 128. Full 12 B-token run ETA ≈ 1.7 days (~3.5 h/year × 12).

Future optimization (not needed for the ~1.7-day ETA): length-bucketing windows before batching would cut padding waste and likely let larger batches win.

Verification

• ContextualEncoder loads, hidden_dim=768, fast tokenizer confirmed.
• Benchmark streams real FineWeb 2018 data and runs forward passes end-to-end (output above).
• config imports cleanly; 116 stopwords loaded.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
language-drift	Ready	Preview, Comment	Jun 6, 2026 6:07pm