Design: Live Contextual Explorer (host frozen BERT over an API for on-the-fly exploration)

[scheuclu]

Live Contextual Explorer — hosting the frozen BERT encoder for on-the-fly exploration

Type: Design / architecture
Status: Proposal
Relates to: #34 (contextual drift pipeline — the data this builds on), #31 (live semantic search), #32 (LanceDB ANN backend), #29 (Context Drift page — existing client-side WASM pattern)
Depends on: the contextual pipeline (PRs #35 → #36 → #37) and a completed multi-year run.

---

1. Summary

The contextual-drift pipeline (#34) produces, for every vocabulary word and year, an average BERT context vector ("centroid") in one shared coordinate space. Today those centroids are precomputed and static — the site can only show words we already processed. This proposal adds a thin live encoder service that embeds arbitrary user text with the same frozen model and drops it into that same 12-year space. That turns a fixed gallery into an open sandbox: search by sentence, disambiguate senses, explore out-of-vocabulary words, and get explained drift with live examples.

2. Background & current state

• How the space is built: a frozen bert-base-uncased encodes FineWeb 2014–2025; we accumulate per-word sum/sumsq/count on the fly → per-year centroid + dispersion, then per-year mean-center (anisotropy fix). Drift = 1 − cos(centroid_year, centroid_2018). A frozen, shared encoder means every year is already in one coordinate system — no alignment needed.
• Artifacts produced: models/contextual/{year}.npy (raw centroid), {year}_centered.npy, {year}_count.npy, {year}_dispersion.npy, drift_vs_2018.parquet, drift_summary.parquet (768-d).
• The limitation: static. OOV words (slang, brand names, neologisms) are invisible; users can only browse, not ask.

3. The one hard constraint

The live API must serve the exact same frozen bert-base-uncased we aggregated with.

A different embedding model — even a "better" one via an AI gateway — produces coordinates that don't line up with our historical centroids. The shared frozen encoder is the coordinate system; live exploration is only coherent if the live vectors come from that identical model (same weights, same layer, same first-subword pooling).

4. Architecture

                         ┌─────────────────────────────────────────┐
   user text  ──────────▶│  /api/embed  (frozen bert-base-uncased)  │
  (word + sentence,      │  pool target word (first-subword) or      │
   phrase, paragraph)    │  mean-pool sentence → 768-d vector        │
                         └───────────────┬───────────────────────────┘
                                         │ raw 768-d vector
                          ┌──────────────▼───────────────┐
                          │  Placement layer (static)     │
                          │  • reference-frame centering   │
                          │  • ANN query in 768-d  ───────┼──▶ nearest historical words
                          │  • 2D projector.transform()    │     + their drift trajectories
                          └──────────────┬────────────────┘
                                         │ {x,y}, neighbors, drift
                          ┌──────────────▼────────────────┐
                          │  Galaxy / Context-Drift UI     │
                          │  beacon on the map + side panel │
                          └────────────────────────────────┘
        precomputed (served as static blobs): centroids, year-means,
        fitted 2D projector, ANN index, per-word sense clusters, example store

4.1 The /api/embed contract

POST /api/embed
{
  "text":   "we hopped on a zoom call",   // required
  "target": "zoom"                          // optional: pool this word; else mean-pool sentence
}
→ 200
{
  "vector":   [768 floats],     // raw last-layer, first-subword (or mean) pooled
  "model":    "bert-base-uncased",
  "pooling":  "first|mean",
  "token_ok": true              // false if target not found in text
}

Stateless, cacheable by (text, target) hash. Returns the raw vector; all placement is done in the static layer so the model service stays trivial.

4.2 Placing a live vector in the historical space (the crux)

The hard part isn't embedding — it's making the live point comparable to 12 years of centroids. Three precomputed pieces handle it:

1. Reference frame. Historical centroids were per-year mean-centered. A live query has no "year," so we center it by a single fixed reference mean (candidates: the anchor-year mean, or a global mean over all years). Open question — needs a small experiment (§7).
2. Nearest historical words via an ANN index over the 768-d centered centroids (this is exactly what refactor: Integrate LanceDB for Fast Nearest-Neighbor Vector Storage #32 / LanceDB is for). Done in full 768-d, not 2D, for accuracy.
3. 2D beacon via a persisted, parametric projector so new points map stably and fast. UMAP's non-parametric .transform() is slow/approximate; prefer PCA-to-2D, parametric UMAP, or a tiny learned MLP fit once on the historical layout. (The static galaxy can keep its prettier non-parametric UMAP for the background; the live beacon uses the parametric map, or we switch the whole galaxy to a parametric projector for consistency.)

4.3 Serving options

Option	Pros	Cons	Verdict
Client-side WASM/ONNX (reuse the #29 Context Drift pattern)	$0 server, private, scales free	bert-base ≈ 400 MB to ship; slower first load	Good for power users / privacy; heavy for casual
Serverless Python fn (Vercel Fluid Compute)	one forward pass, sub-second on CPU; no GPU needed for single sentences; trivial to cache	cold start; per-call cost	Recommended default for the live endpoint
Dedicated GPU box	batch/stream throughput	infra to run	only if we add the live "drift radar" stream (§5F)

Recommendation: serverless /api/embed for interactive use; offer the WASM path as a privacy/offline mode later. The static placement artifacts are served as blobs alongside the existing data/vN (note: contextual artifacts are 768-d, so web packing must be parametrized in dim — currently hardcoded to 300-d / 14400-byte stride).

5. What it unlocks (features)

• A. Drop a sentence into the galaxy. Embed live → beacon on the map + nearest historical words + how that region drifted. Search becomes semantic (overlaps feat: Add Sentence Space Visualizer with Live Semantic Search #31).
• B. Sense disambiguation. Pre-cluster each polysemous word's contexts into senses (dispersion flags which words need it); a live sentence lands in the matching sense bucket ("your 'mouse' = device, which barely drifted; the animal sense did").
• C. Escape the vocabulary. Embed OOV slang / brand names / 2025 coinages and place them relative to known words — the long tail the precomputed vocab misses.
• D. Explainable drift. For a drifted word, embed example sentences on demand and show nearest 2018 vs 2024 usages ("'zoom' 2018 = a lens; 2021 = a call"). Requires a small per-word example-sentence store captured during the pipeline run.
• E. "Does your writing sound like 2015 or 2024?" Embed a paragraph, compare to each year's centroids → a playful, shareable hook.
• F. Live drift radar (stretch). Point /api/embed at a news/social stream, compare to historical centroids, flag emerging shifts in real time (this is the GPU-box case).

6. Precomputed artifacts to publish (static)

• centered_centroids.bin (768-d, anchor or chosen reference frame) + word index
• year_means.npy (per-year mean vectors, for the centering choice)
• projector.{json|onnx} — the parametric 2D map (PCA matrix / parametric-UMAP / MLP)
• ANN index (LanceDB table or hnswlib graph) over centered centroids — refactor: Integrate LanceDB for Fast Nearest-Neighbor Vector Storage #32
• sense_clusters.json — per-word sub-centroids for §5B
• examples/{word}.json — a few representative sentences per word for §5D (sampled during the run)

7. Milestones

1. /api/embed serverless endpoint serving the frozen model + the static vector contract. Verify a live vector reproduces a known centroid when fed enough sentences.
2. Placement layer: pick the reference frame (experiment: does anchor-mean or global-mean give the cleanest nearest-neighbors for held-out sentences?), ship ANN index + parametric projector.
3. Galaxy beacon: "type a sentence → see where it lands + neighbors + trajectory."
4. Sense disambiguation (§5B) + explainable examples (§5D).
5. WASM offline mode (§4.3) and/or live radar (§5F).

8. Risks & open questions

• Reference frame for live queries — the central unknown; needs the §7.2 experiment before the UX is trustworthy.
• Parametric vs non-parametric projection — non-parametric UMAP can't cleanly place new points; committing to a parametric map may slightly change the galaxy's look.
• Model size / latency — bert-base cold starts; mitigate with warm Fluid Compute + response caching. WASM download size for the client path.
• 768-d web packing — the current packer is 300-d-hardcoded; must parametrize dim before publishing contextual blobs.
• Drift ranking noise — unsupervised top-N is dominated by FineWeb vocab noise (foreign tokens, names, SEO/spam); the effect-vs-significance (z) split helps, curated trajectories are the trustworthy view. Live exploration sidesteps this by letting users bring their own query.

9. Out of scope

Retraining/fine-tuning the encoder (must stay the frozen one), cross-lingual (#30), and the full sentence-corpus visualizer (#31, though this shares the live-embed endpoint with it).

---

Static precursor already built for intuition: an interactive "Contextual Drift Galaxy" (UMAP of 2018 vectors, colored by drift percentile, with a top-movers slider and per-word trajectories) generated from the 4-year validation — a scratch artifact, not committed.