feat(contextual): pipeline part 3 — drift analysis + eval - #37
Draft
scheuclu wants to merge 2 commits into
Draft
Conversation
Part 3/3 of the contextual word-drift pipeline (issue #34). - analysis/contextual_drift.py: load centered per-year centroids into one coordinate system (frozen encoder => no alignment), compute cosine drift vs the anchor year, gate every (word,year) on min observation count so noisy low-frequency centroids don't pollute the ranking, and emit a long table + a per-word summary with a dispersion (polysemy) column. Reuses analysis/drift.py::compute_drift_from_base; summary aggregation is local because that drift is keyed by a single `year` column. - scripts/contextual_drift.py: add a `drift` stage (and fold it into `all`) that writes drift_vs_{anchor}.parquet + drift_summary.parquet. - scripts/contextual_eval.py: local eval (plan Verification) — anisotropy, stable words, dispersion (polysemous vs monosemous), drift ranking + a coarse noise-floor/SNR. Works with however many years are present. Verified on the 2-year smoke (2014 vs 2018, models/contextual_smoke): - anisotropy: raw ~0.47 both years -> centered ~0.000 both years. - stable words low drift: house 0.117, water 0.026, mother 0.069. - dispersion: polysemous 104.8 > monosemous 100.2 (weak at smoke scale). - drift: 771 gated words, noise floor 0.034; top words are high-variance topical words (sampling noise between two tiny snapshots). Real drifters (covid/zoom/delve) and SNR-vs-Word2Vec need the full 12-year run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
The cosine total_drift ranking is dominated by high-variance, near-threshold words whose centroids merely wobble. Add a t-like significance score so the ranking reflects real shifts. - analysis/contextual_drift.py: per (word,year), z = ||Δcentroid|| / SE where SE^2 = dispersion_base/count_base + dispersion_year/count_year (a centroid is a sample mean of `count` vectors, so its squared standard error is dispersion/count; SE^2 adds across two independent years). Δ is on the centered centroids. summary now carries total_z/max_z/peak_year alongside the cosine effect-size columns, and is sorted by total_z. - scripts/contextual_eval.py: z-ranked top list + per-year trajectory table for known drifters vs stable controls (the clean, interpretable view). Validated on the 4-year run (2018-2021, 200M tok/yr): trajectories show the expected flat-then-2020-spike — zoom 0.002/0.033/0.053, pandemic 0.006/0.044/0.050, lockdown 0.031/0.059/0.064 — while stable controls (house/water/mother/river) sit at ~0.000 and delve stays flat (its LLM-ese shift is 2023+, outside this window: a correct negative). Tradeoff worth knowing: z (significance) favors ubiquitous high-count words with tiny effects, while cosine (effect) favors low-count noise; both columns are kept so downstream can balance them. Also: a high min_count gates out emergent words that were rare in the 2018 base year (lockdown/pandemic/moderna vanish at min_count=300), so the default 50 is better for catching neologisms. Unsupervised top-N is ultimately limited by FineWeb vocab noise (foreign tokens, names, SEO/spam drift); curated trajectories are the trustworthy view. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contextual word-drift pipeline — Part 3/3: drift analysis + eval
Final slice of the contextual word-drift pipeline (#34). Turns the finalized per-year centroids into drift artifacts and adds the local evaluation from the plan's Verification section.
What's here
analysis/contextual_drift.pyanalysis/drift.py::compute_drift_from_base.scripts/contextual_drift.pydriftstage (also folded intoall) →drift_vs_{anchor}.parquet+drift_summary.parquet.scripts/contextual_eval.pyVerification (2-year smoke: 2014 vs 2018)
uv run python scripts/contextual_drift.py --smokethenuv run python scripts/contextual_eval.py --smoke:Caveats / what the smoke can and can't show
Follow-up (out of scope, as planned)
Web packing (768-d artifacts → a new
data/vNalongside the existing TWEC data) and a/contextual-driftview.🤖 Generated with Claude Code