Readout-aligned dictionary geometry — build the Gram on the basis the model reads through (proposal) - #113
Closed
jascal wants to merge 3 commits into
Closed
Readout-aligned dictionary geometry — build the Gram on the basis the model reads through (proposal)#113jascal wants to merge 3 commits into
jascal wants to merge 3 commits into
Conversation
…asis the model reads through Polygram's geometry-tracks-behaviour claim (Spearman 0.637) is built on RAW decoder vectors (clustered.py:165 k-means input + centroid/variance/gamma-PCA), the "deceptively compressible" basis. The behaviourally-relevant object is the readout-aligned decision geometry — now a measured, model-general fact: powered R2 (tau_star_powered.py, 20k tokens, 3 seeds) shows a trained rank-r projection onto the readout-aligned subspace beats frozen SVD on open-class next-token recovery at every rung (GPT-2 +52pp, Pythia-70m +31pp, 160m +40pp). Proposes a `readout-aligned` geometric profile (new polygram/geometry/readout_aligned.py mirroring clustered.py) that projects decoder vectors onto the top-r SVD subspace of gain*U BEFORE clustering/knob assignment; from_sae_lens(u_matrix=, gain=) threads the readout geometry (required for the profile; caller supplies U/gain, no host load in core). Q-Orca emission transparent. Gate: head-to-head Spearman on GPT-2-small, WIN >= 0.70 vs 0.637, or NO-IMPROVEMENT (basis isn't the limit -> Reckoning #3 prune more likely). Proposal/ tasks/spec only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… (tasks 1-3) Implements the profile from the proposal + addresses the review's edge cases: - polygram/geometry/readout_aligned.py: ReadoutAlignedKnobAssignment + ReadoutAlignedFidelity + readout_aligned() factory. readout_subspace(U, gain, d_model, rank) = top-r right singular dirs of gain*U via np.linalg.svd(..., full_matrices=False)[2][:r]; r capped at min(rank, d_model, vocab) (so readout_rank > rank(gain*U) is safe); gain scalar->broadcast / vector / None->ones. The knob assignment projects (n_features, d_model) -> (n_features, r) then DELEGATES to the calibrated ClusteredKnobAssignment logic in readout space (k-means / beta / per-cluster-PCA gamma), and the fidelity's cosine-overlap target is computed in the same projected space (consistent). readout_rank is tunable (default 64). - readout_variance_captured(): the review's suggested diagnostic — fraction of decoder Frobenius energy in the top-r readout subspace (quantifies how readout-aligned the SAE features already are). - from_sae_lens(u_matrix=, gain=, readout_rank=): inject the caller-supplied host unembed into the registry placeholder strategy+fidelity via dataclasses.replace; u_matrix REQUIRED for the profile (ValueError else), 2-D validated. Polygram loads no host itself. Default (clustered) path byte-identical. - Registered in geometry/__init__.py. tests/test_readout_aligned_profile.py (10): registration, validation, readout!=raw, determinism, edge cases, variance diagnostic. 109 clustered/profile/uniform regression tests unaffected. Q-Orca emission transparent (consumes final knobs). Behavioural head-to-head gate (task 4) next. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…dout basis is wrong for co-firing
Ran the head-to-head behavioural gate (task 4). Extended behavioural_gram_scaleup.py with
--profile {clustered,readout-aligned} --readout-rank R (+ HF-cache SAE fallback, --sae-path), supplying
GPT-2's tied unembed + ln_f gain as u_matrix/gain.
RESULT (blocks.10, Spearman(Polygram, co-firing Jaccard)): clustered 0.640; readout r=64 0.267; r=256 0.067;
r=768(full) 0.640. Readout-alignment STRICTLY HURTS — full rank just recovers the baseline (a full-rank
readout projection is an orthonormal rotation and k-means is rotation-invariant), every truncation degrades.
Mechanism: co-firing is ENCODER-side (which features fire together, full-residual geometry); the readout
subspace is DECODE-side (argmax). Readout-alignment is right for the decode tax (R2 +52/+31/+40pp) but WRONG
for co-firing -> this VINDICATES the raw-decoder geometry here (0.640 is not a basis artifact; Reckoning #3's
basis-limited concern doesn't apply). The profile ships as a correct tool for decode-relevant geometry; a
decode-side behavioural metric is where it would help. Writeup in proposal "Gate RESULT" +
behavioural-scaleup-probe.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owner
Author
|
Closing — not shipping the readout-aligned profile. The behavioural head-to-head FALSIFIED its premise: |
jascal
added a commit
that referenced
this pull request
Jun 13, 2026
…alsified finding (#114) PR #113's readout-aligned geometry profile is not shipped — the behavioural head-to-head FALSIFIED its premise (readout-alignment HURTS the co-firing Spearman 0.640 -> 0.27; full rank just recovers baseline; co-firing is encoder-side, the readout subspace is decode-side -> vindicates the raw-decoder geometry). Archive the proposal + spec as a research record (spec delta NOT applied to canonical specs; implementation stays on the closed PR branch) and keep the head-to-head result in behavioural-scaleup-probe.md so the finding is on main. Co-authored-by: Allan Scott <allans@pop-os.lan> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposal only (OpenSpec
add-readout-aligned-geometry-profile). Turns FABLE direction X1 into a concrete change, now backed by hard cross-model evidence.Why
Polygram's load-bearing claim is behavioural —
Spearman(Polygram-overlap, co-firing Jaccard) = 0.637. But the geometry is built on raw decoder vectors (clustered.py:165k-means input + centroid/variance/γ-PCA), the "deceptively compressible" basis. The behaviourally relevant object is the readout-aligned decision geometry — and that's now a measured, model-general fact: powered R2 (tau_star_powered.py, 20k tokens, 3 seeds) shows a trained rank-rprojection onto the readout-aligned subspace beats frozen SVD on open-class next-token recovery at every rung — GPT-2 +52pp, Pythia-70m +31pp, Pythia-160m +40pp (all 6 model×rank cells). So fitting the Gram in the raw basis means 0.637 partly measures the basis, not the model.What
polygram/geometry/readout_aligned.py(ReadoutAlignedKnobAssignment/readout_aligned()mirroringclustered.py): project decoder vectors onto the top-rSVD subspace ofgain⊙Ubefore clustering + knob assignment. Registered alongsideclustered/uniform_sphere.from_sae_lens(u_matrix=, gain=)threads the readout geometry (required whenprofile="readout-aligned"; caller suppliesU/gain— no host load in the core, stays numpy-light).behavioural-scaleup-probe.md.Gate (descriptive, both ways)
Q-Orca emission is transparent (consumes final knobs, not projections).
clusteredstays the default; opt-in, byte-identical otherwise. Implementation pending review.🤖 Generated with Claude Code