Reproduce the dose-response + layer-sensitivity report card on small models (Qwen 1.5B/0.5B, T4)#17
Merged
Merged
Conversation
Add Qwen2.5-1.5B-Instruct (primary) + 0.5B-Instruct (secondary) to MODELS. Switch GPU A100 -> T4 (16GB fits <=1.5B, cheapest). Distinct GGUF_PATH stem so the small-model vector never overwrites the 7B shared gguf. Parameterize hardcoded plot labels: run_dose depth-fraction (was literal '0.50 depth') and GPU suptitle (was literal 'A100') now derive from actual layer/n_layers/GPU. run_layer_sweep gains model_id/n_layers/stem params and its title derives model + GPU instead of hardcoded 7B/A100.
Canonical transferable 7B dose (alpha 0.044) is too weak on 1.5B to resolve a layer-localized peak. Baseline formality 4.56; all layers land 4.3-4.9 within seed noise (+/-0.07-0.23). Repetition 0.03-0.09 and unsteered ppl ~2.7-4.5 everywhere -> far below any coherence cliff, large headroom. A weak broad back-half plateau (frac ~0.57-0.89) sits above baseline; L17/0.61 is inside it, not uniquely peaked. Kept as the documented below-floor point; the resolved sweep re-runs at a stronger, cliff-justified alpha (mapped by the L17 dose).
Layer 17 = 0.61*28 (the introspection-scaling depth anchor), resid_norm 352.5, 3 seeds, T4. Formality climbs monotonically through the coherent band (alpha -0.07..+0.11, coeff -25..+40: effect 3.05->5.06, rep<0.15, ppl<7). Coherence cliff at alpha ~0.13-0.17 (coeff 60: rep 0.34, ppl 15). Past-cliff (coeff 90/130) the formality proxy inflates on degenerate repetition (rep 0.66) and seed variance explodes (+/-2.42) -- classic effect-reversal artifact, not real steering. Negative doses go casual symmetrically, degenerate by alpha -0.11. Usable positive band ~alpha 0.02-0.11; sweet separation dose ~0.09.
Cliff-justified dose (0.09, from the L17 dose usable band) resolves the peak the 0.044 run could not. Coherent-gated max = layer 17 = frac 0.61 = the introspection-scaling depth anchor (matches the 7B result), formality 4.97+/-0.09, rep 0.066, ppl 5.13. It is a back-half PLATEAU, not a sharp point: L17-25 (frac 0.61-0.93) all land 4.82-4.97 within overlapping seed range -- argmax is 0.61 but L20/21/22/25 are statistically indistinguishable. Dead-spot (honest negative): early-mid layers frac ~0.25-0.54 (L7-15) are DISRUPTIVE at this dose -- formality drops below the 4.56 baseline (L13 3.38, L15 3.22) and coherence breaks (ppl spikes 6->875, seed variance +/-1-2). Injecting the concept mid-network damages the model rather than steering it. 3 seeds [0,1,2], T4, own-direction-per-layer.
RESULTS_small_model.md: layer sweep (0.044 below-floor + 0.09 resolved), dose-response at L17/0.61, usable band + cliff, dead-spot, 0.61 synergy note. reproduce.sh: one command trains vector + runs dose + both sweeps, prints spend. 0.5B section pending its runs.
resid_norm 64.5 (vs 352 on 1.5B) -> the shared coeff grid overshoots into collapse. Usable band alpha -0.12..+0.23 (coeff -8..+15): formality 3.06->5.24, rep<0.16, ppl<11. Sharp cliff at alpha ~0.23-0.39 (rep 0.16->0.55). Past it the 0.5B model COLLAPSES to near-empty output (alpha 0.62 formality 0.81 rep 0; alpha 0.93 formality 0.00+/-0.00) then garbage (alpha 1.4 ppl 347k). Far more fragile than 1.5B's graceful degradation. Strongest clean dose ~alpha 0.124.
baseline 4.53. Coherent concept-steering plateau frac ~0.58-0.83 (L14-20: effect 4.87-5.06, rep<0.10, ppl<6), peaking L15-16 = frac 0.62-0.67 (5.02-5.06). L15/0.62 sits on the 0.61 anchor. Noisier than 1.5B (seed std up to +/-0.7 mid-stack). Dead-spot frac 0.29-0.54 (L7-13): below/near baseline, ppl 6-15, rep spikes. HONEST caveat: the auto coherent-gated argmax is L23/frac 0.96 (5.77), but that is an OUTPUT-LAYER edge artifact -- last layer, coeff 33 (vs ~8 mid-stack) at fixed alpha, directly boosting formal-word logits, not concept steering. It is excluded from the concept-layer claim; the concept peak is the mid-0.6 plateau.
Promote RESULTS to repo-root RESULTS.md covering both scales: headline transfer table (0.5B frac 0.62-0.67, 1.5B frac 0.61, 7B 0.607 -> max-effect layer at frac ~0.61 across a 14x size range), per-model dose + layer curves with seeds[0,1,2]/T4/wall-clock, total GPU spend ~$0.70, and all honest negatives (dose-transfer floor, mid-network dead-spot, past-cliff artifact, 0.5B catastrophic collapse, plateau-not-point, 0.5B output-layer edge artifact). README: keep the real 7B hero (largest dynamic range, anchors the shape narrative) and add a note that the small-model run is the one-command in-repo reproducible companion via experiments/reproduce.sh. Add the 0.61-depth justification to the introspection-scaling paragraph.
Verified (not assumed): a 1-seed inspection at 0.5B L23/coeff 33 produced fluent, coherent, more-formal-than-baseline text -- so the earlier 'output-layer logit artifact' exclusion was wrong. The 0.5B coherent argmax genuinely IS the last layer (frac 0.96, 5.77); 0.61 is the START of its coherent back-half plateau, not its peak. (On 1.5B the last layers sit at/below baseline despite higher coeff, so this is a 0.5B-specific late climb, not a general high-coeff effect.) Reframe the headline to the honest, scrutiny-surviving claim: frac ~0.61 is a high-effect COHERENT injection depth at every scale -- the argmax on 1.5B and 7B, the plateau start on 0.5B -- which still justifies introspection-scaling's 0.61 without overclaiming a universal argmax. L23 now shown in the table with the verified reason inline. README softened to match. Also: parameterize smoke() with model_id/gguf_path (enabled the L23 inspection); fix reproduce.sh to set SWEEP_ALPHA per model (0.09 for 1.5b, 0.124 for 0.5b) so MODEL=0.5b reproduces the committed 0.124 artifact, not 0.09.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
A one-command, in-repo reproducible version of the steerbench report card on cheap small models, alongside the existing (untouched) 7B/8B/9B cross-model CSVs. Everything runs on a Modal serverless T4 (16 GB) instead of an A100.
Models:
Qwen/Qwen2.5-1.5B-Instruct(primary, 28 layers) andQwen/Qwen2.5-0.5B-Instruct(secondary, 24 layers). FormalityControlVectortrained via repeng's native path (69 contrastive pairs, one unit-norm direction per layer).What reproduces
Both small models reproduce the same interior-optimum dose-response shape as 7B — formality climbs with coefficient while perplexity holds, then falls off the coherence cliff — and both give the same layer-sensitivity story:
The 0.61 finding across scales
frac ≈ 0.61 of depthis a high-effect, coherent formality-injection layer on 0.5B, 1.5B and 7B — the coherent argmax on 1.5B (0.61) and 7B (0.607), and on the coherent back-half plateau on 0.5B. This measures, rather than assumes, the depth the companionintrospection-scalingproject injects/reads at, across a ~14× size range.Honest negatives (kept in RESULTS.md, not hidden)
Method / cost
Reproduce
One command per model; sets the per-model cliff-justified α (0.09 / 0.124) automatically, renders CSV+PNG into
results/, and prints the GPU spend.steer-reportthen builds the four-part card from the CSVs on CPU.