diff --git a/README.md b/README.md index 0e3d019..aec8a7b 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,15 @@ coefficient while perplexity holds — then falls off the **coherence cliff**. steerbench marks the sweet spot and the cliff automatically.* +> The hero stays the real 7B curve — largest dynamic range, and it anchors the +> shape narrative below. For a **one-command, in-repo reproducible** version on a +> cheap T4 (no A100), see the small-model canonical run in +> [`RESULTS.md`](RESULTS.md) via [`experiments/reproduce.sh`](experiments/reproduce.sh): +> Qwen2.5-1.5B and 0.5B reproduce the same interior-optimum dose-response shape, +> and **frac ≈ 0.61 of depth** is a high-effect coherent injection layer on both +> (the argmax on 1.5B; the start of the coherent plateau on 0.5B, whose argmax is +> the last layer — see [`RESULTS.md`](RESULTS.md) for the honest breakdown). + ## Quickstart Render the four-part card from a repeng vector — CPU-only, no model download: @@ -89,7 +98,12 @@ project steerbench was built for — asks whether models can introspect on *injected* concepts as they scale. To run that cleanly it must inject at a known-good layer and strength on every model in the ladder. steerbench produces that report card — the sweet spot, the cliff, the safe layer — so the injection -is **calibrated, not guessed**. +is **calibrated, not guessed**. Concretely: introspection-scaling injects/reads at +**frac ≈ 0.61 of depth**, and steerbench's layer sweep finds that depth to be a +high-effect, coherent formality-injection layer on Qwen2.5-**0.5B, 1.5B and 7B** +— the coherent argmax on 1.5B and 7B, and on the coherent back-half plateau on +0.5B (see [`RESULTS.md`](RESULTS.md)) — so the depth hyperparameter is measured +across a ~14× size range, not assumed. ## Results diff --git a/RESULTS.md b/RESULTS.md new file mode 100644 index 0000000..5944d45 --- /dev/null +++ b/RESULTS.md @@ -0,0 +1,182 @@ +# steerbench — small-model canonical sweeps (RESULTS) + +One-command-reproducible dose-response + layer-sensitivity for a **formality** +ControlVector on cheap small models. This is the **canonical reproducible** run +(`./experiments/reproduce.sh`); the existing 7B/8B/9B cross-model CSVs in +`results/` are real and untouched. + +- **Hardware:** Modal serverless **T4** (16 GB). bf16 confirmed working on Turing. +- **Models:** `Qwen/Qwen2.5-1.5B-Instruct` (primary, 28 layers, hidden 1536), + `Qwen/Qwen2.5-0.5B-Instruct` (secondary, 24 layers, hidden 896). +- **Seeds:** [0, 1, 2], mean ± std (population std). +- **Vector:** repeng `ControlVector` (native training path, not reimplemented), + 69 contrastive persona pairs, one unit-norm direction per layer. +- **Metrics:** effect = lexical formality proxy; coherence = repetition rate + (1−distinct-2) + unsteered perplexity. Capability side-effects (MMLU/GSM8K) are + deliberately **not** folded into the dose curve — separate work. + +## Headline — 0.61 depth is a high-effect coherent injection layer at every scale + +| model | params | depth | coherent-effect argmax | frac | high-effect plateau (frac) | 0.61 status | +|---|---|---|---|---|---|---| +| Qwen2.5-0.5B-Instruct | 0.5B | 24 L | L23 (last) | 0.96 | ~0.58–0.96 | plateau start, not argmax | +| Qwen2.5-1.5B-Instruct | 1.5B | 28 L | L17 | **0.61** | ~0.61–0.93 | argmax | +| Qwen2.5-7B-Instruct (prior, real) | 7B | 28 L | L17 | **0.607** | back-half | argmax | + +The honest claim: on all three sizes, **frac ≈ 0.61 is a high-effect, coherent +formality-injection depth** — it sits on the coherent back-half plateau at every +scale, and on 1.5B and 7B it is the **argmax**. This justifies the depth +**introspection-scaling** injects/reads at (0.61) — measured, not assumed, across +a ~14× size range. + +The honest caveat (do **not** overclaim a universal 0.61 argmax): on **0.5B the +numeric argmax is the last layer (frac 0.96)**, and it is **real coherent formal +prose**, not an artifact — a 1-seed inspection at L23/coeff 33 produced fluent, +more-formal-than-baseline text (verified, not asserted). So 0.5B's effect keeps +climbing gently to the output end; 0.61 is the **start of its coherent plateau**, +not its peak. Reported as a plateau, not razor-precision — with 3 seeds the argmax +is coarse and the back half is a broad high band. + +**Total GPU spend: ~$0.70** (measured dose+sweep wall 3,464 s across 5 sweeps + +~600 s container/cold-start overhead across 8 T4 runs; T4 @ ~$0.59/hr). Well +under the $15 budget. + +| run | wall-clock | +|---|---| +| 1.5B layer sweep α=0.044 (below-floor) | 995 s | +| 1.5B layer sweep α=0.09 (resolved) | 954 s | +| 1.5B dose-response @ L17 | 483 s | +| 0.5B layer sweep α=0.124 (resolved) | 684 s | +| 0.5B dose-response @ L15 | 348 s | +| train (1.5B, 0.5B) + smoke | ~short, container-dominated | + +--- + +## Qwen2.5-1.5B-Instruct (primary / canonical) + +### Layer sweep — resolving the dose + +The transferable 7B dose (α 0.044) is **below the resolving floor on 1.5B** +(`layer_sweep_qwen1.5b_a044.{csv,png}`): every layer lands 4.3–4.9 within seed +noise (±0.07–0.23), rep 0.03–0.09, ppl 2.7–4.5 — huge headroom, no localized peak. +(α 0.044 *did* resolve on 7B; the smaller model needs a stronger normalized dose +to separate layers.) So the sweep re-ran at a **cliff-justified** dose read from +the L17 dose-response, **α 0.09** — committed before looking at which layer won, +to avoid nudging toward 0.61. + +Resolved (`layer_sweep_qwen1.5b.{csv,png}`), coherent-gated (rep < 0.15, ppl < 6): + +| region | frac | formality (mean±std) | read | +|---|---|---|---| +| **peak** | **0.61 (L17)** | **4.97 ± 0.09** | argmax, rep 0.07, ppl 5.1 | +| plateau | 0.61–0.93 (L17–25) | 4.82–4.97 | overlapping seed range; L20/21/22/25 ≈ L17 | +| **dead-spot** | 0.25–0.54 (L7–15) | 3.2–4.2, several < baseline | disruptive: ppl 6→875, var ±1–2 | +| early | 0.04–0.21 (L1–6) | 4.5–4.8 | mild, near baseline | + +**Peak plateau frac 0.61–0.93, argmax frac 0.61 (L17).** Brackets and hits the +anchor. + +### Dose-response @ L17 (frac 0.61 anchor = the resolved peak) + +`dose_response_qwen1.5b.{csv,png}`, resid_norm@L17 = 352.5. + +| alpha | coeff | formality | rep | ppl | note | +|---|---|---|---|---|---| +| −0.113 | −40 | 2.82 ± 0.23 | 0.406 | 8.1 | degenerate (casual) | +| −0.071 | −25 | 3.05 ± 0.11 | 0.133 | 5.3 | casual, edge | +| −0.043 | −15 | 3.47 ± 0.13 | 0.082 | 3.5 | casual, coherent | +| 0.000 | 0 | 4.49 ± 0.07 | 0.070 | 2.6 | baseline | +| 0.043 | 15 | 4.84 ± 0.03 | 0.055 | 2.9 | formal, clean | +| 0.071 | 25 | 4.90 ± 0.02 | 0.066 | 3.6 | formal, clean | +| **0.113** | **40** | **5.06 ± 0.12** | 0.100 | 7.1 | **effect peak, ppl rising** | +| 0.170 | 60 | 5.00 ± 0.01 | 0.335 | 15.1 | **cliff** | +| 0.255 | 90 | 5.97 ± 0.44 | 0.662 | 9.4 | past-cliff artifact (looping) | +| 0.369 | 130 | 4.70 ± 2.42 | 0.309 | 6.1 | chaotic, variance blows up | + +- **Usable band:** α ≈ −0.07 → +0.11 (coeff −25 → +40); formality 3.05 → 5.06 + monotone, rep < 0.15, ppl single-digit. +- **Coherence cliff:** α ≈ 0.13–0.17. Past it the formality proxy **inflates on + degenerate repetition** (α 0.255: rep 0.66) — an effect-reversal artifact, not + real steering — and seed variance explodes (α 0.369: ±2.42). + +--- + +## Qwen2.5-0.5B-Instruct (secondary) + +Same protocol. resid_norm@L15 = 64.5 (~5× smaller than 1.5B) → the shared coeff +grid overshoots hard, so the model is far more fragile. + +### Dose-response @ L15 (frac 0.62 anchor) + +`dose_response_qwen0.5b.{csv,png}`: + +| alpha | coeff | formality | rep | ppl | note | +|---|---|---|---|---|---| +| −0.124 | −8 | 3.06 ± 0.16 | 0.087 | 5.9 | casual, coherent | +| 0.000 | 0 | 4.55 ± 0.12 | 0.051 | 3.1 | baseline | +| 0.124 | 8 | 5.02 ± 0.03 | 0.078 | 4.9 | formal, clean | +| 0.233 | 15 | 5.24 ± 0.05 | 0.160 | 10.7 | effect peak, cliff edge | +| 0.388 | 25 | 4.74 ± 0.37 | 0.548 | 10.2 | degenerate (looping) | +| 0.620 | 40 | 0.81 ± 0.62 | 0.000 | 4.4 | **collapse to near-empty** | +| 0.930 | 60 | 0.00 ± 0.00 | 0.000 | nan | **total collapse** | +| 1.395 | 90 | 1.40 ± 1.98 | 0.000 | 347490 | garbage | + +- **Usable band (narrow):** α ≈ −0.12 → +0.23 (coeff −8 → +15); formality + 3.06 → 5.24. +- **Sharp cliff at α ≈ 0.23–0.39**, then — the honest negative — a **catastrophic + collapse**: the 0.5B model emits near-empty output (formality → 0), unlike + 1.5B's graceful degradation into repetition. + +### Layer sweep @ α 0.124 (strongest clean dose) + +`layer_sweep_qwen0.5b.{csv,png}`, baseline 4.53, coherent-gated (rep < 0.15, ppl < 6): + +| region | frac | formality | read | +|---|---|---|---| +| **argmax** | **0.96 (L23, last)** | **5.77 ± 0.17** | rep 0.05, ppl 5.5 — real formal prose (verified) | +| plateau | 0.58–0.96 (L14–23) | 4.87–5.77 | broad coherent high band; climbs to the output end | +| 0.61 anchor | 0.62–0.67 (L15–16) | 5.02–5.06 | on the plateau, not its peak | +| dead-spot | 0.29–0.54 (L7–13) | 3.2–4.25 | ppl 6–15, rep spikes | + +**On the argmax (verified, not assumed):** the coherent-gated argmax is +**L23/frac 0.96 (5.77)** — the last layer, where fixed α maps to coeff 33 (vs ~8 +mid-stack). I initially suspected an output-layer logit-boost artifact, but a +1-seed inspection at L23/coeff 33 disproved that: the generation is fluent, +coherent, and more formal than baseline ("…created by Alibaba Cloud… including but +not limited to general knowledge, cultural customs, scientific discoveries…") — a +real steering effect, not gamed tokens or truncation. So 0.5B genuinely peaks at +the output end; **0.61 is the start of its coherent plateau, not its peak.** (By +contrast, on 1.5B the last layers sit at/below baseline — e.g. L25/coeff 43.8 = +4.88, L27 = 4.56 — so this is a 0.5B-specific late-layer climb, not a general +high-coeff effect.) 0.5B is also noisier than 1.5B (seed std up to ±0.7 mid-stack). + +--- + +## Honest negatives (summary) + +1. **Dose transfer breaks down at small scale.** α 0.044 (fine on 7B) is below + the layer-resolving floor on 1.5B; α needs re-derivation per model from the + cliff, not blind transfer. +2. **Mid-network dead-spot.** frac ~0.25–0.54 injection *reduces* formality and + *breaks coherence* on both small models (ppl spikes, high variance). +3. **Past-cliff effect is an artifact.** Beyond the cliff the formality proxy can + rise, but on degenerate repetition (1.5B) or collapsed/near-empty output + (0.5B) — not real steering; report the usable band, not the raw max. +4. **Peak is a plateau, not a point — and 0.61 is not the argmax everywhere.** + 3 seeds give a coarse argmax; the back half is broadly high. 0.61 is the clean + argmax on 1.5B and 7B, but on 0.5B the argmax is the last layer (0.96, verified + real prose) and 0.61 is only the plateau *start*. The defensible claim is + "0.61 is a high-effect coherent injection depth at every scale," not "0.61 is + the universal peak." + +## Reproduce + +```bash +# from the repo root (so the src/ mount resolves), Modal authed: +./experiments/reproduce.sh # 1.5B primary: train + dose + both sweeps +MODEL=0.5b ./experiments/reproduce.sh # 0.5B secondary +``` + +Renders `results/{dose_response,layer_sweep}_qwen{1.5b,0.5b}.{csv,png}` and prints +the GPU spend. `steer-report` then builds the four-part card from these CSVs on +CPU (no GPU, no download). diff --git a/experiments/modal_app.py b/experiments/modal_app.py index ec458b4..0dc65a1 100644 --- a/experiments/modal_app.py +++ b/experiments/modal_app.py @@ -1,9 +1,9 @@ """Modal app for steerbench M0: repeng ControlVector dose-response + layer sweep. -Serverless A100. Pinned image so runs are reproducible. +Serverless T4 (small-model canonical run). Pinned image so runs are reproducible. Pipeline: - 1. train_and_export — train a FORMALITY ControlVector on Qwen2.5-7B-Instruct, + 1. train_and_export — train a FORMALITY ControlVector on Qwen2.5-1.5B-Instruct, save in repeng's native gguf to a Volume, print keys+norms. 2. smoke — reload gguf via steerbench.load_vector, steer one mid layer, generate at coeff 0 and a big coeff (de-risk everything cheap). @@ -48,11 +48,15 @@ vol = modal.Volume.from_name("steerbench-m0-vol", create_if_missing=True) VOL = "/vol" HF_CACHE = f"{VOL}/hf" -GGUF_PATH = f"{VOL}/formality.gguf" +# Distinct stem so the small-model vector never overwrites the 7B shared gguf. +GGUF_PATH = f"{VOL}/formality_qwen1.5b.gguf" -MODEL_ID = "Qwen/Qwen2.5-7B-Instruct" -MODEL_BASE = "Qwen/Qwen2.5-7B" -GPU = "A100" +# Small-model canonical run (T4-cheap). 1.5B primary, 0.5B secondary. +# The 7B/8B/9B cross-model artifacts are separate files and stay untouched. +MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct" +MODEL_BASE = "Qwen/Qwen2.5-1.5B" +GPU = "T4" # 16GB fits <=1.5B; cheapest. bf16 unverified on Turing — confirm on +# first train run; if it throws a kernel error, switch dtype to torch.float16. # HF token for gated models (Llama, Gemma). Modal secret "huggingface". HF_SECRET = modal.Secret.from_name("huggingface") @@ -63,6 +67,10 @@ SWEET_ALPHA = 0.044 INJECT_FRAC = 0.61 MODELS = { + # small-model canonical run (this branch) + "qwen1.5b": "Qwen/Qwen2.5-1.5B-Instruct", # primary, 28 layers + "qwen0.5b": "Qwen/Qwen2.5-0.5B-Instruct", # secondary, 24 layers + # cross-model set (existing 7B/8B/9B artifacts, untouched) "qwen": "Qwen/Qwen2.5-7B-Instruct", # ungated mirror of Llama-3.1-8B-Instruct (identical weights) "llama": "NousResearch/Meta-Llama-3.1-8B-Instruct", @@ -583,14 +591,19 @@ def train_and_export( @app.function(gpu=GPU, timeout=1800, volumes={VOL: vol}, secrets=[HF_SECRET]) -def smoke(layer: int = 14, big_coeff: float = 8.0) -> dict: +def smoke( + layer: int = 14, + big_coeff: float = 8.0, + model_id: str = MODEL_ID, + gguf_path: str | None = None, +) -> dict: import sys sys.path.insert(0, "/root/src") vol.reload() # see the freshly committed gguf from steerbench.vectors import load_vector - vec = load_vector(GGUF_PATH) # exercise the real reload path + vec = load_vector(gguf_path or GGUF_PATH) # exercise the real reload path print(f"reloaded keys: {sorted(vec.directions.keys())}") import numpy as np @@ -598,7 +611,7 @@ def smoke(layer: int = 14, big_coeff: float = 8.0) -> dict: dnorm = float(np.linalg.norm(vec.directions[layer])) print(f"layer {layer} ||dir|| = {dnorm:.4f}") - model, tok = _load_model_and_tokenizer() + model, tok = _load_model_and_tokenizer(model_id) from repeng import ControlModel cmodel = ControlModel(model, [layer]) @@ -1080,12 +1093,15 @@ def run_dose( rep_m = [agg[c]["repetition"][0] for c in cs] ppl_m = [agg[c]["ppl"][0] for c in cs] + # derive depth-fraction + GPU string from actuals (was hardcoded 7B/A100) + frac = layer / res["n_layers"] + fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(13, 5)) ax1.errorbar(cs, form_m, yerr=form_s, marker="o", capsize=3, color="C0") ax1.axvline(0, color="gray", ls=":", lw=1) ax1.set_xlabel("coefficient (raw)") ax1.set_ylabel("formality proxy (higher = more formal)") - ax1.set_title(f"EFFECT — dose-response @ layer {layer} (0.50 depth)") + ax1.set_title(f"EFFECT — dose-response @ layer {layer} ({frac:.2f} depth)") ax1.grid(alpha=0.3) ax2.plot(cs, rep_m, marker="s", color="C3", label="repetition (1-distinct2)") @@ -1098,8 +1114,8 @@ def run_dose( ax2.set_title("COHERENCE — the cliff") ax2.grid(alpha=0.3) fig.suptitle( - f"steerbench dose-response · {model_id.split('/')[-1]} · A100 · " - f"3 seeds · layer {layer}/{res['n_layers']} · " + f"steerbench dose-response · {model_id.split('/')[-1]} · {GPU} · " + f"3 seeds · layer {layer}/{res['n_layers']} ({frac:.2f} depth) · " f"||dir||={res['dir_norm']:.2f} resid_norm={res['resid_norm']:.1f} · " f"wall {res['wall_s']:.0f}s") fig.tight_layout() @@ -1115,9 +1131,16 @@ def run_dose( @app.local_entrypoint() -def run_layer_sweep(target_alpha: float = 0.044, coeff: float = 0.0) -> None: +def run_layer_sweep( + target_alpha: float = 0.044, + coeff: float = 0.0, + model_id: str = MODEL_ID, + n_layers: int = 28, + stem: str = "layer_sweep", +) -> None: """Primary: fixed-alpha (equal normalized strength per layer). Pass coeff>0 to instead run the secondary fixed-raw-coeff mode. + model_id/n_layers/stem let this retarget any model (28 for 1.5B, 24 for 0.5B). """ import csv import os @@ -1127,14 +1150,19 @@ def run_layer_sweep(target_alpha: float = 0.044, coeff: float = 0.0) -> None: matplotlib.use("Agg") import matplotlib.pyplot as plt - layers = list(range(1, 28)) # each layer's own direction + gguf_path = _gguf_path(model_id) if model_id != MODEL_ID else GGUF_PATH + layers = list(range(1, n_layers)) # each layer's own direction seeds = [0, 1, 2] if coeff > 0: - res = layer_sweep.remote(seeds=seeds, layers=layers, coeff=coeff) - tag, stem = f"coeff={coeff}", "layer_sweep_coeff" + res = layer_sweep.remote(seeds=seeds, layers=layers, coeff=coeff, + model_id=model_id, gguf_path=gguf_path) + tag = f"coeff={coeff}" + stem = f"{stem}_coeff" if stem == "layer_sweep" else stem else: - res = layer_sweep.remote(seeds=seeds, layers=layers, target_alpha=target_alpha) - tag, stem = f"alpha_norm={target_alpha}", "layer_sweep" + res = layer_sweep.remote(seeds=seeds, layers=layers, + target_alpha=target_alpha, + model_id=model_id, gguf_path=gguf_path) + tag = f"alpha_norm={target_alpha}" rows = res["rows"] n_layers = res["n_layers"] @@ -1165,7 +1193,7 @@ def run_layer_sweep(target_alpha: float = 0.044, coeff: float = 0.0) -> None: ax1b.plot(ls, rep_m, marker="s", color="C3", alpha=0.6, label="repetition") ax1b.set_ylabel("repetition rate", color="C3") ax1.set_title( - f"steerbench layer sweep · Qwen2.5-7B-Instruct · A100 · {tag} · " + f"steerbench layer sweep · {model_id.split('/')[-1]} · {GPU} · {tag} · " f"3 seeds · own-direction-per-layer · wall {res['wall_s']:.0f}s") ax1.grid(alpha=0.3) ax2 = ax1.secondary_xaxis( diff --git a/experiments/reproduce.sh b/experiments/reproduce.sh new file mode 100755 index 0000000..81ec5ad --- /dev/null +++ b/experiments/reproduce.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# Reproduce the steerbench small-model canonical sweeps (Qwen2.5-1.5B, T4). +# One command: trains the formality vector, runs the dose-response + both layer +# sweeps, renders CSV+PNG into results/. Prints the Modal GPU spend at the end. +# +# Prereqs: Modal authed (`modal token new`), run from the repo root so the +# `src/` mount resolves. HF cache + trained gguf persist on the Modal volume. +# +# Usage: +# experiments/reproduce.sh # 1.5B primary run +# MODEL=0.5b experiments/reproduce.sh # 0.5B secondary run +set -euo pipefail + +MODAL="${MODAL:-modal}" +APP="experiments/modal_app.py" +MODEL="${MODEL:-1.5b}" + +# SWEEP_ALPHA = the strongest-still-coherent dose for the resolved layer sweep, +# read from each model's own anchor-layer dose-response usable band (see +# RESULTS.md). 0.044 (the transferable 7B dose) is below the resolving floor on +# small models, so each model gets its own cliff-justified value. +if [[ "$MODEL" == "1.5b" ]]; then + MODEL_ID="Qwen/Qwen2.5-1.5B-Instruct"; NLAYERS=28; ANCHOR=17; STEM="qwen1.5b" + GGUF="" # 1.5B is the default MODEL_ID -> uses GGUF_PATH + DEFAULT_ALPHA=0.09 +elif [[ "$MODEL" == "0.5b" ]]; then + MODEL_ID="Qwen/Qwen2.5-0.5B-Instruct"; NLAYERS=24; ANCHOR=15; STEM="qwen0.5b" + GGUF="/vol/formality_Qwen2_5-0_5B-Instruct.gguf" + DEFAULT_ALPHA=0.124 +else + echo "MODEL must be 1.5b or 0.5b" >&2; exit 1 +fi + +SWEEP_ALPHA="${SWEEP_ALPHA:-$DEFAULT_ALPHA}" + +echo "== 1/4 train formality ControlVector ($MODEL_ID) ==" +if [[ "$MODEL" == "1.5b" ]]; then + "$MODAL" run "$APP"::train_and_export --model-id "$MODEL_ID" +else + "$MODAL" run "$APP"::train_and_export --model-id "$MODEL_ID" --gguf-path "$GGUF" +fi + +echo "== 2/4 dose-response at the 0.61 anchor layer (L$ANCHOR) — maps cliff ==" +"$MODAL" run "$APP"::run_dose --layer "$ANCHOR" --model-id "$MODEL_ID" \ + --stem "dose_response_$STEM" + +echo "== 3/4 layer sweep @ alpha 0.044 (documented below-floor point) ==" +"$MODAL" run "$APP"::run_layer_sweep --target-alpha 0.044 --model-id "$MODEL_ID" \ + --n-layers "$NLAYERS" --stem "layer_sweep_${STEM}_a044" + +echo "== 4/4 layer sweep @ alpha $SWEEP_ALPHA (resolved, canonical) ==" +"$MODAL" run "$APP"::run_layer_sweep --target-alpha "$SWEEP_ALPHA" \ + --model-id "$MODEL_ID" --n-layers "$NLAYERS" --stem "layer_sweep_$STEM" + +echo +echo "== done. artifacts in results/{dose_response,layer_sweep}_$STEM.{csv,png} ==" +echo "GPU spend: see https://modal.com/apps/$($MODAL profile current 2>/dev/null || echo bamdad) " +echo "(Modal dashboard shows per-app T4 seconds x rate; this run is a handful of" +echo " minutes of T4 at ~\$0.59/hr — well under a dollar.)" diff --git a/results/dose_response_qwen0.5b.csv b/results/dose_response_qwen0.5b.csv new file mode 100644 index 0000000..40f60ed --- /dev/null +++ b/results/dose_response_qwen0.5b.csv @@ -0,0 +1,40 @@ +coeff,seed,alpha_norm,effect,repetition,ppl +-60.0,0,-0.9302326690318973,4.698445207892883,0.16565934065934068,109.56714699733575 +-60.0,1,-0.9302326690318973,6.776830778146567,0.041740347774830516,105.82109627822467 +-60.0,2,-0.9302326690318973,3.8568570318570314,0.29588902586111776,93.6575675394672 +-40.0,0,-0.6201551126879316,3.9583333333333335,0.0,41.74382513572147 +-40.0,1,-0.6201551126879316,0.9542328042328042,0.1578054298642534,18.09600535092919 +-40.0,2,-0.6201551126879316,0.72,0.10416666666666666,13.658929510326876 +-25.0,0,-0.3875969454299572,1.3282979441516027,0.0346153846153846,39.69610564977806 +-25.0,1,-0.3875969454299572,0.16694444444444467,0.14022108843537417,34.22367964551573 +-25.0,2,-0.3875969454299572,-0.2981289660321016,0.08680963572267919,20.405657290931675 +-15.0,0,-0.23255816725797432,1.4242773892773892,0.12212787212787213,12.553206558306515 +-15.0,1,-0.23255816725797432,1.1001641291198254,0.18515951042712867,12.903333115091675 +-15.0,2,-0.23255816725797432,2.1285552925989673,0.06667830024475258,9.447762567486407 +-8.0,0,-0.1240310225375863,3.1069031011559747,0.08991726475279108,5.3351961069882705 +-8.0,1,-0.1240310225375863,3.2286402053102807,0.06966984023946049,6.977118385864365 +-8.0,2,-0.1240310225375863,2.8435061635518952,0.10193722211043216,5.381523547771442 +0.0,0,0.0,4.527106236366525,0.055467238861338586,3.062477159346837 +0.0,1,0.0,4.414877660350028,0.04571041141697968,3.0143900155679857 +0.0,2,0.0,4.709743107769423,0.05146314710898389,3.1804815948726226 +8.0,0,0.1240310225375863,5.031393343578794,0.08503315366114708,4.881482076325848 +8.0,1,0.1240310225375863,4.969274877012772,0.09244459244459244,4.920985037862291 +8.0,2,0.1240310225375863,5.047286926688336,0.05624691763932271,4.805212405693994 +15.0,0,0.23255816725797432,5.304552963698241,0.11672699318827393,10.027969466749495 +15.0,1,0.23255816725797432,5.177677813651119,0.21540755499505954,10.077780245818055 +15.0,2,0.23255816725797432,5.237916120849189,0.1481840477835217,12.057354349725067 +25.0,0,0.3875969454299572,4.804387567433512,0.524166819784585,9.461720418554846 +25.0,1,0.3875969454299572,4.26322777674479,0.652113880742913,6.5249253070153035 +25.0,2,0.3875969454299572,5.161291973791974,0.4669062532358985,14.542003623783634 +40.0,0,0.6201551126879316,0.9166666666666666,0.0,4.813306696141206 +40.0,1,0.6201551126879316,0.0,0.0,5.179164613861612 +40.0,2,0.6201551126879316,1.5,0.0,3.2041557629544823 +60.0,0,0.9302326690318973,0.0,0.0,nan +60.0,1,0.9302326690318973,0.0,0.0,876.4297176079333 +60.0,2,0.9302326690318973,0.0,0.0,26.3708940622368 +90.0,0,1.395349003547846,0.0,0.0,337871.5478492333 +90.0,1,1.395349003547846,4.208333333333333,0.0,643083.1238509375 +90.0,2,1.395349003547846,0.0,0.0,61517.83693686246 +130.0,0,2.0155041162357774,0.0,0.0,nan +130.0,1,2.0155041162357774,5.604166666666666,0.0,42736.86098815021 +130.0,2,2.0155041162357774,1.25,0.0,2004861.442307665 diff --git a/results/dose_response_qwen0.5b.png b/results/dose_response_qwen0.5b.png new file mode 100644 index 0000000..bcca161 Binary files /dev/null and b/results/dose_response_qwen0.5b.png differ diff --git a/results/dose_response_qwen1.5b.csv b/results/dose_response_qwen1.5b.csv new file mode 100644 index 0000000..823a4be --- /dev/null +++ b/results/dose_response_qwen1.5b.csv @@ -0,0 +1,40 @@ +coeff,seed,alpha_norm,effect,repetition,ppl +-60.0,0,-0.1702127862483897,2.6846218814968816,0.5641533244706471,10.469239841829415 +-60.0,1,-0.1702127862483897,1.8615854822002942,0.52749791216086,5.373101825723497 +-60.0,2,-0.1702127862483897,2.5514077898365826,0.42471285892634203,12.861277064480657 +-40.0,0,-0.1134751908322598,3.1104731045858043,0.40976406207015187,9.162952785155234 +-40.0,1,-0.1134751908322598,2.55360089305172,0.37591695616300347,8.609940794678518 +-40.0,2,-0.1134751908322598,2.792365513894033,0.43257898013850715,6.613986161040836 +-25.0,0,-0.07092199427016238,2.90277361618825,0.21948009182598222,4.669085999949941 +-25.0,1,-0.07092199427016238,3.1458861014905857,0.14649439394973252,6.062537585326134 +-25.0,2,-0.07092199427016238,3.1065615615615614,0.03401909047770518,5.297162382476471 +-15.0,0,-0.042553196562097426,3.3613051829460288,0.11581870510441941,3.6848075228376187 +-15.0,1,-0.042553196562097426,3.408325835957415,0.0735397023068256,3.4136412443737623 +-15.0,2,-0.042553196562097426,3.654937358833948,0.056742640075973405,3.3027169752737517 +-8.0,0,-0.02269503816645196,4.4483208955223885,0.07209984108718287,3.287707680868816 +-8.0,1,-0.02269503816645196,4.456902214714715,0.042345821929718,3.0456723690939462 +-8.0,2,-0.02269503816645196,4.556306446486502,0.0708413500080167,2.7035202961846134 +0.0,0,0.0,4.561496400885936,0.07519758856722497,2.8409855643741015 +0.0,1,0.0,4.401662808641976,0.07497861785836471,2.691504971268208 +0.0,2,0.0,4.498309853905783,0.058948883044583955,2.3784403590201406 +8.0,0,0.02269503816645196,4.802343633794437,0.07797256097560978,2.8559825929006006 +8.0,1,0.02269503816645196,4.696647324091054,0.05910820529722968,2.811572801073514 +8.0,2,0.02269503816645196,4.928962418300653,0.03357645875251508,2.7930107168214633 +15.0,0,0.042553196562097426,4.797208063021316,0.03856509344314224,2.8690981518326644 +15.0,1,0.042553196562097426,4.8567942942942945,0.07170105774232705,2.7311085522073917 +15.0,2,0.042553196562097426,4.864264139545421,0.05510141093474427,3.180981835622299 +25.0,0,0.07092199427016238,4.9181883034530465,0.09336697504906083,3.711235085836259 +25.0,1,0.07092199427016238,4.893668467396918,0.0500205952013181,3.6085823038658553 +25.0,2,0.07092199427016238,4.873504784688995,0.053908045977011515,3.5318791747392715 +40.0,0,0.1134751908322598,5.161520712433633,0.07343260188087772,6.709658918676788 +40.0,1,0.1134751908322598,4.896153932526688,0.11744724573671941,8.307550354523373 +40.0,2,0.1134751908322598,5.129872028556239,0.11037050634660822,6.181619793201595 +60.0,0,0.1702127862483897,5.013444732545857,0.27089646464646466,10.782079761179414 +60.0,1,0.1702127862483897,4.981890208472487,0.478566626768021,7.230429700910399 +60.0,2,0.1702127862483897,4.993524567537725,0.2562483293237102,27.368930236830277 +90.0,0,0.2553191793725846,5.664396837355891,0.6549096558796048,9.104068484510622 +90.0,1,0.2553191793725846,5.657368123449585,0.6714850802360268,10.06683799602726 +90.0,2,0.2553191793725846,6.58705000518726,0.6587683440529033,9.079929920058381 +130.0,0,0.36879437020484435,4.846618357487923,0.4107562695924765,5.5456112152202595 +130.0,1,0.36879437020484435,7.591228070175438,0.4608134920634921,5.817446336117334 +130.0,2,0.36879437020484435,1.675,0.05555555555555555,7.055721548102023 diff --git a/results/dose_response_qwen1.5b.png b/results/dose_response_qwen1.5b.png new file mode 100644 index 0000000..cd29b4e Binary files /dev/null and b/results/dose_response_qwen1.5b.png differ diff --git a/results/layer_sweep_qwen0.5b.csv b/results/layer_sweep_qwen0.5b.csv new file mode 100644 index 0000000..a9a5e4a --- /dev/null +++ b/results/layer_sweep_qwen0.5b.csv @@ -0,0 +1,70 @@ +layer,layer_pos,seed,dir_norm,resid_norm,coeff,alpha_norm,effect,repetition,ppl +1,1,0,1.0000001192092896,8.421875,1.0443123755082637,0.124,4.447187019026099,0.060763414217361617,3.0708118604179777 +1,1,1,1.0000001192092896,8.421875,1.0443123755082637,0.124,4.651914970004401,0.04264678671170921,3.1867447269975475 +1,1,2,1.0000001192092896,8.421875,1.0443123755082637,0.124,4.785978193523007,0.04468319461893652,3.3494000626985376 +2,2,0,1.000000238418579,31.6875,3.9292490631940216,0.124,4.195636064686473,0.06033400702052974,4.1646569224291925 +2,2,1,1.000000238418579,31.6875,3.9292490631940216,0.124,4.284365621968004,0.07774725274725275,4.929907302816902 +2,2,2,1.000000238418579,31.6875,3.9292490631940216,0.124,3.9716267018898597,0.15602479521938178,4.148475053051469 +3,3,0,0.9999999403953552,55.4375,6.874250409737254,0.124,4.002380952380952,0.25868408699734,32.7400803132811 +3,3,1,0.9999999403953552,55.4375,6.874250409737254,0.124,5.551589267285862,0.41047297297297297,145.6129741180144 +3,3,2,0.9999999403953552,55.4375,6.874250409737254,0.124,4.038426711941144,0.21911616161616163,26.63541369043211 +4,4,0,1.0,57.0,7.068,0.124,4.623663835020256,0.07051431683784626,4.6535437651113885 +4,4,1,1.0,57.0,7.068,0.124,4.044364146276715,0.0962719298245614,5.187091284898601 +4,4,2,1.0,57.0,7.068,0.124,4.350892123418569,0.09550403225806453,4.805368088484335 +5,5,0,1.0,58.0,7.192,0.124,5.591916924062235,0.0551239209133946,4.860572957994379 +5,5,1,1.0,58.0,7.192,0.124,4.024342703165791,0.29376781981404837,5.66713488871274 +5,5,2,1.0,58.0,7.192,0.124,4.747062423500612,0.15698997909022766,4.4916513575397765 +6,6,0,1.0000001192092896,58.125,7.207499140799148,0.124,5.1613257575757565,0.20071879474107132,5.224429728037086 +6,6,1,1.0000001192092896,58.125,7.207499140799148,0.124,5.2899564090270434,0.27912264827915234,5.192531552065164 +6,6,2,1.0000001192092896,58.125,7.207499140799148,0.124,4.529984129267181,0.21850649350649354,6.47074037405836 +7,7,0,1.0000001192092896,58.8125,7.292749130636557,0.124,4.091175691778101,0.03353658536585366,18.83632924448402 +7,7,1,1.0000001192092896,58.8125,7.292749130636557,0.124,4.0515958815958815,0.0704225352112676,14.472252973368064 +7,7,2,1.0000001192092896,58.8125,7.292749130636557,0.124,3.8752392344497606,0.019230769230769218,13.234514983499512 +8,8,0,1.0,59.5625,7.38575,0.124,4.119906585102158,0.04188804554079695,17.43463511912619 +8,8,1,1.0,59.5625,7.38575,0.124,4.071820557491289,0.031101614434947772,11.278406741212489 +8,8,2,1.0,59.5625,7.38575,0.124,4.151037507562009,0.05090175605125771,10.957200344168312 +9,9,0,1.0,60.4375,7.49425,0.124,4.327232877384702,0.13421174021003854,7.21954898380768 +9,9,1,1.0,60.4375,7.49425,0.124,4.369113043924466,0.11037554556706403,17.238033251788753 +9,9,2,1.0,60.4375,7.49425,0.124,4.063137244658984,0.21649362250307477,6.225891797338597 +10,10,0,1.0,60.5625,7.50975,0.12400000000000001,2.514185542718151,0.03,9.988547003430146 +10,10,1,1.0,60.5625,7.50975,0.12400000000000001,3.5135808270676687,0.06,11.88289996991588 +10,10,2,1.0,60.5625,7.50975,0.12400000000000001,3.6246161014125478,0.08171791443850268,6.646763246529082 +11,11,0,0.9999999403953552,60.375,7.4865004462302,0.12400000000000001,4.759097607239703,0.2502973782150284,12.534934613337919 +11,11,1,0.9999999403953552,60.375,7.4865004462302,0.12400000000000001,4.35046336726321,0.3983833324031181,8.86943997042291 +11,11,2,0.9999999403953552,60.375,7.4865004462302,0.12400000000000001,4.301368868693476,0.4472454816704645,6.935362600042912 +12,12,0,0.9999998807907104,61.0,7.564000901699174,0.124,4.555343986362033,0.20266483516483516,10.140073120440526 +12,12,1,0.9999998807907104,61.0,7.564000901699174,0.124,4.17801655000322,0.2108002448068595,9.824940917782323 +12,12,2,0.9999998807907104,61.0,7.564000901699174,0.124,4.132602710493542,0.21387360377265482,8.762172204645621 +13,13,0,0.9999999403953552,61.9375,7.6802504577786,0.124,4.377037704982496,0.19449664124909224,6.033550884032971 +13,13,1,0.9999999403953552,61.9375,7.6802504577786,0.124,3.796327184720042,0.06439393939393939,7.020895203625925 +13,13,2,0.9999999403953552,61.9375,7.6802504577786,0.124,4.448075043971077,0.061834733893557425,6.052301209596841 +14,14,0,0.9999999403953552,63.0,7.812000465631513,0.124,4.736043620319545,0.18782727850065312,4.124602889078219 +14,14,1,0.9999999403953552,63.0,7.812000465631513,0.124,4.70128769263439,0.050679467346134044,4.595576966443948 +14,14,2,0.9999999403953552,63.0,7.812000465631513,0.124,5.183195491511032,0.04943772728325574,4.672780486660067 +15,15,0,1.0000001192092896,64.5,7.997999046564216,0.124,5.0511190380781095,0.06973720679192438,4.952306449655121 +15,15,1,1.0000001192092896,64.5,7.997999046564216,0.124,5.123907055789967,0.10144230769230766,5.474701948028992 +15,15,2,1.0000001192092896,64.5,7.997999046564216,0.124,4.877554411396609,0.046893420259509516,4.796434286880102 +16,16,0,1.0000003576278687,68.125,8.44749697893966,0.124,5.261550491201654,0.06609906457143916,5.037973642417829 +16,16,1,1.0000003576278687,68.125,8.44749697893966,0.124,5.049177501970055,0.06524992345376607,5.374092076490848 +16,16,2,1.0000003576278687,68.125,8.44749697893966,0.124,4.878893716577541,0.07689612234427184,4.410209588623184 +17,17,0,1.0,72.875,9.0365,0.124,5.145570595056896,0.0862941880516416,4.7399950122229235 +17,17,1,1.0,72.875,9.0365,0.124,4.834600305985848,0.047909777205696724,3.9981299190279396 +17,17,2,1.0,72.875,9.0365,0.124,5.0090287995339144,0.05963560122844319,4.212388784253915 +18,18,0,0.9999997615814209,76.75,9.517002269030158,0.124,4.957851890756302,0.06854738318001938,4.565846204073926 +18,18,1,0.9999997615814209,76.75,9.517002269030158,0.124,4.895820632556308,0.07119991289198607,4.264541131502773 +18,18,2,0.9999997615814209,76.75,9.517002269030158,0.124,5.066362546374245,0.07830511263768969,4.353682585732777 +19,19,0,1.0,82.25,10.199,0.124,5.357747712445741,0.08028117715617716,3.707529963427291 +19,19,1,1.0,82.25,10.199,0.124,4.776786874593892,0.03671942640692641,3.8565722458289433 +19,19,2,1.0,82.25,10.199,0.124,4.74618789171028,0.046372202095093656,5.257808634330184 +20,20,0,1.0,92.625,11.4855,0.124,5.043626693250715,0.04581395409946343,3.430468433792775 +20,20,1,1.0,92.625,11.4855,0.124,4.6670108695652175,0.05948260248334708,3.777761819426255 +20,20,2,1.0,92.625,11.4855,0.124,5.155962445707027,0.14593187427409987,3.2302509750115265 +21,21,0,1.0,62.1875,7.71125,0.124,4.954071705670378,0.09789820967146551,3.5992680617758714 +21,21,1,1.0,62.1875,7.71125,0.124,4.945276239583751,0.04413176865507373,3.6023631482211838 +21,21,2,1.0,62.1875,7.71125,0.124,4.574329542156547,0.05796073972001364,3.4486317916564904 +22,22,0,1.0,69.5,8.618,0.124,4.786409415937084,0.06604462245602846,3.238300556502737 +22,22,1,1.0,69.5,8.618,0.124,5.049344286801478,0.03542403929041332,2.9344214595290943 +22,22,2,1.0,69.5,8.618,0.124,4.676400017466817,0.061291737845681316,3.2369449547878246 +23,23,0,1.0000001192092896,269.5,33.41799601626444,0.124,5.963417969940857,0.027543238727449265,5.730668625502126 +23,23,1,1.0000001192092896,269.5,33.41799601626444,0.124,5.807998668876834,0.0711111061253977,5.421494657570518 +23,23,2,1.0000001192092896,269.5,33.41799601626444,0.124,5.54967512730824,0.04801274515507467,5.236424886028548 diff --git a/results/layer_sweep_qwen0.5b.png b/results/layer_sweep_qwen0.5b.png new file mode 100644 index 0000000..b6f8141 Binary files /dev/null and b/results/layer_sweep_qwen0.5b.png differ diff --git a/results/layer_sweep_qwen1.5b.csv b/results/layer_sweep_qwen1.5b.csv new file mode 100644 index 0000000..52e8c95 --- /dev/null +++ b/results/layer_sweep_qwen1.5b.csv @@ -0,0 +1,82 @@ +layer,layer_pos,seed,dir_norm,resid_norm,coeff,alpha_norm,effect,repetition,ppl +1,1,0,0.9999998807907104,155.0,13.950001662969786,0.09,4.7887449051922735,0.03807782466010315,3.718446206357423 +1,1,1,0.9999998807907104,155.0,13.950001662969786,0.09,4.370292666166916,0.09678307554723278,3.3023119325503374 +1,1,2,0.9999998807907104,155.0,13.950001662969786,0.09,4.821289002436544,0.05521478328173374,3.242404958531417 +2,2,0,1.0000001192092896,310.5,27.944996668696803,0.09,4.662496010072351,0.03641493403593177,4.25392179787295 +2,2,1,1.0000001192092896,310.5,27.944996668696803,0.09,4.85896448467783,0.03338744588744588,4.957672978166435 +2,2,2,1.0000001192092896,310.5,27.944996668696803,0.09,4.993421521362244,0.039464679168693395,5.721872945647776 +3,3,0,1.0000001192092896,319.5,28.754996572137287,0.09,4.59732482651625,0.027372134038800733,4.3070139557934874 +3,3,1,1.0000001192092896,319.5,28.754996572137287,0.09,4.251919955044955,0.03823434629886241,4.548854616460117 +3,3,2,1.0000001192092896,319.5,28.754996572137287,0.09,4.754880908869396,0.055039682539682555,3.474045568151263 +4,4,0,1.0000001192092896,328.0,29.51999648094219,0.09,4.095420650417452,0.1633221850613155,42.21524794954013 +4,4,1,1.0000001192092896,328.0,29.51999648094219,0.09,5.034583940424154,0.2134249471458774,2503.933539190293 +4,4,2,1.0000001192092896,328.0,29.51999648094219,0.09,4.271825396825397,0.022727272727272735,77.56185849056862 +5,5,0,0.9999998807907104,332.0,29.880003561973997,0.09,4.301522206340345,0.26303039575683396,4.3590508609987575 +5,5,1,0.9999998807907104,332.0,29.880003561973997,0.09,4.894728191541557,0.15780339162692103,4.881169237767776 +5,5,2,0.9999998807907104,332.0,29.880003561973997,0.09,4.641976195160339,0.12899564002505182,4.2510929394696335 +6,6,0,1.0,334.0,30.06,0.09,4.7102542081758845,0.08290419935534879,3.956339877110753 +6,6,1,1.0,334.0,30.06,0.09,4.575033875338754,0.04783189880020866,3.8345128260341625 +6,6,2,1.0,334.0,30.06,0.09,4.61474709106616,0.031745741313377124,4.633565734583072 +7,7,0,1.000000238418579,338.0,30.41999274730855,0.09,4.696835391275944,0.1376495091865721,7.904825841242319 +7,7,1,1.000000238418579,338.0,30.41999274730855,0.09,0.0,0.0,9.404971470249603 +7,7,2,1.000000238418579,338.0,30.41999274730855,0.09,4.556965563506261,0.08700815193741476,7.105778582260645 +8,8,0,1.0000001192092896,339.5,30.55499635756059,0.09,5.487577639751553,0.012500000000000011,21.502035516005606 +8,8,1,1.0000001192092896,339.5,30.55499635756059,0.09,2.9541933840714325,0.27163355839154923,30.879738417766042 +8,8,2,1.0000001192092896,339.5,30.55499635756059,0.09,4.2585227272727275,0.07500000000000001,172.80977881340087 +9,9,0,1.000000238418579,342.0,30.779992661477884,0.09,4.4060095041843645,0.28542677497319596,6.388653370551469 +9,9,1,1.000000238418579,342.0,30.779992661477884,0.09,4.117701575532902,0.15612131770668358,13.293629504105338 +9,9,2,1.000000238418579,342.0,30.779992661477884,0.09,4.18578943772631,0.1808729627504728,9.082811553296473 +10,10,0,1.0000001192092896,343.0,30.86999632000967,0.09,3.737843137254902,0.03800675675675674,5.059050137629561 +10,10,1,1.0000001192092896,343.0,30.86999632000967,0.09,4.2501344830803625,0.037192118226600984,9.876992998965806 +10,10,2,1.0000001192092896,343.0,30.86999632000967,0.09,3.9557648995148997,0.03616474849351564,4.755204093107657 +11,11,0,0.9999998807907104,342.5,30.82500367462679,0.09,4.6887273731000825,0.10862013386397654,8.3440807101635 +11,11,1,0.9999998807907104,342.5,30.82500367462679,0.09,4.870660507792861,0.08444988110481069,9.216997454074937 +11,11,2,0.9999998807907104,342.5,30.82500367462679,0.09,4.6499977181453085,0.09605262638347606,8.762056486165434 +12,12,0,1.0000001192092896,343.0,30.86999632000967,0.09,3.831257383086651,0.05440750723769594,5.731052925305031 +12,12,1,1.0000001192092896,343.0,30.86999632000967,0.09,3.6945737761495168,0.06901927437641722,6.379995743221593 +12,12,2,1.0000001192092896,343.0,30.86999632000967,0.09,3.4051987426987425,0.07466616783699781,7.164849288625384 +13,13,0,1.0000001192092896,346.5,31.184996282458748,0.09,3.349281832298137,0.008995619524405501,8.068597232832639 +13,13,1,1.0000001192092896,346.5,31.184996282458748,0.09,3.626214483343313,0.05538825757575758,7.984623526450665 +13,13,2,1.0000001192092896,346.5,31.184996282458748,0.09,3.169722222222222,0.022448979591836726,10.099165108409625 +14,14,0,1.0000001192092896,348.5,31.364996261001078,0.09,3.7150076277650648,0.027396645405149994,6.559027481547114 +14,14,1,1.0000001192092896,348.5,31.364996261001078,0.09,3.9686710551884956,0.05334195435749753,7.437865038079242 +14,14,2,1.0000001192092896,348.5,31.364996261001078,0.09,3.49769069664903,0.1365699955557967,5.359235919492754 +15,15,0,1.0000001192092896,349.5,31.45499625027224,0.09,3.246633312486971,0.022727272727272735,7.780090969814809 +15,15,1,1.0000001192092896,349.5,31.45499625027224,0.09,3.37056058228909,0.022664835164835168,7.369201336555114 +15,15,2,1.0000001192092896,349.5,31.45499625027224,0.09,3.044376222619893,0.04496031394765573,6.68607471741803 +16,16,0,1.0000001192092896,350.0,31.499996244907827,0.09,4.696490759934724,0.06528238075857123,4.538694245448534 +16,16,1,1.0000001192092896,350.0,31.499996244907827,0.09,4.401397060488559,0.07533936651583709,7.99394576790138 +16,16,2,1.0000001192092896,350.0,31.499996244907827,0.09,4.622694011815252,0.05130801687763714,4.774719390105837 +17,17,0,1.0000001192092896,352.5,31.72499621808574,0.09,5.09747528099588,0.09650974025974027,4.7801566489439455 +17,17,1,1.0000001192092896,352.5,31.72499621808574,0.09,4.908710127591706,0.03432088091415053,6.508150717819918 +17,17,2,1.0000001192092896,352.5,31.72499621808574,0.09,4.8976742706919945,0.0658307210031348,4.09822958247917 +18,18,0,1.0,357.5,32.175,0.09,4.7687793630518085,0.04819666353383459,4.04412422580369 +18,18,1,1.0,357.5,32.175,0.09,4.734928080841505,0.0607937622214241,5.089942037069323 +18,18,2,1.0,357.5,32.175,0.09,4.775569618071514,0.08648391118414486,3.9296110968761364 +19,19,0,1.0000003576278687,367.0,33.02998818755572,0.09,4.673453443649017,0.08247875092387288,4.173872327302847 +19,19,1,1.0000003576278687,367.0,33.02998818755572,0.09,4.479636040876313,0.23734272755357094,3.258374782262907 +19,19,2,1.0000003576278687,367.0,33.02998818755572,0.09,4.948367521142514,0.04319752402069474,4.635582238941297 +20,20,0,0.9999998807907104,372.5,33.525003996491904,0.09,4.927852577718027,0.029586834733893563,3.5138558574494145 +20,20,1,0.9999998807907104,372.5,33.525003996491904,0.09,4.8532125760700735,0.04931410931410932,4.052647250165374 +20,20,2,0.9999998807907104,372.5,33.525003996491904,0.09,4.778777957394442,0.09201365277966664,3.396667395565254 +21,21,0,1.0,385.0,34.65,0.09,4.847266068361672,0.05759591285907073,3.226060995089128 +21,21,1,1.0,385.0,34.65,0.09,4.849617700077434,0.06165144596651445,3.5219686768600003 +21,21,2,1.0,385.0,34.65,0.09,5.0725378787878785,0.02676208265556504,3.6228500974236884 +22,22,0,0.9999996423721313,404.0,36.360013003353956,0.09,4.985378614916287,0.0418171910249375,3.0834762605964334 +22,22,1,0.9999996423721313,404.0,36.360013003353956,0.09,4.853195488721805,0.05531645569620253,3.1855951092466888 +22,22,2,0.9999996423721313,404.0,36.360013003353956,0.09,4.812727326070374,0.04097799349737333,2.8930849352915198 +23,23,0,1.0000001192092896,428.0,38.51999540805871,0.09,4.989139674857285,0.057669736366919466,3.072118596100018 +23,23,1,1.0000001192092896,428.0,38.51999540805871,0.09,4.652219413728864,0.0372000823808053,3.175801418518994 +23,23,2,1.0000001192092896,428.0,38.51999540805871,0.09,4.7591804029304035,0.03777562989092653,3.2277056584066566 +24,24,0,1.0,455.0,40.949999999999996,0.09,4.663872717304756,0.0712860310421286,3.376043404957147 +24,24,1,1.0,455.0,40.949999999999996,0.09,4.914493639921722,0.04738045007032349,3.1391844252223926 +24,24,2,1.0,455.0,40.949999999999996,0.09,4.892835324578623,0.02707914151576124,3.089828099301758 +25,25,0,1.000000238418579,487.0,43.82998955011617,0.09,4.974509803921569,0.07130961444341727,2.949397643388827 +25,25,1,1.000000238418579,487.0,43.82998955011617,0.09,4.707390873015873,0.06558521481182353,3.1271265724399657 +25,25,2,1.000000238418579,487.0,43.82998955011617,0.09,4.9555379188712525,0.026371460746460712,2.806456991821833 +26,26,0,1.000000238418579,251.75,22.65749459803233,0.09,4.630627226089133,0.08020334059549747,2.742394493947959 +26,26,1,1.000000238418579,251.75,22.65749459803233,0.09,4.94628887852228,0.04537210756722951,3.0207245811768497 +26,26,2,1.000000238418579,251.75,22.65749459803233,0.09,4.881859827264239,0.06705819696881823,2.5910404506890874 +27,27,0,0.9999997615814209,160.5,14.445003443957196,0.09,4.318996737601389,0.06625777889964371,2.709027395109521 +27,27,1,0.9999997615814209,160.5,14.445003443957196,0.09,4.6490267134122405,0.03703703703703706,3.0154149109828134 +27,27,2,0.9999997615814209,160.5,14.445003443957196,0.09,4.704949199930839,0.07637611816265852,2.6128393448138096 diff --git a/results/layer_sweep_qwen1.5b.png b/results/layer_sweep_qwen1.5b.png new file mode 100644 index 0000000..79ab095 Binary files /dev/null and b/results/layer_sweep_qwen1.5b.png differ diff --git a/results/layer_sweep_qwen1.5b_a044.csv b/results/layer_sweep_qwen1.5b_a044.csv new file mode 100644 index 0000000..4b72f05 --- /dev/null +++ b/results/layer_sweep_qwen1.5b_a044.csv @@ -0,0 +1,82 @@ +layer,layer_pos,seed,dir_norm,resid_norm,coeff,alpha_norm,effect,repetition,ppl +1,1,0,0.9999998807907104,155.0,6.820000813007451,0.044,4.698345624937606,0.03565992304172874,2.921675748136177 +1,1,1,0.9999998807907104,155.0,6.820000813007451,0.044,4.609980682427329,0.028579840805718015,2.9389491688518827 +1,1,2,0.9999998807907104,155.0,6.820000813007451,0.044,4.7754356783429595,0.013544891640866863,2.774849881051268 +2,2,0,1.0000001192092896,310.5,13.661998371362879,0.044,4.630326396960311,0.05489399011637175,3.078034079444241 +2,2,1,1.0000001192092896,310.5,13.661998371362879,0.044,4.818720804137471,0.06333337803046557,2.9580739481795355 +2,2,2,1.0000001192092896,310.5,13.661998371362879,0.044,5.06470256949709,0.017361111111111105,3.2776192804702116 +3,3,0,1.0000001192092896,319.5,14.057998324156006,0.044,4.810811064228786,0.043319040194040176,3.2836151142686933 +3,3,1,1.0000001192092896,319.5,14.057998324156006,0.044,4.4871697984749455,0.04299664651426413,2.9173695699751407 +3,3,2,1.0000001192092896,319.5,14.057998324156006,0.044,4.828828688944675,0.04348003960808838,3.0733320992745865 +4,4,0,1.0000001192092896,328.0,14.431998279571737,0.044,5.061603613777526,0.04624613003095976,3.1058017971767353 +4,4,1,1.0000001192092896,328.0,14.431998279571737,0.044,4.574571165716999,0.03425085943468295,2.7442906874412634 +4,4,2,1.0000001192092896,328.0,14.431998279571737,0.044,4.664445003353454,0.023321858864027545,2.921511766401444 +5,5,0,0.9999998807907104,332.0,14.608001741409508,0.044,4.710178676950829,0.049227679950571535,3.33794755488465 +5,5,1,0.9999998807907104,332.0,14.608001741409508,0.044,4.614936053292217,0.06688596491228072,3.4705476202816268 +5,5,2,0.9999998807907104,332.0,14.608001741409508,0.044,4.66093903935343,0.0393264454165593,3.1686386266315747 +6,6,0,1.0,334.0,14.696,0.044,4.800542717086834,0.05249056529601384,3.2537773347894086 +6,6,1,1.0,334.0,14.696,0.044,4.75473810728745,0.04904761904761906,3.0439760591983713 +6,6,2,1.0,334.0,14.696,0.044,4.732743350623551,0.042487696442498724,3.107500543734059 +7,7,0,1.000000238418579,338.0,14.871996454239737,0.044,4.223649001192605,0.0728323453273888,2.677443690631895 +7,7,1,1.000000238418579,338.0,14.871996454239737,0.044,4.6283500369385315,0.07206838833094212,3.081120896086915 +7,7,2,1.000000238418579,338.0,14.871996454239737,0.044,4.627857142857143,0.08307399972795734,2.8329268151859073 +8,8,0,1.0000001192092896,339.5,14.937998219251844,0.044,4.581906189630001,0.11649891941957394,2.6803620882661896 +8,8,1,1.0000001192092896,339.5,14.937998219251844,0.044,4.072001105583196,0.05729166666666666,4.649749561589611 +8,8,2,1.0000001192092896,339.5,14.937998219251844,0.044,4.533201282633945,0.1031940330795415,2.5944952296096675 +9,9,0,1.000000238418579,342.0,15.047996412278076,0.044,4.54909363927059,0.09317954488622154,2.749974244902541 +9,9,1,1.000000238418579,342.0,15.047996412278076,0.044,4.557703754578755,0.06982083535020656,2.8550285789366874 +9,9,2,1.000000238418579,342.0,15.047996412278076,0.044,5.002533016299063,0.054591948270109186,3.210352571700033 +10,10,0,1.0000001192092896,343.0,15.091998200893615,0.044,4.423847831571409,0.09536282858200668,2.8147114120275885 +10,10,1,1.0000001192092896,343.0,15.091998200893615,0.044,4.387517704633858,0.0261284722222222,3.1223647769866987 +10,10,2,1.0000001192092896,343.0,15.091998200893615,0.044,4.651504668440449,0.03911456655359094,3.123549945456891 +11,11,0,0.9999998807907104,342.5,15.070001796484206,0.044,4.676387939232468,0.048878468168432915,3.59952872104712 +11,11,1,0.9999998807907104,342.5,15.070001796484206,0.044,4.753635937133474,0.046648550724637666,3.4499469880751494 +11,11,2,0.9999998807907104,342.5,15.070001796484206,0.044,4.815183404942336,0.0560813660643496,3.0179319690105655 +12,12,0,1.0000001192092896,343.0,15.091998200893615,0.044,4.306137046383081,0.07006452655186832,3.3578010762201327 +12,12,1,1.0000001192092896,343.0,15.091998200893615,0.044,4.3108909998970235,0.05112548907636508,3.043269097224595 +12,12,2,1.0000001192092896,343.0,15.091998200893615,0.044,4.54810106473438,0.06027554535017221,3.1616915032606583 +13,13,0,1.0000001192092896,346.5,15.245998182535386,0.044,4.539289263771451,0.03887085137085139,3.0327348427843823 +13,13,1,1.0000001192092896,346.5,15.245998182535386,0.044,4.333460365853659,0.07225503984997658,2.810588337524298 +13,13,2,1.0000001192092896,346.5,15.245998182535386,0.044,4.833562256572542,0.05797733217088055,3.174006789442833 +14,14,0,1.0000001192092896,348.5,15.333998172044971,0.044,4.393555554670406,0.05294605183845691,2.867484938507812 +14,14,1,1.0000001192092896,348.5,15.333998172044971,0.044,4.405484931443289,0.04970867208672086,2.9559447082486785 +14,14,2,1.0000001192092896,348.5,15.333998172044971,0.044,4.172847706155633,0.05950525025559342,2.927854839833901 +15,15,0,1.0000001192092896,349.5,15.377998166799761,0.044,4.473214285714286,0.042495429019198144,2.8905068569355574 +15,15,1,1.0000001192092896,349.5,15.377998166799761,0.044,4.236222284272359,0.05116674716199576,4.207227178104722 +15,15,2,1.0000001192092896,349.5,15.377998166799761,0.044,4.055876127722335,0.05235762245682157,2.9521172949886667 +16,16,0,1.0000001192092896,350.0,15.399998164177159,0.044,4.818655580892339,0.046993464052287603,3.267073917517873 +16,16,1,1.0000001192092896,350.0,15.399998164177159,0.044,4.709594248720265,0.04535504397091156,5.3406828904013866 +16,16,2,1.0000001192092896,350.0,15.399998164177159,0.044,4.762849755360778,0.0650516795865633,4.867675945305414 +17,17,0,1.0000001192092896,352.5,15.509998151064138,0.044,4.882606863263866,0.0407246538218807,2.829565119531776 +17,17,1,1.0000001192092896,352.5,15.509998151064138,0.044,4.853839006259781,0.05421666940654285,2.8744148891408976 +17,17,2,1.0000001192092896,352.5,15.509998151064138,0.044,5.006779984721161,0.032062460367786955,3.3553041255551017 +18,18,0,1.0,357.5,15.729999999999999,0.044,4.852978413038334,0.05345238095238097,2.9453869268157797 +18,18,1,1.0,357.5,15.729999999999999,0.044,4.723904962829013,0.062171763978992906,3.054091369287573 +18,18,2,1.0,357.5,15.729999999999999,0.044,4.914453467113619,0.038927043217800605,2.8981034700780492 +19,19,0,1.0000003576278687,367.0,16.147994225027244,0.044,4.755043985465672,0.05661392133287532,2.920111036026399 +19,19,1,1.0000003576278687,367.0,16.147994225027244,0.044,4.764738841405508,0.05579260266760269,2.86343292199795 +19,19,2,1.0000003576278687,367.0,16.147994225027244,0.044,4.647120840748078,0.047307343818971725,2.922867171150801 +20,20,0,0.9999998807907104,372.5,16.39000195384049,0.044000000000000004,4.614834994145339,0.04671590753941823,2.7642856221585888 +20,20,1,0.9999998807907104,372.5,16.39000195384049,0.044000000000000004,4.886464570072057,0.034340326613533395,2.8870614123549228 +20,20,2,0.9999998807907104,372.5,16.39000195384049,0.044000000000000004,4.977153590371981,0.03209421790379102,2.885827121941678 +21,21,0,1.0,385.0,16.939999999999998,0.044,4.822999105101198,0.05980300187617263,2.843527218884529 +21,21,1,1.0,385.0,16.939999999999998,0.044,4.974018739994349,0.07082482025404399,3.111442888063518 +21,21,2,1.0,385.0,16.939999999999998,0.044,4.9536921820010065,0.036545027742210856,3.0622734888041445 +22,22,0,0.9999996423721313,404.0,17.776006357195268,0.044,4.656788472964944,0.06120002347828302,3.0418823787663096 +22,22,1,0.9999996423721313,404.0,17.776006357195268,0.044,4.916346742688206,0.048935997400909675,2.9260603423985145 +22,22,2,0.9999996423721313,404.0,17.776006357195268,0.044,4.904580574710101,0.036230477039300574,2.835445481233668 +23,23,0,1.0000001192092896,428.0,18.831997755050924,0.04399999999999999,4.612926854467788,0.06557372859698443,3.0729090248897117 +23,23,1,1.0000001192092896,428.0,18.831997755050924,0.04399999999999999,4.94312881185715,0.057709887886550126,2.940214674146125 +23,23,2,1.0000001192092896,428.0,18.831997755050924,0.04399999999999999,4.780120901194426,0.028484750543574078,3.047117208019285 +24,24,0,1.0,455.0,20.02,0.044,4.752519688924595,0.06612766318648675,3.004739927448633 +24,24,1,1.0,455.0,20.02,0.044,4.656731878280599,0.09016485998193316,2.929747096316215 +24,24,2,1.0,455.0,20.02,0.044,4.742326417326418,0.05284776581865622,2.743702116932994 +25,25,0,1.000000238418579,487.0,21.427994891167902,0.044,4.78752676413031,0.06652187902187903,3.0581097412566454 +25,25,1,1.000000238418579,487.0,21.427994891167902,0.044,4.878740014524329,0.0593242694805195,3.0865001700036925 +25,25,2,1.000000238418579,487.0,21.427994891167902,0.044,4.821345643520566,0.029036144578313244,2.8736740687117077 +26,26,0,1.000000238418579,251.75,11.076997359038028,0.044,4.6482112095484185,0.11346748147419503,2.8741891346708344 +26,26,1,1.000000238418579,251.75,11.076997359038028,0.044,4.7344720214716824,0.04790242039166001,3.0101615027928643 +26,26,2,1.000000238418579,251.75,11.076997359038028,0.044,4.568096405228758,0.05711160904822876,2.726165241502152 +27,27,0,0.9999997615814209,160.5,7.062001683712406,0.044,4.354103745092117,0.05432131000783319,2.73822287404439 +27,27,1,0.9999997615814209,160.5,7.062001683712406,0.044,4.544271586799277,0.05600677136821716,2.8249534789378337 +27,27,2,0.9999997615814209,160.5,7.062001683712406,0.044,4.498531899487198,0.07261615213818765,2.4576621732650565 diff --git a/results/layer_sweep_qwen1.5b_a044.png b/results/layer_sweep_qwen1.5b_a044.png new file mode 100644 index 0000000..b78c9a5 Binary files /dev/null and b/results/layer_sweep_qwen1.5b_a044.png differ