Fine-Tuning Silently Breaks Linear Safety Monitors

The same direction at the same checkpoint gets opposite verdicts from steering and ablation.

Code and data for "Fine-Tuning Silently Breaks Linear Safety Monitors" (Iyer 2026). See also the blog post on vaiyr.com.

What we found

We trained Qwen2.5-Coder-7B on CodeContests with iterative SFT that rewards solutions passing visible tests (whether or not they're correct), tracked a linear shortcut-detection probe across 5 rounds, and found a compound failure mode that standard evaluation misses:

1. The probe direction rotates within the first SFT round. cos(base, round_k) drops to about 0.37 and settles between 0.38 and 0.40. Fresh probes refit at each checkpoint still recover AUROC ≥ 0.92 — the feature is there, just not where the frozen probe is looking.
2. Additive steering still moves behavior, but the robust signal is the α-sweep, not a clean per-round curve. Round 4 at α=10 gives −17.1% shortcut reduction (CI [−25.8%, −7.4%]), and the round-2 α-sweep excludes zero at α=7 and α=10. The α=5 per-round trajectory is noisier: its CIs cross zero at base, r0, r1, r3, and r4. Standard eval still reports the monitor "still works."
3. But the frozen direction fails the ablation necessity test. Projecting it out of the residual stream at round 4: Δ = −0.9%, p = 0.57. Sufficient to push behavior via magnitude, but not the causal axis the model uses.

A frozen monitor reports "still works" when the model has already routed around it. A probe with near-perfect AUROC can be causally disconnected from the feature it classifies.

What to do about it

1. Refit probes whenever retraining occurs. Freezing directions across training steps is the core bug.
2. Run projective ablation alongside steering. Sufficiency ≠ necessity. Additive steering alone can't tell them apart.
3. Treat AUROC as calibration, not causal faithfulness. A separating hyperplane is not a causal handle.

Of the three extraction methods tested, only difference-of-means ever showed significant causal necessity, and only transiently (round 2; by round 4 its CI crosses zero). See NEXT.md for a proposed self-recalibrating monitor that catches this specific failure mode.

Follow-on: the same study under RL (GRPO)

A GRPO rerun (same model, reward signal, and eval battery; trl.GRPOTrainer in place of rejection-sampling SFT) sharpens the dissociation: frozen detection holds at AUROC 0.96 the whole run while the direction rotates to the refit-noise floor within 50 optimizer steps, and at matched power a freshly refit direction recovers a strong causal handle under SFT (−15.9pp, −36.5% relative) but not under RL (−2.3pp, −6.7% relative). Full results, controls (including a split-half refit-stability floor and a matched random-direction control), and one disclosed measurement-bug fix in results/paper/grpo/RESULTS_GRPO.md; reproduce with python scripts/compare_rl_vs_sft.py.

Repo layout

preprint/         Paper (main.pdf, main.tex), figures/
src/              Probe extraction, steering, rejection-sampling SFT, env wrappers
scripts/          Bootstrap CIs, reproduce numbers, regenerate figures
deploy/           Modal entrypoints (app.py) for the full pipeline
results/paper/raw/  Per-condition shortcut counts, ablation deltas, probe cosines (JSON, ~108 KB)

Note: results/data/problems.jsonl and results/feasibility/generations.jsonl are superseded APPS-era pilot files kept for provenance; the paper's 1000-problem CodeContests set and its 16k generations live on the Modal volume (control-results, see RESULTS.md "Key files and artifacts").

Committed artifacts are summary statistics only — raw activations (~GB-scale) and model checkpoints are not in the repo. results/paper/raw/*.json is enough to reproduce every number and figure in the paper; regenerating the artifacts themselves requires the pipeline below.

Reproduce the paper numbers

pip install -e '.[analysis]'
python scripts/bootstrap_cis.py
python scripts/reproduce_key_numbers.py

Runs offline in under 30 seconds. No GPU needed.

To re-run from scratch (Modal + ~1 day of 4×A100-80GB):

pip install -e '.[train,extract,analysis]'
modal token new
modal run deploy/app.py --mode freeze-cc --n-problems 1000
modal run --detach deploy/app.py --mode train-rft-cc --max-steps 5
# See RESULTS.md ("Running the full pipeline from scratch") for the full pipeline

The training loop is rejection-sampling SFT (src/train_rft.py); true RL such as PPO or GRPO is left to future work.

Citation

@misc{iyer2026finetuning,
  title  = {Fine-Tuning Silently Breaks Linear Safety Monitors},
  author = {Iyer, Varun},
  year   = {2026},
  note   = {Preprint},
}

MIT licensed. See LICENSE.