Linear Safety Probes Cannot Silence Features They Detect

Code for the paper "Linear Safety Probes Cannot Silence Features They Detect" (Iyer 2026).

Paper: preprint/main.pdf

Overview

Linear probes on a language model's residual stream are deployed as safety monitors, where one direction is asked to do three jobs: detect a feature, steer by subtracting it, and silence by projecting it out. Only silencing tests that the direction is the causal axis, yet detection and steering are what get reported. Across four open-weights chat models (Llama-3-8B, Gemma-2-9B, Mistral-7B-v0.3, OLMo-2-7B) and two features (refusal, sycophancy), we find probes that detect at AUROC ≥ 0.91 and steer cleanly yet fail to silence. Probe accuracy is not evidence of a causal handle.

1. Detect, steer, and silence come apart at calibration. Difference-of-means (DoM) and regularized logistic regression (LR-CV) have disjoint failure sets across families, so no fixed recipe is safe. Mistral-7B at L14 is the cleanest existence proof: across the recipes and body layers we search (DoM and LR-CV at layers {8,10,12,14,16,18}), no direction silences refusal until the most-trained checkpoint. A Llama-3 sycophancy sign-flip (ablating DoM raises sycophancy by +10 pp; ablating LR does nothing) rules out the objection that the split is noise.

2. A within-class confound explains the failure, with a calibration-time cosine signature. Plain DoM decomposes as α · (causal axis) + β · (within-class confound) with α, β varying by (model, feature). Whitening the top-k within-class PCs before refitting DoM produces the predicted regimes: recovery when β dominates, preservation when α dominates, near-noop at β → 0. The cosine between plain and whitened DoM tracks silencing success monotonically across 35 cells spanning cos ∈ [0.12, 1.00] in-sample.

3. A calibration-time necessity check catches silent intervention failure. For each candidate recipe (DoM and LR-CV) the check fits a direction, projects it out, and measures the silencing effect with a Newcombe 95% CI; it keeps the recipe whose CI excludes zero in the canonical direction (tie-break by effect size) and alarms NECESSITY_LOST when neither does. Cost is one probe fit and one projection per candidate recipe. On the 22-cell four-family battery the picker recovers all 17 causal handles that exist and correctly alarms on the remaining 5 Mistral cells, which have no silencing handle under either recipe; fixed DoM recovers 10/22, fixed LR 13/22. A held-out Qwen2.5-7B replication adds 5 confound-light cells where both recipes silence and the picker selects the larger-effect DoM on all five. The cosine of item 2 is an in-sample correlate of the winner and a candidate cheaper screen, not the selector behind these counts (src/monitor.py runs no cosine threshold in the picker; the separate online drift monitor uses 0.70, exercised only in unit tests).

Repository Structure

├── preprint/                        # Paper source and figures
│   ├── main.tex, main.pdf           # Paper
│   ├── references.bib, compile.sh
│   ├── figure_scripts/              # Regenerate each figure
│   │   ├── _style.py
│   │   ├── figure_three_jobs.py                # Fig 1 (synthetic schematic)
│   │   ├── figure_cross_model_hero_2x2.py      # Fig 2 (cross-model hero)
│   │   ├── figure_cos_vs_silencing.py          # Fig 3 (cosine scatter)
│   │   └── figure_k_dose_response.py           # Fig 4 (whitening dose-response)
│   └── figures/                     # Compiled PDF/PNG pairs
├── src/                             # Core library
│   ├── monitor.py                   # AblationCell, necessity-gated picker, RecalibratingMonitor
│   ├── probe.py                     # DoM / LR-CV fits, projection utilities
│   ├── stats_ci.py                  # Wilson / Newcombe difference-of-proportions CI
│   ├── steer.py                     # Additive steering + directional-ablation hooks
│   ├── refusal_{data,judge,substrate}.py  # AdvBench+Alpaca buffer, Haiku judge, prefix match
│   ├── sycophancy_{data,judge}.py   # MWE buffer, Sonnet judge
│   └── registry.py                  # Frozen constants
├── deploy/                          # Modal apps
│   ├── refusal_app.py               # Refusal replication pipeline
│   └── sycophancy_app.py            # Sycophancy pipeline
├── scripts/
│   ├── run_autopicker_battery.py    # CPU replay — reproduces body numbers
│   ├── refusal_fit_probes.py        # LR-CV / LR-det / DoM fits on cached activations
│   ├── fit_dom_pcwhitened.py        # Within-class PC whitening for the §4 mechanism
│   ├── fit_random_baseline.py       # Seeded Gaussian direction per (model, layer)
│   ├── build_random_baseline_directions.py
│   ├── summarize_whitening_{mistral,all}.py    # Produces results/paper/raw/*_whitening.json
│   ├── check_refusal_chat_template.py          # CPU-only pre-Modal sanity
│   ├── run_refusal_pipeline{,_llama3}.sh       # End-to-end Modal pipelines
│   └── launch_*.sh                  # Per-family Modal orchestrators (whitening, layer sweep, random baseline)
├── results/
│   ├── ablation_refusal/                       # Qwen-Instruct refusal (appendix)
│   ├── ablation_refusal_{llama3,gemma2,mistral,olmo}/
│   ├── ablation_refusal_gemma2_layer_sweep/    # Gemma-2 L24/L30/L36 follow-up
│   ├── ablation_sycophancy_llama3/             # Llama-3 sycophancy
│   ├── paper/raw/                              # Whitening JSONs (one per family) + preregistration
│   └── monitor/autopicker_battery.json         # Picker output
├── tests/                           # Unit tests (pytest)
├── pyproject.toml
└── README.md

Setup

Requirements

• Python 3.11+
• CPU is enough to reproduce every paper number from the shipped results
• GPU (H100 via Modal; 1× for extraction/ablation, 2× for SFT) required only to regenerate the raw data from scratch

Installation

# Minimal: CPU replay + figure regeneration
pip install -e '.[analysis]'

# Full: add Modal + training + extraction dependencies
pip install -e '.[modal,analysis]'

Modal setup (only for re-running experiments)

modal token new                     # authenticate; one-time per workspace

The Modal apps (deploy/refusal_app.py, deploy/sycophancy_app.py) read two secrets by name. Create them in your workspace:

# HuggingFace token (for gated-model downloads)
modal secret create huggingface-token HF_TOKEN=hf_xxx

# Anthropic API key (for Claude-based judges: refusal classifier, sycophancy classifier)
modal secret create anthropic-token ANTHROPIC_API_KEY=sk-ant-xxx

Gated models. Llama-3-8B-Instruct, Gemma-2-9B-it, and Mistral-7B-Instruct-v0.3 each require accepting a license on HuggingFace Hub before the HF token can pull them. OLMo-2 and Qwen2.5 are open-weights.

Volumes. Each family isolates results onto its own Modal volume to avoid cross-family clobbering. Volumes auto-create on the first modal run; the commands in Reproducing Results set MONITOR_DRIFT_REFUSAL_VOLUME per family — pick any names you like, just keep them distinct.

Models

Five (model, feature) conditions span the paper. Probe layer is chosen per family by base-checkpoint sweep.

Model	Feature	Layer	Labeled buffer
Llama-3-8B-Instruct	refusal	L12	AdvBench + Alpaca
Gemma-2-9B-it	refusal	L18	AdvBench + Alpaca
Mistral-7B-Instruct-v0.3	refusal	L14	AdvBench + Alpaca
OLMo-2-1124-7B-Instruct	refusal	L30	AdvBench + Alpaca
Llama-3-8B-Instruct	sycophancy	L12	Anthropic MWE

Plus Qwen2.5-7B-Instruct refusal as a held-out replication (Appendix).

Training substrate: UltraChat LoRA SFT (r=16, α=16, lr=2e-5), four rounds, safety-gated per round.

Reproducing Results

Step 1: CPU replay of the body numbers (§3)

Every number in Table 2 of the paper. Reads the shipped per-cell result.json files:

python3 -m scripts.run_autopicker_battery

Prints per-cell decisions (DoM vs LR-CV vs NECESSITY_LOST) per condition, then aggregate totals: auto 22/27, dom 15/27, lr_cv 18/27. Of the 27 cells (22 body + 5 held-out Qwen) the picker recovers a handle on 22; the 5 it does not are the Mistral cells with no silencing handle under either recipe, where it correctly emits NECESSITY_LOST. Those are correct alarms, not silencing failures, so they match the paper's 17/22 body handles plus the 5 Qwen handles. Writes results/monitor/autopicker_battery.json. Takes < 1 second, no GPU.

Step 2: Regenerate figures

cd preprint/figure_scripts
python3 figure_three_jobs.py              # Fig 1: synthetic three-jobs schematic
python3 figure_cross_model_hero_2x2.py    # Fig 2: cross-model hero
python3 figure_cos_vs_silencing.py        # Fig 3: cosine vs silencing scatter
python3 figure_k_dose_response.py         # Fig 4: whitening dose-response

# Compile the PDF
cd .. && bash compile.sh

Step 3: Refusal replication from scratch (§3, Tables 3-4, 7-8)

Each family runs the same end-to-end pipeline (freeze buffer → base eval → extract → fit probes → SFT rounds → ablation battery) on Modal. Rough Modal cost per family at H100 pricing: ~$60 for refusal, ~$35 for sycophancy, ~$80 for Gemma-2 (larger model). Wall-clock 12–24 h; chain rounds via --detach so the launcher can exit.

# Llama-3-8B refusal (body)
bash scripts/run_refusal_pipeline_llama3.sh

# Gemma-2-9B refusal (body)
MONITOR_DRIFT_REFUSAL_BASE_MODEL=google/gemma-2-9b-it \
MONITOR_DRIFT_REFUSAL_VOLUME=refusal-results-gemma2 \
MONITOR_DRIFT_REFUSAL_APP=refusal-dynamics-gemma2 \
REFUSAL_ABL_LAYER=18 \
  bash scripts/run_refusal_pipeline.sh

# Mistral-7B-v0.3 refusal (body)
MONITOR_DRIFT_REFUSAL_BASE_MODEL=mistralai/Mistral-7B-Instruct-v0.3 \
MONITOR_DRIFT_REFUSAL_VOLUME=refusal-results-mistral \
MONITOR_DRIFT_REFUSAL_APP=refusal-dynamics-mistral \
MONITOR_DRIFT_REFUSAL_HARMFUL_GATE=0.55 \
REFUSAL_ABL_LAYER=14 \
  bash scripts/run_refusal_pipeline.sh

# OLMo-2-7B refusal (body; peak layer L30)
MONITOR_DRIFT_REFUSAL_BASE_MODEL=allenai/OLMo-2-1124-7B-Instruct \
MONITOR_DRIFT_REFUSAL_VOLUME=refusal-results-olmo \
REFUSAL_SWEEP_LAYERS=8,10,12,14,16,18,20,24,25,28,30,31 \
REFUSAL_ABL_LAYER=30 \
  bash scripts/run_refusal_pipeline.sh

# Qwen2.5-7B-Instruct refusal (appendix; confound-light control)
bash scripts/run_refusal_pipeline.sh

Step 4: Sycophancy replication (§3, Table 6)

# Llama-3 sycophancy (MWE buffer, 4 rounds, r0–r4 cells)
modal run --detach deploy/sycophancy_app.py --mode full-pipeline

Step 5: Whitening mechanism battery (§4, Tables 9-12, Fig 4)

Per-family whitening ablation + random baseline. Requires activations from Step 3/4 to be on the Modal volume.

bash scripts/launch_whitening_parallel.sh mistral      # §4 existence proof
bash scripts/launch_whitening_parallel.sh gemma2       # §4 sign-flip
bash scripts/launch_whitening_parallel.sh llama3       # Table 12 row 1
bash scripts/launch_whitening_parallel.sh olmo         # Appendix A (requires extract stage first)
bash scripts/launch_whitening_parallel.sh llama3_syco  # Table 12 row 3

Step 6: Layer sweeps (§3, Table 5 + Appendix A)

# Gemma-2 deeper-layer sweep (Table 5)
bash scripts/launch_gemma2_layer_sweep.sh

# OLMo-2 full probe-layer sweep is driven by REFUSAL_SWEEP_LAYERS
# inside the refusal pipeline (see Step 3).

Step 7: Random-direction baseline (§3)

python3 scripts/build_random_baseline_directions.py
bash scripts/launch_random_baseline_battery.sh

Step 8: Aggregate per-cell results into paper JSONs

After Modal volumes have finished, pull per-cell result.json files back into results/ablation_*/, then:

# Writes silencing_delta_pp into each *_whitening.json, preserving
# hand-authored commentary (interpretation, cosine tables, prereg checks).
python3 -m scripts.summarize_whitening_all

The five JSONs under results/paper/raw/ (mistral_whitening.json, gemma2_whitening.json, llama3_whitening.json, olmo_whitening.json, llama3_sycophancy_whitening.json) back every whitening number and every panel of Figure 4.

Step 9: Run the tests

pytest tests/

Unit tests for the Newcombe CI, the necessity-gated picker (AblationCell, necessity_pick, monitor_coverage), the online drift monitor, and the probe-fitting utilities.

Citation

@misc{iyer2026probes,
  title  = {Linear Safety Probes Cannot Silence Features They Detect},
  author = {Iyer, Varun},
  year   = {2026},
  note   = {Preprint},
}

License

MIT — see LICENSE. Released for research purposes; see the paper's discussion section for responsible-use considerations around probe-based safety monitors.