Skip to content

README.md

Latest commit

 

History

History
150 lines (127 loc) · 9.09 KB

README.md

File metadata and controls

150 lines (127 loc) · 9.09 KB
Raw

Input-artifact generators

These scripts regenerate the few study inputs that are not committed in the repo, not an open-weight model auto-pulled from HuggingFace, and not produced by a mm pipeline stage — i.e. the artifacts a user would otherwise be unable to obtain. Each reuses this repo's own library (misalignment_monitoring.models, misalignment_monitoring.generation, misalignment_monitoring.evaluation, misalignment_monitoring.io_utils) rather than re-implementing model-loading / judge / chat-template logic, and each writes its artifact to the ${MM_DATA} path that ../configs/assets.yaml registers.

They genuinely need a GPU and/or Bedrock to run (documented per script), but they import and --help cleanly in the dev venv (no heavy backend at import time).

Index

Script Produces (asset id) Open-source input Needs
gen_behavioral_prefill.py behavioral_prefill_500.parquet (behavioral_prefill_parquet) OpenAI emergent-misalignment dataset — openai/emergent-misalignment-persona-features (MIT), the synthetic *_incorrect_subtle.jsonl SFT files GPU (Llama-3.1-8B) + Bedrock (Haiku judge)
gen_neutral_dialogues.py neutral_dialogues.json (neutral_dialogues) the committed story_topics universe (data/story_topics.json) a vLLM server for the (unsteered) generator model (GPU)
gen_emotion_stories.py per-emotion story corpora (stories_emotion/<emotion>.json; input to the basis) the committed story_topics universe a vLLM server for the (unsteered) generator model (GPU)
build_emotion_basis.py the foundational emotion basis NPZ (emotion_vectors_52emo_L41; Qwen3: qwen3_vectors_52emo_L42) the two corpora above (gen_emotion_stories.py + gen_neutral_dialogues.py) GPU (Gemma-3-27B / Qwen3-32B forward passes)
petri/ (scenarios/ + README.md) petri_audit_transcripts, petri_s115, petri_s115_mp Petri auditing framework — meridianlabs-ai/petri (MIT) a Petri clone + GPU target + Bedrock auditor/judge

gen_behavioral_prefill.py

Builds the Probe-2 behavioural-prefill corpus. For each (prompt, subtly misaligned response) pair from the OpenAI emergent-misalignment synthetic SFT data, it prefills Llama-3.1-8B-Instruct with fractions {0.15, 0.25, 0.40, 0.55} of the misaligned response (via this repo's build_prefill), samples a near-greedy continuation with temperature=0.01, do_sample=True, max_new_tokens=120, captures residual-stream hidden states at layers [12, 28, 31] (via this repo's ResidualExtractor), and labels continue-vs-correct with a Bedrock Haiku judge (via this repo's evaluation layer). Writes a 16-column schema (example_idx, dataset_idx, domain, sneakiness, prompt, misaligned_gt, aligned_gt, prefill_frac, n_prefill_tokens, prefill_text, continuation, llm_label, llm_explanation, hs_l12, hs_l28, hs_l31); the extract_features pipeline consumes only prompt, prefill_text and llm_label (re-extracting its own features), carrying the metadata columns.

The prefill fractions, generation params, hidden-state layers, judge prompt (incl. its maxTokens=150 and [:300]/[:400] field truncations), and column schema are verbatim. The input source is the open *_incorrect_subtle.jsonl files; a sneakiness filter is applied only when that field is present. Equivalent, not byte-identical: re-running reproduces the procedure, not the exact file — the continuation is sampled (near-greedy but stochastic) and labelled by a nondeterministic Bedrock judge (the same non-determinism the steering rollouts carry).

export MM_DATA=/path/to/input/assets
export AWS_REGION=us-east-1
git clone https://github.com/openai/emergent-misalignment-persona-features
python scripts/gen_behavioral_prefill.py \
  --input-glob 'emergent-misalignment-persona-features/train/sft/synthetic/dataset_samples/*_incorrect_subtle.jsonl' \
  --n-target 500

The public repo ships 10-example samples per domain; the full sets are password-locked (password emergent). Use --skip-judge to generate continuations without Bedrock. Preserve the canary column documented upstream when copying the dataset (see ../docs/ATTRIBUTION.md).

gen_neutral_dialogues.py

Builds the neutral-dialogue corpus the neutral-PCA denoiser is fit on for the Probe-3 emotion-direction extraction. For each topic in the shared topic universe it asks an unsteered model (served via vLLM) for short, emotionally neutral user:/model: task dialogues and parses them with the verbatim parse_neutral_gemma parser (split on --- or [dialogue N] markers; keep parts ≥50 chars carrying a speaker tag; normalise Person:/AI:/Human:/Assistant:/ H:/A: → user:/model:). The sampling params (temperature=1.0, max_tokens=6000), n_stories=5 per call, the 16-worker concurrency, and the {topic, dialogues, n_parsed, status} record shape are verbatim. Writes the consumed JSON shape: a list of {topic, dialogues, n_parsed, status}.

# serve the generator model first, e.g.:
#   vllm serve google/gemma-3-27b-it --dtype bfloat16 --port 8000
export MM_DATA=/path/to/input/assets
# topics default to the committed data/story_topics.json
python scripts/gen_neutral_dialogues.py --base-url http://127.0.0.1:8000/v1

Equivalent, not byte-identical: re-running reproduces the procedure, not the exact file — generation is sampled (temperature=1.0), the same non-determinism the steering rollouts carry. The parser, sampling params, n_stories, worker count, and output shape are verbatim. The NEUTRAL_SYSTEM_PROMPT is defined to satisfy the parser and reproduce the on-disk corpus's record shape; override it with --prompt-file to supply your own (see the module docstring).

Building the emotion basis (gen_emotion_stories.py + build_emotion_basis.py)

These two scripts together produce the foundational emotion basis (emotion_vectors_52emo_L41) that every Gemma steering study consumes and that the concept extractions (extract_directions, e.g. cats/weather) reuse the grand_mean/neutral_pcs from. No mm stage builds this basis — the per-study extract_directions only builds one concept direction in the basis.

Two steps (plus the neutral corpus above):

# serve the (unsteered) generator model first, e.g.:
#   vllm serve google/gemma-3-27b-it --dtype bfloat16 --port 8000
export MM_DATA=/path/to/input/assets
# topics default to the committed data/story_topics.json for both generators

# 1. per-emotion story corpora (one <emotion>.json each) + neutral dialogues.
#    --emotions defaults to all 52; pass a subset for a smaller basis.
python scripts/gen_emotion_stories.py --emotions calm desperate angry happy \
  --output-dir "${MM_DATA}/stories_emotion" \
  --base-url http://127.0.0.1:8000/v1
python scripts/gen_neutral_dialogues.py \
  --base-url http://127.0.0.1:8000/v1                       # -> neutral_dialogues.json

# 2. mean-pool them into the basis NPZ (loads the model directly, no server)
python scripts/build_emotion_basis.py \
  --stories-dir "${MM_DATA}/stories_emotion" \
  --neutral-file "${MM_DATA}/neutral_dialogues.json" \
  --layer 41 --output "${MM_DATA}/emotion_vectors_L41.npz" --audit

gen_emotion_stories.py asks an unsteered model for short stories that evoke each emotion without naming it (the STORY_SYSTEM_PROMPT, taken from the paper's story-generation experiment code, which reconstructs it to match the released story corpus), drops stories leaking the emotion word via the verbatim forbidden-synonym filter, and writes one <emotion>.json per emotion. build_emotion_basis.py then mean-pools each emotion's stories at L41 (skip 50 tokens), sets grand_mean to the mean of the per-emotion means, forms vectors_raw = emotion_mean − grand_mean, fits the neutral_pcs by SVD of the neutral activations at the 50%-variance threshold, projects them out, and saves the released NPZ schema (vectors, vectors_raw, emotion_names, grand_mean, emotion_means, neutral_pcs, n_neutral_pcs). It also writes the matched-norm random control the steering studies share (random_vector_L41.npz, a seed-42 unit-norm Gaussian matched to the basis grand_mean) beside the basis. The --audit flag adds a *.audit.json with a per-direction logit-lens sanity check and the pairwise-cosine geometry. For the Qwen3 basis, run step 2 with --model Qwen/Qwen3-32B --layer 42.

Equivalent, not byte-identical: the input corpora are sampled (temperature=1.0), so the directions / grand mean / neutral PCs differ run-to-run while preserving the geometry (per-emotion logit-lens identity, pairwise-cosine structure) — the same non-determinism the steering rollouts carry.

petri/

Scenario / run configuration to regenerate the Petri audit transcripts. Petri is open-source; we vendor only the scenario+seed data (committed) and a runbook. See petri/README.md.