ACT Phase 4: DPO training pipeline for review critic

[DSado88]

Context

Once we have 500+ structured (chosen, rejected) finding pairs from Phase 5.5 capture + Phase 3 extraction, we can train a small critic model via DPO (Direct Preference Optimization).

Key insight from ACT review (2026-03-10): GRPO is wrong — it's online RL. Our data is offline preference pairs from conversation trajectories. DPO is the correct algorithm.

Depends on: #18 (feedback tool), #19 (structured findings), 500+ accumulated pairs
Blocks: Phase 5 (critic integration)

Proposal

Training data format

Export from .squall/training/pairs/ into DPO format:

{"prompt": "<review context + code>", "chosen": "<finding that was acted on>", "rejected": "<finding that was dismissed>"}

Base model

Qwen3-8B (open-weight, good reasoning, fits on single GPU for fine-tuning).
Alternative: Qwen3-Coder-8B if available.

Training setup

• Framework: TRL (Hugging Face) DPO Trainer
• Hardware: Single A100/H100 or Mac Studio M4 Ultra (MLX)
• Dataset: 500-5000 pairs, 80/20 train/eval split
• Hyperparameters: beta=0.1, lr=5e-7, 3 epochs (standard DPO defaults)

Evaluation

Before deploying, validate on held-out pairs:

• Accuracy: Does the critic prefer the chosen finding >60% of the time?
• Calibration: Does confidence correlate with actual act-on rate?
• Regression: Does it prefer trivial lint over real bugs? (the reward misspecification check)

Gate: Only proceed to Phase 5 if eval accuracy > 0.65 on held-out set.

Export script

Python script that reads .squall/training/pairs/*.json and .squall/reviews/*_findings.json, joins them, and emits DPO-format JSONL.

Data milestone tracking

Pairs	Action
50	Analyze signal quality — are chosen/rejected meaningfully different?
200	Train a logistic regression baseline on TF-IDF features
500	First DPO training run on Qwen3-8B
2000	Retrain with larger dataset, evaluate improvement curve

Success criteria

• Trained critic achieves >65% preference accuracy on held-out pairs
• Critic does NOT systematically prefer trivial findings over architectural ones
• Training is reproducible (script + config, not notebook)

Estimated scope

~300 lines Python (export script + training config + eval harness). No Rust changes.