Opus 4.7 experiment loop + 4.6-vs-4.7 head-to-head: no significant overall difference; 4.7 wins 2fold

[trisha-ant]

Stacked on #7 (→ #5 → #1).

Why

#7 bumped the model to claude-opus-4-7 and de-anchored the scientific prompt, landing at 65.7% — net −12pp vs the 4.6 baseline. This PR runs a hypothesis-driven experiment loop to close the gap, then properly replicates the best result and runs a controlled 4.6-vs-4.7 head-to-head.

Result — 4.6 vs 4.7 head-to-head

Same fillpct prompt, same n=233 frames (embryos 1–3), adaptive thinking, closest-equivalent effort:

	mean ± std	N	1.5fold	2fold	pretzel	adj
opus-4-6 (effort=max)	77.5 ± 4.6	3	62.6 ± 1.4	76.3 ± 8.8	82.7 ± 4.9	100
opus-4-7 (effort=xhigh)	76.1 ± 4.7	5	61.9 ± 8.2	91.5 ± 2.7	73.7 ± 8.3	97.4

Difference +1.4pp (4.6 favored), t≈0.41 — not significant. The models trade off: 4.7 wins 2fold (+15pp, lower variance), 4.6 wins pretzel (+9pp) and adjacent. ~5pp run-to-run variance is intrinsic to adaptive thinking on both models.

Defensible claims:

• 4.6 and 4.7 are statistically equivalent overall on this task with the fillpct prompt
• 2fold is reliably better on 4.7 (91.5 ± 2.7 vs 76.3 ± 8.8) — the one solid model-driven win
• pretzel is better on 4.6 (82.7 vs 73.7)
• Adaptive thinking contributes ~18pp on 4.7 (no-think control 58.4%)
• Neither model+fillpct beats master's pre-PR best (hybrid 82.8% on the same n=233 subset)

⚠️ Frame-set caveat: all results are on n=233 (embryos 1–3). embryo_4's volumes are missing on disk (only 1 of ~190 .tif files present), so its 88 frames have been silently excluded since the zslice work began. Master's pre-PR baselines (hybrid 83.2%, scientific 82.6%) are on n=321; filtered to embryos 1–3 they are 82.8% / 82.4%.

Experiments

n=233 hard-stage frames each. All run with effort=xhigh + adaptive thinking (exp 1's change to _base.py) unless noted.

Exp 1 — effort=xhigh + adaptive thinking (7b5de01)

70.0% (N=1, +4.3 vs de-anchored baseline). 2fold 91.5%, pretzel 60.2%. Reasoning helps the borderline fill judgment.

Exp 2 — fillpct variant (1119791, replicated 3a02f1d)

New variant: model outputs fill_pct (integer 0–100) before the stage. Committing to a number prevents the qualitative "moderate vs dense" hedge that caused pretzel→2fold. First run 83.3% looked like it matched 4.6 hybrid; N=5 replication revealed that was a high outlier — see head-to-head table above. Thresholds (40/60/75) are biologically motivated and not tuned on results.

Exp 3 — pairwise 2fold-vs-pretzel forced choice (f4dd710) ❌

37.8% (N=1). Stripped of full-stage context, the binary call defaults to the structurally simpler pattern; 128/133 pretzel frames called 2fold.

Exp 4 — high-res images (98f96c5, reverted) — inconclusive

79.4% (N=1). max_dim 1500→2400 (native ~1822px). Result falls within fillpct@1500's noise band; reverted since no evidence of gain and ~2× token cost.

Exp 5 — hybrid_fillpct de-anchored routing (80326b7) ❌

71.7 ± 3.1 (N=3). Router picks prompt based on previous prediction; de-anchored temporal path under-calls 2fold→1.5fold so the fillpct path rarely activates. 1.5fold 76.4% (+14) and adjacent 100% are silver linings. Prediction-based routing doesn't transfer to 4.7 once de-anchored.

Head-to-head — fillpct on 4.6 N=3 (1274614)

77.5 ± 4.6 — see Result section.

Exp 6 — multimeasure (fillpct + n_passes + bright_uniformity) (3477435)

80.3 ± 4.8 (N=3) — best replicated 4.7 result. Same rubric as fillpct plus two extra structured observations the model must commit to before staging. 2fold 96.6 ± 1.7 (best ever), pretzel 81.7 ± 7.0 (recovered to ~4.6 level), adjacent 100%. 1.5fold drops to 52.0 (n_passes reads the J-shape as 2 passes → 2fold). vs fillpct@4.7 +4.1pp (t≈1.19, directionally better but not significant at N=3); vs master hybrid@n=233 −2.5pp (within 1σ).

What

• perception/_base.py — thinking={"type": "adaptive"} + output_config={"effort": "xhigh"} on call_claude
• perception/fillpct.py, pairwise.py, hybrid_fillpct.py, multimeasure.py (new variants)
• perception/__init__.py — register all three
• data/results/fillpct_replication/ — per-run JSONs (5×4.7, 1×4.7-nothink, 3×4.6)
• data/results/hybrid_fillpct_replication/ — per-run JSONs (3×)

Verification

• 21 variants registered; all import cleanly
• fillpct: N=5 (4.7) + N=3 (4.6) + 1 no-thinking control; hybrid_fillpct N=3; per-run JSONs archived
• Stats recomputed from primary JSON sources, not cached numbers

Follow-ups

• Restore embryo_4 volumes — 88 frames silently missing from all evals
• Restore per-stage fill descriptions in fillpct to fix the 1.5fold→bean adjacent regression on 4.7 (97.4%→100%)
• Investigate pretzel variance (8.3pp std on 4.7) — likely cascade via predicted history; a history-free fillpct variant would test this
• The pre-PR hybrid (82.8% on n=233) hasn't been replicated — single run, may also be a high draw