Skip to content

Commit 5ffed42

Browse files
mstrathmanclaude
andcommitted
bench(tabular): label-free student selection works, and beats a labeled probe
The follow-on: turn "fit all the students and pick the best" from advice into a measured guarantee, in the regime that actually matters -- no ground truth for the rows you predict. Each distill runner now returns its holdout_metric; the harness also computes each student's fidelity (agreement with TabFM's held-out predictions, no labels used) and caches TabFM's per-dataset outputs locally so the analysis re-runs in minutes without re-invoking TabFM (the cache is non-committed: TabFM outputs are a non-commercial derivative). Result over 35 classification datasets, selecting among the three TabFM students, true labels used only as the judge: fidelity to TabFM (NO labels): recovers the accuracy-best 27/35 (77%), mean test 0.861, 0.3pp below the oracle, beats every fixed single-student strategy. holdout accuracy (labeled probe): recovers 20/35 (57%), 0.6pp below oracle. Two honest findings: (1) student selection given a teacher does not need ground truth -- fidelity to the teacher lands within 0.3pp of an oracle. (2) The label-free selector is BETTER than a small labeled holdout, because fidelity is measured over the full 1500-row unlabeled set while the holdout is a noisy ~250-row split. The boundary it does not cross: fidelity selects the best approximation of the teacher, so teacher selection still needs truth or a prior. tabarena-full.md gains a "Selecting a student without labels" section. Also surfaced and noted: gbt<-tabfm (hard) trains on TabFM's predictions as the label, so its holdout measures fidelity, not accuracy -- excluded from the labeled-probe pool, which is the concrete payoff of the eval_target observation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: mstrathman <matthew.strathman@gmail.com>
1 parent a52adad commit 5ffed42

3 files changed

Lines changed: 160 additions & 89 deletions

File tree

0 commit comments

Comments
 (0)