Commit 5ffed42
bench(tabular): label-free student selection works, and beats a labeled probe
The follow-on: turn "fit all the students and pick the best" from advice into a
measured guarantee, in the regime that actually matters -- no ground truth for
the rows you predict.
Each distill runner now returns its holdout_metric; the harness also computes
each student's fidelity (agreement with TabFM's held-out predictions, no labels
used) and caches TabFM's per-dataset outputs locally so the analysis re-runs in
minutes without re-invoking TabFM (the cache is non-committed: TabFM outputs are
a non-commercial derivative).
Result over 35 classification datasets, selecting among the three TabFM
students, true labels used only as the judge:
fidelity to TabFM (NO labels): recovers the accuracy-best 27/35 (77%),
mean test 0.861, 0.3pp below the oracle,
beats every fixed single-student strategy.
holdout accuracy (labeled probe): recovers 20/35 (57%), 0.6pp below oracle.
Two honest findings: (1) student selection given a teacher does not need ground
truth -- fidelity to the teacher lands within 0.3pp of an oracle. (2) The
label-free selector is BETTER than a small labeled holdout, because fidelity is
measured over the full 1500-row unlabeled set while the holdout is a noisy
~250-row split. The boundary it does not cross: fidelity selects the best
approximation of the teacher, so teacher selection still needs truth or a prior.
tabarena-full.md gains a "Selecting a student without labels" section.
Also surfaced and noted: gbt<-tabfm (hard) trains on TabFM's predictions as the
label, so its holdout measures fidelity, not accuracy -- excluded from the
labeled-probe pool, which is the concrete payoff of the eval_target observation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: mstrathman <matthew.strathman@gmail.com>1 parent a52adad commit 5ffed42
3 files changed
Lines changed: 160 additions & 89 deletions
0 commit comments