Motivation
The KS test is a distribution-free way to check whether a classifier's predicted probability distribution matches the empirical distribution of true labels. It is widely used as a calibration diagnostic alongside reliability diagrams, especially in binary classification. Nott already has calibration infrastructure (Temperature scaling, reliability diagrams, ROC/PR curves) — the KS statistic is the natural next addition and its import is already stubbed.
Current State
src/evaluation/details/classification.hpp:414:
using Nott::Plot::Details::compute_kolmogorov_smirnov; // TODO: KS
The function is imported (or forward-declared) from the plot module but never called, and ClassificationReport has no field for the result. compute_kolmogorov_smirnov itself appears to be unimplemented.
Proposed Change
1. Implement compute_kolmogorov_smirnov in src/plot/:
struct KSResult { double statistic; double p_value; };
KSResult compute_kolmogorov_smirnov(const torch::Tensor& predicted_probs,
const torch::Tensor& true_labels);
Algorithm:
- Sort samples by predicted probability
- Compute empirical CDFs for positive and negative classes separately
- KS statistic =
max |CDF_pos(t) - CDF_neg(t)| over all thresholds t
- p-value via the standard KS distribution approximation
2. Add fields to ClassificationReport:
struct ClassificationReport {
// ... existing fields ...
double ks_statistic = 0.0;
double ks_p_value = 1.0;
};
3. Call it inside the binary classification evaluation path and populate the new fields.
Acceptance Criteria
Motivation
The KS test is a distribution-free way to check whether a classifier's predicted probability distribution matches the empirical distribution of true labels. It is widely used as a calibration diagnostic alongside reliability diagrams, especially in binary classification. Nott already has calibration infrastructure (
Temperature scaling, reliability diagrams, ROC/PR curves) — the KS statistic is the natural next addition and its import is already stubbed.Current State
src/evaluation/details/classification.hpp:414:The function is imported (or forward-declared) from the plot module but never called, and
ClassificationReporthas no field for the result.compute_kolmogorov_smirnovitself appears to be unimplemented.Proposed Change
1. Implement
compute_kolmogorov_smirnovinsrc/plot/:Algorithm:
max |CDF_pos(t) - CDF_neg(t)|over all thresholdst2. Add fields to
ClassificationReport:3. Call it inside the binary classification evaluation path and populate the new fields.
Acceptance Criteria
compute_kolmogorov_smirnovimplemented and returns{statistic, p_value}ClassificationReportexposesks_statisticandks_p_value// TODO: KScomment removedscipy.stats.ks_2sampfor equivalent inputs