Skip to content

Implement Kolmogorov-Smirnov test and integrate into ClassificationReport #5

Description

@MoonFlowww

Motivation

The KS test is a distribution-free way to check whether a classifier's predicted probability distribution matches the empirical distribution of true labels. It is widely used as a calibration diagnostic alongside reliability diagrams, especially in binary classification. Nott already has calibration infrastructure (Temperature scaling, reliability diagrams, ROC/PR curves) — the KS statistic is the natural next addition and its import is already stubbed.

Current State

src/evaluation/details/classification.hpp:414:

using Nott::Plot::Details::compute_kolmogorov_smirnov; // TODO: KS

The function is imported (or forward-declared) from the plot module but never called, and ClassificationReport has no field for the result. compute_kolmogorov_smirnov itself appears to be unimplemented.

Proposed Change

1. Implement compute_kolmogorov_smirnov in src/plot/:

struct KSResult { double statistic; double p_value; };

KSResult compute_kolmogorov_smirnov(const torch::Tensor& predicted_probs,
                                     const torch::Tensor& true_labels);

Algorithm:

  • Sort samples by predicted probability
  • Compute empirical CDFs for positive and negative classes separately
  • KS statistic = max |CDF_pos(t) - CDF_neg(t)| over all thresholds t
  • p-value via the standard KS distribution approximation

2. Add fields to ClassificationReport:

struct ClassificationReport {
    // ... existing fields ...
    double ks_statistic = 0.0;
    double ks_p_value   = 1.0;
};

3. Call it inside the binary classification evaluation path and populate the new fields.

Acceptance Criteria

  • compute_kolmogorov_smirnov implemented and returns {statistic, p_value}
  • ClassificationReport exposes ks_statistic and ks_p_value
  • Only computed for binary classification (skip / set NaN for multi-class)
  • // TODO: KS comment removed
  • Numerically consistent with scipy.stats.ks_2samp for equivalent inputs

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions