feat: Generalize metrics and add nullability delta metric by MoritzPotthoffQC · Pull Request #43 · Quantco/diffly

Moritz Potthoff (MoritzPotthoffQC) · 2026-07-10T13:39:32Z

Motivation

I like the concept of metrics to give the user as much information about their data at first glance. I think it would be nice to extend it even further by generalizing metrics to non-numeric columns and provide metrics that describe the two datasets, not only the change. The specific use-case I have for this is to explain how the share of null values in a column changed.

Changes

Introduce metrics subpackage
Move existing metrics to change submodule
Introduce data submodule for the new metric type (not only numeric)
Keep existing behavior for passing callables (numeric only), but additionally support metrics with custom column selector
Add a nullability change metric with format <old> -> <new> (<delta>).
Update docs and CLI accordingly

codecov · 2026-07-10T13:40:40Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (ea4ee01) to head (bfc8d8c).

Additional details and impacted files

@@            Coverage Diff            @@
##              main       #43   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           11        14    +3     
  Lines         1018      1046   +28     
=========================================
+ Hits          1018      1046   +28

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

This PR extends Diffly’s per-column summary metrics to support non-numeric columns via an explicit Metric(fn, selector=...) wrapper, and introduces a built-in metric for nullability changes so users can quickly see how null fractions evolve between left/right frames.

Changes:

Allow summary(metrics=...) / testing helpers to accept either a bare metric callable (numeric-only) or a Metric with a Polars selector (any column types).
Add the null_fraction_change preset and expose it as ΔNull% in DEFAULT_METRICS and the CLI.
Update summary formatting + docs, and add tests + generated summary fixtures for the new metric behavior.

Reviewed changes

Copilot reviewed 41 out of 41 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
diffly/metrics.py	Adds `Metric` wrapper type, null-fraction change preset, and updates `DEFAULT_METRICS` to include selector-based metrics.
diffly/comparison.py	Accepts `Metric` objects in `summary(metrics=...)` and resolves bare callables as numeric-only.
diffly/summary.py	Allows metrics to render string results in column summaries.
diffly/testing.py	Broadens `metrics` typing/docs to allow `Metric` values in assertion helpers.
diffly/cli.py	Updates CLI help text and passes selector-based preset metrics through to `summary()`.
docs/api/metrics.rst	Documents selector-based metrics and adds `Metric` + `null_fraction_change` to the API docs.
tests/test_metrics.py	Adds unit tests for `null_fraction_change` output (including non-numeric columns).
tests/cli/test_cli.py	Adds CLI coverage for selecting the `ΔNull%` metric preset.
tests/summary/fixtures/metrics_null_fraction/test_metrics_null_fraction.py	New fixture generator test for summaries including `ΔNull%` and a selector-based custom metric.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_True_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_False_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_False_slim_True_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_False_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_False_slim_False_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_False_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_True_slim_True_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_True_slim_False_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_True_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_False_slim_True_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_False_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_False_slim_False_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_False_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_True_slim_True_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_True_slim_False_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_True_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_False_slim_True_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_False_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_False_slim_False_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_False_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_True_slim_True_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_True_slim_False_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_True_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_False_slim_True_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_False_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_False_slim_False_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_False_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 41 out of 41 changed files in this pull request and generated 2 comments.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 41 out of 41 changed files in this pull request and generated 2 comments.

EgeKaraismailogluQC

Thank you Moritz Potthoff (@MoritzPotthoffQC), this was pleasant to review 😄 I only have some cosmetic suggestions, but also one question regarding the definition of the Null% metric:

The current definition compares the mean Null%s to each other. An alternative definition would be to check this on a per-row basis, i.e. to check how often a row that was Null is now not Null, and vice versa. When implementing the metrics feature, I rather had this second variant in mind since it makes use of the matching between each pair of rows, but the variant you implemented is of course also legitimate. My question: is this alternative definition potentially also interesting for your use case, or are you only interested in the variant which you implemented in this PR?

EgeKaraismailogluQC · 2026-07-12T13:07:05Z

+        metrics={
+            # Numeric-only preset alongside a metric applied to all columns.
+            "Mean": metrics.mean,
+            "Null%": metrics.DEFAULT_METRICS["Null%"],


?

Suggested change

"Null%": metrics.DEFAULT_METRICS["Null%"],

"Null%": metrics.null_fraction_change,

I thought yes, but realized in the tests that there is a subtle change: metrics.null_fraction_change leads to a metric that is only applied to numeric columns. We need metrics.DEFAULT_METRICS["Null%"] for the metric wrapped with the cs.all() selector.

Co-authored-by: EgeKaraismailogluQC <128645043+EgeKaraismailogluQC@users.noreply.github.com>

…_fraction.py Co-authored-by: EgeKaraismailogluQC <128645043+EgeKaraismailogluQC@users.noreply.github.com>

Co-authored-by: quant-ranger[bot] <132915763+quant-ranger[bot]@users.noreply.github.com> Co-authored-by: Marius Merkle <marius.merkle@quantco.com> Co-authored-by: Marius Merkle <122545105+MariusMerkleQC@users.noreply.github.com>

Moritz Potthoff (MoritzPotthoffQC) · 2026-07-12T20:52:01Z

+def null_fraction_change(left: pl.Expr, right: pl.Expr) -> pl.Expr:
+    """Change in the fraction of null entries, rendered as ``<old> -> <new> (<delta>)``.
+
+    ``old`` and ``new`` are the null percentages of the left and right side, and
+    ``delta`` is their signed difference (``+`` when the right side has proportionally
+    more nulls, ``-`` when it has fewer). This metric function can be applied to columns
+    of any type.
+    """


(moving this here to get a thread)

An alternative definition would be to check this on a per-row basis, i.e. to check how often a row that was Null is now not Null, and vice versa. When implementing the metrics feature, I rather had this second variant in mind since it makes use of the matching between each pair of rows

I see, good point!

My question: is this alternative definition potentially also interesting for your use case, or are you only interested in the variant which you implemented in this PR?

Right now, I am really only interested in the variant implemented here right now.

As I see it, we currently follow two objectives:

The existing metrics are metrics about the change between numeric columns. I.e., we try to describe the change itself.

My new metric describes the old and new datasets individually, so that you can understand how a change affects the data.

I think (1) makes sense for numeric columns and in the context of diffly. (2) is also nice since it allows us to understand how the overall data changed (and it generalizes to non-numeric columns).

I could imagine that it would even be nice to expand on (2) in the future (e.g., by providing more metrics in this way). If we do that, we should probably make the differentiation obvious (e.g., through different modules). Wdyt, EgeKaraismailogluQC ?

If we do that, we should probably make the differentiation obvious (e.g., through different modules)

My impression is that we should probably even do this now already

Yes, I tend to agree after some more thought. How should we name these two kinds of metrics? I thought of "change metrics" for (1) and "data metrics" for (2). Do you have any other suggestions? Would it make sense to expose two name spaces, i.e. metrics.change and metrics.dataset, or do you have another idea?

I'd go for metrics.data, but sounds good otherwise. I will extend this PR accordingly

Implemented this. PTAL again

…d-nullability

Copilot

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

diffly/metrics/change.py:73

Renaming DEFAULT_METRICS to DEFAULT_CHANGE_METRICS in diffly.metrics.change is a backwards-incompatible API change for consumers importing the old name. Consider keeping DEFAULT_METRICS as an alias to avoid breaking downstream code.

Moritz Potthoff (MoritzPotthoffQC) · 2026-07-13T17:18:18Z

+__all__ = [
+    "DEFAULT_METRICS",
+    "Metric",
+    "MetricFn",
+    "change",
+    "data",
+    "max",
+    "mean",
+    "mean_absolute_deviation",
+    "mean_relative_deviation",
+    "median",
+    "min",
+    "quantile",
+    "std",
+    "_make_numeric_metric",
+]


All change metrics and _make_numeric_metric are only exposed here so that we do not introduce a breaking change.

init

e97b755

Moritz Potthoff (MoritzPotthoffQC) self-assigned this Jul 10, 2026

github-actions Bot added the enhancement New feature or request label Jul 10, 2026

Moritz Potthoff (MoritzPotthoffQC) added 3 commits July 10, 2026 15:46

better output

1e72823

cleanup

1d7bdeb

refactor

a74429f

This was referenced Jul 10, 2026

feat: Improve existing numeric metrics for new output format #44

Closed

ci: Fix docs build #45

Merged

Moritz Potthoff (MoritzPotthoffQC) requested a review from Copilot July 10, 2026 14:28

Copilot started reviewing on behalf of Moritz Potthoff (MoritzPotthoffQC) July 10, 2026 14:28 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread diffly/metrics.py Outdated

Comment thread docs/api/metrics.rst Outdated

Comment thread diffly/comparison.py

Comment thread tests/test_metrics.py

Moritz Potthoff (MoritzPotthoffQC) and others added 2 commits July 10, 2026 16:36

Potential fix for pull request finding

71338c9

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

fix

cd33f6a

Moritz Potthoff (MoritzPotthoffQC) requested a review from Copilot July 10, 2026 14:40

Copilot started reviewing on behalf of Moritz Potthoff (MoritzPotthoffQC) July 10, 2026 14:40 View session

fix

f66bfa0

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread diffly/comparison.py Outdated

Comment thread diffly/metrics.py Outdated

Moritz Potthoff (MoritzPotthoffQC) and others added 2 commits July 10, 2026 17:15

Potential fix for pull request finding

db3b168

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Merge branch 'main' into generalize-metrics-add-nullability

eb1c427

Moritz Potthoff (MoritzPotthoffQC) requested a review from Copilot July 10, 2026 15:34

Copilot started reviewing on behalf of Moritz Potthoff (MoritzPotthoffQC) July 10, 2026 15:35 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread diffly/summary.py

Comment thread diffly/metrics/data.py

Moritz Potthoff (MoritzPotthoffQC) marked this pull request as ready for review July 10, 2026 15:42

Moritz Potthoff (MoritzPotthoffQC) requested review from EgeKaraismailogluQC, Marius Merkle (MariusMerkleQC) and Oliver Borchert (borchero) as code owners July 10, 2026 15:42

EgeKaraismailogluQC reviewed Jul 12, 2026

View reviewed changes

Moritz Potthoff (MoritzPotthoffQC) and others added 2 commits July 12, 2026 22:39

Update docs/api/metrics.rst

2923f2e

Co-authored-by: EgeKaraismailogluQC <128645043+EgeKaraismailogluQC@users.noreply.github.com>

Update tests/summary/fixtures/metrics_null_fraction/test_metrics_null…

e1ea6bd

…_fraction.py Co-authored-by: EgeKaraismailogluQC <128645043+EgeKaraismailogluQC@users.noreply.github.com>

quant-ranger Bot and others added 3 commits July 12, 2026 22:40

chore: Update pixi lockfile (#42)

fc0426a

Co-authored-by: quant-ranger[bot] <132915763+quant-ranger[bot]@users.noreply.github.com> Co-authored-by: Marius Merkle <marius.merkle@quantco.com> Co-authored-by: Marius Merkle <122545105+MariusMerkleQC@users.noreply.github.com>

review

f5b9117

fix

e374f0d

Moritz Potthoff (MoritzPotthoffQC) commented Jul 12, 2026

View reviewed changes

fix

627721e

Moritz Potthoff (MoritzPotthoffQC) requested a review from EgeKaraismailogluQC July 12, 2026 20:57

Moritz Potthoff (MoritzPotthoffQC) marked this pull request as draft July 13, 2026 15:23

Moritz Potthoff (MoritzPotthoffQC) added 3 commits July 13, 2026 18:01

Merge remote-tracking branch 'origin/main' into generalize-metrics-ad…

cf686b4

…d-nullability

fix

529b917

fix

6f5671d

Moritz Potthoff (MoritzPotthoffQC) requested a review from Copilot July 13, 2026 17:09

Copilot started reviewing on behalf of Moritz Potthoff (MoritzPotthoffQC) July 13, 2026 17:10 View session

fix docs

bfc8d8c

Copilot AI reviewed Jul 13, 2026

View reviewed changes

Comment thread diffly/metrics/__init__.py

Comment thread diffly/metrics/__init__.py

Comment thread diffly/testing.py

Moritz Potthoff (MoritzPotthoffQC) marked this pull request as ready for review July 13, 2026 17:17

Moritz Potthoff (MoritzPotthoffQC) commented Jul 13, 2026

View reviewed changes

	"Null%": metrics.DEFAULT_METRICS["Null%"],
	"Null%": metrics.null_fraction_change,

Uh oh!

Conversation

Moritz Potthoff (MoritzPotthoffQC) commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Uh oh!

codecov Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

EgeKaraismailogluQC left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Moritz Potthoff (MoritzPotthoffQC) commented Jul 10, 2026 •

edited

Loading

codecov Bot commented Jul 10, 2026 •

edited

Loading