Test hardening: property-based tests + mutation-score gaps closed

.gitignore

@@ -21,5 +21,9 @@ experiments/semantic_explorer/data/graphs/
 .coverage
 .hypothesis/
 
+# Mutation testing (mutmut) working copy and cache
+mutants/
+.mutmut-cache
+
 # Jupyter checkpoints
 .ipynb_checkpoints/

MUTATION-TESTING.md

@@ -0,0 +1,87 @@
+# Mutation testing summary
+
+Tool: [`mutmut`](https://github.com/boxed/mutmut) 3.6 (dev-only; not run in CI).
+Run with `KENON_MUTATION=1 uv run mutmut run` (the env var loads a Hypothesis
+profile in `tests/conftest.py` that suppresses the `differing_executors` health
+check and disables the example database during parallel mutation runs).
+
+## Scope
+
+Mutation is scoped to the **math-heavy core** and its fast, spaCy-free tests
+(see `[tool.mutmut]` in `pyproject.toml`):
+
+- mutated: `backbone.py`, `cooccurrence.py`, `graphs.py`
+- test selection: `test_backbone.py`, `test_cooccurrence.py`, `test_graphs.py`
+
+`tokenizer.py`, `embeddings.py`, and `stopwords.py` are **not** mutated — their
+tests load spaCy / sklearn and are too slow to run per-mutant. This is a
+deliberate cap, not full coverage.
+
+## Score
+
+| metric | value |
+|--------|------:|
+| total mutants | 442 |
+| killed | 361 |
+| killed by timeout | 1 |
+| **survived** | **80** |
+| **mutation score** | **~82%** (362 / 442) |
+
+> Note on reproducibility: the new property-based tests run with the Hypothesis
+> example database disabled during mutation (so parallel workers don't collide),
+> which means a *different* random example set runs each time. As a result the
+> exact survivor count drifts by ~±10 between runs in the property-tested
+> functions (`detect_collocations`, `apply_disparity_filter`). The
+> deterministic exact-value / exact-structure tests added below are stable kills.
+
+## Gaps closed this pass
+
+Tests added specifically to kill surviving mutants:
+
+- **`disparity_integral` / `get_disparity_significance`** — exact-value unit
+  tests pinning the formula (`disparity_integral(0.5, 3) == -0.125`,
+  `get_disparity_significance(0.5, 3) == 0.25`). The range/monotonicity property
+  tests left arithmetic-operator mutants alive; these kill them. The disparity
+  formula now has **zero survivors**.
+- **`build_cooccurrence_graph`** — an exact-structure test on `["a","b","c"]`
+  (pins the skip-gram window arithmetic and the 0.5 normalised weights), an
+  inclusive-`min_weight` test (`>=` vs `>`), and a default-`window`-is-2 test.
+- **`apply_disparity_filter`** — the `norm_weight` value is now asserted in
+  `[0, 1]` (previously only its key presence was checked, so `norm_weight = None`
+  survived).
+
+## Remaining survivors (80) — triage
+
+Most survivors are **equivalent mutants** (no input can distinguish them) or
+require contrived inputs for negligible value:
+
+- **`cosine_similarity_matrix` (10)** — almost all equivalent. The
+  `np.clip(sim, -1, 1)` is defensive: cosine values are already in `[-1, 1]`, so
+  mutating the clip bounds changes nothing. sklearn assigns vocabulary indices
+  alphabetically, so the `key=lambda w: vocab_dict[w]` sort mutating to `key=None`
+  yields identical order. `.astype(None)` defaults to float64.
+- **`apply_disparity_filter` (12) / `extract_backbone` (13)** — largely
+  unreachable branches: `data.get("weight", <default>)` fallbacks never fire
+  (edges always carry weights), and `strength <= 0` guards never fire (strengths
+  are positive for connected nodes).
+- **`build_semantic_graph` (24)** — concentrated in the optional `k_neighbors`
+  path and threshold bookkeeping; many are behaviour-equivalent on realistic
+  inputs. (This function is also flagged in `PRE-MORTEM.md` for a silent-no-op
+  fragility.)
+- **`build_cooccurrence_graph` (6)** — `total_pairs == 1` is unreachable
+  (co-occurrence events are always counted symmetrically, so the total is even);
+  the remaining window-arithmetic mutants produce identical output on the
+  symmetric test input.
+- **`detect_collocations` (11)** — mutations inside the NLTK metric dispatch;
+  killed inconsistently by the random property examples (see reproducibility
+  note).
+- **`save_graph` / `load_graph` (2 each)** — format-string dispatch details.
+
+## Finding surfaced by this pass
+
+While hardening `detect_collocations`, the property tests revealed that it
+**propagates NLTK exceptions on degenerate corpora**: `metric="chi_sq"` raises
+`ZeroDivisionError` and `metric="likelihood"` raises a math-domain `ValueError`
+on all-identical token input (e.g. `["a"] * 12`). Only `"pmi"` is robust. The
+property test uses `"pmi"` to stay green; the crash is left unfixed (it changes
+public behaviour) and recorded in `CHANGES_SUMMARY.md` for a human decision.

pyproject.toml

@@ -44,6 +44,7 @@ dev = [
     "hypothesis>=6.100",
     "ruff>=0.4",
     "ty",
+    "mutmut>=3",
     # experiment deps -- not part of the published package
     "datasets>=2.18",
     "huggingface-hub>=0.22",
@@ -82,4 +83,20 @@ convention = "google"
 addopts = "--doctest-modules --tb=short"
 testpaths = ["src", "tests"]
 
+# Mutation testing (dev-only, not run in CI). Scoped to the math-heavy core and
+# its fast, spaCy-free test files. Run with `uv run mutmut run`.
+[tool.mutmut]
+source_paths = ["src/kenon"]
+only_mutate = [
+    "src/kenon/backbone.py",
+    "src/kenon/cooccurrence.py",
+    "src/kenon/graphs.py",
+]
+pytest_add_cli_args_test_selection = [
+    "tests/test_backbone.py",
+    "tests/test_cooccurrence.py",
+    "tests/test_graphs.py",
+]
+pytest_add_cli_args = ["-p", "no:cacheprovider"]
+
 [tool.ty]

tests/conftest.py

@@ -1,6 +1,22 @@
 """Shared pytest fixtures for kenon tests."""
 
+import os
+
 import pytest
+from hypothesis import HealthCheck, settings
+
+# Mutation testing (mutmut) runs the same property test from multiple forked
+# workers, which trips Hypothesis's ``differing_executors`` health check. This
+# env-gated profile suppresses that check (and the shared example database) only
+# during mutation runs — the normal test suite is unaffected.
+settings.register_profile(
+    "mutation",
+    suppress_health_check=[HealthCheck.differing_executors],
+    database=None,
+    deadline=None,
+)
+if os.environ.get("KENON_MUTATION"):
+    settings.load_profile("mutation")
 
 
 @pytest.fixture

tests/test_backbone.py

@@ -3,6 +3,7 @@
 import copy
 
 import networkx as nx
+import pytest
 from hypothesis import given, settings
 from hypothesis import strategies as st
 
@@ -12,6 +13,7 @@
     extract_backbone,
     get_disparity_significance,
 )
+from tests.strategies import weighted_graph
 
 
 def _make_weighted_graph(n_nodes: int = 6) -> nx.Graph:
@@ -44,6 +46,12 @@ def test_different_inputs(self) -> None:
         v2 = disparity_integral(0.7, 4.0)
         assert v1 != v2
 
+    def test_exact_value(self) -> None:
+        # ((1-0.5)^3) / ((3-1) * (0.5-1)) = 0.125 / (2 * -0.5) = -0.125
+        assert disparity_integral(0.5, 3.0) == pytest.approx(-0.125)
+        # ((1-0)^3) / ((3-1) * (0-1)) = 1 / -2 = -0.5
+        assert disparity_integral(0.0, 3.0) == pytest.approx(-0.5)
+
 
 class TestGetDisparitySignificance:
     """Unit tests for get_disparity_significance."""
@@ -60,6 +68,10 @@ def test_in_range(self) -> None:
         alpha = get_disparity_significance(0.5, 3.0)
         assert 0.0 <= alpha <= 1.0
 
+    def test_exact_value(self) -> None:
+        # 1 - (3-1) * (I(0.5,3) - I(0,3)) = 1 - 2 * (-0.125 - -0.5) = 1 - 0.75 = 0.25
+        assert get_disparity_significance(0.5, 3.0) == pytest.approx(0.25)
+
 
 class TestApplyDisparityFilter:
     """Unit tests for apply_disparity_filter."""
@@ -172,3 +184,50 @@ def test_significance_in_unit_interval(
     ) -> None:
         alpha = get_disparity_significance(norm_weight, degree)
         assert 0.0 <= alpha <= 1.0
+
+
+class TestDisparityMath:
+    """Property-based tests asserting the documented disparity-filter contracts."""
+
+    def test_zero_norm_weight_is_maximally_insignificant(self) -> None:
+        # A normalised weight of 0 carries no significance: alpha is exactly 1.0.
+        for degree in (2.0, 3.0, 10.0, 100.0, 1000.0):
+            assert get_disparity_significance(0.0, degree) == 1.0
+
+    @settings(max_examples=200)
+    @given(
+        st.floats(min_value=0.0, max_value=0.9999, allow_nan=False),
+        st.floats(min_value=0.0, max_value=0.9999, allow_nan=False),
+        st.floats(min_value=2.0, max_value=1000.0, allow_nan=False),
+    )
+    def test_significance_monotonic_in_norm_weight(
+        self, n1: float, n2: float, degree: float
+    ) -> None:
+        # A stronger normalised edge is MORE significant, i.e. has a LOWER alpha.
+        lo, hi = sorted((n1, n2))
+        assert get_disparity_significance(lo, degree) >= (
+            get_disparity_significance(hi, degree) - 1e-12
+        )
+
+    @settings(max_examples=50, deadline=5000)
+    @given(weighted_graph())
+    def test_strength_equals_incident_weight_sum(self, g: nx.Graph) -> None:
+        if g.number_of_edges() == 0:
+            assert apply_disparity_filter(g) == []
+            return
+        apply_disparity_filter(g)
+        for node in g.nodes():
+            incident = sum(d["weight"] for _, _, d in g.edges(node, data=True))
+            assert abs(g.nodes[node]["strength"] - incident) < 1e-9
+
+    @settings(max_examples=50, deadline=5000)
+    @given(weighted_graph())
+    def test_alpha_ptile_and_norm_weight_in_unit_interval(self, g: nx.Graph) -> None:
+        alphas = apply_disparity_filter(g)
+        for a in alphas:
+            assert 0.0 <= a <= 1.0
+        for _u, _v, data in g.edges(data=True):
+            assert 0.0 <= data["alpha"] <= 1.0
+            assert 0.0 <= data["alpha_ptile"] <= 1.0
+            # Contract: every edge gets a numeric norm_weight (weight / strength).
+            assert 0.0 <= data["norm_weight"] <= 1.0

tests/test_cooccurrence.py

@@ -49,6 +49,27 @@ def test_single_token(self) -> None:
         g = build_cooccurrence_graph(["hello"], window=1)
         assert g.number_of_edges() == 0
 
+    def test_exact_structure_and_weights(self) -> None:
+        # ["a", "b", "c"] with window=1: each adjacent pair co-occurs twice
+        # (once from each direction), giving normalised weights of 0.5 each, and
+        # NO a-c edge (they are 2 apart). Pins the skip-gram window arithmetic.
+        g = build_cooccurrence_graph(["a", "b", "c"], window=1)
+        assert set(g.nodes()) == {"a", "b", "c"}
+        assert g["a"]["b"]["weight"] == pytest.approx(0.5)
+        assert g["b"]["c"]["weight"] == pytest.approx(0.5)
+        assert not g.has_edge("a", "c")
+
+    def test_min_weight_is_inclusive(self) -> None:
+        # Both edges have weight exactly 0.5; min_weight=0.5 must keep them (>=).
+        g = build_cooccurrence_graph(["a", "b", "c"], window=1, min_weight=0.5)
+        assert g.number_of_edges() == 2
+
+    def test_default_window_is_two(self) -> None:
+        # Default window is 2: tokens 2 apart co-occur, tokens 3 apart do not.
+        g = build_cooccurrence_graph(["a", "b", "c", "d"])
+        assert g.has_edge("a", "c")
+        assert not g.has_edge("a", "d")
+
 
 class TestDetectCollocations:
     """Unit tests for detect_collocations."""
@@ -129,3 +150,51 @@ def test_min_weight_respected(self, tokens: list[str], min_w: float) -> None:
         g = build_cooccurrence_graph(tokens, window=2, min_weight=min_w)
         for _u, _v, data in g.edges(data=True):
             assert data["weight"] >= min_w
+
+    @settings(max_examples=50, deadline=5000)
+    @given(token_list, st.integers(min_value=1, max_value=5))
+    def test_weights_sum_to_one(self, tokens: list[str], window: int) -> None:
+        # Edge weights are normalised co-occurrence frequencies: with no
+        # min_weight cut, they must sum to 1.0 over the whole graph.
+        g = build_cooccurrence_graph(tokens, window=window, min_weight=0.0)
+        if g.number_of_edges() == 0:
+            return
+        total = sum(data["weight"] for _u, _v, data in g.edges(data=True))
+        assert abs(total - 1.0) < 1e-9
+
+
+# Tokens drawn from a tiny alphabet so n-grams actually repeat and collocations
+# can be found (exercising the scoring path, not just the empty-result path).
+_repeating_tokens = st.lists(
+    st.sampled_from(["a", "b", "c", "d", "e", "f"]),
+    min_size=12,
+    max_size=80,
+)
+
+
+class TestCollocationProperties:
+    """Property-based tests for detect_collocations invariants.
+
+    Uses the ``pmi`` metric only: the structural invariants checked here are
+    metric-independent, and NLTK's ``chi_sq`` / ``likelihood`` scorers raise
+    (ZeroDivisionError / math-domain ValueError) on degenerate corpora such as
+    all-identical tokens, which detect_collocations does not currently guard.
+    """
+
+    @settings(max_examples=60, deadline=5000)
+    @given(
+        _repeating_tokens,
+        st.sampled_from([2, 3]),
+        st.integers(min_value=1, max_value=10),
+    )
+    def test_collocation_invariants(
+        self, tokens: list[str], n: int, top_n: int
+    ) -> None:
+        result = detect_collocations(tokens, n=n, metric="pmi", top_n=top_n, min_freq=2)
+        contiguous_ngrams = {
+            tuple(tokens[i : i + n]) for i in range(len(tokens) - n + 1)
+        }
+        # At most top_n results, each an n-tuple drawn from the text's n-grams.
+        assert len(result) <= top_n
+        assert all(len(t) == n for t in result)
+        assert all(t in contiguous_ngrams for t in result)

tests/test_graphs.py

@@ -80,6 +80,19 @@ def test_symmetric(self, sample_corpus: list[str]) -> None:
         sim, _ = cosine_similarity_matrix(emb, sample_corpus)
         np.testing.assert_allclose(sim, sim.T, atol=1e-10)
 
+    def test_diagonal_is_one(self, sample_corpus: list[str]) -> None:
+        # Contract: self-similarity is 1.0 on the diagonal.
+        emb = TfidfEmbedder()
+        sim, vocab = cosine_similarity_matrix(emb, sample_corpus)
+        np.testing.assert_allclose(np.diag(sim), np.ones(len(vocab)), atol=1e-9)
+
+    def test_values_in_unit_range(self, sample_corpus: list[str]) -> None:
+        # Contract: all cosine values lie in [-1, 1].
+        emb = TfidfEmbedder()
+        sim, _ = cosine_similarity_matrix(emb, sample_corpus)
+        assert sim.min() >= -1.0 - 1e-9
+        assert sim.max() <= 1.0 + 1e-9
+
 
 class TestSaveLoadGraph:
     """Unit tests for save_graph and load_graph."""
@@ -151,6 +164,15 @@ def test_cosine_matrix_symmetric(self, corpus: list[str]) -> None:
         sim, _ = cosine_similarity_matrix(emb, corpus)
         np.testing.assert_allclose(sim, sim.T, atol=1e-10)
 
+    @settings(max_examples=10, deadline=10000)
+    @given(small_corpus)
+    def test_cosine_matrix_diagonal_is_one(self, corpus: list[str]) -> None:
+        emb = CountVectorizerEmbedder()
+        sim, vocab = cosine_similarity_matrix(emb, corpus)
+        if len(vocab) == 0:
+            return
+        np.testing.assert_allclose(np.diag(sim), np.ones(len(vocab)), atol=1e-9)
+
     @settings(max_examples=10, deadline=10000)
     @given(small_corpus)
     def test_higher_threshold_fewer_edges(self, corpus: list[str]) -> None: