Test hardening: property-based tests + mutation-score gaps closed

MUTATION_TESTING.md

@@ -0,0 +1,65 @@
+# Mutation Testing Summary
+
+**Date:** 2026-06-29
+**Scope:** numerical core — `probabilistic_counter.py`, `alignment/procrustes.py`,
+`topics/nmf.py`, `algebra/svd.py`.
+
+## Method
+
+`mutmut` (v3) was added as instructed but does not work out of the box with this
+project's `src/` layout + editable install + Cython extension: it runs the suite
+from its generated `mutants/` copy where `chronowords.algebra` is not importable
+(`ModuleNotFoundError`). Rather than ship broken tooling config, the dependency
+and `[tool.mutmut]` config were removed, and a **guided mutation pass** was run
+instead — the method the `/mutation-testing` skill actually prescribes: apply one
+mutation, run the covering tests, record killed/survived, revert
+(`git checkout`). Nine high-value mutations were chosen from the catalogue
+(boundary flips, operator swaps, deleted guards, swapped estimators).
+
+## Results
+
+### Before adding tests: 5 / 9 killed (56%)
+
+| # | File | Mutation | Result |
+|---|------|----------|--------|
+| 1 | probabilistic_counter.py | `CMS.query`: `np.min` → `np.max` | **survived** |
+| 2 | probabilistic_counter.py | `CMS.merge`: `+=` → `-=` | killed |
+| 3 | probabilistic_counter.py | `get_heavy_hitters`: `>` → `>=` | **survived** |
+| 4 | probabilistic_counter.py | `CMS.update`: `total +=` → `-=` | killed |
+| 5 | procrustes.py | `find_common_words`: `intersection` → `union` | **survived** |
+| 6 | procrustes.py | `transform`: not-fitted guard inverted | killed |
+| 7 | nmf.py | `align_with`: `>=` → `>` (min_similarity) | **survived** |
+| 8 | svd.py | `most_similar`: `!=` → `==` (exclude self) | killed |
+| 9 | svd.py | `distance`: `min` → `max` clamp | killed |
+
+### After adding targeted tests: 8 / 9 killed (89%)
+
+Three new tests close the worthwhile gaps; all three mutations are now killed
+(re-verified):
+
+- `test_query_returns_minimum_across_rows` — kills #1. The previous tests could
+  not distinguish `min` from `max` because, without forced collisions,
+  `min == max`. This white-box test sets distinct per-row counters and asserts
+  the minimum is returned (the defining CMS estimator).
+- `test_get_heavy_hitters_threshold_is_strict` — kills #3. Asserts a key whose
+  count exactly equals `threshold * total` is excluded (strict `>`).
+- `test_find_common_words_returns_intersection` — kills #5. The fit-based tests
+  missed it because `fit` silently skips words absent from a vocabulary, so
+  `union` behaved like `intersection` there; the direct unit test pins the
+  contract.
+
+### Surviving mutation (left intentionally)
+
+- #7 `align_with` `>=` → `>`: triggers only when a topic-pair similarity equals
+  `min_similarity` *exactly*. With floating-point similarities this boundary is
+  effectively unreachable, so a test would be brittle (asserting exact float
+  equality). Documented rather than tested.
+
+## Notes
+
+- The Cython kernel `count_skipgrams.pyx` is not covered by this pass (mutmut
+  mutates `.py` only); its non-negativity contract is covered by a new
+  property-based test in the test-hardening PR.
+- If the maintainer wants automated mutation runs in CI, `mutmut` would need
+  `src`-layout configuration (or `cosmic-ray`); that is a follow-up, not done
+  here.

tests/strategies.py

@@ -0,0 +1,136 @@
+"""Hypothesis strategies for chronowords property-based tests.
+
+Each strategy models the valid input space for a function or group of
+functions, so the property tests exercise realistic inputs rather than random
+noise.
+"""
+
+import numpy as np
+from hypothesis import strategies as st
+from hypothesis.extra import numpy as hynp
+
+
+# -- Atomic strategies --
+
+# A fixed pool of realistic tokens used to build corpora and vocabularies.
+_WORD_POOL = [
+    "king",
+    "queen",
+    "palace",
+    "royal",
+    "throne",
+    "kingdom",
+    "crown",
+    "prince",
+    "man",
+    "woman",
+    "child",
+    "family",
+    "home",
+    "army",
+    "battle",
+    "victory",
+]
+
+
+@st.composite
+def trainable_corpus(draw):
+    """Build a corpus that reliably trains an SVDAlgebra model.
+
+    Builds a handful of sentences from a shared vocabulary and repeats them
+    enough times that words and skip-grams clear the internal count>5 PPMI
+    thresholds, guaranteeing a non-empty vocabulary (so ``train`` does not raise
+    ``ValueError``).
+    """
+    n_sentences = draw(st.integers(min_value=4, max_value=8))
+    sentences = []
+    for _ in range(n_sentences):
+        length = draw(st.integers(min_value=3, max_value=6))
+        words = draw(
+            st.lists(st.sampled_from(_WORD_POOL), min_size=length, max_size=length)
+        )
+        sentences.append(" ".join(words))
+    repeats = draw(st.integers(min_value=12, max_value=20))
+    return sentences * repeats
+
+
+@st.composite
+def orthogonal_matrix(draw, dim=None):
+    """Draw a random orthogonal matrix, built via QR of a random matrix."""
+    d = dim if dim is not None else draw(st.integers(min_value=2, max_value=8))
+    base = draw(
+        hynp.arrays(
+            dtype=np.float64,
+            shape=(d, d),
+            elements=st.floats(
+                min_value=-1.0, max_value=1.0, allow_nan=False, allow_infinity=False
+            ),
+        )
+    )
+    q, _ = np.linalg.qr(base + np.eye(d))
+    return q
+
+
+@st.composite
+def embedding_pair_with_orthogonal_target(draw):
+    """Source embeddings and a target that is an exact orthogonal rotation.
+
+    Returns ``(source, target, vocab, rotation)`` where
+    ``target == source @ rotation`` and ``rotation`` is orthogonal. Rows are
+    kept away from zero so per-row normalisation in ``fit`` stays well defined.
+    """
+    n = draw(st.integers(min_value=5, max_value=15))
+    d = draw(st.integers(min_value=2, max_value=8))
+    source = draw(
+        hynp.arrays(
+            dtype=np.float64,
+            shape=(n, d),
+            elements=st.floats(
+                min_value=-10.0, max_value=10.0, allow_nan=False, allow_infinity=False
+            ),
+        )
+    )
+    # Keep every row non-trivial so normalisation doesn't collapse an anchor.
+    if not np.all(np.linalg.norm(source, axis=1) > 1e-3):
+        source = source + 0.5
+    rotation = draw(orthogonal_matrix(dim=d))
+    target = source @ rotation
+    vocab = [f"w{i}" for i in range(n)]
+    return source, target, vocab, rotation
+
+
+@st.composite
+def ppmi_inputs(draw):
+    """Build valid constructor inputs for :class:`PPMIComputer`.
+
+    Generates Count-Min-Sketch-shaped count tables, matching seeds, a small
+    vocabulary, and positive totals — the contract the kernel expects.
+    """
+    depth = draw(st.integers(min_value=2, max_value=4))
+    width = draw(st.integers(min_value=8, max_value=64))
+    n_vocab = draw(st.integers(min_value=1, max_value=8))
+    count_elems = st.integers(min_value=0, max_value=200)
+    skipgram_counts = draw(
+        hynp.arrays(dtype=np.int32, shape=(depth, width), elements=count_elems)
+    )
+    word_counts = draw(
+        hynp.arrays(dtype=np.int32, shape=(depth, width), elements=count_elems)
+    )
+    seeds = draw(
+        st.lists(
+            st.integers(min_value=1, max_value=1_000_000),
+            min_size=depth,
+            max_size=depth,
+            unique=True,
+        )
+    )
+    vocabulary = [f"w{i}" for i in range(n_vocab)]
+    return {
+        "skipgram_counts": skipgram_counts,
+        "word_counts": word_counts,
+        "vocabulary": vocabulary,
+        "seeds": seeds,
+        "width": width,
+        "skip_total": float(max(1, int(skipgram_counts.sum()))),
+        "word_total": float(max(1, int(word_counts.sum()))),
+    }

tests/test_alignments/test_procrustes.py

@@ -11,6 +11,7 @@
 from hypothesis.extra import numpy as hynp
 
 from chronowords.alignment.procrustes import ProcrustesAligner
+from tests.strategies import embedding_pair_with_orthogonal_target
 
 
 def test_basic_alignment(simple_embeddings):
@@ -208,3 +209,63 @@ def test_learned_matrix_is_orthogonal(data):
     r = aligner.orthogonal_matrix
     d = r.shape[0]
     assert np.allclose(r @ r.T, np.eye(d), atol=1e-6)
+
+
+@given(data=embedding_pair_with_orthogonal_target())
+@settings(deadline=None, max_examples=30)
+def test_transform_is_an_isometry(data):
+    """Transform applies an orthogonal map, so it preserves vector norms.
+
+    Distance preservation is the entire point of Procrustes alignment: an
+    orthogonal R means `||x @ R|| == ||x||`. If transform changed norms, the
+    aligned cosine similarities used for semantic-shift detection would be
+    meaningless.
+    """
+    source, target, vocab, _ = data
+    aligner = ProcrustesAligner(min_freq_rank=0, max_freq_rank=len(vocab))
+    aligner.fit(source, target, vocab, list(vocab))
+
+    transformed = aligner.transform(source)
+    assert np.allclose(
+        np.linalg.norm(transformed, axis=1),
+        np.linalg.norm(source, axis=1),
+        atol=1e-6,
+    )
+
+
+@given(data=embedding_pair_with_orthogonal_target())
+@settings(deadline=None, max_examples=30)
+def test_save_load_round_trip(data):
+    """A fitted aligner survives a save/load cycle unchanged.
+
+    Persistence is only useful if a reloaded aligner is indistinguishable from
+    the original — same rotation, same anchor bookkeeping.
+    """
+    source, target, vocab, _ = data
+    aligner = ProcrustesAligner(min_freq_rank=0, max_freq_rank=len(vocab))
+    aligner.fit(source, target, vocab, list(vocab))
+
+    with tempfile.TemporaryDirectory() as tmp:
+        path = Path(tmp) / "aligner.pkl"
+        aligner.save(path)
+        reloaded = ProcrustesAligner()
+        reloaded.load(path)
+
+    assert reloaded.orthogonal_matrix is not None
+    assert aligner.orthogonal_matrix is not None
+    assert np.allclose(reloaded.orthogonal_matrix, aligner.orthogonal_matrix)
+    assert reloaded.source_words == aligner.source_words
+    assert reloaded.target_words == aligner.target_words
+    assert reloaded.anchors == aligner.anchors
+
+
+def test_find_common_words_returns_intersection():
+    """find_common_words returns only words present in BOTH vocabularies.
+
+    Mutation-testing gap: intersection->union survived (fit silently skips the
+    extra words) until this direct test.
+    """
+    aligner = ProcrustesAligner(min_freq_rank=0, max_freq_rank=10)
+    source = ["king", "queen", "man"]
+    target = ["queen", "man", "woman"]
+    assert aligner.find_common_words(source, target) == ["man", "queen"]

tests/test_embeddings/test_svd.py

@@ -1,3 +1,5 @@
+import tempfile
+
 import numpy as np
 import pytest
 from hypothesis import given
@@ -7,6 +9,7 @@
 
 from chronowords.algebra.svd import SVDAlgebra
 from chronowords.utils.count_skipgrams import PPMIComputer  # ty: ignore
+from tests.strategies import trainable_corpus
 
 
 def test_ppmi_computation():
@@ -285,3 +288,25 @@ def test_most_similar_output_contract(trained_model, data, n):
 
     sims = [item.similarity for item in results]
     assert sims == sorted(sims, reverse=True)
+
+
+@given(corpus=trainable_corpus())
+@settings(deadline=None, max_examples=15)
+def test_save_load_round_trip(corpus):
+    """save_model then load_model reproduces the vocabulary and embeddings.
+
+    Models are trained once and reloaded many times downstream, so a reloaded
+    model must be indistinguishable from the original.
+    """
+    model = SVDAlgebra(n_components=5, cms_width=2000, cms_depth=3, min_word_length=2)
+    model.train(iter(corpus))
+
+    with tempfile.TemporaryDirectory() as tmp:
+        model.save_model(tmp)
+        reloaded = SVDAlgebra()
+        reloaded.load_model(tmp)
+
+    assert reloaded.vocabulary == model.vocabulary
+    assert model.embeddings is not None
+    assert reloaded.embeddings is not None
+    assert np.allclose(reloaded.embeddings, model.embeddings)

tests/utils/test_count_skipgrams.py

@@ -1,7 +1,10 @@
 import numpy as np
 import pytest
+from hypothesis import given
+from hypothesis import settings
 
 from chronowords.utils.count_skipgrams import PPMIComputer  # ty: ignore
+from tests.strategies import ppmi_inputs
 
 
 @pytest.fixture
@@ -124,3 +127,19 @@ def test_ppmi_invalid_seeds():
             skip_total=1.0,
             word_total=1.0,
         )
+
+
+@given(inputs=ppmi_inputs())
+@settings(deadline=None, max_examples=60)
+def test_ppmi_values_are_non_negative(inputs):
+    """Every PPMI entry is non-negative for any valid count inputs.
+
+    PPMI keeps only strictly-positive PMI values, so the resulting sparse
+    matrix must never contain a negative entry. Non-negativity is a hard
+    precondition for the downstream NMF topic model.
+    """
+    computer = PPMIComputer(**inputs)
+    matrix = computer.compute_ppmi_matrix_with_sketch()
+    n = len(inputs["vocabulary"])
+    assert matrix.shape == (n, n)
+    assert np.all(matrix.data >= 0)

tests/utils/test_probabilistic_counter.py

@@ -117,3 +117,34 @@ def test_merge_equals_single_combined_stream(stream_a, stream_b):
     assert merged.total == combined.total
     for key in set(stream_a) | set(stream_b):
         assert merged.query(key) == combined.query(key)
+
+
+def test_query_returns_minimum_across_rows():
+    """Query must return the MIN across rows — the defining CMS estimator.
+
+    Returning any other row's value (e.g. the max) would inflate counts under
+    hash collisions and break the tightness of the never-underestimate bound.
+    Mutation-testing gap: min->max survived until this test.
+    """
+    cms = CountMinSketch(width=10, depth=3, seed=42)
+    key = b"x"
+    indices = cms._hash_indices(key)
+    for row, idx in enumerate(indices):
+        cms.counts[row, idx] = (row + 1) * 10  # rows hold 10, 20, 30
+    assert cms.query(key) == 10  # the minimum, not 20 or 30
+
+
+def test_get_heavy_hitters_threshold_is_strict():
+    """A key whose count equals exactly threshold*total is excluded.
+
+    The cutoff is strict (`>`), so boundary items are not heavy hitters.
+    Mutation-testing gap: `>`->`>=` survived until this test.
+    """
+    cms = CountMinSketch(width=1000, depth=5, seed=42)
+    for _ in range(90):
+        cms.update("filler")
+    for _ in range(10):
+        cms.update("border")  # count 10; total 100; threshold_count == 10
+    hitters = dict(cms.get_heavy_hitters(threshold=0.1))
+    assert "border" not in hitters  # 10 is not > 10
+    assert "filler" in hitters