Skip to content

add vibe coded regression test suite#40

Merged
jonmarks12 merged 10 commits into
mainfrom
feature/deterministic_tests
Jun 18, 2026
Merged

add vibe coded regression test suite#40
jonmarks12 merged 10 commits into
mainfrom
feature/deterministic_tests

Conversation

@jonmarks12

Copy link
Copy Markdown
Collaborator

Add golden-output regression tests and raise coverage

Why

The FSM is fully deterministic given the same inputs and the EMT calculator, but
the test suite never validated its output — the existing tests only ran
fsm_example.py in a subprocess and asserted the exit code. That left two gaps:

  1. A change could silently alter energies or geometries and still pass.
  2. Subprocess runs don't count toward --cov=src/mlfsm, so coverage was stuck.

This PR adds a golden-output regression test that makes it trivial to tell
whether a change is purely efficiency/flags or actually changes numerical output,
and adds targeted unit tests to raise coverage.

What

Golden regression suite

  • For each interpolation style (cart, lst, ric), a small deterministic EMT
    run produces fsm.out, compared against a committed golden file.
  • Comparison is token- and numeric-tolerance aware (abs_tol=1e-5), not raw
    byte equality — CI runs on both linux-64 and osx-arm64, where the last
    printed digit can drift. Structure and text changes remain hard failures, with
    line-numbered diffs on mismatch.
  • The only non-deterministic line (Date/Time:) is masked before comparison.
  • Goldens regenerate via pytest tests/test_golden.py --update-goldens
    (or UPDATE_GOLDENS=1); review the diff in git diff tests/data/golden.

Coverage

  • Converted the existing subprocess tests (test_script, test_cartesian,
    test_stepsize) to in-process run_fsm calls so they count toward coverage.
  • Added unit tests for utils, output, interp, and cos, plus tests for the
    comparison helpers themselves.
  • Coverage rose to ~78% (interp.py 100%, output.py 93%, utils.py 98%).

Files

  • New: tests/conftest.py, tests/golden_utils.py, tests/test_golden.py,
    tests/test_golden_utils.py, tests/test_utils.py, tests/test_output.py,
    tests/test_interp.py, tests/test_cos.py
  • New data: tests/data/reactions/diels_alder/ (frozen inputs) and
    tests/data/golden/diels_alder_{cart,lst,ric}/fsm.out
  • Modified: tests/test_script.py, tests/test_cartesian.py,
    tests/test_stepsize.py (subprocess → in-process)
  • pyproject.toml: per-file ruff ignores for test code

No src/mlfsm behavior changes — the goldens exist precisely to detect such changes.

Notes

  • utils.float_check's docstring claims 0-D ndarrays are supported, but the code
    raises TypeError on them. Tests assert the actual behavior; aligning the two
    is a possible follow-up.
  • Remaining low-coverage areas: opt.py (the in-development InternalsOptimizer
    / optcoords="ric" path) and cos.py/coords.py frontier-growth edge cases.

Verification

  • pytest tests/ — 44 passed
  • pytest tests/ --cov=src/mlfsm — ~78%
  • ruff check ., ruff format --check ., mypy . — all clean
  • Regenerating goldens is byte-identical (confirms determinism end-to-end)

@joegomes joegomes self-requested a review June 18, 2026 17:08

@joegomes joegomes left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests and example update LGTM!

@jonmarks12 jonmarks12 merged commit f1f53a0 into main Jun 18, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants