Skip to content

CI red on main: test_wrong_theta_length_fails not updated for the warning-only size check (#150) #160

Description

@philluiz2323

Bug

tests/test_validate_submission.py::test_wrong_theta_length_fails fails on main, so the test-router CI job is red on every PR (CI runs the PR merged with the latest main).

FAILED tests/test_validate_submission.py::test_wrong_theta_length_fails - assert not True
  where True = ValidationResult(ok=True, errors=[], warnings=['summary.json missing recommended keys: pool'],
  info=[... 'best_theta.npy shape=(128,) dtype=float64', 'theta length matches n_total=128']).ok
1 failed, 7 passed

Root cause

#150 ("fix: make submission size check warning-only", commit f56e7af) intentionally changed src/trinity/validate_submission.py so that a θ-length mismatch is reported as a warning instead of a hard error, and prefers the bundle's own summary.json n_total:

Layout mismatches are reported as warnings instead of blocking PR submission.

But it did not update test_wrong_theta_length_fails, which still asserts the old hard-error behaviour:

def test_wrong_theta_length_fails(tmp_path):
    _write_valid_bundle(tmp_path, n_total=128)
    (tmp_path / "summary.json").write_text(
        json.dumps({"benchmark": "math500", "n_total": 128, "best_fitness": 0.1}),
    )
    result = validate_bundle(tmp_path)
    assert not result.ok                                  # <- now True: warning-only
    assert any("length 128" in e for e in result.errors)  # <- now a warning, not an error

Two things make the assertion stale under the new code:

  1. The summary now sets the expected length (n_total=128), which matches the 128-long θ, so there is no mismatch at all.
  2. Even with a genuine mismatch, it is an add_warning, so result.ok stays True.

Impact

main's own test suite is red, so no PR can get a green test-router check until this is fixed. This is a test-vs-behaviour drift from #150, not a regression in validate_bundle itself (the warning-only behaviour is the intended one).

Fix

Update the test to the intended behaviour: give it a real mismatch (summary n_total different from the θ length) and assert the mismatch is reported as a warning while the bundle still validates (result.ok is True). Rename it accordingly (e.g. test_wrong_theta_length_warns).

Location

tests/test_validate_submission.py::test_wrong_theta_length_fails (behaviour set by src/trinity/validate_submission.py, PR #150).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions