Skip to content

ci: test every interpreter the project claims to support - #1089

Closed
doublewhy wants to merge 2 commits into
devfrom
ci-python-version-matrix
Closed

ci: test every interpreter the project claims to support#1089
doublewhy wants to merge 2 commits into
devfrom
ci-python-version-matrix

Conversation

@doublewhy

@doublewhy doublewhy commented Aug 12, 2026

Copy link
Copy Markdown

Plain-language summary

  • Context: OpenRAE declares support for multiple Python versions, so CI must run on the interpreter it labels and expose unsupported versions early.
  • Problem: The existing workflow tested only part of the declared range, and the current PR covers only the basic 3.11–3.14 matrix rather than the complete qualification required by ci: qualify CPython 3.14 and bound interpreter support #1097.
  • Fix: Add the basic interpreter matrix and classifiers here; the fully qualified successor adds exact-runtime checks, build/install smoke tests, a 3.14t preview, and an explicit upper bound.

Related issues

Part of #1097

Depends on #1087. The accepted #1097 scope also requires an explicit <3.15 bound, exact-runtime selection, distribution build and clean-install smoke, and a separately labelled 3.14t preview; those are not yet in this PR.


GitHub currently targets this branch directly at dev, so its diff also includes #1087's MCP test commit. It remains blocked and does not satisfy all #1097 acceptance criteria; the focused qualification successor will replace this combined branch.

What breaks

The project publishes to interpreters nothing tests.

requires-python = ">=3.11" is open-ended, so pip install raes succeeds on 3.13, 3.14, and 3.15 when it arrives. Every workflow pins 3.12. So a user on 3.13 installs a version whose test suite has never been run on their interpreter.

That is not hypothetical: it is exactly how the bug in #1087 survived. asyncio.get_event_loop() has been deprecated since 3.12 and raises on 3.14, so tests/test_mcp_server.py was already broken on every version above the pin — invisibly, until uv happened to pick 3.14 on a machine of mine and ~20 tests failed at once.

The trove classifiers advertise 3.11 and 3.12 only, which contradicts requires-python in the other direction. The comment directly above them already asks for the two to agree:

# Keep this list in step with `requires-python`.

The fix

A new interpreters job runs the hermetic unit suite on 3.11, 3.12, 3.13 and 3.14, with fail-fast: false so one version failing still reports the rest. Classifiers gain 3.13 and 3.14 to match.

Why not just matrix verify, or move the pin

verify deliberately stays single-version. It owns the coverage-report artifact that the sonar job downloads, and three jobs writing one artifact name collide — Sonar would ingest coverage from an arbitrary interpreter. It also acquires and replays Isabelle, whose result cannot vary by interpreter, and it is pinned to ubuntu-22.04 specifically so bubblewrap can enforce the proof sandbox. Fanning all of that out costs CI time and risks the governed parts for no signal.

Raising the pin instead of adding a matrix would just relocate the blind spot to whichever versions the new pin excludes.

How I know

I ran the hermetic suite on each end of the range before wiring it up, using the locked dependency set:

Interpreter Result
3.11.15 6313 passed, 1 skipped
3.12 already green in CI
3.13.5 green (my development interpreter all session)
3.14.4 6313 passed, 1 skipped

uv sync --frozen resolved every dependency on both 3.11 and 3.14, including the governed z3-solver==4.16.0.0 pin that reproducible satisfiability evidence depends on.

3.14 passes only with #1087 applied; on dev today it fails.

One risk worth naming

Python 3.14 makes PEP 649 deferred annotation evaluation the default, and this project has ~158 files of Pydantic contracts that lean on annotation introspection. The runs above are reassuring, but they are one commit. This job is what keeps that true as the contract layer grows — which is the argument for adding it now rather than at the point someone wants to bump the pin.

If you would rather start advisory, adding continue-on-error: true to the job makes it non-blocking in one line.

🤖 Generated with Claude Code

Yernat Yestekov and others added 2 commits August 11, 2026 19:35
`asyncio.get_event_loop()` outside a running loop has warned since 3.12
and raises `RuntimeError: There is no current event loop` on 3.14, so the
whole module fails there — 20+ tests — while only emitting a
DeprecationWarning on the interpreters CI currently pins. `asyncio.run`
is the supported spelling for driving a coroutine from sync test code.

Verified by running the module under `-W error::DeprecationWarning`,
which reproduces the 3.14 failure: the previous spelling errors out,
this one passes all 85 tests.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`requires-python = ">=3.11"` is open-ended and the trove classifiers
advertise 3.11 and 3.12, but every workflow pins 3.12, so the project
publishes to interpreters nothing exercises. That is how the
`asyncio.get_event_loop()` call fixed in the parent commit stayed
invisible: it was latent on every version above 3.12 and only surfaced
when uv happened to select 3.14.

A new `interpreters` job runs the hermetic unit suite on 3.11, 3.12, 3.13
and 3.14, with `fail-fast: false` so one version failing still reports the
others. `verify` deliberately stays single-version: it owns the coverage
artifact Sonar consumes and the Isabelle replay, and neither varies by
interpreter.

Classifiers now list 3.13 and 3.14, which the file's own comment asks for
("Keep this list in step with `requires-python`").

Verified locally before wiring it up: the hermetic suite is 6313 passed on
3.11.15 and on 3.14.4, both with the full locked dependency set including
the governed z3-solver 4.16.0.0 pin.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@doublewhy
doublewhy changed the base branch from test-mcp-event-loop to dev August 12, 2026 03:38
@Brad-Edwards

Copy link
Copy Markdown
Collaborator

Please note in the comments or body which issue this PR closes. If no issue exists, please create one and link it.

Thank you!

@doublewhy

Copy link
Copy Markdown
Author

Linked the compatibility issue in the PR body: this PR closes #1097 and retains its dependency on #1087. Thank you.

@doublewhy

Copy link
Copy Markdown
Author

Correction after reconciling the accepted issue scope: this basic matrix is only part of #1097, so I changed the body from Closes to Part of. I will link the qualified successor that carries the metadata bound, exact-runtime check, clean distribution smoke, and 3.14t preview before closing this PR.

@doublewhy

Copy link
Copy Markdown
Author

Superseded by #1134, which closes #1097 and #1100. The successor includes the 3.11–3.14 matrix plus the missing <3.15 metadata bound, exact-interpreter assertions, frozen full suite, wheel/sdist and clean-install smoke tests, and the separate nonblocking 3.14t preview.

@doublewhy doublewhy closed this Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants