Skip to content

Commit 0c09063

Browse files
committed
tests/interaction: manifest invariants and README for the era-axis model
test_coverage.py gains five manifest-level invariants (vacuously green until entries start using the new fields): - SPEC_VERSIONS subset of KNOWN_PROTOCOL_VERSIONS and includes LATEST - CONNECTABLE_TRANSPORTS == conftest._FACTORIES keys - supersedes / superseded_by links are bidirectional and versioned - every arm_exclusion / known_failure targets a reachable cell README.md: the requirements-manifest field list documents the new fields, and the spec-revision section is rewritten as 'Spec versions and the era axis' covering compute_cells() intersection semantics, the transports-is-metadata-only rule, and the checklist for landing a new revision. Full gate: 2299 passed, 100.00% coverage, pyright/ruff clean, 411 connect cells with unchanged ids. Claude-Session: https://claude.ai/code/session_01NF95tmzF6RhVPktJdL6fK5
1 parent f28d0cc commit 0c09063

2 files changed

Lines changed: 104 additions & 9 deletions

File tree

tests/interaction/README.md

Lines changed: 47 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,14 @@ clients can share one session manager.
9595
would require real-time waits the suite refuses.
9696
- **`transports`** names the transports a behaviour applies to; omitted means transport-independent.
9797
- **`issue`** carries the tracking link for a recorded gap once one is filed.
98+
- **`note`** carries free-form context that does not fit `divergence` or `deferred`.
99+
- **`added_in`** / **`removed_in`** bound the spec versions the behaviour exists in, as a half-open
100+
`[added_in, removed_in)` window.
101+
- **`supersedes`** / **`superseded_by`** link a retired entry to its replacement; the link is
102+
bidirectional and both ends must be versioned.
103+
- **`arm_exclusions`** carve specific `(transport, spec_version)` matrix cells out with a typed
104+
`ArmExclusionReason`.
105+
- **`known_failures`** mark specific `(transport, spec_version)` cells as strict xfail.
98106

99107
Tests link themselves to the manifest with a decorator:
100108

@@ -126,15 +134,45 @@ This is also the triage key for any rewrite: a test that fails on the new code p
126134
divergence note (the rewrite accidentally fixed a known gap — decide whether to keep the fix) or
127135
it does not (the rewrite broke something that was correct — fix the rewrite).
128136

129-
### When a new spec revision is released
130-
131-
1. Update `SPEC_REVISION` and walk the new revision's changelog.
132-
2. For each changed interaction, find its requirements (the IDs use the wire method strings the
133-
changelog speaks in), re-audit the tests against the new text, and update `source` links and
134-
assertions where behaviour legitimately changed.
135-
3. New interactions get new requirements and new tests; removed interactions get their
136-
requirements deleted along with their tests.
137-
4. A behaviour that is correct under both revisions needs no change beyond the `source` link.
137+
### Spec versions and the era axis
138+
139+
`SPEC_VERSIONS` in `_requirements.py` is the ordered tuple of protocol revisions the suite
140+
exercises; `SPEC_REVISION = SPEC_VERSIONS[-1]` is the newest. The `connect` fixture fans out over
141+
`CONNECTABLE_TRANSPORTS × SPEC_VERSIONS`, but the grid is filtered per test:
142+
`pytest_generate_tests` reads the test's stacked `@requirement` marks and calls `compute_cells()`,
143+
which intersects the admissible cells across every cited requirement — a cell survives only if
144+
**all** of the test's requirements admit it.
145+
146+
What admits or excludes a cell:
147+
148+
- **`added_in` / `removed_in`** gate which spec versions a requirement exists in, as a half-open
149+
`[added_in, removed_in)` window. A test runs only on versions inside every cited requirement's
150+
window.
151+
- **`arm_exclusions`** carve specific `(transport, spec_version)` cells out with a typed
152+
`ArmExclusionReason`. The reason vocabulary doubles as a re-admission checklist: when the gap
153+
closes, grep for the reason string to find every cell to re-admit.
154+
- **`known_failures`** keep a cell in the grid but mark it as a strict xfail — the test runs and
155+
must fail; an unexpected pass fails the suite.
156+
- **`transports`** is descriptive metadata for the non-`connect` transport-specific suites under
157+
`transports/` and does **not** drive cell generation. Only `arm_exclusions`, `added_in`, and
158+
`removed_in` filter the grid.
159+
- **`supersedes` / `superseded_by`** link a retired entry to its replacement. `test_coverage.py`
160+
enforces that links are bidirectional and versioned: the retired entry carries `removed_in`, the
161+
replacement carries `added_in`.
162+
163+
Node IDs stay `[transport]` while `len(SPEC_VERSIONS) == 1`, so today's test IDs are
164+
byte-identical to before the era axis existed. They become `[transport-version]` the moment a
165+
second version is appended to `SPEC_VERSIONS`.
166+
167+
When a new spec revision lands:
168+
169+
1. Append the version string to `SPEC_VERSIONS` (and to the `SpecVersion` `Literal`).
170+
2. Walk the new revision's changelog.
171+
3. For each affected requirement: set `removed_in` on retired behaviour, add a new entry with
172+
`added_in` for its replacement, and link the pair with `supersedes` / `superseded_by`.
173+
Behaviour that survives unchanged needs nothing beyond a re-audit of its `source` URL.
174+
4. For requirements that cannot run on the new era's path, add an `arm_exclusions` entry with the
175+
appropriate `ArmExclusionReason`.
138176

139177
## Writing a test
140178

tests/interaction/test_coverage.py

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,12 @@
1515

1616
import pytest
1717

18+
from mcp.shared.version import KNOWN_PROTOCOL_VERSIONS
19+
from mcp.types import LATEST_PROTOCOL_VERSION
1820
from tests.interaction._requirements import (
1921
CONNECTABLE_TRANSPORTS,
2022
REQUIREMENTS,
23+
SPEC_VERSIONS,
2124
ArmExclusion,
2225
KnownFailure,
2326
Requirement,
@@ -28,6 +31,7 @@
2831
covered_by,
2932
requirement,
3033
)
34+
from tests.interaction.conftest import _FACTORIES
3135

3236
_SUITE_ROOT = Path(__file__).parent
3337
_REPO_ROOT = _SUITE_ROOT.parent.parent
@@ -106,6 +110,59 @@ def test_deferral_reasons_cite_existing_paths() -> None:
106110
assert not missing, f"Deferral reasons citing paths that do not exist: {missing}"
107111

108112

113+
def test_spec_versions_are_known_and_include_latest() -> None:
114+
"""Every active spec version is one the SDK knows about, and the SDK's latest is on the active axis."""
115+
assert set(SPEC_VERSIONS) <= set(KNOWN_PROTOCOL_VERSIONS)
116+
assert LATEST_PROTOCOL_VERSION in SPEC_VERSIONS
117+
118+
119+
def test_connectable_transports_match_connect_factories() -> None:
120+
"""CONNECTABLE_TRANSPORTS and the conftest factory map name exactly the same transports."""
121+
assert set(CONNECTABLE_TRANSPORTS) == set(_FACTORIES)
122+
123+
124+
def test_supersession_links_are_symmetric_and_versioned() -> None:
125+
"""``supersedes``/``superseded_by`` reference real entries, agree in both directions, and carry version bounds."""
126+
broken = [
127+
f"{req_id} -> {target}"
128+
for req_id, req in REQUIREMENTS.items()
129+
for target in req.supersedes
130+
if target not in REQUIREMENTS or REQUIREMENTS[target].superseded_by != req_id or req.added_in is None
131+
] + [
132+
f"{req_id} <- {req.superseded_by}"
133+
for req_id, req in REQUIREMENTS.items()
134+
if req.superseded_by is not None
135+
if req.superseded_by not in REQUIREMENTS
136+
or req_id not in REQUIREMENTS[req.superseded_by].supersedes
137+
or req.removed_in is None
138+
]
139+
assert not broken, f"Broken supersession links (forward '->' or back '<-'): {broken}"
140+
141+
142+
def test_every_arm_exclusion_targets_a_reachable_cell() -> None:
143+
"""Every arm exclusion names a connectable transport and an active spec version (or wildcards)."""
144+
unreachable = [
145+
f"{req_id}: {exclusion}"
146+
for req_id, req in REQUIREMENTS.items()
147+
for exclusion in req.arm_exclusions
148+
if (exclusion.transport is not None and exclusion.transport not in CONNECTABLE_TRANSPORTS)
149+
or (exclusion.spec_version is not None and exclusion.spec_version not in SPEC_VERSIONS)
150+
]
151+
assert not unreachable, f"Arm exclusions targeting unreachable cells: {unreachable}"
152+
153+
154+
def test_every_known_failure_targets_a_reachable_cell() -> None:
155+
"""Every known failure names a connectable transport and an active spec version (or wildcards)."""
156+
unreachable = [
157+
f"{req_id}: {failure}"
158+
for req_id, req in REQUIREMENTS.items()
159+
for failure in req.known_failures
160+
if (failure.transport is not None and failure.transport not in CONNECTABLE_TRANSPORTS)
161+
or (failure.spec_version is not None and failure.spec_version not in SPEC_VERSIONS)
162+
]
163+
assert not unreachable, f"Known failures targeting unreachable cells: {unreachable}"
164+
165+
109166
def test_unknown_requirement_id_is_rejected() -> None:
110167
"""Marking a test with an ID that is not in the manifest fails at decoration time."""
111168
with pytest.raises(KeyError, match="Unknown requirement id 'tools:call:does-not-exist'"):

0 commit comments

Comments
 (0)