test(conformance): pin the conformance suite to 0.2.0-alpha.2 and reconcile the expected-failures baseline#2279
Merged
Conversation
…oncile the baseline Bumps the published conformance pin from 0.2.0-alpha.1 to 0.2.0-alpha.2 and reconciles expected-failures.yaml against it: 45 -> 33 entries. The twelve removed client auth entries failed only on the SEP-837 application_type check, which 0.2.0-alpha.2 re-scopes to draft-version runs; the dated-version scenarios now pass. The auth/iss-* scenarios do not exist in this release yet (reported unknown/failed; covered by their baseline entries). Disputed draft error-ladder cells stay listed pending conformance #336. All three CI legs verified green with zero stale entries.
|
@modelcontextprotocol/client
@modelcontextprotocol/codemod
@modelcontextprotocol/server
@modelcontextprotocol/server-legacy
@modelcontextprotocol/express
@modelcontextprotocol/fastify
@modelcontextprotocol/hono
@modelcontextprotocol/node
commit: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bumps the pinned
@modelcontextprotocol/conformancerelease from 0.2.0-alpha.1 to 0.2.0-alpha.2 and reconcilesexpected-failures.yamlagainst it: 45 → 33 entries, zero stale.Motivation and Context
The conformance baseline was established against 0.2.0-alpha.1. Since then, the conformance suite re-scoped the SEP-837
application_typecheck to draft-version runs only (shipped in 0.2.0-alpha.2), so twelve dated-version client auth scenarios that failed only on that check now pass — those twelve entries existed purely as artifacts of the stale referee and are removed. Keeping the baseline honest matters because the runner fails CI on stale entries, makingexpected-failures.yamlthe per-PR burn-down ledger for draft-spec work.Referee policy made explicit in the yaml header: the referee is always a published conformance release; newer releases are adopted by deliberately bumping the pin and reconciling the baseline in the same change.
How Has This Been Tested?
All three CI conformance legs run green against 0.2.0-alpha.2 with zero stale entries:
test:conformance:client:all(16 expected client failures)test:conformance:server(42/42 green)test:conformance:server:draft(17 expected server failures)Notes from the reconcile, verified by running both referees:
createAuthServerSEP-837 check now draft-scoped).auth/scope-step-upstays red for a new reason (SEP-2350 scope-union WARNING) — entry kept, comment updated.auth/iss-*scenarios are not present in 0.2.0-alpha.2 (reported unknown → failed); their baseline entries cover them until a release ships them.Breaking Changes
None. Test/CI configuration only; no runtime or API changes.
Types of changes
Checklist
Additional context
First of a series: subsequent PRs lean on this baseline as their conformance burn-down referee. No changeset (test-only, per repo convention).